A background job is a unit of work that runs asynchronously — outside the request/response cycle — via a queue and worker pool.
Background jobs are units of work executed asynchronously outside the user-facing request path — sending emails, processing webhooks, generating exports, running scheduled tasks, calling slow third-party APIs. The pattern: user request enqueues a job; a pool of workers pulls jobs off the queue and executes them; success and failure are tracked separately from the original request. Get this layer right early — Sidekiq (Ruby), Celery (Python), BullMQ (Node), Que (Postgres-based) — and it hums for years; get it wrong and it produces a steady stream of mysterious late-night failures.
Every background job will run twice at some point — worker crashes, network partitions, retry storms. Jobs must be idempotent: running a job with the same input twice must produce the same outcome as running it once. Common techniques: an operation_id or idempotency_key stored on the target record; SELECT-then-INSERT patterns replaced with ON CONFLICT DO NOTHING; external API calls made with idempotency keys the vendor supports (Stripe, most modern APIs support this). Non-idempotent jobs are a category of latent bug — they don't cause daily pain but produce corrupted state during any incident.
Default retry strategy: exponential backoff with jitter, up to some maximum count (typically 5-10 attempts), then dead-letter to a manual queue for human review. Failing jobs should not retry forever — they hide problems and consume worker capacity. Categorize failures: transient (network, rate limits — retry aggressively), permanent (validation error, missing record — fail fast, don't retry), and requires-attention (unknown state — move to dead-letter). Blindly retrying every exception is the primary cause of retry storms taking down downstream systems.
A single queue is fine at first but becomes a bottleneck as job types diverge. Separate queues by priority (critical, default, low) and by resource profile (CPU-heavy vs. I/O-bound). This prevents a burst of low-priority export jobs from starving user-facing email sends. Common pattern: 3-5 queues, dedicated worker pools per queue with sized concurrency. Avoid dozens of queues — the operational complexity outweighs the isolation benefit for most teams.
Scheduled jobs (nightly reports, weekly cleanups) are usually run via cron or a scheduler bolted onto the job system (Sidekiq-cron, Celery Beat, Cloud Scheduler). Failure modes: the scheduler node dies and jobs stop running silently; the scheduler runs the same job twice during network partitions; long-running scheduled jobs overlap with the next scheduled run. Mitigations: monitor 'last successful run' timestamps per job with alerts; use distributed locks or idempotency to prevent double-runs; skip runs when the previous instance is still executing.
Instrument every job with: enqueue timestamp, start timestamp, completion timestamp, retry count, and outcome. Dashboards should show queue depth per queue (rising depth = capacity problem), job latency percentiles (P99 latency spikes = downstream slowdown), and failure rate. Dead-letter queues need a review process — jobs sitting in DLQ for weeks are usually customer-visible bugs no one has noticed. Weekly triage of the DLQ is the highest-leverage 30 minutes a backend team can spend.
Investor directory · Fundraising library · Articles A–Z · Company funding database