background-jobs

# Background Jobs ## Purpose Move work out of the request path without losing it. A job system is a distributed system with a friendly interface; the failure modes are the same, and they are hidden until they are not. ## When to Use - Work that is slow, retriable, or must survive a request timeout: emails, exports, webhooks, media processing. - Scheduled and recurring work. - Diagnosing duplicate side effects, stuck queues, or jobs that silently disappear. ## Capabilities - Queue and worker design, including priority and isolation. - Idempotency and exactly-once effects. - Retry policy: backoff, jitter, attempt limits, dead-letter handling. - Scheduling: cron jobs, delayed jobs, and deduplicating them across instances. - Observability: queue depth, age, failure rate. ## Inputs - The work to be done and whether it can safely run twice. - Latency expectations: seconds, minutes, or overnight. - Volume, and how bursty it is. ## Outputs - Job handlers that are idempotent by construction. - Explicit retry and DLQ policies per job type. - Alerts on queue depth and oldest-message age. ## Workflow 1. **Make it idempotent first** — Before anything else. A job will run twice; design so that this is harmless. Key the side effect, not the invocation. 2. **Pass identifiers, not objects** — Enqueue `{"order_id": "..."}`, not a serialized order. By the time the job runs, the object may be stale. 3. **Set the retry policy per job** — A transient network failure deserves exponentia...