loom-background-jobs

# Background Jobs ## Overview Reliable async task execution: enqueue work, process it in workers decoupled from the request cycle, and survive crashes/retries without corrupting state. This file covers job queues, retries/backoff, DLQs, scheduling, worker pools, and delivery guarantees. For pub/sub, event sourcing, CQRS, sagas, and streaming brokers (Kafka/Pulsar), see **loom-event-driven** — don't reimplement those here. ## The two invariants everything hangs off 1. **At-least-once is the default. Design every handler to be idempotent.** Redis-backed queues (Sidekiq, BullMQ, Celery+Redis), SQS standard, and a queue retrying after a crash can deliver work more than once. FIFO producer deduplication does not make a consumer's external side effect exactly once. The achievable goal is exactly-once *effect* = durable idempotency at the sink plus retry-safe handling. 2. **Ack after success, never before.** The job must stay owned by the worker until the side effect is durably committed. Ack-then-process = at-most-once = silent data loss on crash. Process-then-ack = at-least-once = duplicates you dedup away. Always choose the latter. ### Idempotency: the non-negotiable pattern Derive a stable key from the job's *business identity* (not a random UUID per enqueue), and make the **durable sink** reject duplicates. A separate “done” flag cannot atomically cover an external side effect and a crash. ```python def handle(job): key = job["idempotency_key"] # e.g. f"rec...