Stage 7 · Master
Background Processing
Job Queues, Retries, and Scheduling
The first planned Meridian queue is a billing-service concern: tenant-scoped payment and invoice jobs with explicit retry and schedule metadata.
Why a Queue Is Needed
Worker pools solve bounded concurrency, but they still need a source of work. In Meridian that source should be an explicit billing-service queue rather than fire-and-forget goroutines launched from the HTTP path. A queue records intent durably, supports retry after transient failures, and allows scheduled work such as monthly dues generation to execute even when no user request is in flight.
Workflow
Enqueue: billing-service records tenant-scoped work with an idempotency key and optional RunAfter timestamp.
Step 1 / 5 — Enqueue
No Redis queue, Postgres queue table, or scheduler exists in the reference repo today. The design below is a planned execution model for billing-service once asynchronous payment and invoicing workflows are implemented.
Job Envelope
The first useful job types are closely aligned with billing-service ownership: payment.webhook.process, invoices.recurring.generate, and receipts.render. That bounded vocabulary helps keep the queue honest. A queue that starts collecting unrelated cross-service work is a signal that the service boundary is drifting or that a new bounded context is being smuggled in without an architectural decision.
Retries and Backoff
| Failure class | Retry? | Reason |
|---|---|---|
| Temporary payment-provider timeout | Yes | Transient network or provider instability |
| Redis unavailable while dequeuing | Yes | Infrastructure fault outside the job payload |
| Validation failure in the payload | No | The job is malformed and should dead-letter |
| Tenant not found or invoice already finalized incompatibly | Usually no | Permanent domain failure unless a follow-up write changes the facts |
Scheduled Work
Recurring dues are a scheduling problem as much as they are a queueing problem. The cleanest design is to enqueue scheduled jobs with a RunAfter timestamp and let the worker claim only ready jobs. That keeps invoice generation and webhook processing inside one consistent dispatch mechanism. It also avoids a separate cron-only code path that would need its own observability, locking, and failure handling.
apps/billing-service/internal/queue/redis.gocreateResponsibility: Stores queued jobs, claims ready jobs, requeues retries, and moves permanent failures to a dead-letter stream.
Why now: Queue mechanics should be isolated from billing workflow logic and reusable across several job types in the same module.
Connects to: Uses the Redis phase introduced in Chapter 8 and feeds the worker pool from this module's worker binary.
apps/billing-service/internal/jobs/scheduler.gocreateResponsibility: Calculates recurring billing periods and enqueues future invoice-generation jobs with tenant-aware RunAfter timestamps.
Why now: Scheduling policy is business logic; it should not be embedded in a generic Redis adapter.
Connects to: Pairs with apps/billing-service/internal/jobs/job.go and apps/billing-service/cmd/worker/main.go.
The Decision
- The first queue belongs to billing-service and carries invoice, payment, and receipt work only.
- Every job envelope includes tenant_id, retry metadata, schedule metadata, and an idempotency key.
- Retries are reserved for transient infrastructure or provider failures; permanent domain failures should dead-letter quickly.
- Scheduled work should reuse the same queue contract via RunAfter rather than creating a parallel cron-only pipeline.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.