Stage 7 · Master
Resilience Patterns
Timeouts, Context Deadlines, and Retries
Meridian already applies transport-level HTTP timeouts in every binary; this chapter specifies the next layer of resilience: one request budget, propagated from the gateway through every tenant-scoped operation.
Current Baseline
The scaffolding phase already establishes a useful lower bound: each service starts an http.Server with explicit ReadTimeout and WriteTimeout values. gateway, identity-service, community-service, and billing-service therefore cannot hang forever at the TCP edge. That protection is real, but it is not yet an end-to-end request budget. Once the gateway begins proxying tenant-scoped traffic, the platform needs one parent deadline that every downstream hop spends from, not four independent timers that reset at each boundary.
No reverse-proxy routes, tenant lookup, database calls, or retry helpers exist yet in meridian. The remaining sections specify the design for the implementation phase; they do not describe working code in the current repository state.
Single Request Budget
The request budget must begin at the only public entrypoint: apps/gateway. A public call such as GET /v1/:tenantSlug/residents or POST /v1/:tenantSlug/invoices first spends time on tenant slug resolution, then on reverse proxying to an internal service, and later on database work inside that service. The budget therefore belongs to the whole request, not to any individual dependency. Local transport limits still remain, but they must never outlive the parent context created at the gateway.
apps/gateway/internal/middleware/request_deadline.gocreateResponsibility: Attach a bounded parent context to every public request before routing or reverse proxying begins.
Why now: The budget has to start at the first network boundary; starting it inside a downstream service already wastes time.
Connects to: Wraps the router returned by apps/gateway/internal/router/router.go and composes with the existing recovery middleware.
apps/gateway/internal/router/router.gomodifyResponsibility: Preserve the caller context when proxying to identity-service, community-service, and billing-service.
Why now: A gateway-owned deadline only matters if downstream requests inherit it instead of creating a fresh timeout budget.
Connects to: Uses the downstream base URLs already loaded by apps/gateway/internal/config/config.go.
apps/identity-service/internal/http/middleware.gocreateResponsibility: Reject requests whose propagated context is already expired before any tenant-scoped work begins.
Why now: Late work should fail fast rather than continue consuming CPU or database time after the caller has already timed out.
Connects to: Works alongside tenancy.TenantIDHeader once gateway tenant propagation is implemented.
Context Propagation
Meridian already standardizes one cross-service concern in libs/platform/tenancy: the gateway resolves a tenant slug once and forwards the tenant UUID through X-Tenant-Id. Deadline propagation should follow the same discipline. The gateway owns the public concern, downstream services consume it, and no repository or client is allowed to substitute context.Background(). Inside a service, tenant identity and remaining time travel together through context.Context.
| Model | Behavior | Operational consequence |
|---|---|---|
| Per-hop timeout reset | Each client package starts its own timer | Slow requests keep beginning expensive work after the original caller has already waited too long |
| Parent deadline propagated through context | Every hop spends from the same shrinking budget | Chosen. Expired work is refused consistently across gateway and internal services. |
Retry Safety
Retries are only valid when two conditions hold simultaneously: the operation is semantically safe to repeat, and the remaining context deadline still leaves enough time for another attempt. In Meridian, transient reads such as tenant-slug lookup or an idempotent GET proxied through the gateway may qualify. Commands such as creating invoices, applying payments, or mutating memberships do not become retry-safe merely because the transport failed. Until idempotency keys and persistence semantics are designed explicitly, those writes should fail once and surface a structured APIError.
- Safe candidates: idempotent tenant resolution, internal GET calls, health probes, metadata reads.
- Unsafe without extra design: invoice creation, payment capture, membership writes, complaint creation, announcement publication.
- Every retry loop must stop when ctx.Err() becomes non-nil; retry policy may not outlive the caller budget.
The Decision
Meridian keeps the implemented server-level timeouts in each binary and specifies a second layer for the implementation phase: gateway-owned request deadlines, context propagation across every internal hop, and retries limited to idempotent operations that still have remaining budget. The resulting rule is simple: tenant identity is resolved once, time is budgeted once, and both travel through the system together.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.