Stage 7 · Master
Phase 15 — Redis Integration
TTL
Fix the multi-replica SMS budget gap from Notification with a shared Redis counter and TTL, then choose TTLs deliberately per data volatility instead of picking one number everywhere.
Revisit the Per-Process SMS Budget
The Notification module's PerProcessBudget was explicit about its limitation: running two replicas of notification-service doubles the effective SMS budget per organization, because each replica counts independently. That gap is now worth closing — not because Redis is available, but because the specific cost-control problem it was flagged for is real and measured in provider billing.
The primitive itself is not new. API Gateway already runs a per-org token bucket in Redis to stop one noisy tenant starving the others, and the window counter here works the same way: one key per subject per window, INCR to claim, compare against a limit. What differs is what a rejection means. The gateway sheds load and the client retries a second later; an SMS budget rejection means a resident's payment reminder is never sent at all. That difference decides the failure mode below, and it is the reason this counter does not simply reuse the gateway limiter as-is.
package redisadapter
import (
"context"
"time"
"github.com/redis/go-redis/v9"
)
type SharedSMSBudget struct {
redis *redis.Client
limit int64
window time.Duration
}
func NewSharedSMSBudget(client *redis.Client, limit int64, window time.Duration) *SharedSMSBudget {
return &SharedSMSBudget{redis: client, limit: limit, window: window}
}
func budgetKey(organizationID string, windowStart time.Time) string {
return "sms:budget:" + organizationID + ":" + windowStart.Format("2006010215")
}
// Allow is safe across any number of notification-service replicas: every
// replica increments the same key, and Redis serializes INCR operations
// against a single key regardless of how many clients call it concurrently.
func (b *SharedSMSBudget) Allow(ctx context.Context, organizationID string) (bool, error) {
key := budgetKey(organizationID, time.Now().UTC())
count, err := b.redis.Incr(ctx, key).Result()
if err != nil {
return false, err
}
if count == 1 {
// Only the first increment in this window sets the expiry — every
// later INCR in the same window must not reset the countdown.
if err := b.redis.Expire(ctx, key, b.window).Err(); err != nil {
// A key without a TTL would cap this organization forever.
return false, err
}
}
return count <= b.limit, nil
}Two details carry the lesson. The count == 1 guard is the correctness detail — calling EXPIRE on every increment slides the deadline forward and the budget never resets on schedule. The error return is the design detail: unlike the gateway limiter, which can fail open because a dropped request is retried, this returns (false, err) so a Redis outage suppresses SMS rather than uncapping spend, and the caller decides whether a suppressed reminder is escalated.
Keying on the hour string means the budget resets on wall-clock hour boundaries, so an organization that exhausts its quota at 10:59 gets a fresh allowance sixty seconds later. A sliding window would be fairer and materially more expensive: it needs a sorted set per organization and a range trim on every send. Fixed windows are the right trade here because the budget exists to bound a bill, not to smooth traffic.
Not Every Key Deserves the Same TTL
| Data | Volatility | TTL | Reasoning |
|---|---|---|---|
| Resident profile cache | Changes rarely (move-out, edits) | 15 minutes | Eager invalidation handles real changes; TTL is only a safety net |
| Flat list cache (versioned) | Changes on move-in/move-out | 5 minutes | Shorter, because stale entries are abandoned, not actively cleaned up |
| SMS budget counter | Resets on a fixed schedule | Exactly 1 hour | TTL IS the reset mechanism here, not a safety net |
| Revocation epoch (next lesson) | Written once, read constantly | Exactly the access-token lifetime | Shorter reopens the gap; longer wastes memory on a key no live token can predate |
Identical TTLs Expire Together, on Purpose or Not
If every resident profile cached during one busy morning gets exactly a 15-minute TTL, all of them expire within the same narrow window, and the resulting wave of simultaneous cache misses hits PostgreSQL all at once — the exact stampede a cache exists to prevent. A small random jitter spreads expirations out.
package cacheadapter
import (
"math/rand"
"time"
)
// withJitter spreads expirations across a ±10% window so that a burst of
// cache writes at the same moment does not also expire at the same moment.
func withJitter(base time.Duration) time.Duration {
jitterRange := int64(base) / 10
offset := rand.Int63n(2*jitterRange) - jitterRange
return base + time.Duration(offset)
}A ±10% spread on a 15-minute TTL means expirations land anywhere between roughly 13:30 and 16:30 minutes out — enough to break the synchronization without meaningfully changing how fresh the data is expected to be.
Cache the Absence of a Resident, Briefly
A resident ID that does not exist — a typo, an old bookmarked link, a probing request — still costs a full PostgreSQL round trip on every request if nothing is cached for it. A short negative-cache entry protects the database from repeated lookups of the same nonexistent ID without risking long-lived staleness if the ID becomes valid later.
const notFoundSentinel = "__not_found__"
const notFoundTTL = 30 * time.Second
// GetProfile checks for the negative sentinel before treating a cache miss
// as "go to Postgres" — this keeps a nonexistent ID from ever repeatedly
// reaching the database while still expiring quickly if the ID starts
// resolving to a real resident (e.g. after a delayed provisioning step).
func (c *CachedResidentReader) checkNegativeCache(ctx context.Context, key string) bool {
value, err := c.redis.Get(ctx, key+":absent").Result()
return err == nil && value == notFoundSentinel
}30 seconds is deliberately short relative to the 15-minute positive TTL — a negative cache entry that outlives that window would risk hiding a resident record created moments after the miss was first cached.
Confirm the Budget Holds Across Two Processes
Applied exercise
Choose a TTL and jitter for the notification template cache
Templates change rarely — an organization updates its payment receipt wording maybe once a quarter — but the current design queries PostgreSQL on every send.
- Propose a TTL for a cached template resolution, justified against how often templates actually change.
- State whether jitter is worth adding for this specific cache, given how many keys would realistically expire at once.
- Decide how invalidation should work when an organization updates its own template — eager delete, or TTL-only.
- Identify the one property (from the ownership lesson) this new cache must still respect.
Deliverable
A short written TTL/invalidation design for the template cache.
Completion checks
- The TTL choice is justified by template change frequency, not copied from the resident profile's 15 minutes without reasoning.
- The design still names the owning service responsible for invalidating this cache.
In SharedSMSBudget.Allow, why does the code call Expire only when `count == 1`, rather than on every call?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.