Stage 7 · Master
Phase 15 — Redis Integration
Distributed Locking
Guard a critical section across multiple replicas with a Redis lock and a monotonic fencing token, so a paused or slow process can't overwrite work done after its lock should have expired.
Two Replicas Racing for the Same Job
Complaint's SLA sweep and Notice's publisher already solved single-runner safety with a PostgreSQL advisory lock, and for scheduled work that answer is still correct: the lock lives in the same transaction as the writes it protects, so it cannot be held by a process whose database session has died. The case this lesson addresses is different. Vendor assignment and monthly charge generation are triggered by requests, not schedules, and the work they protect spans an external call — so the holder can be alive, still believe it holds the lock, and be wrong.
SET NX PX Gets the Lock; INCR Gets a Token That Outlives It
A plain 'acquire a Redis key with an expiry' lock has a subtle gap: if the process holding the lock pauses — a long garbage-collection cycle, a slow network call, a suspended container — for longer than the lock's expiry, the lock can be granted to a second process while the first one is still, from its own perspective, mid-critical-section. A fencing token turns that race into something the downstream write can detect and reject.
package redisx
import (
"context"
"errors"
"time"
"github.com/redis/go-redis/v9"
)
var ErrLockHeld = errors.New("lock is currently held by another process")
type Lock struct {
redis *redis.Client
}
func NewLock(client *redis.Client) *Lock {
return &Lock{redis: client}
}
// Acquire returns a monotonically increasing fencing token alongside the
// lock itself. The token comes from a separate INCR counter that survives
// independently of the lock key's own expiry.
func (l *Lock) Acquire(ctx context.Context, resource string, ttl time.Duration) (token int64, release func(), err error) {
ok, err := l.redis.SetNX(ctx, "lock:"+resource, "held", ttl).Result()
if err != nil {
return 0, nil, err
}
if !ok {
return 0, nil, ErrLockHeld
}
token, err = l.redis.Incr(ctx, "lock:"+resource+":fence").Result()
if err != nil {
l.redis.Del(ctx, "lock:"+resource)
return 0, nil, err
}
release = func() {
l.redis.Del(ctx, "lock:"+resource)
}
return token, release, nil
}The fence counter is never deleted by release() — its whole value comes from continuing to increase across every acquisition of this resource, forever, so a token from an earlier acquisition is always provably smaller than the current one.
The Lock Alone Never Protected Anything — the Check Does
Holding the lock is not the safety property; comparing the fencing token at the moment of the actual write is. A stalled process that thinks it still holds the lock, and later wakes up to make its write, gets rejected because a newer process's higher token is already recorded against the resource it's writing to.
ALTER TABLE monthly_charge_runs
ADD COLUMN last_fence_token bigint NOT NULL DEFAULT 0;monthly_charge_runs already exists from the Maintenance phase; this lesson only adds the one column the fencing check needs.
package application
import (
"context"
"errors"
"time"
"github.com/hoa-platform/backend/pkg/redisx"
)
var ErrStaleFencingToken = errors.New("a newer process already ran this charge generation")
func (u *GenerateMonthlyCharges) Execute(ctx context.Context, organizationID string, month time.Time) error {
token, release, err := u.lock.Acquire(ctx, "monthly-charges:"+organizationID, 5*time.Minute)
if errors.Is(err, redisx.ErrLockHeld) {
return nil // another replica already has this; not an error worth surfacing
}
if err != nil {
return err
}
defer release()
return u.db.WithTx(ctx, func(tx Tx) error {
lastToken, err := tx.CurrentFenceToken(ctx, organizationID)
if err != nil {
return err
}
if token <= lastToken {
// This process's lock was granted after a slower run's TTL
// expired, but that slower run has now caught up and already
// committed with a higher token. Reject rather than double-run.
return ErrStaleFencingToken
}
if err := tx.InsertMonthlyCharges(ctx, organizationID, month); err != nil {
return err
}
return tx.SetFenceToken(ctx, organizationID, token)
})
}token <= lastToken is the actual correctness guarantee. Everything before it — SetNX, the TTL, the whole lock — only reduces how often this branch is hit; this comparison is what makes a double-run structurally impossible rather than merely unlikely.
A well-tuned TTL makes the stale-lock race rare in practice. The fencing token is what makes it safe even on the rare occasion the race actually happens — the two mechanisms solve different halves of the same problem.
Simulate a Paused Process and Confirm It Gets Rejected
Applied exercise
Apply fencing to vendor assignment
Two staff-service replicas could both attempt to assign the same vendor to the same complaint within milliseconds of each other, before Module 13's contract-window check even runs.
- Identify the resource name you would lock on for a single complaint's vendor assignment.
- Decide where the fencing token comparison belongs — on the vendor_assignments table, or the complaint's own record.
- State what should happen to the losing (stale-token) request from the caller's point of view — a 409, a silent no-op, something else.
- Write the one test that would prove two concurrent assignment attempts never both succeed.
Deliverable
A short design naming the lock resource, the fencing check's location, and the described test.
Completion checks
- The chosen resource name is specific enough to not accidentally lock unrelated complaints against each other.
- The losing request's behavior is explicit, not left as 'whatever the database happens to do'.
Why does the fencing token comparison happen inside the same database transaction that performs the actual write, rather than as a separate check right after Acquire returns?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.