Stage 7 · Master
Phase 10 — Complaint Service
Deployment
Deploy complaint automation safely by preventing duplicate SLA sweeps across replicas, rolling out additive migrations in order, and wiring environment-specific policy controls through Helm.
Why does an in-process SLA ticker become a correctness bug once you scale replicas?
A ticker running inside one process looks fine in local development. In Kubernetes, three replicas mean three independent ticker loops, and each loop can scan the same due complaints and emit the same breach notification. The data update may still converge because sla_breached flips true, but duplicate notifications, duplicate audit events, and duplicate pager noise are still correctness bugs. Complaint workflows are operational systems, so duplicates are not cosmetic.
You have two viable deployment shapes: run the sweep out-of-band from a CronJob that hits an internal endpoint, or keep the code in-process but guard it with a Postgres advisory lock so only one replica actually performs the scan at a time. The lock approach is useful when you want the same binary to serve HTTP and own its own maintenance loop without introducing another workload type.
If one breached complaint produces three alerts or three 'escalated' side effects, operators stop trusting the system exactly when it is supposed to help them respond faster.
How does a Postgres advisory lock keep exactly one sweep active?
package complaint
import (
"context"
"github.com/jackc/pgx/v5/pgxpool"
)
const complaintSweepLockKey int64 = 10036
func (w *Watcher) SweepWithAdvisoryLock(ctx context.Context, pool *pgxpool.Pool) error {
conn, err := pool.Acquire(ctx)
if err != nil {
return err
}
defer conn.Release()
var locked bool
if err := conn.QueryRow(ctx, "SELECT pg_try_advisory_lock($1)", complaintSweepLockKey).Scan(&locked); err != nil {
return err
}
if !locked {
return nil
}
defer func() {
_, _ = conn.Exec(ctx, "SELECT pg_advisory_unlock($1)", complaintSweepLockKey)
}()
return w.Sweep(ctx)
}The lock is session-scoped to the acquired connection. One replica wins, the others observe 'not locked' and exit quietly instead of double-processing the same queue.
If you choose the CronJob path instead, protect /internal/sla/sweep as an internal-only endpoint. Keep it on the cluster network, require a shared secret header or service-to-service auth, and never expose it through the public gateway. A public sweep endpoint would let an attacker create needless scan load or repeatedly trigger side effects.
Which rollout order keeps complaint migrations safe during a rolling deploy?
Migration-before-traffic is settled policy by now — User Service made the Job wait, and Payment Service showed what happens when a retrying caller meets a missing table. Complaints add one wrinkle those lessons did not have: the SLA policy rows are *data* the sweep depends on, not just schema. A cluster that gets 0037's columns but not 0040's seed rows will run the sweeper against an empty policy table and quietly breach nothing, forever. So seed data ships as its own additive migration, in order, rather than as application startup code that only runs where someone remembered to deploy it.
- 0036 complaints, 0037 SLA policy plus columns, 0038 assignments, 0039 events, 0040 policy seed — additive, in that order, before handlers serve traffic.
- Seed defaults belong in a migration, not in startup code, so restored databases and fresh environments agree.
- Rollback order inverts: code first, then remove unused additive objects once traffic is stable.
INSERT INTO complaint_sla_policies (org_id, category, severity, acknowledge_within, resolve_within)
SELECT
o.id,
seeded.category,
seeded.severity,
seeded.acknowledge_within,
seeded.resolve_within
FROM organizations o
CROSS JOIN (
VALUES
('security', 'critical', INTERVAL '2 hours', INTERVAL '24 hours'),
('security', 'high', INTERVAL '4 hours', INTERVAL '36 hours'),
('plumbing', 'high', INTERVAL '8 hours', INTERVAL '72 hours'),
('electrical', 'high', INTERVAL '8 hours', INTERVAL '72 hours'),
('noise', 'medium', INTERVAL '12 hours', INTERVAL '48 hours'),
('common_area', 'medium', INTERVAL '12 hours', INTERVAL '96 hours'),
('parking', 'medium', INTERVAL '24 hours', INTERVAL '72 hours')
) AS seeded(category, severity, acknowledge_within, resolve_within)
ON CONFLICT (org_id, category, severity) DO NOTHING;Seeding in a migration makes defaults reproducible across local, staging, and restored environments.
DELETE FROM complaint_sla_policies
WHERE (category, severity, acknowledge_within, resolve_within) IN (
('security', 'critical', INTERVAL '2 hours', INTERVAL '24 hours'),
('security', 'high', INTERVAL '4 hours', INTERVAL '36 hours'),
('plumbing', 'high', INTERVAL '8 hours', INTERVAL '72 hours'),
('electrical', 'high', INTERVAL '8 hours', INTERVAL '72 hours'),
('noise', 'medium', INTERVAL '12 hours', INTERVAL '48 hours'),
('common_area', 'medium', INTERVAL '12 hours', INTERVAL '96 hours'),
('parking', 'medium', INTERVAL '24 hours', INTERVAL '72 hours')
);How do Helm values and an internal CronJob make SLA sweep behavior environment-specific?
Helm should not duplicate the SLA policy table itself, but it can control how the sweep runs in each environment: enabled or disabled, schedule, endpoint token secret, and whether you rely on CronJob triggering or in-process locks. That keeps staging conservative and production explicit. The template below shows the CronJob variant; a values snippet then overrides schedule and secrets per environment without recompiling the service.
{{- if .Values.complaint.slaSweep.enabled }}
apiVersion: batch/v1
kind: CronJob
metadata:
name: {{ include "complaint-service.fullname" . }}-complaint-sla-sweep
labels:
app.kubernetes.io/name: {{ include "complaint-service.name" . }}
spec:
schedule: {{ .Values.complaint.slaSweep.schedule | quote }}
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "complaint-service.name" . }}
spec:
restartPolicy: OnFailure
containers:
- name: sweep
image: curlimages/curl:8.11.1
command:
- /bin/sh
- -ec
- >-
curl --fail --show-error
-X POST
-H "X-Internal-Token: ${INTERNAL_SWEEP_TOKEN}"
http://{{ include "complaint-service.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local:8080/internal/sla/sweep
env:
- name: INTERNAL_SWEEP_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.complaint.slaSweep.internalTokenSecretName }}
key: token
{{- end }}Forbid concurrency prevents overlapping CronJob runs if one sweep takes longer than expected.
complaint:
slaSweep:
enabled: true
schedule: "*/5 * * * *"
internalTokenSecretName: "complaint-service-internal-sweep"Values keep environment tuning outside the binary while the authoritative per-org SLA numbers still live in the database.
How do you verify the deployment shape before production traffic depends on it?
Applied exercise
Eliminate duplicate breach notifications in a three-replica rollout
Production now runs three complaint-service replicas, and operations reports each breached security complaint shows up three times in Slack after the complaint module deploy.
- Explain why an in-process ticker per replica causes duplicate sweep side effects even if sla_breached eventually becomes true.
- Choose either the advisory-lock path or the CronJob path and justify it for this service.
- Describe the rollout order for migrations, code, and Helm changes so no replica starts with half the complaint schema.
- Add one verification step that would prove duplicates are gone before declaring the fix complete.
Deliverable
A deployment note describing the chosen single-run strategy, rollout order, and verification evidence.
Completion checks
- The plan addresses duplicate side effects, not just duplicate CPU work.
- The rollout order keeps schema changes additive and ahead of code that depends on them.
- The internal sweep trigger is not exposed as a public endpoint.
Why is a plain time.Ticker inside every complaint-service replica unsafe for the SLA watcher in production?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.