Stage 7 · Master
Phase 11 — Notice Service
Deployment
Ship notice publishing safely by rolling out object storage, concurrent search indexes, and replica-safe workers through Helm.
Which Helm values make object storage production-safe?
Attachments require more than an endpoint and bucket name. The chart needs private bucket defaults, a lifecycle age for archived files, and a clear distinction between local MinIO and production object stores. The cost failure mode is predictable: if archived notice attachments live forever, the bucket grows quietly while the active feed stays small.
notice:
storage:
provider: s3
endpoint: http://minio.default.svc.cluster.local:9000
region: us-east-1
bucket: hoa-notice-assets
forcePathStyle: true
accessKeySecretRef:
name: notice-storage
key: access-key
secretKeySecretRef:
name: notice-storage
key: secret-key
archiveRetentionDays: 90
allowedContentTypes:
- application/pdf
- image/jpeg
- image/png
maxUploadBytes: 10485760
publisher:
schedule: "*/1 * * * *"
batchSize: 100
orgScanLimit: 25
advisoryLockKey: notice-publisher-v1
search:
referenceTimeHeader: X-Search-Evaluated-AtSecrets stay in Kubernetes Secret objects, not inline in the values file; the values file only declares where the chart should read them from.
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "notice-service.fullname" . }}-notice-storage-bootstrap
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: mc
image: minio/mc:RELEASE.2026-07-05T15-07-57Z
command:
- /bin/sh
- -ec
- |
mc alias set notice {{ .Values.notice.storage.endpoint }} "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
mc mb --ignore-existing notice/{{ .Values.notice.storage.bucket }}
cat <<'POLICY' >/work/lifecycle.json
{
"Rules": [
{
"ID": "expire-archived-notice-attachments",
"Status": "Enabled",
"Filter": { "Prefix": "notices/" },
"Expiration": { "Days": {{ .Values.notice.storage.archiveRetentionDays }} }
}
]
}
POLICY
mc ilm import notice/{{ .Values.notice.storage.bucket }} < /work/lifecycle.json
env:
- name: MINIO_ROOT_USER
valueFrom:
secretKeyRef:
name: {{ .Values.notice.storage.accessKeySecretRef.name }}
key: {{ .Values.notice.storage.accessKeySecretRef.key }}
- name: MINIO_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.notice.storage.secretKeySecretRef.name }}
key: {{ .Values.notice.storage.secretKeySecretRef.key }}The bootstrap job creates the bucket idempotently and reapplies lifecycle policy on upgrade, which is safer than relying on one manual kubectl session that nobody remembers six months later.
How do you roll out the search index without locking writers?
The Search lesson wrote the generated column and its GIN index as migration 0045, and on an empty local table that migration is instant. On a production notices table with two years of rows, the same statement takes a write lock long enough that editors watch their save button hang. The production-safe form is CREATE INDEX CONCURRENTLY — but PostgreSQL forbids it inside a transaction, and the repository's make migrate wraps every migration transactionally. So the operational rollout does not re-run 0045; it splits it.
ALTER TABLE notices
- ADD COLUMN search_vector tsvector GENERATED ALWAYS AS (...) STORED;
+ ADD COLUMN IF NOT EXISTS search_vector tsvector GENERATED ALWAYS AS (...) STORED;
- CREATE INDEX notices_search_vector_idx
+ CREATE INDEX CONCURRENTLY IF NOT EXISTS notices_search_vector_idx
ON notices USING GIN (search_vector);Two words carry the entire production difference. The column definition itself is unchanged from the Search lesson and is not restated here — adding IF NOT EXISTS only makes the step re-runnable after a failed attempt, and CONCURRENTLY moves the index build off the write path. The tradeoff is that the concurrent build must run outside the transactional migrate batch, as its own pre-deploy step.
"Migration command exited 0" is not proof the index is healthy. An INVALID index is worse than no index: it costs writes on every insert while contributing nothing to reads, and search latency will look exactly like the pre-index baseline.
Why must only one publisher replica act on a schedule?
The complaint SLA watcher already established the pattern: schedule the work with Kubernetes, then guard the critical section with a PostgreSQL advisory lock. Notice publishing reuses that same idea because scaling the notice-service deployment from one to three pods should not triple-publish scheduled notices. The lock is the final safety net even if two CronJob pods overlap during rollout.
apiVersion: batch/v1
kind: CronJob
metadata:
name: {{ include "notice-service.fullname" . }}-notice-publisher
spec:
schedule: {{ .Values.notice.publisher.schedule | quote }}
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 2
failedJobsHistoryLimit: 5
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: publisher
image: {{ include "notice-service.image" . }}
imagePullPolicy: IfNotPresent
args:
- publish-notices-once
env:
- name: NOTICE_PUBLISHER_BATCH_SIZE
value: {{ .Values.notice.publisher.batchSize | quote }}
- name: NOTICE_PUBLISHER_ORG_SCAN_LIMIT
value: {{ .Values.notice.publisher.orgScanLimit | quote }}
- name: NOTICE_PUBLISHER_LOCK_KEY
value: {{ .Values.notice.publisher.advisoryLockKey | quote }}
- name: NOTICE_STORAGE_BUCKET
value: {{ .Values.notice.storage.bucket | quote }}
- name: NOTICE_STORAGE_ENDPOINT
value: {{ .Values.notice.storage.endpoint | quote }}
envFrom:
- secretRef:
name: {{ .Values.notice.storage.accessKeySecretRef.name }}concurrencyPolicy: Forbid reduces overlap at the Kubernetes layer, while the advisory lock inside the worker still protects against retries, overlapping rollouts, or manual job runs.
What do you validate before promoting the chart?
Validate three things locally: MinIO is reachable, the Helm templates render the lifecycle and CronJob resources you expect, and the publish worker still behaves correctly when invoked as a one-shot command. One security concern to keep in mind is bucket exposure: notice attachments are private resident documents and should never be placed behind a world-readable bucket policy just to make local testing easier.
Applied exercise
Plan a zero-surprise Notice rollout
Production already contains thousands of published notices from internal seed scripts, and you are about to enable full-text search plus object-storage-backed attachments. Your deploy plan must avoid write locks, prevent duplicate publishers, and keep storage cost bounded.
- Sequence the rollout steps for the generated search column, the concurrent index build, and the Helm chart changes.
- Describe why the concurrent index statement cannot run inside the default transactional migrate path.
- Explain how the MinIO or S3 lifecycle rule contains cost for archived attachments without deleting active records prematurely.
- State which two layers prevent duplicate scheduled publication when the CronJob overlaps during a rollout.
Deliverable
A deployment checklist that orders schema, storage, and worker changes correctly and names the specific safety mechanism for each risk.
Completion checks
- The index build is explicitly called out as a no-transaction operation.
- The plan mentions both CronJob concurrencyPolicy and the advisory lock inside the worker.
- Archived attachment retention is framed as a lifecycle policy in object storage, not as a database cleanup job.
Why use CREATE INDEX CONCURRENTLY for the notice search index in production?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.