Stage 7 · Master
Phase 19 — Production Deployment
ConfigMaps
Write the templates/configmap.yaml the Helm lesson's checksum annotation already referenced, prove that editing a ConfigMap alone never restarts a running Pod, and choose between environment variables and mounted files based on whether a setting needs to hot-reload.
The Helm Lesson Added an Annotation Whose Purpose It Could Not Yet Demonstrate
charts/organization/templates/deployment.yaml carries checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}, and the Helm lesson shipped a three-key ConfigMap alongside it so the chart would actually render. What that lesson could not do was justify the annotation: with nothing yet changing in the ConfigMap, a checksum that never moves looks like ceremony. This lesson supplies the missing half: it changes the ConfigMap out-of-band, shows that nothing restarts, and only then is the annotation's presence in that template obviously load-bearing rather than decorative.
Changing a ConfigMap's Content Does Not Restart Any Pod Reading It as Environment Variables
envFrom: configMapRef values are read exactly once, when a container process starts, and copied into that process's environment. Kubernetes has no mechanism that watches a ConfigMap and restarts a Deployment's Pods when it changes — from the Deployment's perspective, only its own pod template spec matters, and updating a ConfigMap in isolation never touches that spec.
$ kubectl exec deploy/organization-api -n hoa-staging -- printenv LOG_LEVEL
info
# ...even though the ConfigMap itself now reports:
$ kubectl get configmap organization-config -n hoa-staging -o jsonpath='{.data.LOG_LEVEL}'
debugThe ConfigMap object and the running Pod's actual environment have silently diverged — an on-call engineer trusting kubectl get configmap output while debugging a live incident would draw an entirely wrong conclusion about what the running process is actually configured to do.
Fold the ConfigMap's Content Into the Pod Template Itself
A rollout only happens when the Pod template spec changes. The fix is not a Kubernetes feature that watches ConfigMaps — it is making the ConfigMap's own content part of the Pod template's hash indirectly, via an annotation whose value is a checksum of the rendered ConfigMap. Change the ConfigMap's content through Helm, and the checksum — therefore the Pod template, therefore the Deployment — changes too.
data:
LOG_LEVEL: {{ .Values.logLevel | default "info" | quote }}
ORGANIZATION_HTTP_ADDR: ":8081"
ORGANIZATION_INTERNAL_ADDR: ":9090"
+ HTTP_READ_TIMEOUT: "10s"
+ HTTP_WRITE_TIMEOUT: "15s"
ENVIRONMENT: {{ .Values.environment | quote }}Two timeouts join the settings the Helm lesson's minimal ConfigMap already carried. LOG_LEVEL is shown as unchanged context deliberately — it is the value the reproduction above just patched, and the one an operator will plausibly reach for at 3am, which is exactly when discovering that changing it does nothing would be most expensive. The trust-category rule has not moved: nothing here is sensitive, and the password stays in the Secret.
Environment Variables and Mounted Volumes Solve Different Problems
| Delivery method | Updates without a restart? | Right for |
|---|---|---|
| envFrom: configMapRef | No — read once at process start; requires the checksum-annotation rollout above to pick up a change | Settings that are safe to apply only on a controlled rollout: timeouts, addresses, structural config |
| volumeMount from a ConfigMap | Yes — kubelet syncs mounted ConfigMap files to the container's filesystem within roughly a minute of a change, with no Pod restart | A value that genuinely benefits from live tuning without a deploy: this platform's log level, watched by a small file-watcher goroutine |
// WatchLogLevel polls the mounted ConfigMap file every 5s and adjusts the
// running process's log level in place — no restart, no rollout, no
// checksum annotation involved, because this one setting is deliberately
// delivered by mounted file instead of environment variable.
// zerolog.SetGlobalLevel is what makes this achievable in one line: it is
// a single, process-wide filter every zerolog.Logger in this binary reads
// on each call, so flipping it here takes effect immediately for every
// logger already constructed, with nothing to plumb through and no
// per-logger handler to rebuild.
func WatchLogLevel(ctx context.Context, path string, logger zerolog.Logger) {
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
var current string
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
raw, err := os.ReadFile(path)
if err != nil {
continue // a transient read during kubelet's atomic symlink swap; next tick retries
}
level := strings.TrimSpace(string(raw))
if level == current {
continue
}
parsed, err := zerolog.ParseLevel(level)
if err != nil {
logger.Warn().Str("value", level).Msg("log level file contained an unrecognized value, ignoring")
continue
}
zerolog.SetGlobalLevel(parsed)
logger.Info().Str("from", current).Str("to", level).Msg("log level changed via mounted ConfigMap")
current = level
}
}
}Reading the file on a ticker, rather than trying to watch it with fsnotify, sidesteps a real kubelet detail: ConfigMap volume updates happen via an atomic symlink swap, which can briefly confuse a naive file-watcher expecting a simple write event.
Every value delivered via envFrom is, by design, only ever appliable through a controlled rollout — which is exactly the safety property most configuration should have. A mounted, hot-reloadable file should be reserved deliberately for the small number of settings where live tuning during an incident (raising log verbosity without a deploy) outweighs losing that rollout-gated safety.
Applied exercise
Give services/gateway-service a ConfigMap with the same checksum discipline
gateway has hardcoded its upstream timeout values directly in Go source, and a recent incident needed a quick timeout increase that required a full code change and redeploy instead of a config value.
- Write charts/gateway/templates/configmap.yaml exposing the upstream timeout as a value.
- Add the checksum/config annotation to gateway's Deployment template, referencing this new file.
- Change only the timeout value via helm upgrade --set and confirm a new rollout occurs via kubectl rollout history.
- State in one sentence why this value belongs in a ConfigMap delivered via envFrom, rather than a mounted, hot-reloadable file.
Deliverable
A committed gateway ConfigMap template, a checksum annotation wired into its Deployment, a confirmed triggered rollout, and a written envFrom-vs-mount justification.
Completion checks
- Changing only the ConfigMap-backed value via helm upgrade produces a new Deployment revision.
- kubectl exec into a new Pod shows the updated timeout value in its environment.
- The justification correctly identifies that a timeout affecting request-handling behavior should be rollout-gated, not silently hot-reloaded mid-request.
Why does patching a ConfigMap directly with kubectl patch not cause organization-api's running Pods to see the new value, when the ConfigMap is consumed via envFrom?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.