Stage 7 · Master
Security Hardening
Securing Secrets & Sanitizing Input
Move credentials out of the repo, keep them out of process dumps, and close the input bugs parameterized SQL never touched.
Why This Stopped Being a Cleanup Task
Early in Fieldwork, secrets looked like a boring configuration problem: get Postgres credentials into the gateway, get Redis credentials into the tasks service, move on. That assumption survived exactly until the first time we dumped process environment during a bad deploy and realized we had made the incident log itself sensitive. The issue was not only where secrets came from. It was where they could leak after the app started.
We rejected the easiest setup — a checked-in .env file copied into local shells and CI variables mirrored directly into container environment — because it made every debugging surface dangerous. kubectl describe, panic reports that included config, shell history, and copied support bundles all became places credentials could escape. In a multi-service system, that isn't one leak. It's a leak multiplier.
The mistake is treating env vars as a security boundary. They improve deploy-time configuration, but they still show up in crash tooling, process inspection, and habit-driven debugging. For long-lived credentials, Fieldwork moved from 'secret in env' to 'reference in env, secret in mounted file or secret store'.
Getting Secrets Out of Source Control
The decision we kept was simple: non-secret config stays in environment variables, secret material is loaded from files mounted by the runtime when possible, and only local development is allowed to fall back to direct env values. That gave us one code path that worked in Kubernetes, one obvious place to rotate credentials, and fewer reasons to ever print raw secret values during support work.
We also refused to bundle unrelated secrets into one giant blob. Gateway signing keys, Kafka credentials, and database URLs rotate on different schedules and have different blast radii. Grouping them because 'it's all secret anyway' makes every rotation larger and every RBAC rule broader than it needs to be.
apiVersion: apps/v1
kind: Deployment
metadata:
name: tasks-service
spec:
template:
spec:
containers:
- name: tasks-service
image: ghcr.io/thesyscoder/fieldwork/tasks-service:1.0.0
env:
- name: HTTP_PORT
value: "8080"
- name: REDIS_ADDR
value: fieldwork-redis:6379
- name: JWT_ISSUER
value: https://api.fieldwork.internal
- name: POSTGRES_URL_FILE
value: /var/run/secrets/fieldwork/postgres-url
- name: KAFKA_SASL_PASSWORD_FILE
value: /var/run/secrets/fieldwork/kafka-password
- name: JWT_SIGNING_KEY_FILE
value: /var/run/secrets/fieldwork/jwt-signing-key.pem
volumeMounts:
- name: app-secrets
mountPath: /var/run/secrets/fieldwork
readOnly: true
volumes:
- name: app-secrets
secret:
secretName: tasks-service-secrets
defaultMode: 0400This keeps operationally useful settings visible and sensitive values out of common env inspection paths. The runtime still receives the secret; the difference is who can casually see it later.
| Decision | Rejected alternative | Why we rejected it |
|---|---|---|
| Mounted files or secret store references for long-lived credentials | Everything in plain env vars | Too easy to leak through support bundles, process inspection, and habit-based debugging |
| Separate secrets per concern | One global FIELDWORK_SECRETS blob | Rotation and RBAC become broad, risky, and impossible to reason about during incidents |
| Local env fallback only for development | Force production-like secret tooling everywhere | Developer setup cost becomes high enough that people start bypassing the intended path |
Sanitizing the Request Boundary
Secrets handling solved only half the lesson. The other half was admitting that 'we use parameterized queries' was not an input-safety strategy. Fieldwork accepts tenant-scoped JSON requests from browsers, internal service calls from the gateway, webhook payloads, and query parameters used for sorting and filtering. The dangerous part is not only SQL literals. The dangerous part is every field that changes program behavior.
We kept validation and normalization at the edge of each service: parse once, trim once, reject impossible states immediately, and pass a typed command deeper into the application. We rejected the 'sanitize later in the repository' approach because it lets invalid state survive across handlers, auth checks, and logging before anyone decides whether it was acceptable.
Fieldwork does not strip Markdown or HTML-looking text out of task descriptions at write time because storage should preserve user intent. What we do sanitize at write time is structural risk: impossible lengths, control characters in names, duplicate labels, invalid enums, and fields that would alter query behavior. Output escaping still belongs at the renderer.
The Injections Parameterized Queries Don't Stop
Parameterized values close the obvious SQL injection hole in WHERE task_id = $1. They do not protect identifier injection, unbounded sort keys, or application-level behavior switches. The easiest Fieldwork example was task listing: users could choose sort order, direction, and a search term. If those became string concatenation, the query was still injectable even while all literal values used placeholders.
| Risk | Why parameters don't solve it | What Fieldwork does instead |
|---|---|---|
| ORDER BY / direction injection | Database placeholders cannot bind identifiers or SQL keywords | Map client-facing fields to fixed server-owned columns and directions |
| LIKE wildcard abuse | The parameter is safe syntactically but still changes query breadth and cost | Escape % and _, cap lengths, and keep tenant/project filters mandatory |
| Mass assignment | A placeholder won't stop a client from setting a field they should never control | Decode into request-specific DTOs, then copy only allowed fields into commands |
| Log injection | Parameterized SQL says nothing about multiline user input landing in logs | Normalize control characters before logging titles, labels, and actor metadata |
What We Shipped and What We Refused
The final rule set was opinionated on purpose. Secrets are references, not literals, in deploy manifests. Config loaders know which fields are sensitive. Handlers normalize input into commands before business logic runs. Repositories never accept raw client sort fields, raw SQL fragments, or unconstrained search behavior. Those rules are repetitive, but repetition is cheaper than one exploit or one leaked support bundle.
- Treat secret handling as an observability problem as much as a storage problem: ask where values can leak after process start.
- Keep secret rotation units small. Different systems should not share a single mega-secret just because they are all sensitive.
- Normalize at the edge: trim, bound, parse, and de-duplicate before auth, logging, and persistence scatter the bad input around the service.
- Audit every user-controlled field that changes query behavior. Parameterization closes only one class of injection.
The actual decision was narrower and more useful: Fieldwork stores secrets outside source control and outside routine env inspection paths, then treats every request field that can alter behavior as a thing to constrain explicitly. That is concrete enough to implement, review, and keep consistent across services.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.