Structured Logs & Error Taxonomies

Logs Are for Queries, Not Reading Top to Bottom

Fieldwork's earliest logs were readable in the way a small side project log is readable: nice sentences, a few IDs sprinkled in, and just enough context for one engineer watching one terminal. That style collapsed the moment requests began bouncing through the gateway, task service, membership service, and async workers. Production debugging is not reading one stream from top to bottom. It is asking targeted questions like "show me all authorization failures for tenant_42 in the last fifteen minutes" or "group create-task latency spikes by upstream route and status class." Free-form text is hostile to that kind of work.

So we made an opinionated choice: every request-path log in Fieldwork is structured first and human-friendly second. We rejected the alternative of logging plain English strings with embedded IDs because it scales poorly under cardinality and drift. One handler writes tenant_id, another writes tenant, a third writes org_id because the author came from a different codebase, and suddenly your logs are technically present but operationally useless. Queryability depends on consistent keys more than witty message text.

A Small Error Taxonomy Beats Free-Form Panic

Structured logs alone did not solve another problem: every failure still looked semantically different. A missing project, an expired request deadline, a broken downstream dependency, and a malformed payload all surfaced as arbitrary wrapped errors. Developers understood them locally, but operations could not reliably group them. We did not want a giant enterprise error framework, so we chose a smaller thing: a handful of stable error kinds that reflect how the system should respond, not every possible root cause string.

That distinction matters. We rejected taxonomies based on package names or exception ancestry because they mirror code layout rather than production meaning. Fieldwork's taxonomy groups failures into kinds like invalid_argument, not_found, forbidden, conflict, dependency_unavailable, and internal. Those kinds drive HTTP translation, log level, and alerting. A Postgres unique violation and a domain-level duplicate membership request can both become conflict if that is the behavior the caller needs to see.

How Handlers Translate Errors Into Logs

The next decision was where logging should happen. We rejected "log every error at every layer" almost immediately because it creates duplicates with different context and different wording. One failed membership lookup turned into four entries: client error, service error, handler error, and middleware error. That did not add observability; it added noise. Fieldwork's rule became log once at the boundary that has the full request context and is actually making a user-visible decision. Usually that is the HTTP handler or the async worker loop. Inner layers return wrapped errors with operation names and kinds; they do not all log.

That boundary decision also let us tie log level to behavior. invalid_argument and not_found stay at info because they are expected user-space outcomes. forbidden is typically warn because it may indicate misuse or an attack path worth review. dependency_unavailable and internal reach error because they represent system degradation. The important point is consistency: if severity is based on how annoyed a developer felt while writing the message, filters become meaningless.

What We Decided to Log and Skip

The final policy was deliberately constrained. We always log request_id, trace_id, tenant_id when present, workspace_id when present, route, method, status_code, duration, and error_kind for failures. We do not log entire request bodies, access tokens, or free-form SQL fragments because they create security and cost problems with little diagnostic value. For task titles and descriptions we log IDs, counts, and sizes rather than content. That keeps logs safe enough to centralize and cheap enough to retain.

The bigger lesson is that observability quality came from consistency, not volume. Once logs were structured and errors were classified by behavior, finding all dependency_unavailable failures for one tenant during a deploy became a simple query instead of an archaeology session. That is why this lesson's decision was not "use Zap" or "prefer JSON." The decision was to treat logs and errors as operational data models with stable fields and stable meaning.

• Standardize field names for identifiers like tenant_id, workspace_id, project_id, request_id, and trace_id.
• Classify errors by operational behavior, not by package origin or raw message text.
• Log once at the boundary that can attach route, tenant, trace, and final status information.
• Use log level consistently: expected client failures are lower severity than dependency or internal failures.
• Never put secrets or full request payloads into centralized logs just because they were convenient during local debugging.