Prometheus Metrics & Health Checks

We Measured the Shape of Work

The first temptation with metrics is to instrument everything visible and hope dashboards will tell you what matters later. We did not do that in Fieldwork because uncurated metrics become a tax: more cardinality, more storage, more confusion, and still not enough signal during an incident. Instead we started from the questions operators would ask. How many requests hit each service? How long do the critical routes take? Which routes are failing, and is the failure local or downstream? Is the service ready to receive production traffic right now? Those questions shaped the metric set.

We rejected label-heavy, payload-derived metrics early. In a multi-tenant API, it is tempting to label every counter by tenant_id or workspace_id because those dimensions feel operationally important. They are important for logs and traces; they are disastrous for Prometheus cardinality. A metric that explodes by tenant, workspace, project, task, and user is not observability. It is a denial-of-storage attack you launched against yourself. So Fieldwork kept metrics low-cardinality and used logs or traces when the investigation needed per-tenant detail.

Health Checks Need Real Dependency State

Health endpoints were another area where we rejected the easy version. A liveness check that returns 200 because the process is not dead tells Kubernetes almost nothing about whether the service can handle traffic. A readiness check that only says "HTTP listener started" is equally misleading. Fieldwork's task service is not ready if it cannot borrow a Postgres connection or if its membership dependency is in a failed-open breaker state. Saying ready in that condition just moves failure from startup to user traffic.

So we split concerns cleanly. Liveness asks whether the process should be restarted at all. That stays shallow because a dependency outage should not cause restart loops. Readiness asks whether this instance should receive traffic right now. That probes dependencies with short budgets and returns degraded state when the service cannot do useful work. Startup checks, where needed, guard slow initialization so readiness does not flap during boot. The cost of this design is a little more code; the payoff is that deployment and autoscaling decisions are based on reality instead of on an empty 200.

Instrumentation Lives at the Edges

We also learned to keep most instrumentation at boundaries: HTTP middleware, repository wrappers, and worker loops. That gives stable, reusable measurement points without contaminating every inner function with counter increments. The task service itself should talk about business rules. The repository wrapper can record query latency. The outbox worker can record batch size, publish attempts, and lag. Instrumenting every helper function created a lot of numbers and very little understanding. Instrumenting edges made the measurements composable.

Where we did add domain metrics, we kept them intentional. For example, task creations_total by result was useful because it tied directly to a product workflow and helped compare traffic patterns against request failure rates. But we still kept the label set tight: service, result, and maybe source path. Not actor, not task title, not workspace ID. The discipline here was to ask whether the label improves a dashboard or just satisfies curiosity in the moment.

What Went on the Dashboard

The final dashboard for each Fieldwork service ended up small on purpose. At the top: request rate, error rate, and p50/p95/p99 latency by stable route. Then dependency panels: Postgres pool usage, readiness failures by dependency, breaker open counts where relevant, and outbox backlog for async workers. Finally, business-adjacent counters like task creations_total and membership authorization denials. We did not try to make one dashboard answer every forensic question. It exists to tell an operator whether the service is healthy, what is getting worse, and which subsystem deserves the next click.

That is why the decision here was narrower than "add Prometheus." We standardized low-cardinality service metrics, made readiness depend on actual ability to serve traffic, and kept liveness shallow enough to avoid self-inflicted restart storms. In other words: metrics for trend detection, health checks for traffic control, and enough discipline that the numbers still mean the same thing a month later.

• Count requests and measure latency by stable route template, method, and status class.
• Keep label sets bounded; use logs or traces for high-cardinality identifiers like tenant_id.
• Make readiness reflect critical dependency state, not just process existence.
• Keep liveness shallow so dependency outages do not turn into restart loops.
• Instrument boundaries and a few high-value domain workflows instead of every helper function.