Stage 7 · Master
Phase 17 — Observability
Dashboards
Build the one dashboard no single service's own panels can show — a cross-service view of the event pipeline Part 16 built — from metrics that already exist, without inventing a single new one.
Three Separate Dashboards, No Single View of the Pipeline Between Them
Phase 2's organization-dashboard.json answers questions about one service's own HTTP traffic. This module's pool dashboard answers questions about one service's own database connections. Neither, alone or together, answers the question an on-call engineer actually has at 2am when a customer reports a payment confirmation email never arrived: is the event pipeline connecting payment-service to notification-service healthy right now? That question spans two services and Part 16's own Kafka metrics — nothing built so far assembles them into one place.
Backlog, Lag, and Dead-Letter Rate — All Already Emitted, Never Assembled
Part 16's Lesson 6 already built notification_dlq_messages_total. Kafka itself already tracks consumer offsets, which a standard exporter can turn into a lag metric without this course writing a single line of Go for it. The only genuinely new metric this lesson introduces is outbox backlog — how many payment_outbox rows are waiting to be published — since nothing in Part 16 ever counted it.
kafka-exporter:
image: danielqsj/kafka-exporter:v1.7.0
restart: unless-stopped
command: ["--kafka.server=kafka:9092"]
ports:
- "9308:9308"
depends_on:
- kafka
networks:
- hoa-net
- job_name: kafka-lag
static_configs:
- targets: ["kafka-exporter:9308"]
package outbox
import "github.com/prometheus/client_golang/prometheus"
// BacklogGauge is a plain Gauge, not a Collector, because computing its
// value requires a database query — unlike pgxmetrics' Collect, which
// reads an in-memory struct, this must be refreshed by RelayOnce itself
// after every poll rather than on every Prometheus scrape.
var BacklogGauge = prometheus.NewGauge(prometheus.GaugeOpts{
Name: "hoa_payment_outbox_backlog",
Help: "Number of payment_outbox rows not yet published, sampled once per relay poll.",
})
func RegisterOutboxMetrics(reg prometheus.Registerer) {
reg.MustRegister(BacklogGauge)
}
func RelayOnce(ctx context.Context, pool *pgxpool.Pool, writer *kafkago.Writer, batchSize int) error {
tx, err := pool.Begin(ctx)
if err != nil {
return err
}
defer tx.Rollback(ctx)
var backlog int
if err := tx.QueryRow(ctx, "SELECT count(*) FROM payment_outbox WHERE published_at IS NULL").Scan(&backlog); err != nil {
return err
}
BacklogGauge.Set(float64(backlog))
// existing FOR UPDATE SKIP LOCKED select, publish loop, and commit
// from Part 16 Lesson 2 are unchanged below this line
The backlog count runs inside the same transaction as the row selection that follows it, so the gauge and the actual publish batch are always computed against a consistent snapshot rather than two separately-timed reads.
One New Dashboard, Assembled Entirely From Metrics That Already Existed Before This Lesson Began
{
"title": "Event Pipeline — payment-service to notification-service",
"panels": [
{
"title": "Outbox backlog (rows not yet published)",
"targets": [
{ "expr": "hoa_payment_outbox_backlog" }
]
},
{
"title": "Consumer group lag (payment.completed)",
"targets": [
{ "expr": "sum(kafka_consumergroup_lag{consumergroup=\"notification-service\",topic=\"payment.completed\"}) by (partition)" }
]
},
{
"title": "Dead-letter rate (5m)",
"targets": [
{ "expr": "sum(rate(notification_dlq_messages_total[5m])) by (error_kind)" }
]
}
]
}
hoa_payment_outbox_backlog, kafka_consumergroup_lag, and notification_dlq_messages_total were each defined for a purely local reason — a relay's own health, Kafka's own offset bookkeeping, one consumer's own failure accounting. None of them was built with this dashboard in mind. Assembling them into one view is exactly the value a cross-service dashboard adds over any single service's own panels: it costs nothing to instrument further, only the judgment to notice which existing signals, read together, answer a question no one signal answers alone.
A Flat Line at Zero and No Line At All Are Not the Same Failure
If kafka-exporter itself goes down, sum(kafka_consumergroup_lag{...}) by (partition) does not return zero — Prometheus's rate() and aggregation functions over a metric with no data points simply return no data, rendered as a gap in the panel, not a flat line at zero. Reading a gap as "lag is zero, everything is fine" instead of "the exporter itself may be down" is a real misdiagnosis risk this dashboard's viewer needs to know to distinguish, which is precisely what Lesson 7's absent()-based meta-alert exists to catch instead of relying on a human noticing a gap.
Confirm the Backlog Gauge Actually Moves When the Relay Falls Behind
Applied exercise
Confirm the dashboard shows a gap, not a zero, when kafka-exporter is unreachable
This lesson claims a missing exporter produces a gap rather than a misleading zero. See it directly in Grafana rather than trusting the PromQL explanation alone.
- Open the Event Pipeline dashboard in Grafana and confirm the Consumer group lag panel currently shows real data.
- Stop the kafka-exporter container (docker compose stop kafka-exporter) and wait roughly one scrape interval (15s).
- Refresh the dashboard and confirm the lag panel now shows a visible gap in the line, not a drop to zero.
- Restart kafka-exporter and confirm the panel resumes showing data once Prometheus successfully scrapes it again.
Deliverable
Screenshots or a written description of the panel before, during, and after the outage.
Completion checks
- The panel is clearly shown as a gap (no line) during the outage, not a line at value zero.
- The panel resumes showing real data once kafka-exporter is restarted.
Why does this lesson introduce exactly one new metric (outbox backlog) while reusing Part 16's DLQ metric and a standard Kafka exporter's lag metric unchanged?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.