Monitoring

Which complaint metrics matter more than generic request counts?

HTTP latency and status codes are necessary, but they do not answer the questions complaint operations actually ask: are tickets moving, which categories are breaching, and how large is the open queue right now? Business metrics make the workflow observable as a workflow. complaint_transitions_total tracks movement through the state machine, complaint_sla_breaches_total makes missed promises visible by category, and complaint_open_by_category shows the size of the unresolved queue that staff is carrying.

These metrics also close the loop on earlier design choices. Because categories do not fork the state machine, transition labels stay compact and comparable across all complaint types. Because due_at and breach detection live in the complaint service, the service itself can emit authoritative SLA breach metrics instead of reconstructing them from logs later.

How do you instrument complaint metrics without leaking tenant identity into labels?

Do not put org_id, complaint_id, flat_id, or user_id in Prometheus labels. That would create unbounded cardinality and can turn the monitoring system itself into the next incident. complaint_sla_breaches_total{category} and complaint_transitions_total{from,to} stay bounded because the label sets are tiny and known ahead of time. Tenant-specific drill-down belongs in SQL or logs, not in primary metrics series.

package complaint

import (
	"context"

	"github.com/prometheus/client_golang/prometheus"
)

type OpenCountRow struct {
	Category Category
	Total    int
}

type OpenCountRepository interface {
	CountOpenByCategory(ctx context.Context) ([]OpenCountRow, error)
}

type Metrics struct {
	transitions    *prometheus.CounterVec
	slaBreaches    *prometheus.CounterVec
	openByCategory *prometheus.GaugeVec
}

func NewMetrics(reg prometheus.Registerer) *Metrics {
	m := &Metrics{
		transitions: prometheus.NewCounterVec(
			prometheus.CounterOpts{
				Namespace: "society",
				Subsystem: "complaint",
				Name:      "transitions_total",
				Help:      "Total complaint status transitions by from and to status.",
			},
			[]string{"from", "to"},
		),
		slaBreaches: prometheus.NewCounterVec(
			prometheus.CounterOpts{
				Namespace: "society",
				Subsystem: "complaint",
				Name:      "sla_breaches_total",
				Help:      "Total complaint SLA breaches by category.",
			},
			[]string{"category"},
		),
		openByCategory: prometheus.NewGaugeVec(
			prometheus.GaugeOpts{
				Namespace: "society",
				Subsystem: "complaint",
				Name:      "open_by_category",
				Help:      "Current open complaint count by category.",
			},
			[]string{"category"},
		),
	}

	reg.MustRegister(m.transitions, m.slaBreaches, m.openByCategory)
	return m
}

func (m *Metrics) ObserveTransition(from, to Status) {
	m.transitions.WithLabelValues(string(from), string(to)).Inc()
}

func (m *Metrics) ObserveSLABreach(category Category) {
	m.slaBreaches.WithLabelValues(string(category)).Inc()
}

func (m *Metrics) RefreshOpenByCategory(ctx context.Context, repo OpenCountRepository) error {
	rows, err := repo.CountOpenByCategory(ctx)
	if err != nil {
		return err
	}

	m.openByCategory.Reset()
	for _, row := range rows {
		m.openByCategory.WithLabelValues(string(row.Category)).Set(float64(row.Total))
	}
	return nil
}

Which alert and Grafana views turn raw complaint metrics into operator action?

The most useful alert is category-specific breach acceleration. A total breach counter climbing slowly over a week might be ordinary background noise; a sudden rate spike in security or plumbing means an operational system is failing right now. Grafana should then make two views obvious: a queue-size panel for open complaints by category, and a transition-rate panel that shows whether tickets are entering resolved/closed fast enough to drain the backlog.

groups:
  - name: complaint-sla
    rules:
      - alert: ComplaintSLABreachSpike
        expr: |
          sum by (category) (rate(complaint_sla_breaches_total[15m]))
            >
          3 * clamp_min(sum by (category) (rate(complaint_sla_breaches_total[6h])), 0.001)
        for: 20m
        labels:
          severity: warning
          service: complaint-service
        annotations:
          summary: "Complaint SLA breaches spiking for {{ $labels.category }}"
          description: "The 15-minute breach rate for complaint category {{ $labels.category }} is more than triple its 6-hour baseline."

• Grafana panel 1: a stacked bar or area chart of complaint_open_by_category so backlog growth is visible by complaint type, not just in aggregate.
• Grafana panel 2: rate(complaint_transitions_total{to=~"resolved|closed"}[15m]) split by destination so you can see whether work is merely being marked resolved or actually being closed by residents/admins.
• Grafana panel 3: increase(complaint_sla_breaches_total[24h]) by category beside the open queue, so a growing backlog can be correlated with missed promises.

How do you verify the complaint metrics path locally without waiting for production traffic?

Emit a few controlled transitions and then inspect /metrics. You should see bounded labels only: from, to, and category. If org_id or complaint_id appears anywhere in the metric output, fix that before deployment. Monitoring bugs are easiest to remove before they seed a high-cardinality time series in a shared Prometheus.

Which monitoring mistakes distort complaint operations even when the service code is correct?

The main failure modes are stale gauges and noisy labels. If open_by_category is only incremented and never refreshed from authoritative counts, a drained queue can still look full in Grafana. If breach metrics are labeled by org_id, Prometheus can become overloaded long before the complaint service itself does. Monitoring has its own security angle too: label cardinality is a resource-exhaustion vector whether caused by accident or abuse.