Step 01 of 06
Understand the idea
An alert only helps if it reaches someone who can respond. Alertmanager routes alerts to the right place, groups related alerts, and can hold back less urgent alerts when a larger incident is already active.
Key technical ideas
Grouping reduces duplicate panic
If ten Beacon replicas all report the same SLO burn, responders need one incident, not ten messages. Alertmanager groups alerts by labels such as alertname and service, waits briefly for related alerts, then sends a combined notification. Good grouping preserves scope while avoiding notification storms. Bad grouping hides separate incidents in one blob.
Inhibition encodes which alerts are secondary
When a high-level Beacon SLO page is firing, a dozen lower-level latency or pod alerts may add no value. Inhibition lets Alertmanager suppress those secondary notifications when labels match. It does not delete alerts; it stops duplicate messages. Use it carefully: inhibiting the only symptom alert with a flaky cause alert is how silence becomes outage camouflage.
Silences are for expected noise, not shame
A silence says: this alert is expected between these times for this label set. It should have a creator, comment and expiry. Permanent silences are abandoned policy. During maintenance, silences keep trust in the pager by making planned work explicit rather than forcing responders to mentally subtract known noise.
Annotations are written for a tired responder
At 3am, expr=(1 - ratio) > 0.0144 is not a message. A useful alert says the impact, when it started, where to look and what runbook to follow. Include dashboard and runbook links. Put the metric name in the details, not as the entire explanation. The receiver should know the first move without reading PromQL.
annotations:
summary: Beacon check success is burning budget
impact: Users may receive late or missing endpoint failure and certificate-expiry checks.
dashboard: http://localhost:3000/d/beacon-overview
runbook: docs/runbooks/beacon-slo-burn.md
first_action: Check the RED dashboard, then pause risky deploys if the burn is real.The metric is not the message. The annotation tells the responder why they were contacted and what to open first.
Alertmanager tools
| Tool | Purpose | Danger |
|---|---|---|
Route | Choose receiver | Labels too vague |
Group | Combine related alerts | Hide separate failures |
Inhibit | Suppress secondary noise | Mask useful symptoms |
Silence | Mute expected alerts | Permanent black hole |
Technical terms used at work
Receiver — A destination such as Discord, email, PagerDuty or a webhook.
Route — A label-matching branch in Alertmanager's notification tree.
Inhibition — Suppressing one alert notification while another related alert is active.
