Project challenges / verified progress
Beacon: run it like an SRE

The engineering notebook

See the running system

How do you know what a running system is doing before it wakes you up?

Loading Vault account...
Phase 1 — Make Beacon visible

Step 01 of 06

Understand the idea

Beacon can check other websites, but it does not yet report its own health. This course adds metrics, dashboards, alerts, and incident practices so an operator can understand how Beacon is working.

ONE EVENT, THREE SIGNALSBeacon requestone check handledLogswhy this one failedrich, priceyMetricshow many and how fastcheap, aggregateTraceswhere time wentsampled path
The same request can leave three kinds of evidence. Logs preserve details, metrics preserve cheap totals, and traces preserve causality across components. Keeping those jobs separate prevents the classic mistake: trying to make Prometheus a log database with worse search.
Step 01

Key technical ideas

Monitoring starts with questions you already know

Monitoring checks known conditions, such as whether a process is running or errors are increasing. Observability helps you investigate a problem you did not predict, such as slow requests in one region. Services need both: monitoring can alert you, and logs, metrics, and traces can help explain what happened.

Metrics summarize groups of requests

A metric such as beacon_checks_total{result="ok"} stores a running count for that label set. It does not store a separate record for every request. This keeps metrics compact, but they cannot tell you the details of one specific request. Use logs or traces for that. Metrics are best for questions about groups of requests, such as rate, error ratio, and latency.

Metric labels can create too many time series

Every unique combination of metric labels creates a separate time series. If a metric has 5 regions and 4,000 endpoints, it can create 20,000 series. Adding a label with a different value for every user can create far more. This growth is called cardinality. Avoid labels with many possible values, such as user IDs.

Prometheus collects metrics by scraping

Prometheus usually collects metrics by sending an HTTP request to each target; this is called scraping. If a scrape fails, Prometheus records that the target is down. Short-lived jobs may need to push their metrics because they finish before the next scrape. Beacon runs continuously, so scraping is a good fit.

Cardinality arithmetic on a shop checkout
metric: checkout_requests_total
labels: method=2, status=5, route=12
series: 2 * 5 * 12 = 120

add store_id=800
series: 120 * 800 = 96,000

add customer_id=2,000,000
series: 96,000 * 2,000,000 = 192,000,000,000

The metric name did not change. The code may have added one innocent label. The storage bill and query cost changed by nine orders of magnitude. That is why SREs ask about cardinality before they ask about dashboard colours.

The three signals do different work

SignalBest atCost trap

Logs

One event with context

Higlab volume and retention

Metrics

Trends, ratios, alerts

High-cardinality labels

Traces

Cross-service latency paths

Sampling and storage cost

Profiles

CPU and memory hotspots

Usually not always-on

Technical terms used at work

  • Time series — One stream of samples for one metric name and one exact set of labels.

  • Cardinality — The count of distinct label combinations a metric can produce.

  • Scrape — One Prometheus HTTP collection attempt against a target's metrics endpoint.

  • Known-unknown — A question you expected to ask, such as whether error rate is above 1%.