Grafana

A Dashboard Built Through Grafana's UI Lives Only Inside That Container

Phase 2 already authored organization-dashboard.json as a plain JSON file, anticipating a future Grafana instance without ever standing one up. The tempting next step — opening Grafana's UI, clicking through panel editors, and saving — would recreate exactly the failure mode this course has avoided everywhere else: a fact that lives only inside a running container's database, gone the moment that container is recreated, invisible to code review, impossible to diff. Grafana's provisioning system reads datasources and dashboards from files on disk at startup instead, so the entire Grafana configuration is exactly as reviewable and disposable as organization-alerts.yml already is.

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

apiVersion: 1

providers:
  - name: organization-platform
    folder: ""
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/provisioning/dashboards/json

Phase 2's organization-dashboard.json moves into this directory unchanged — no content in that file needs to change, only its location, so it is picked up by the provider above instead of waiting to be imported by hand into a Grafana that never existed until this lesson.

The One Panel Phase 2 Could Not Have Written

Phase 2's four panels — request rate, latency, error-budget burn, replica count — predate this module's hoa_organization_postgres_pool_* metrics entirely; there was nothing for a fifth panel to graph. This lesson adds exactly that panel, as its own dashboard file rather than a fifth panel bolted onto Phase 2's existing organization-dashboard.json, since pool pressure is a distinct operational question ("is the database connection pool the bottleneck") from anything the original four panels answer.

{
  "title": "Organization Service — Postgres Pool",
  "panels": [
    {
      "title": "Acquired / Max connections",
      "targets": [
        { "expr": "hoa_organization_postgres_pool_acquired_connections" },
        { "expr": "hoa_organization_postgres_pool_max_connections" }
      ]
    },
    {
      "title": "Acquired / Max ratio",
      "targets": [
        { "expr": "hoa_organization_postgres_pool_acquired_connections / hoa_organization_postgres_pool_max_connections" }
      ]
    },
    {
      "title": "Cumulative acquire wait time (rate, 5m)",
      "targets": [
        { "expr": "rate(hoa_organization_postgres_pool_acquire_duration_seconds_total[5m])" }
      ]
    }
  ]
}

docker-compose.yml Gains a Grafana Service, Mounting the Whole Provisioning Tree

  grafana:
    image: grafana/grafana:11.2.0
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_PASSWORD: hoa_grafana_dev_password
      GF_AUTH_ANONYMOUS_ENABLED: "false"
    volumes:
      - ./deploy/observability/grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
    networks:
      - hoa-net

No Dashboard Variable Ever Filters By Organization

Grafana supports template variables that populate a dropdown from a live Prometheus label query — a tempting way to let an operator 'pick an organization' and filter every panel to it. Every metric this module and Phase 2 together expose is deliberately unlabeled by org_id specifically to keep cardinality bounded; building a variable that queries for organization values would either return nothing (the label doesn't exist) or, worse, encourage someone to add an org_id label to a metric later just to make the dropdown work, reintroducing the exact unbounded-cardinality problem Phase 2's Metrics lesson and this module's Prometheus lesson both rule out. Investigating one specific organization's incident is Lesson 5's job (Jaeger, searchable by trace attribute), not a dashboard's.

Confirm Both Dashboards Load Automatically, With No Manual Import Step