Stage 4 · Provision

Observability-Driven Remediation

Job Output Observability

Export AWX job events, callback plugins, structured logs, and metrics for automation runs.

7 min readAnsible AutomationProvision

AWX Job Events

AWX records every event in a job run — task start, task complete, host play, runner on started, runner on ok, runner on changed. These events provide full observability into automation execution.

Query AWX job events


# Get job events via AWX API
- name: Fetch job events
  ansible.builtin.uri:
    url: "https://awx.example.com/api/v2/jobs/{{ job_id }}/job_events/"
    method: GET
    user: "{{ awx_user }}"
    password: "{{ awx_password }}"
    force_basic_auth: yes
    return_content: yes
  register: job_events

- name: Show task results
  ansible.builtin.debug:
    msg: "{{ item.event }}: {{ item.event_data.task }} on {{ item.event_data.host }}"
  loop: "{{ job_events.json.results }}"
  when: item.event == "runner_on_ok"

AWX job events contain detailed information about every task execution. Use them for debugging, auditing, and building observability dashboards.

Callback Plugins

Enable callback plugins in ansible.cfg


[defaults]
callback_whitelist = timer, profile_tasks, json
callback_plugins = /path/to/plugins/callback

# json callback outputs structured JSON
# Useful for CI/CD and log aggregation
stdout_callback = json

Callback plugins transform Ansible output. The json plugin outputs structured data that is easy to parse and store.

Custom callback for logging


# Custom callback plugin structure
# plugins/callback/centralized_logger.py

# The callback captures:
# - Playbook name and start time
# - Each task name and execution time
# - Host-level results (ok, changed, failed)
# - Total play recap
# - All registered variables

# Output destinations:
# - JSON file for local analysis
# - Syslog for centralized logging
# - HTTP endpoint for API ingestion
# - Kafka for streaming

Custom callback plugins let you send Ansible output to any destination. This integrates automation with your observability stack.

Structured Logging

Structured log output


# JSON callback output structure
{
  "plays": [
    {
      "play": "Configure web servers",
      "start_time": "2024-01-15T10:30:00Z",
      "hosts": {
        "web1.example.com": {
          "tasks": [
            {
              "task": "Install nginx",
              "status": "changed",
              "duration": 12.5,
              "changed": true
            }
          ]
        }
      }
    }
  ],
  "stats": {
    "web1.example.com": {
      "ok": 15,
      "changed": 3,
      "failed": 0,
      "unreachable": 0
    }
  }
}

Structured logs enable searching, filtering, and alerting on automation events. Query by host, task, status, or duration.

Metrics Export

Prometheus metrics from Ansible


# Export metrics to Prometheus
# Metrics to track:
# - ansible_playbook_duration_seconds
# - ansible_task_duration_seconds
# - ansible_tasks_total{status="ok|changed|failed"}
# - ansible_hosts_total{status="reachable|unreachable"}
# - ansible_jobs_total{status="success|failure"}

# Custom metrics endpoint
- name: Push metrics to Prometheus
  ansible.builtin.uri:
    url: "http://pushgateway:9091/metrics/job/ansible/instance/{{ inventory_hostname }}"
    method: PUT
    body: |
      ansible_playbook_duration_seconds{{playbook="deploy"}} 45.2
      ansible_tasks_total{{status="changed"}} 3
      ansible_tasks_total{{status="ok"}} 12
    headers:
      Content-Type: text/plain

Metrics enable alerting on automation failures, tracking deployment frequency, and measuring mean time to remediate.

Building Dashboards

Job success rate over time
Average playbook execution duration
Most frequently changed tasks
Host-level failure distribution
Remediation frequency by alert type
Mean time from alert to remediation

Use AWX API for dashboards

AWX exposes a REST API with all job data. Query it from Grafana, Datadog, or any dashboarding tool to build automation observability dashboards.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.