Stage 4 · Provision
Observability-Driven Remediation
Rollback Playbooks
Design reversible tasks with backups, handlers, versioned templates, and tested restore procedures.
Designing Reversible Tasks
A rollback is only possible if you know what changed. Every remediation playbook should record its changes and provide a way to undo them. Design tasks to be reversible from the start.
- name: Record pre-change state
ansible.builtin.set_fact:
rollback_data:
previous_version: "{{ current_version | default('unknown') }}"
changed_files: []
timestamp: "{{ ansible_date_time.iso8601 }}"
host: "{{ inventory_hostname }}"
- name: Deploy new config
ansible.builtin.template:
src: app.conf.j2
dest: /etc/app/config.yml
register: config_result
- name: Record changed file
ansible.builtin.set_fact:
rollback_data: "{{ rollback_data | combine({'changed_files': rollback_data.changed_files + [config_result.dest]}) }}"
when: config_result.changed
- name: Save rollback data
ansible.builtin.copy:
content: "{{ rollback_data | to_nice_json }}"
dest: "/var/log/ansible/rollback-{{ ansible_date_time.iso8601_basic_short }}.json"
mode: "0600"Recording changes before making them ensures you can always undo. Store rollback data alongside the automation logs.
Backup Strategy
- name: Backup before changes
ansible.builtin.copy:
src: "{{ item }}"
dest: "{{ backup_dir }}/{{ ansible_date_time.iso8601_basic_short }}/{{ item | basename }}"
remote_src: yes
loop:
- /etc/app/config.yml
- /etc/nginx/nginx.conf
- /etc/systemd/system/myapp.service
register: backup_result
- name: Create backup manifest
ansible.builtin.copy:
content: |
{
"timestamp": "{{ ansible_date_time.iso8601 }}",
"host": "{{ inventory_hostname }}",
"files": {{ backup_result.results | map(attribute='dest') | list }},
"version": "{{ deploy_version }}"
}
dest: "{{ backup_dir }}/{{ ansible_date_time.iso8601_basic_short }}/manifest.json"Back up every file before modifying it. The manifest records what was backed up, when, and from which version.
Versioned Templates
# Directory structure for versioned configs
# templates/
# v1/
# app.conf.j2
# nginx.conf.j2
# v2/
# app.conf.j2
# nginx.conf.j2
- name: Deploy versioned config
ansible.builtin.template:
src: "templates/v{{ deploy_version }}/app.conf.j2"
dest: /etc/app/config.yml
notify: Restart app
# Rollback uses previous version
- name: Rollback config
ansible.builtin.template:
src: "templates/v{{ rollback_version }}/app.conf.j2"
dest: /etc/app/config.yml
notify: Restart appVersion templates alongside code. Rollback means deploying the previous version's template, not guessing what changed.
Rollback Playbook
---
- name: Rollback application
hosts: "{{ target_hosts }}"
serial: 1
max_fail_percentage: 0
vars:
rollback_version: "{{ lookup('file', '/var/log/ansible/latest-rollback.json') | from_json | json_query('previous_version') }}"
tasks:
- name: Show rollback info
ansible.builtin.debug:
msg: "Rolling back from {{ current_version }} to {{ rollback_version }}"
- name: Deploy previous version
ansible.builtin.git:
repo: https://github.com/app/repo.git
dest: /opt/app
version: "{{ rollback_version }}"
- name: Restore previous config
ansible.builtin.copy:
src: "{{ backup_dir }}/{{ latest_backup }}/config.yml"
dest: /etc/app/config.yml
remote_src: yes
- name: Restart application
ansible.builtin.service:
name: myapp
state: restarted
- name: Health check
ansible.builtin.uri:
url: http://localhost:8080/health
status_code: 200
retries: 5
delay: 10The rollback playbook restores the previous version and configuration. It is the safety net for every deployment.
Testing Rollback Procedures
---
- name: Test rollback
hosts: all
become: yes
tasks:
- name: Deploy v2
ansible.builtin.git:
repo: https://github.com/app/repo.git
dest: /opt/app
version: v2
- name: Verify v2 running
ansible.builtin.command: /opt/app/version
register: result
assert:
that: result.stdout == "v2"
- name: Rollback to v1
ansible.builtin.git:
repo: https://github.com/app/repo.git
dest: /opt/app
version: v1
- name: Verify v1 running
ansible.builtin.command: /opt/app/version
register: result
assert:
that: result.stdout == "v1"Test rollback procedures in Molecule. If rollback does not work, it is not a rollback — it is a hope.
Schedule regular rollback drills. Deploy a version, then roll it back. If you cannot roll back cleanly in staging, you cannot do it in production.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.