Stage 6 · Operate
Toil Reduction Strategies
Identifying Toil
Manual, repetitive, automatable, scales with service — the four criteria that define toil.
What Is Toil?
Toil is manual, repetitive work that provides no lasting value. It scales linearly with service growth — more services means more toil. It is the opposite of engineering work, which creates durable, scalable solutions. The goal of SRE is to reduce toil to under 50% of time spent.
If you have 10 services and each requires 1 hour of weekly maintenance, you have 10 hours of toil per week. When you grow to 100 services, you have 100 hours. Toil grows exactly as fast as your service footprint. Automation scales flat.
The Four Criteria
A task is toil if it meets all four criteria. If it fails any one criterion, it is not toil — it may be valuable operational work or engineering.
- Manual — Requires human intervention. A human must trigger, execute, or verify the task.
- Repetitive — Occurs regularly. If it happens once a year during audits, it may not be toil.
- Automatable — Could be done by a machine. If it requires human judgment, it is not toil.
- Scales linearly — More services or traffic means more of this task. It does not scale sublinearly.
Common Toil Examples
Toil hides in common operational tasks. Identifying it is the first step toward elimination. Here are the most common sources of toil in SRE teams.
toil_inventory:
- task: "Manual certificate renewal"
frequency: "Every 90 days"
time_spent: "2 hours per renewal"
criteria:
manual: true
repetitive: true
automatable: true
scales_with_services: true
toil_score: 4/4
- task: "Responding to disk space alerts"
frequency: "Weekly"
time_spent: "30 minutes per alert"
criteria:
manual: true
repetitive: true
automatable: true
scales_with_services: true
toil_score: 4/4
- task: "Capacity planning review"
frequency: "Monthly"
time_spent: "4 hours per review"
criteria:
manual: true
repetitive: true
automatable: false # Requires judgment
scales_with_services: true
toil_score: 3/4What Is Not Toil
Many valuable SRE activities are not toil. Debugging novel issues, designing systems, writing automation, mentoring teammates — these are engineering work. They create lasting value and scale sublinearly.
| Toil | Not Toil |
|---|---|
| Restarting a service after a crash | Building a self-healing controller |
| Scaling a deployment manually | Implementing autoscaling |
| Responding to repetitive alerts | Tuning alerts to reduce noise |
| Manually updating configurations | Building a configuration pipeline |
| Rotating secrets by hand | Implementing automated secret rotation |
Some tasks should not be automated. Capacity planning, architecture decisions, and incident investigation require human judgment. Focus automation on tasks that are clearly toil — manual, repetitive, automatable, and scaling linearly.
Tracking Toil
You cannot reduce what you do not measure. Track toil systematically by logging time spent on operational tasks. Use a simple spreadsheet or issue tracker to categorize time as engineering, toil, or meetings.
categories:
engineering:
- "Building automation"
- "Designing systems"
- "Writing documentation"
- "Improving reliability"
toil:
- "Manual restarts"
- "Certificate renewals"
- "Config updates"
- "Responding to repetitive alerts"
- "Log rotation"
- "Backup verification"
operational:
- "Incident response"
- "Postmortem reviews"
- "On-call rotations"
- "Capacity reviews"Toil Budget
Set a toil budget for your team. The Google SRE book recommends under 50% of engineering time spent on toil. Track this metric monthly. When toil exceeds the budget, prioritize automation over feature work.
# Track time spent on toil vs engineering
# Using a simple counter that team members increment
groups:
- name: toil_tracking
rules:
# Toil ratio over 30 days
- record: team:toil:ratio
expr: |
sum(increase(toil_time_hours_total[30d]))
/
sum(increase(total_work_hours_total[30d]))Create a toil dashboard that shows time spent on toil vs engineering. Make it visible to leadership. When leadership sees 60% of engineering time going to toil, automation gets prioritized.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.