Retry Strategies

Why Retry?

Transient failures — network blips, momentary overload, connection resets — are common in distributed systems. Retrying the failed operation often succeeds. However, retries can amplify load and cause cascading failures if not implemented carefully.

Exponential Backoff

Exponential backoff increases the delay between retries exponentially: 1s, 2s, 4s, 8s, 16s. This gives the downstream service time to recover. Without backoff, retries hit the struggling service at the same rate, making recovery impossible.

import time
import random

def retry_with_backoff(func, max_retries=3, base_delay=1.0, max_delay=30.0):
    for attempt in range(max_retries + 1):
        try:
            return func()
        except (ConnectionError, TimeoutError) as e:
            if attempt == max_retries:
                raise
            
            # Exponential backoff with jitter
            delay = min(base_delay * (2 ** attempt), max_delay)
            jitter = delay * random.uniform(0.5, 1.0)  # 50-100% of delay
            time.sleep(jitter)

Adding Jitter

# Full jitter (recommended by AWS)
delay = random.uniform(0, min(cap, base * 2 ** attempt))

# Equal jitter
temp = min(cap, base * 2 ** attempt)
delay = temp / 2 + random.uniform(0, temp / 2)

# Decorrelated jitter
delay = min(cap, random.uniform(base, prev_delay * 3))

Retry Budgets

A retry budget limits the total number of retries per second. Without a budget, retries can double the load on a struggling service. A common budget: retries cannot exceed 10% of normal request volume.

retry_budget:
  budget_percent:
    value: 20.0    # Max 20% of requests can be retries
  min_retry_concurrency: 3  # At least 3 concurrent retries allowed

When NOT to Retry

• Non-idempotent operations — POST, DELETE, PUT without idempotency keys.
• 4xx errors (except 429) — Client errors are not transient.
• Business logic failures — Payment declined, insufficient inventory.
• Authentication failures — Retrying with the same credentials will fail.

Retry Configuration