thesyscoder / writing
Field notes
Engineering notes and case studies from the workbench.
- Platform Engineering 7 min read
Graceful Shutdown Isn't Optional: The Rolling Deploy That Dropped 3% of Requests
Every deploy, every day, for months, we silently dropped a small percentage of real requests — and nobody noticed until we actually measured it, because 3% doesn't look like an outage.
Kubernetesgraceful shutdownrolling deploys - Platform Engineering 7 min read
The Day a `kubectl drain` Took Down Payments (and the PDB We Didn't Have)
A routine node drain during a cluster upgrade evicted every replica of a critical service within seconds of each other, because nothing told Kubernetes it wasn't allowed to.
KubernetesPodDisruptionBudgetnode maintenance - Backend 8 min read
The Idempotency Bug That Only Showed Up Under Retries
Our payments API had an idempotency key on every write. It still double-charged a customer, because the bug wasn't in the idea — it was in the four-line race condition between checking the key and writing the result.
idempotencyrace conditionsdistributed systems - SRE 9 min read
Multi-Window Multi-Burn-Rate Alerts, Explained With the Incident That Made Me Actually Understand Them
Simple threshold alerts either page you for nothing or miss real outages. Here's the incident that forced me to actually understand burn-rate math instead of copy-pasting a Prometheus rule.
SLOerror budgetsalerting - SRE 7 min read
The 3am Page That Taught Me Alerts Need Runbooks, Not Just Thresholds
The alert fired correctly. The threshold was right. I still spent 40 minutes at 3am guessing, because nobody had written down what to actually do about it.
on-callrunbookstoil - DevOps 8 min read
Why Our CI Pipeline Took 40 Minutes and What We Actually Cut to Get to 8
No magic caching trick fixed this. It was four separate, boring bottlenecks — and finding them required actually timing every step instead of guessing.
CI/CDGitHub Actionsbuild performance - DevOps 8 min read
The Rollback That Made It Worse: A Postmortem on Assuming Rollback = Safe
We reverted a bad deploy in 4 minutes, exactly like the runbook said. That's when the real outage started, because the deploy had already changed the database schema and the old code couldn't read the new shape.
deploymentsdatabase migrationsrollback - Backend 8 min read
Connection Pool Exhaustion: A Case Study in a 2am Death Spiral
One slow downstream dependency turned into a full outage in six minutes — not because of the slowness itself, but because of what a fixed-size connection pool does when every connection is stuck waiting.
connection poolingdatabasescascading failure