❯ war stories · true stories and on-call nights

War stories.

What really happened when real systems hit real limits, in two minutes each, with the sources. Then take the page yourself: incidents played out in the channel, modelled on real postmortems, where you make the calls.

read the guide →

80 real stories · 30 with a short video · 29 war rooms

▶ swipe through 80 shorts

real stories · two minutes eachIt really happened

Money · 37signals2 min37signals left the cloud and cut its yearly bill from $3.2 million to $1.3 millionThe makers of Basecamp and HEY bought about $700,000 of servers and, by their own math, got it back within a year.2022–2024▶ 3-min video
Outage · GitHub2 minA 43-second network blip gave GitHub 24 hours of troubleThe link came back in 43 seconds. By then, GitHub's databases had moved across the country, and couldn't safely move back.2018▶ 3-min video
Strange but true · Boeing2 minA Boeing 787 left powered on for 248 days could lose all AC power, so the FAA ordered a restartA counter inside the 787's generator controllers would overflow after 248 days. The FAA's fix: switch the plane off and on again.2015
Money · Amazon Web Services2 minA brand-new S3 test bucket ran up a $1,300 bill in a dayA developer made a storage bucket for a quick test. Strangers' machines sent it nearly 100 million requests, and he got the bill.2024▶ 3-min video
Outage · Amazon Web Services2 minA brief network blip pushed DynamoDB's error rate to 55%, because its answers had crept up to a timeoutA popular new feature quietly made DynamoDB's metadata answers bigger, until they barely fit the timeout. Then the network blinked.2015
People · Atlassian2 minA cleanup script deleted 883 Atlassian sites, and some took 14 days to come backThe script was meant to remove one old app. It was handed the IDs of whole customer sites, and deleted them in 23 minutes.2022▶ 3-min video
Live migration · Netflix2 minA corrupted database pushed Netflix into a seven-year move to the cloudIn 2008 Netflix couldn't ship DVDs for three days. Its answer took seven years and rebuilt almost all of its technology.2008–2016▶ 3-min video
People · Capital One2 minA misconfigured firewall let an outsider copy Capital One data on about 106 million peopleThree commands: get a cloud account's keys, list the buckets, copy them. A stranger's email months later raised the alarm.2019
Outage · US Army (Patriot)2 minA Patriot battery missed a Scud because it had been running for 100 hoursIts clock error grew every hour it stayed on. After 100 hours, the radar looked about 687 metres away from the incoming missile.1991
Money · Amazon Prime Video2 minA Prime Video team cut a tool's cost by 90% by turning it into a monolithNot all of Prime Video, just one monitoring tool, whose pieces turned out far cheaper living together in one program.2023▶ 3-min video
Outage · Google Cloud2 minA routine maintenance job switched off the brains of Google's network in several places at onceMaintenance in one Google data center stopped the network's control software in other locations too. Minutes later, routes began to vanish.2019
Outage · Datadog2 minA security update ran at the same hour in every Datadog region, and took them all downDatadog's regions shared no network and ran on three clouds. They still fell together, at 06:00 UTC, to one automatic OS update.2023

showing 12 of 80

war room · you're on callTake the page

Fictional company and people; each night is modelled on a real public postmortem, named at the end.all 29 war rooms →

29 rooms

showing 6 of 29