The loop — a procedure you can run frightened
Every class before this one assumed the systems were up. This one begins at 2:14 on a Saturday afternoon when they are not, and it begins with a physiological fact: under an outage, with revenue burning and someone senior typing "???" into the channel, your working intelligence drops — sharply, measurably, for everyone. You cannot fix that. What you can do is what aviation and medicine did — replace improvisation with a loop so practised it survives the adrenaline. Incident response is four verbs in a fixed order: detect, triage, mitigate, learn — and most of the damage done during real incidents comes from running them in the wrong order, usually by diving for root cause while the store is still dark.
- Detect
- The Class 30 machinery notices before a customer does: the availability test fails, the alert fires, the action group pages a named owner. If detection arrived by angry email, that is the first postmortem finding, logged before anything else happens.
- Triage
- Three questions, five minutes, no heroics: what is actually broken for whom (symptom, not cause)? how wide (one store, one region, everyone)? what changed (the question §2 answers)? Output: a severity call and a declared incident with one owner.
- Mitigate
- Make it stop hurting by the fastest safe path — roll back, scale out, fail over, flip the flag. Explicitly not the same as fixing it, which is §3's whole argument.
- Learn
- The postmortem (§5): what the timeline was, which guardrail was missing, what gets built so this class of failure retires. The only phase with no time pressure — and the one most often skipped, which is why the same incident happens twice.
Two rules make the loop run under pressure. First, one incident, one commander: somebody — not necessarily the most senior person, and on a Saturday it is whoever the action group paged — owns the incident, makes the mitigation calls, and is the single voice in the status channel. Committees debug nothing. Second, write everything down as it happens — a running log in the incident channel, timestamped, ugly. At 2:30 it feels like bureaucracy; at the Wednesday postmortem it is the only honest witness, because memory under adrenaline is fiction with confidence.
Reading the estate under pressure — what changed?
Nearly every incident that is not hardware weather is a change — a deploy, a config edit, a scale event, a certificate quietly expiring, a dependency shifting under you. So triage's sharpest question is "what changed in the last hour?", and Azure keeps a ledger that answers it: the Activity Log, the control-plane record of every create, update, delete, and role assignment in the subscription — who, what, when, from where — kept automatically, whether or not you configured anything.1 It is Class Sixteen's ARM layer writing its diary, and at 2:20 on a Saturday, filtering it to the last two hours is worth more than any dashboard: deploys, setting changes, and scaling operations line up against the moment the error rate bent.
The data-plane story — what the application is doing — is the Class Twenty-Eight-to-Thirty toolkit read at speed. The discipline under pressure is to check a fixed shortlist rather than wander: the availability test (is it still failing?), the failed-request chart and its bend time, Live Metrics for the right-now pulse, and one Class Twenty-Nine query — errors by operation, last hour, binned by minute — to see which operation broke and precisely when. Bend time from telemetry plus change ledger from the Activity Log is the whole diagnostic method: find when it bent, then find what changed just before. It resolves a remarkable share of incidents in minutes, and it is why engineers who cannot write KQL from memory still survive Saturdays — the two views they need are bookmarked from the Class Thirty dashboard.
Mitigate first, root-cause second
Here is the instinct this section exists to overwrite: you found the suspicious deploy, and every engineering cell in you wants to know why it broke — open the diff, read the code, understand. Meanwhile the store is dark at hundreds of dollars a minute — November Saturday prices. The professional move is the one that feels like defeat: roll it back without understanding it. Understanding is Wednesday's job, at a desk, with coffee. Saturday's job is to make the harm stop by the fastest reversible path — and Phase Three built you an armoury of them: the pipeline redeploys the previous version (Class Twenty-Two), the slot swaps back (Class Eleven), the scale set adds instances (Class Two's elasticity, remembered under stress), the feature flag flips off.
Saturday restores service. Wednesday explains it.
The corollary rule guards the other flank: mitigation must be reversible and singular. One change at a time, watch the metrics, write it in the log. The classic secondary disaster — a routine outage becoming a long one — is almost always a mitigation stampede: three people changing four things at once, no record, and now the estate is in a configuration nobody has ever seen before, with the original fault still in it somewhere. The Class Twenty lesson holds at 2:30 as it held in the pipeline: state you cannot explain is worse than a fault you can see. And know the moment to stop mitigating: when the reversible options are exhausted and the fault points at the platform itself, you open the support case and check the Azure status feed — Class One's line, remembered mid-fire: some of the stack is Microsoft's half, and no amount of your adrenaline fixes their region.2
Communication — the update cadence
The second system that fails during an incident is not technical: it is the channel filling with "any update?" — each one a context switch for the person doing the work, each unanswered minute breeding another. The fix is mechanical, and it is the commander's job, not the debugger's: a status update on a fixed clock, whether or not anything changed. "No change, next update 3:10" is a real update; it is the silence that costs you, because leadership fills silence with imagination, and imagination during an outage runs dark. The cadence buys the debugging engineer the only currency that matters mid-incident: uninterrupted minutes.
| Audience | Cadence | What it contains |
|---|---|---|
| Incident channel | Continuous | The raw log: timestamps, findings, changes made, by whom. Written for Wednesday as much as for now. |
| Leadership | Every 20–30 min, fixed | Four sentences: impact in business terms, what we know, what we are doing, when the next update comes. Never a promise you have not already tested. |
| Customers / status page | On state change | Honest, plain, no internals: "Checkout is currently failing; we are working on it" beats silence and beats fiction. Written by the commander, not by marketing at hour three. |
Note what the leadership row refuses to contain: an invented ETA. The §5 case and Situation 01 both turn on this — under pressure, a guessed time reads as a commitment, gets forwarded upward, and detonates when it passes. The honest structure is a conditional: "the rollback completes in ten minutes; if it restores service we are done, if not the next path is X and the next update is at 3:10." That sentence gives leadership what they actually need — evidence you have a process — without borrowing against a future you do not control.
The blameless postmortem
The incident is not over when service returns; it is over when the organisation has extracted the lesson — and whether that happens is decided by one design choice made long before Saturday: whether your postmortems are blameless. The logic is not kindness; it is engineering. If postmortems assign fault to people, people optimise for the postmortem — evidence gets thinner, timelines vaguer, "no idea what happened" becomes the safe answer — and the organisation goes blind exactly where it needs sight. If instead the ground rule is that any failure a tired competent human can trigger is a missing guardrail, then the incident's full anatomy comes out on the table, because nobody at the table is defending themselves.3 Class Thirty-One's postmortem already made this call — the guardrails were blamed, the human kept working there — and that sentence was doing more for Campux's security than the vault was.
- Postmortem
- A written account of an incident — timeline, impact, contributing causes, and the guardrails now being built — whose purpose is to retire a class of failure, and whose ground rule is that systems, not people, appear in the findings.
The document is one page and five headings, and the desk lab below makes you write one: Impact (duration, users, money — Class One's native units); Timeline (timestamped, from the §1 running log, detection to resolution); Contributing causes (plural on purpose — real incidents are a chain, and "root cause" is usually just the last link anyone bothered to name); What went well / what got lucky (the alert that worked; the fact that it was Saturday and not Black Friday — luck is a finding, because luck runs out); and Actions, each with an owner and a date, reviewed like any other work. An action item without an owner is a wish. The test of a good postmortem is brutal and simple: six months later, can this incident happen again? If yes, the meeting was theatre.
Twenty-three minutes on a November Saturday
2:14pm, second Saturday of November, checkout starts returning 500s — the same faceless failure that once cost Campux nine hours. What happens instead is the loop, and it is almost boring. 2:14: the Class Thirty availability test fails twice and pages the on-call engineer; total customers who reported it first: zero. 2:19: triage — checkout only, all regions, and the Class Twenty-Nine bookmarked query shows failures bending sharply at 2:08. 2:22: the Activity Log's last-hour filter shows one change in the window: a storefront deploy at 2:07 — a Saturday hotfix for a cosmetic bug, approved through the emergency lane. 2:24: first leadership update goes out — impact, suspicion, plan, next update 2:45. 2:26: the commander makes the §3 call: no diff-reading, roll it back. The Class Twenty-Three pipeline redeploys the previous version. 2:31: deploy completes. 2:34: error rate at baseline; availability test green. 2:37: declared resolved. Twenty-three minutes, one change, one voice, a written log.
Wednesday's postmortem finds what Saturday declined to chase: the hotfix read an app setting that existed in staging but not production — a configuration drift Class Twenty-One warned about — and the emergency lane, by design, skipped the slot-swap smoke test that would have caught it. Findings: two guardrails, not one culprit. Actions: production settings move into the Bicep template so drift is impossible (owner, date), and the emergency lane keeps its speed but gains the one smoke test that runs in ninety seconds (owner, date). The engineer who shipped the hotfix presents the timeline personally — safely, because the ground rules held. And the number goes in the monthly report beside Class Thirty-Two's: the last outage like this one: nine hours. This one: twenty-three minutes. The delta is not talent. It is the machinery, and the loop, and the fact that nobody had to be brave.
The official pages, and a CAMPUX overview
Azure Monitor Activity Log
learn.microsoft.com/azure/azure-monitor/essentials/activity-log
Azure status and Service Health
learn.microsoft.com/azure/service-health/overview
The 23-minute incident replayed against a real dashboard — the bend in the chart, the Activity Log filter, the rollback, and the first leadership update written live — will live here. Video to be added.
Fight the incident on paper, then write its postmortem
Fire drills work because the building is not actually burning. Run the case-file incident yourself, from the page, making each call before you read how Campux made it — then write the one-page postmortem.
Cover the case file. You are paged at 2:14: availability test failing on checkout. Write your first three moves — the actual clicks and queries, in order.
Check yourself against §2: failed-request chart for the bend time, one errors-by-operation query, Activity Log filtered to the last hour. If your first move was "open the code," you dove for Wednesday's job at 2:14 on Saturday.The Activity Log shows a 2:07 deploy. Write the exact sentence you post to leadership at 2:24 — four sentences, per Table 1, including when the next update comes.
Check yourself: impact in business terms, known facts, current action, next update time. No ETA you have not tested. If your draft contains the word "probably," rewrite it as a conditional.Decide: roll back now, or read the diff first? Write one sentence defending your call.
Check yourself against §3: the store is dark and the rollback is reversible in four minutes; the diff is not going anywhere. If you chose the diff, price the choice: reading time × dollars per minute, against a rollback you can undo.Now write the postmortem — one page, the five §5 headings, using the case-file timeline. Give every action an owner (invent names) and a date.
The lesson: notice which heading was hardest. It is almost always Contributing Causes — the discipline of writing two or more systemic causes where your brain offers one human name. That discipline, practised here at zero stakes, is the entire difference between a postmortem and a trial.
Calm under fire, practiced
The page goes off at 2am: production is down. You do not flail. You follow the muscle memory — assess impact, communicate status, mitigate before you diagnose, then find the root cause — and you write the timeline as you go. The outage ends, and the blameless review turns a bad night into a fix that prevents the next one. Calm under fire is a skill, and you practiced it here.
Examination
Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored; this is between you and the page.
B — mitigate first; understanding is Wednesday's job. The loop's order exists precisely for this moment, because A is what your engineering instincts scream for, and A has a price tag: every minute in the diff is a minute of outage, and the diff will read exactly the same on Monday. Run the tape on A: twenty minutes of code archaeology, a theory, a forward-fix written under adrenaline, deployed untested — now there are two changes in production and one of them is brand new. C is a mitigation, but a superstitious one — nothing observed points at instance state, and restarting everything destroys the evidence while changing the estate randomly; reach for it when the telemetry suggests it, not as a reflex. D confuses the loop's last verb with its third. The interview version: "walk me through your first fifteen minutes of an outage" — and the answer that gets hired has a rollback in it before it has a theory.
The Activity Log — Class 16's control plane, writing its diary whether you asked or not. Every ARM operation — create, update, delete, role assignment — lands there with identity, timestamp, and source, no configuration required; at 2:20 on a Saturday it is the "what changed" answer in three clicks. The distractors each miss a plane: App Insights sees the application's behaviour (the bend, not the cause) — a deploy appears there only as its consequences; the workspace holds what Class 28 taught you flows only after you say so — and if you did wire the Activity Log into it, better still, because then Class 29's KQL can join changes against errors, which is the grown-up version of this drill. Advisor is a weekly gardener, not a flight recorder. The pairing to memorise: telemetry tells you when it bent; the Activity Log tells you what changed just before. One without the other is half a diagnosis.
Impact, action-with-conditional, and the next-update time. Those three give leadership what they are actually starved of — evidence of a process — and the third one is the sleeper: a promised update time is what stops the "any update?" drumbeat that costs the debugging engineer their concentration. The rejects are the two classic self-inflicted wounds. An invented ETA feels kind and costs double: it gets forwarded upward as a commitment, and when it passes unmet, the second update opens in deficit — trust, once spent there, does not refund. And a name in a status update is a verdict published before the trial: it poisons Wednesday's blameless ground rules four days early, teaches every engineer watching that visibility equals exposure, and — per §5 — points at the wrong thing anyway, since the finding will be the guardrail that let a tired human ship it. Say what changed; never, mid-incident, say who.
# postmortem: checkout outage, Nov 14 — DRAFT
1. Impact: checkout unavailable 23 min; ~$95K of
carts abandoned; zero customer reports preceded
our detection.
2. Contributing causes: emergency deploy lane skips
the smoke test; production app settings drift
from staging because they live outside IaC.
3. Contributing causes (cont.): the incident was
caused by Priya rushing a hotfix without checking
the settings, contrary to team guidance.
4. Actions: settings into Bicep (owner: M.O., Dec 1);
smoke test added to emergency lane (owner: T.K.,
Nov 28).
Line three — and notice it adds no information; it only adds a defendant. Everything actionable in line three is already in line two: the lane skips the test, the settings drift. Adding Priya's name and "contrary to guidance" retires no failure mode — hire Priya's replacement and the same lane ships the same drift on their first tired Saturday. What the name does accomplish is corrosive and permanent: this document gets read by every engineer on the team, and each reads the same lesson — be invisible near incidents. The next timeline you collect will be vaguer, the next "what changed?" will meet more shrugs, and the organisation will have purchased one moment of accountability theatre at the price of its own eyesight.
The rewrite is mechanical, which is the practical skill this drill installs: delete the name, keep the mechanism — "the emergency lane permitted a deploy without the settings check that the standard lane enforces." The distractors mark the opposite discipline: dollar figures are exactly what Impact is for (A — Class 1's native units, and what makes leadership fund the actions); naming the broken process is the entire point (B confuses blaming systems, which is required, with blaming people, which is banned); and actions without owners and dates are wishes (D). The test for any postmortem sentence: does it still retire the failure if every human in it is replaced? Line three fails; line two is the finding.
The trap is the premise — that the only acceptable answer is a time. What the VP actually needs for the board is not a timestamp; it is evidence that the situation is controlled and a truthful thing to say upward. An invented ETA provides neither: it is a coin-flip that, if missed, converts "outage" into "outage plus a team that missed its own estimate" — and it will be missed, because forty minutes in with no root cause is precisely when estimates are fiction. The discipline is to refuse the number without refusing the person.
Send the structured answer, in business terms, in four sentences. What is true: checkout has been down since 2:08; the most likely cause was ruled out at 2:46 when the rollback did not restore service. What is happening: two paths are being worked in parallel — the configuration diff between staging and production, and failover of the checkout backend to the standby tier — with the first result expected within fifteen minutes. What the board can be told: "the team has a controlled process, has eliminated the primary suspect, and reports again at 3:20." And the commitment you can make honestly: the next update's time, which you will hit even if the content is "no change." A conditional plus a cadence is a professional answer; a guessed clock time is a loan against your credibility at the worst possible interest rate.
Then protect the machine that produces real ETAs. The moment a rollback fails is the moment pressure peaks and the §3 stampede risk returns — so the commander restates the rules in the channel: one change at a time, log everything, updates at :20 and :40. If the VP pushes again — they sometimes do — the sentence that ends it is Class 1's language, not engineering's: "Every minute is costing us about four thousand dollars, which is exactly why I won't spend the next ten of them manufacturing a number. You will have facts at 3:20." Executives respect the arithmetic of their own money; give them that instead of a guess.
A strong answer concedes before it argues. The director is not wrong that something failed and that accountability exists; steamrolling that instinct loses the room and the sponsor in one move. So concede the true half out loud: "You're right — a change reached production on a Saturday without the check that would have caught it, and this document will say exactly that." Then the pivot, stated as method rather than mercy: the document names mechanisms, because its job is to make this failure impossible, and the mechanism is fully identified without a name — the emergency lane permits deploys that skip the smoke test. Who approved it matters less than the finding that the lane exists and approving through it was normal; if guidance was truly violated, that is a management conversation with a manager, in private, on a different day — and saying that sentence plainly, in front of the room, is what gives the accountability question a legitimate home instead of denying it one.
Then price the alternative, because directors respond to consequences. If the name goes in the document, this is the last complete timeline the team ever produces: the next incident's log will be thin, the next "what changed?" will meet lawyers' answers, and detection-to-mitigation time — the number that just fell from nine hours to twenty-three minutes — will climb back up for reasons no dashboard will show. Blameless is not a culture perk; it is the price of the data. The twenty-three minutes the director is proud of was purchased with exactly the ground rule they are about to break.
Close by redirecting the energy at the guardrails, with dates. "Here is what makes Saturday impossible to repeat: settings into IaC by December 1, smoke test in the emergency lane by November 28 — and I'd like your name on the review of both." A director who arrived wanting a culprit and leaves owning a guardrail has been converted, not defeated — and the team watched the ground rules survive contact with power, which is the only demonstration of blamelessness that counts. The postmortem's authority was never the template; it was whether the rules held on the day someone senior tested them.
Five things worth carrying out of this class
- The loop is detect, triage, mitigate, learn — in that order, because order is what survives adrenaline. One commander, one voice, everything written down as it happens.
- Diagnosis is two questions joined: telemetry says when it bent; the Activity Log says what changed just before. The control plane keeps that ledger automatically — who, what, when.
- Mitigate first, root-cause second: the fastest reversible path, one change at a time, watched and logged. Saturday restores service; Wednesday explains it.
- Communication runs on a fixed clock: impact in business terms, a conditional instead of an invented ETA, and the next update's time stated even when nothing changed. Silence is what leadership fills with imagination.
- Postmortems are blameless because blame poisons the data: any failure a tired competent human can trigger is a missing guardrail. Five headings, plural causes, every action with an owner and a date — and six months later, the same incident cannot recur.
- The Activity Log's built-in retention is limited — around ninety days — and it records the control plane only; what happened inside your application is the data plane's story, told by the Class 28 diagnostic settings you configured. Estates that take incidents seriously route the Activity Log into the Log Analytics workspace too, so Class 29's KQL can join "what changed" against "what broke" in one query. Retention numbers and blade names shift; the two-plane distinction does not. ↩
- Treat the "roughly a third" cognitive claim as folklore with good references rather than a measured constant — the studies vary and the number is not the point. The direction is settled and you will feel it personally: under incident stress, working memory shrinks, tunnel vision arrives, and checklists outperform cleverness. Every practice in this class — the loop, the fixed cadence, the written log — is a prosthetic for a brain that is temporarily worse than the one that read this page. ↩
- "Blameless" has an honest boundary: it covers competent people failing inside bad systems, which is nearly every incident — it does not cover malice, recklessness repeated after the guardrail existed, or covering up. Teams that pretend the boundary does not exist eventually meet a case that discredits the whole practice; teams that name it up front ("this protects mistakes, not misconduct") keep the ground rules trusted precisely because they are known to have edges. ↩