The second system that fails is the channel
Part B left you mid-incident with the store coming back up. While you were reading the Activity Log, a second failure was building — not in the estate, in the channel. It fills with "any update?", and each one is a context switch for the person doing the work, each unanswered minute breeding two more. Nobody in the channel is trying to hurt you; they are frightened, and frightened people fill silence with imagination, and imagination during an outage runs dark. The fix is mechanical, and — this is the whole point of the part — it is the commander's job, not the debugging engineer's: a status update on a fixed clock, whether or not anything has changed.
“No change to the confirmed impact; rollback is still being checked; next update at 3:10” is a useful update. It states what is known, what is happening, and when people will hear more. Choose a cadence the team can keep; publish sooner if impact or status changes. A small team may have one person handling both recovery and updates. As the incident grows, assign a communications lead so the engineer making changes can focus on the service.1
- Commander
- Coordinates the response: confirms impact and severity, assigns work, approves risky changes, and keeps the timeline. The first person paged may start this role and hand it to a designated commander.
- Comms lead
- Prepares consistent internal and customer updates from confirmed facts. On a small incident, the commander may also do this work.
Two roles, one voice. The channel hears from the comms lead and only the comms lead; committees debug nothing and they broadcast even less. The failure mode this prevents is the one where three people post three half-true updates in four minutes and leadership, now certain the team is not in control, escalates — pulling the engineer off the fix to answer a director who was reassured by nobody. Fixed clock, single voice, and the estate gets its quiet back.
What a good update contains — by audience
Not everyone gets the same update, and getting that wrong is its own small disaster: the raw engineering log dumped on the board reads as panic; the four-sentence summary handed to the on-call engineer starves them of detail. Three audiences, three cadences, three contents. The table is worth committing to memory before any Saturday, because deciding "who am I writing for?" at 2:19 is a tax you pay in the currency you have least of.
| Audience | Cadence | What it contains |
|---|---|---|
| Incident channel | Continuous | The raw log — timestamps, findings, changes made, by whom. Written for Wednesday's postmortem as much as for now. |
| Leadership | Every 20–30 min, fixed | Four sentences: impact in business terms, what we know, what we are doing, when the next update comes. Never a promise you have not already tested. |
| Customers / status page | At agreed intervals and meaningful state changes | Plain impact and next update: “Some customers cannot check out. We are investigating and will update this page at 3:10.” Follow your published incident policy. |
Do not invent a restoration time. Separate the next update time, which you can commit to, from a recovery estimate, which may be uncertain. State the current impact, confirmed facts, action in progress, and next update. If you have an evidence-based estimate, label it as an estimate and update it when the evidence changes.
A guessed clock is a loan against your credibility.
Keep broad status updates focused on the system and customer impact. “The 2:07 release is being investigated” is more useful than publicly assigning blame before the cause is known. The internal incident timeline can still record who performed each action so the team can reconstruct events accurately.
The blameless postmortem
Service is back. The incident is not over — it is over when the organisation has extracted the lesson, and whether that happens is decided by one design choice made long before Saturday: whether your postmortems are blameless. The logic is not kindness; it is engineering. If postmortems assign fault to people, people optimise for the postmortem — evidence gets thinner, timelines vaguer, "no idea what happened" becomes the safe answer — and the organisation goes blind exactly where it most needs sight.
Turn the rule around and it does real work. If the ground rule is that any failure a tired, competent human can trigger is a missing guardrail, then the incident's full anatomy comes out on the table, because nobody at the table is defending themselves.2 "Human error" stops being an answer and becomes the first question: a competent person did the reasonable-looking thing and the system let it through — so what let it through? The name of the person who shipped the change tells you nothing you can fix. Replace them with their equally competent successor and the same lane ships the same drift on their first tired Saturday. The guardrail is the finding; the human is just where the missing guardrail happened to surface.
- Blameless
- The ground rule that a postmortem's findings name systems, not people — because the goal is to retire a class of failure, and a name retires nothing. It protects mistakes made in good faith; it does not cover malice or a hazard ignored after the fix was known.
The document is one page and five headings, and the pattern is worth having by heart. It is a hang list, not prose, because each heading resists a different temptation.
- Impact
- Duration, users, money — Class One's native units. What makes leadership fund the actions rather than nod at them.
- Timeline
- Timestamped, lifted straight from Part A's running log, detection to resolution. The one honest witness; memory under adrenaline is fiction with confidence.
- Contributing causes
- Plural on purpose. Real incidents are a chain; "root cause" is usually just the last link anyone bothered to name. This is the heading that resists collapsing into one human's name — the hardest discipline in the document.
- What went well / got lucky
- The alert that fired; the fact it was a quiet Saturday and not Black Friday. Luck is a finding, because luck runs out.
- Actions
- Each with an owner and a date, reviewed like any other work. An action item without an owner is a wish.
The test of a good postmortem is brutal and simple: six months later, can this incident happen again? If yes, the meeting was theatre. And there is a second test, the one that decides whether the practice survives contact with power — does every sentence still retire the failure if you replace every human named in it? The finding that survives that swap is the one worth writing down.
Play it through
Three minutes. The channel is filling with "any update?" while checkout is still being fixed. Post the fixed-clock update before anyone has to ask, then bin the guessed ETA before it gets forwarded upward. Then sort two postmortem lines: one retires the failure, one only adds a defendant, using the test you just read above — does it still retire the failure if you replace every name in it?
Check the recovery objectives agreed beforehand
Before an incident, business and service owners should agree how much data loss and downtime each service can tolerate. The recovery plan is designed and tested against those objectives. After an incident, the review compares what actually happened with the targets and may recommend revising the plan or targets. A postmortem should not be the first time anyone chooses them.
- RPO
- Recovery Point Objective — the maximum acceptable gap between the latest recoverable data and the disruption. A one-hour RPO requires a recovery method that can restore data to within one hour of the event. Backup frequency, replication, and recovery testing all matter.
- RTO
- Recovery Time Objective — the target maximum time to restore the service after disruption. A 30-minute RTO requires a tested recovery path that can meet that target; a written number alone does not make recovery fast.
RPO concerns recoverable data; RTO concerns restoration time. Tighter targets can require more frequent recovery points, replication, automation, or ready capacity, all with costs and operational work. Choose targets by service and business impact, then test whether the architecture meets them.3
Two numbers turn recovery into a budget.
Part D shows the objectives on a timeline and tests what a backup can actually restore. If a service has no agreed RPO or RTO, the review should assign an owner to conduct that planning before the next incident. If targets already exist, compare the actual data loss and restoration time with them and record the gap.
Campux holds its first blameless postmortem
In this fictional case, checkout failed at 2:07 and recovered at 2:37: 30 minutes of impact, including seven minutes before the alert reached on-call. The team holds a review afterward to understand the conditions that allowed the failure and to check whether its response met the service's existing recovery target.
This time the team keeps an accurate timeline and asks which release and configuration controls failed. It also checks the incident against recovery objectives agreed before Saturday and assigns an owner to close gaps.
The review after a 30-minute checkout outage
The room holds the engineer who shipped the 2:07 hotfix, the commander, and a director who arrives believing the meeting is short: someone deployed on a Saturday without testing, name the name, done. The person running the meeting concedes the true half out loud — a change reached production without the check that would have caught it, and the document will say exactly that — then holds the ground rule as method, not mercy: the findings name mechanisms. There are two. The emergency lane, by design, skips the smoke test the standard lane enforces. And the production app settings drift from staging because they live outside the Bicep template. Two guardrails, not one culprit; the hotfix engineer presents the timeline personally, safely, because the rules held on the day power tested them.
The action list assigns owners and dates to put the relevant settings under version control, add a checkout smoke test to the emergency release path, and test the on-call alert route. Version control reduces drift but does not make it impossible. The team compares 30 minutes of observed checkout impact with the RTO the business had agreed before the incident; this scenario does not give a measured data-loss value, so it cannot claim that the RPO was met. If Campux had never set those objectives, the action is to convene the business and service owners to define and test them before the next disruption.
The words are a role, not an afterthought
As communications lead, you post confirmed impact, the current action, and the next update time. You avoid public blame while keeping an accurate internal timeline, including who made changes. In the review, you record contributing conditions and assign fixes with owners and dates. You also compare observed recovery with pre-agreed objectives, or assign ownership to define them if they are missing.
Write the update from facts
Use the fictional evidence below. This is a desk exercise; it does not provision Azure resources or measure a live service.
09:00 checkout failures begin 09:12 incident acknowledged; cause unconfirmed 09:20 rollback did not restore service Next update agreed: 09:30
Write a two-sentence update at 09:20.
Name the observation needed before saying recovered.
Draft one postmortem question about detection rather than assigning blame.
On your own: it is 09:30 and checkout errors have been at normal levels for ten minutes. Write the 09:30 update in your own words.
Check your reasoning after you finish
On your own: say service is recovering and you are watching it, not that it is resolved, until the agreed observation window passes; give the next update time. “Checkout is still failing for the affected customers. Rollback has not restored service; we are checking database connection failures and will update at 09:30.” Recovery requires customer-facing success evidence over an agreed observation window.
Examination
Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored; this is between you and the page.
B — the silence is the cost, not the lack of news. A fixed-clock "no change" tells everyone senior the one thing they need: a process is running and they are inside it. A is the trap that feels responsible and detonates — the moment you go quiet, leadership fills the gap with imagination, and imagination during an outage runs dark; the "any update?" drumbeat starts, and each one is a context switch stolen from the fix. C manufactures a commitment you cannot keep and opens the next update in deficit. D collapses the two roles this part exists to separate: the engineer with their hands in the estate is precisely who must be shielded from the channel, not fed to it. Send the boring update on the clock.
B — RPO then RTO. The five-minute RPO is the target maximum gap in recoverable orders. The 30-minute RTO is the target time to restore checkout. The business agrees these before a disruption; engineers design and test recovery methods against them. A postmortem compares actual results with the targets and recommends changes if needed. Neither target guarantees the outcome.
Confirmed impact, current action, and next update time. These let leadership communicate accurately while the team works. A firm restoration time without evidence can mislead; state uncertainty and revise an evidence-based estimate when needed. A broad status update should focus on the service rather than blame a person. The internal incident log can still record who performed each action.
# postmortem: checkout outage, Nov 14 — DRAFT
1. Impact: checkout unavailable 30 min; request
errors rose at 2:07; affected order count is
still being confirmed.
2. Contributing causes: emergency deploy lane skips
the smoke test; production app settings drift
from staging because they live outside IaC.
3. Contributing causes (cont.): the incident was
caused by Priya rushing a hotfix without checking
the settings, contrary to team guidance.
4. Actions: version settings in Bicep (owner: M.O.,
Dec 1); test emergency release path (owner: T.K.,
Nov 28); compare actual recovery with the agreed
RTO and RPO (owner: V.T., Nov 24).
Line three. It blames a person before explaining why the release path allowed the failure. The incident timeline can record who approved and deployed changes, but the contributing-cause section should describe the missing check and configuration gap. That gives the team a specific control to improve. If conduct concerns exist, handle them through a separate, fair process.
Rewrite line three as a mechanism: “The emergency release path did not verify the production setting required by the new code.” Keep the internal action timeline accurate, including who made changes, without turning the contributing-cause statement into a verdict. Line one states a measured 30-minute impact and marks the order count as unconfirmed. Line two names the process gaps. Line four assigns owners and dates, including a comparison with recovery targets agreed before the incident.
Give confirmed facts and the next update time. “Checkout remains unavailable for the affected customers. The rollback completed but did not restore it. We are checking dependency health and the current production configuration. We cannot give a reliable restoration time yet; the next update is at 3:20, sooner if status changes.”
Keep internal coordination explicit. The incident owner records the failed rollback, assigns the next investigation and any safe containment action, and makes sure customer-facing updates match the known impact. Do not present an untested failover or invented revenue estimate as a committed recovery plan.
Explain why this helps the listener. A store manager needs to know whether customers can check out and when they will hear from you again. They cannot act on a guessed root cause. Separate “rollback completed” from “service restored”: the first is an action, the second needs a customer-path check. When that check passes, publish the recovery status and continue investigating why rollback failed.
Separate the event record from the causal finding. The director may need to know who approved the release, and that fact can be checked in the internal timeline. The postmortem's contributing-cause section should explain how the emergency path lacked the check that would have caught the missing setting. If someone knowingly bypassed an established requirement, address that through the appropriate management process without losing the technical finding.
Keep the timeline accurate and the findings useful. Who approved the release can be recorded as an event. The analysis asks why the emergency path omitted a production-settings check and why the new code required a missing value. These are controls the team can change. Avoid claiming that naming a person or adding one guardrail will certainly determine the outcome of the next incident.
Assign and review the actions. Put settings under version control, test the emergency release path, and validate that the alert and rollback procedures work, each with an owner and due date. Compare the observed 30-minute outage with the service's pre-agreed RTO. If no objectives existed, assign business and service owners to set and test them before another incident.
Five things worth carrying out of this part
- Two systems fail in every incident. The channel is the second one; a fixed-clock update — even "no change" — is what keeps leadership out of the engineer's way. Silence is what they fill with imagination.
- Split the roles the moment you can: a commander who owns the calls and a comms lead who owns the words. One voice in the channel; committees debug nothing and broadcast less.
- Keep an internal timeline, give leadership confirmed impact and next steps, and update customers on an agreed cadence and meaningful state changes. Avoid invented restoration times and public blame.
- Postmortems are blameless because blame poisons the data. "Human error" is the first question, not the answer: any failure a tired competent human can trigger is a missing guardrail. Five headings, plural causes, every action owned and dated.
- RPO and RTO are agreed before an incident. The review compares actual data loss and restoration time with those targets and assigns changes when needed.
- On a two-person incident nobody splits anything — the same person mitigates and posts, and that is fine. The role split is a threshold, not a law: once the updates are stealing minutes from the fix, or once the audiences diverge enough that one message cannot serve all three, separate the words from the hands. Team sizes and titles vary; the principle — protect the person touching the estate — does not. ↩
- "Blameless" has an honest boundary: it covers competent people failing inside bad systems, which is nearly every incident — it does not cover malice, recklessness repeated after the guardrail existed, or covering up. Teams that pretend the boundary does not exist eventually meet a case that discredits the whole practice; teams that name it up front — "this protects mistakes, not misconduct" — keep the ground rules trusted precisely because they are known to have edges. ↩
- Treat any specific RPO/RTO figures in this part as illustration, not prescription — the right numbers fall out of what an hour of lost orders or an hour of downtime actually costs a given business, and that is Part F's tiering work, not a constant to memorise. What is settled is the shape: two independent numbers, priced against real loss, agreed per tier before the incident that tests them. ↩
Write a status update people trust and a blameless postmortem people act on.
Play the lesson above to stamp this class.