Two disciplines that share one bad Saturday
Everything you have built in this bootcamp assumed the good day: the deploy that worked, the cluster that scaled, the bill that stayed sane. This track is about the other day — the one where checkout returns a blank page at eleven-fifteen on the busiest Saturday of the year, and the room turns to look at you. There are two jobs to do when that happens, and they are not the same job. One is to respond: stop the bleeding, keep people informed, and learn from it without a witch-hunt. The other is to recover: get the data and the service back from wherever they went, on a timetable the business agreed to in advance. This class teaches both, because on the day they arrive together.
The two jobs connect through recovery targets. Before an outage, business owners and engineers agree on the maximum acceptable data loss, RPO, and maximum acceptable downtime, RTO, for each important workload. They choose and test a recovery design against those targets. During an incident, the team uses the plan and records what actually happened. The postmortem can recommend changing the targets or design, but it does not create the first targets after the failure.
One half stops the bleeding. The other brings it back.
The shape of the whole thing, drawn once
Before the six parts, hold the picture they all hang off. A disaster is a single fixed moment. Two lanes run away from it.1 The respond lane runs forward in the next minutes and hours: you detect the fault, triage what changed, mitigate to restore service, communicate on a cadence, and — days later — hold a blameless postmortem. The recover lane answers a different question that the same moment forces: is the data intact, and is the service still somewhere? If a disk or a database is gone, you restore it from a vault; if a whole region is gone, you fail over to the copy waiting in another one. How much data you may lose and how long you may be down are the two numbers — RPO and RTO — that size every choice in that lane.
The track — six parts, alert to all-clear
The rest of Class Thirty-Six is six pages, each a full class in its own right with its own drills, situations, and lab notes. Read them in order the first time; the recovery half leans on vocabulary the response half sets, and the two numbers introduced at the seam run through everything after. Later they work as reference — the page you reopen the week you are actually on call, or the afternoon you finally write the BCDR plan.
- 36a · Fundamentals & detection
- What counts as an incident, how severity is set, and the loop you run under pressure — plus the signals that tell you something broke before a customer does, and what being on call actually asks of you.
- 36b · Triage & mitigation
- The one question that finds most outages — what changed? — read against the Activity Log and the estate, and the discipline that saves the day: mitigate first, root-cause second. Rollbacks, feature flags, and the safe fast fix.
- 36c · Communication & postmortems
- The update cadence that keeps leadership out of the incident channel, and the blameless postmortem that turns an outage into a system fix instead of a name. Where the response half hands the recovery half its two numbers.
- 36d · Azure Backup
- Choose a vault and policy for a supported workload, understand retention, and test a restore. Cross-region restore and soft-delete behavior depend on the protected item and configuration.
- 36e · Azure Site Recovery
- The standby in another region: continuous replication, the full drill of test failover, failover, reprotect, and failback, and why a failover you have never rehearsed is a hope, not a plan.
- 36f · BCDR strategy
- The judgement no service configures for you: region pairs and zone redundancy, sorting workloads into honest tiers, and writing the BCDR plan — including, in writing, what you chose not to protect. Carries the lab.
The one habit that carries the whole track
Beginners meet an incident by improvising and meet recovery by hoping. The engineers who are calm in both have internalised a single habit: decide the hard things while nothing is on fire, and rehearse them until they are boring. The severity levels, the comms cadence, the rollback command, the failover runbook, the RPO and RTO each tier is owed — all of it is cheap to settle on a Tuesday afternoon and impossibly expensive to invent at three on Sunday morning. The whole track is, in a sense, one long argument for doing the thinking in advance and leaving only the execution for the bad day.
This is why the response half gives you a loop you can run frightened, and why the recovery half makes you test the restore and the failover before you trust them. A backup you have never restored is a rumour; a failover you have never run is a slide. The senior signal in this whole domain is not heroics during the outage — it is the visible, unglamorous evidence that you removed the need for heroics beforehand. Learn that disposition here and the six parts stop being a checklist and become a temperament.
The nine-hour outage that started all of this
Before this bootcamp existed, Campux was dark for nine hours when a single disk failed in the server closet and the only backup had never been tested. That outage is the reason the reader — Campux's first cloud engineer — has a job at all. This class is where it stops being possible from both directions. The response half gives Campux a loop so the next fault costs minutes of confusion, not hours; the recovery half gives it recovery points, a paired-region standby, and a plan that prices each workload's protection against what losing it actually costs.
This is a fictional case study. The nine-hour outage and later twenty-three-minute incident are examples for comparing a missed recovery plan with a rehearsed one; they are not measured CAMPUX production results or promised recovery times.2 The drills below ask you to judge the plan against the actual service and data loss.
Why this track pays for the year
Incident response and recovery are the least glamorous work a cloud engineer does and the work that most reliably keeps them employed. Nobody is promoted for the outage that photographs well; people are trusted with more when the outage that could have been a headline was a footnote instead. The hiring signal is specific and rare: you can describe the loop without notes, you name mitigation before root cause, you run a postmortem that produces a fix and not a culprit, and you can price resilience in the two numbers a business actually signs. That combination — calm under load plus planning before it — is what a hiring manager is straining to detect and what the drills in this hub rehearse.
So treat the six parts as an investment with a legible return. Each one is a paragraph you can say in an interview and a task you can do on the job, and together they move you from "I hope nothing breaks" to "I know exactly what we do when it does, and I have tested it." The parts build the hands; the hub, below, rehearses the judgement they serve.
The engineer the outage does not rattle
Checkout is down and the channel is filling with questions. The team has already agreed on severity levels, update cadence, recovery targets, and tested options. You confirm what failed, choose a safe way to reduce impact, and tell people what is known. Afterward, the postmortem compares actual downtime and data loss with the targets and assigns concrete improvements.
Examination
Four drills, then two situations. These test the judgement the track is built on; the hands are built in the six parts. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B. Response reduces impact and keeps people informed; recovery restores data and service. Business owners agree on RPO and RTO before the outage. The team designs and tests against them, then uses an incident review to compare actual results with those targets. Backup and Site Recovery are possible recovery tools, not the names of the two disciplines.
B. Severity levels, comms cadence, the rollback command, the failover runbook, each tier's RPO and RTO — all cheap to settle on a Tuesday and ruinous to invent at 3am. A sounds streetwise and gets people hurt: improvisation is what the loop and the runbook exist to replace. D is the specific expensive mistake the strategy part dismantles — protecting analytics scratch data to checkout grade is a finding, not prudence, because resilience is bought in real money. The whole track argues for doing the thinking in advance and leaving only execution for the bad day.
The loop without notes, the tested restore and failover, and the blameless postmortem. Each is evidence that the thinking happened before the outage and is shared, not hoarded. The two rejects describe the same anti-pattern wearing a cape: the indispensable hero who fixes everything personally and keeps the runbook in their head is a single point of failure the company cannot promote and cannot let take a holiday. A backup you have never restored is a rumour; a failover you have never run is a slide; a runbook only you can read is a liability. Seniority here is the removal of drama, not the supply of it.
# how I'll handle the next Saturday outage
1. Detect from alerts, set a severity, start the loop.
2. Find the root cause first; do not touch anything
until I fully understand why it broke.
3. Send a leadership update on a fixed cadence.
4. After it's over, hold a blameless postmortem and
compare actual loss and downtime with the targets.
Line two. Do not require a complete root-cause analysis before considering a safe, reversible mitigation. If the evidence points to a recent deploy and the rollback path is tested, restore service while the team continues diagnosis. Preserve useful evidence and assess rollback risks first; an unsafe change can deepen the outage. The other lines establish severity, a regular update cadence, and a postmortem that checks actual results against recovery targets agreed in advance.
Concede the tool before you question the plan. Backup being on is genuinely good; open by saying so, or you sound like you are defending your own headcount. Then put the weight on the word doing the damage: "just restore." A restore is only the recovery half, and only its slowest tool — it brings back last night's data after however long the restore takes, which is fine for a catalogue and useless for checkout in a regional outage. "Just restore" quietly assumes the data is the only thing that breaks, the backup has been tested, someone knows which recovery point to pick, and no customer or executive needs managing while it runs. None of those are free.
Price the gap in the currency that moves a roadmap. The nine-hour outage in the case file was a restore that had never been tested, run by people improvising under pressure with no comms and no severity call. Every hour of that was revenue and trust. The quarter buys the difference between that and a rehearsed twenty-three minutes: a loop, an update cadence, a tested failover for the tier that cannot wait for a restore, and a plan that says in writing what each workload is owed. That is not process for its own sake; it is the conversion of a random disaster into a decision the business already made.
Offer the smaller yes. You do not need the whole quarter before any of it pays off. Propose sequencing: settle severity and comms and one tested restore first — days of work that would have saved most of the nine hours — then the failover and the full BCDR plan for the revenue tier. That reframes the ask from "a quarter of overhead" to "the cheapest insurance we will ever buy, staged so it earns its keep as we go." Managers fund that; they defund "process".
Name the structure before the steps. Open with the frame, so they know you are not improvising: "Two things happen at once — I respond to the incident and I recover the data, and I keep them separate in my head." Then walk the respond lane: detect and confirm the symptom and scope, set a severity, and start communicating on a cadence so leadership stays out of my way. Crucially, mitigate before I diagnose — if I can fail reads over to a replica or put the site in a safe degraded mode, I buy time before I care why the database corrupted.
Then walk the recover lane against the two numbers. "In parallel I ask what the tier's RPO and RTO are, because that decides the tool. If it's minutes and minutes, I fail over to the zone-redundant or replicated copy — I'm not waiting on a restore. If a restore is the right call, I pick the last clean recovery point before the corruption, not the newest, and I know which one because retention was set for exactly this." Naming that you would test-failover quarterly, so this is rehearsed, is the sentence that lands.
Close on the postmortem, because it proves you learn. "When it's back, I record the actual recovery time against the objective, and if the gap is uncomfortable I tighten the plan before the next one — blamelessly, because I want the honest timeline, not a name." Two minutes, two lanes, the two numbers, and a loop that improves itself: that is the answer that ends the reliability portion of the interview and tells them you have done this, not read about it.
Five things worth carrying out of this hub
- An outage demands two separate jobs: respond (stop the bleeding, keep people informed, learn without blame) and recover (bring the data and service back on an agreed timetable). This track teaches both because they arrive together.
- The two halves meet at two time-based targets. RPO is the maximum acceptable data loss; RTO is the maximum acceptable downtime. Agree on both before an incident, test the plan, then compare actual results with the targets.
- The habit that makes it all cheap: decide the hard things while nothing is on fire, and rehearse them until they are boring. A backup you have never restored is a rumour; a failover you have never run is a slide.
- The track is six parts, alert to all-clear: fundamentals & detection, triage & mitigation, communication & postmortems, Azure Backup, Azure Site Recovery, and BCDR strategy. Read them in order once, then keep them as reference.
- Seniority here is the removal of drama, not the supply of it. The reward is trust — the outage that could have been a headline becomes a footnote, and that is what closes the pay gap.
- The clean split into "respond" and "recover" is a teaching device; in a real severe incident the lanes interleave, and the same engineer often runs both while a second manages comms. Do not read the two lanes as two people or two phases in strict sequence — read them as two questions ("is it stopping?" and "is it coming back?") you hold at once. The six parts separate them only so you can learn each without the other's noise. ↩
- The nine-hour and twenty-three-minute figures are fictional teaching numbers. They are not a benchmark or guarantee. Recovery time and data loss depend on the failure, workload, plan, and tested implementation. ↩
Run an incident from detection to postmortem without losing the plot or the evidence.