Design for the failure you need to survive
The shop manager can tolerate tomorrow’s reporting job starting late. She cannot tolerate losing the orders customers have just paid for. Ada writes those two sentences before drawing another region. Recovery time and acceptable data loss belong to a workload and its business owner. Zones, replicas and backups are possible ways to meet those requirements; their presence does not prove that recovery will work.
Zones share a region, so they do not cover a region-wide outage.1 For that, design recovery in another region: copy the relevant data, prepare the application and its dependencies there, decide how traffic will switch, and test the result. Azure has designated region pairs used by some platform features, such as geo-redundant storage. A pair is a placement relationship, not automatic replication or failover for every workload. Some services let you choose a different target region. Check the service, region support, data residency and actual recovery behavior before selecting one.
- Zone
- Availability zone — a separate group of datacentres inside a region. A workload survives a zone failure only if all critical components are configured to do so.
- Region pair
- Two Azure regions linked for certain platform behaviors. The pairing does not, by itself, create a copy of your application or route users after an outage.
Use zone resilience for local failures and a tested cross-region design where the business must survive a regional outage. A service may need a separate setting for each. For example, a zone-redundant Azure SQL Database can survive a zone failure; a failover group or geo-replica is a separate choice for regional recovery. The replica is asynchronous, so a forced failover may lose recent writes. These are workload design decisions, not benefits inherited merely by choosing a paired region.2
Honest tiers — sort the estate, then price each one
A workload is the complete service people use: its application, data, identity, network and outside connections. Group workloads by the harm their loss would cause. For each, ask the business to agree on an RPO (maximum acceptable age of lost data) and an RTO (maximum acceptable time until the service works again). Then price designs that could meet those targets and verify them with recovery tests. A product name alone cannot promise either number.
Protect everything, and you have decided nothing.
The three tiers below are fictional Campux targets, not Azure defaults. Tier 1 is checkout: minutes of tolerable data loss and less than an hour to restore service. Tier 2 is the catalogue: up to a day of data loss and a few hours to recover. Tier 3 is scratch data: it can be rebuilt, so Campux accepts its loss and records how to regenerate it. A daily backup may meet a one-day data-loss target in the primary region; whether it meets the recovery-time target requires a timed restore. A backup copied to another region can be older than a day, so evaluate regional recovery separately.
| Tier | RPO | RTO | Mechanism | Campux example |
|---|---|---|---|---|
| Tier 1 · revenue-critical | Target: minutes | Target: under 1 hour | Zone-resilient design; ASR for supported VMs; separate database regional recovery; rehearse full checkout | Checkout application and orders database |
| Tier 2 · important | Target: 24 hours locally | Target: 4 hours | Daily backup plus a timed restore; check secondary-region point age if regional loss is in scope | Product catalogue and marketing site |
| Tier 3 · regenerable | Loss accepted | Rebuild when needed | No backup of derived data; protect its source and document the rebuild | Analytics scratch data |
Read the table as a set of proposed targets that still need evidence. Tier 1 must include the database and customer routing, not just a replicated VM. Tier 2 needs a restore test to demonstrate four-hour recovery. Tier 3 can skip a backup of derived data only if its inputs survive and the rebuild time is acceptable. Record cost and test results beside each target before calling it met.
Play it through
Three minutes. Assign the fictional workloads to proposed tiers, then identify what evidence would show whether their targets are achievable.
The plan — and the sentence that makes it real
A BCDR plan records who owns each workload, its agreed RPO and RTO, what can fail, and the recovery steps for every critical component. It also names who declares an incident, how customers will be routed, where credentials and runbooks are kept, who communicates, and how often the plan is tested. A one-page tier table is a useful summary; the runbook behind it contains the steps an on-call engineer can execute.
- BCDR
- Business Continuity and Disaster Recovery — the decisions and tested steps that keep essential work going or restore it after an outage. The plan includes acceptable loss, recovery order, owners, dependencies, communication, test evidence and accepted gaps.
Record accepted gaps explicitly. For the fictional scratch dataset: We accept loss of the derived copy because the source data is protected and we can rebuild it overnight; the analytics owner accepts that delay. If the source is not protected, or the rebuild takes longer than the business can wait, the decision changes. The point is a specific, owned trade-off, not a blanket rule that low-tier data never needs protection.
Test the plan on a schedule and after important changes. A successful backup job does not prove restoration, and a VM failover test does not prove that payments and customer routing work. Measure recovery-point age, the time until the service is usable, and the steps that failed. Update the runbook and repeat the test.
A fictional plan, with gaps to verify
After a fictional nine-hour disk outage, Campux asks its business owners for acceptable loss and downtime. Checkout targets minutes of data loss and less than one hour of interruption. Site Recovery protects its supported VMs; the managed orders database needs its own zone and cross-region design, such as Azure SQL Database zone redundancy and a failover group. The team must check replication lag and test the full order path. The catalogue targets 24 hours of loss and four hours to restore, subject to a timed restore and, if regional loss matters, a check of the secondary backup point. Scratch calculations can be regenerated from protected inputs.
In the fictional drill, the VM starts, test identity and payment services respond, and a synthetic order completes in a network verified as separate from production. The team records what was tested and what remains outside the drill. In a later fictional outage, checkout returns in forty minutes after the team verifies dependencies and routes traffic. The plan compares that measured time with the one-hour target; it does not claim zero data loss without checking the recovered orders against the source and the selected recovery point.
The exercise — sort the estate, sign the numbers, write the sentence
Parts 36d and 36e explained backup and Site Recovery and let you practise decisions in simulations. They did not create Azure resources or demonstrate a live recovery. This exercise asks you to write the targets, design and evidence you would need before building a real plan.
The exercise below asks you to draft a small plan on paper. Decide a tier, proposed RPO and RTO, recovery design and test for each workload. Then compare your answer with the example under the steps. The sample is conditional: the business owner must agree to the targets, and a test must show whether the design can meet them.
Three workloads, no tiers assigned yet — sort them
A small logistics company hands you this estate on your first day. For each workload, write down the tier, the RPO, the RTO, and the mechanism from this class that actually produces those numbers.
Dispatch board — the live screen every driver checks before a route. Down for more than fifteen minutes, trucks sit idle and the business loses money by the hour.
Customer invoice history — read constantly, changed rarely. Losing a day of it means re-keying yesterday's invoices from paper copies: annoying, not fatal.
Route-planning scratch calculations — a nightly job re-derives them from the dispatch board and the day's orders. Nobody has ever needed yesterday's copy.
Then state which derived copy you would leave unbacked, which source data must survive, how long rebuilding takes, and who accepts that delay.
Compare a sample answer
Dispatch board — Tier 1 candidate. The story sets a fifteen-minute maximum interruption, so the example one-hour Tier 1 target is too loose. Ask how many minutes of dispatch changes may be lost. Design every critical component for the agreed targets, then run a full failover test and time when drivers can use the board again.
Invoice history — Tier 2 candidate. If the owner accepts re-keying one day of invoices, propose a 24-hour local RPO. A daily backup is a candidate. Agree on the downtime target, restore a representative copy and time it. If regional recovery is required, check the age of the secondary copy too.
Scratch calculations — Tier 3 candidate. Accept loss of the derived copy only if dispatch and orders remain available and the nightly rebuild is documented. A useful acceptance sentence is: “We do not back up the scratch calculations; the operations owner accepts the rebuild delay because protected source data lets us recreate them.”
The recovery nobody had to be a hero for
A partner deletes a critical folder, or a region becomes unavailable. Open the documented runbook, identify the latest usable recovery point, and follow the recovery order. Verify that the service and its dependencies work before routing users to it. Record the actual point age and elapsed recovery time. If either misses its agreed limit, tell the owner and improve the design before the next incident.
Resilience is a number you buy, not a feature you enable
This class has shown how to compare failure scenarios and propose recovery targets. The work is complete only when the plan has been tested. A backup job can succeed while the wrong data is protected. A VM can start in a test region while its database or payment connection fails. Trial restores and full application failovers reveal those gaps before a real outage.
Write recovery steps so another engineer can follow them. Name the decision maker, dependencies, required access, communication channel, traffic switch, data checks and rollback. Keep the latest test result with the runbook. Class Thirty-Seven goes deeper into the databases whose recovery needs a plan of its own.
Choose protection per workload
Use the fictional evidence below. This is a desk exercise; it does not provision Azure resources or measure a live service.
Paid orders: target 15-minute recovery, 1-minute data loss Daily report: target next morning, 24-hour data loss Targets supplied by fictional owners
Explain why the two workloads need different protection.
Complete a report-job example before designing the orders path.
Name what a rehearsal must measure for each.
On your own: the report owner now needs it by 09:00 every day, even after a bad night. Restate the report's targets and what a rehearsal must show, before naming any Azure service.
Check your reasoning after you finish
On your own: the recovery time target tightens to "ready by 09:00", while the data-loss tolerance may stay at a day. The rehearsal must show a restore finishing before 09:00 from the latest usable point. A daily report can tolerate a slower restore from a suitable backup if it meets the supplied targets. Orders need a design evaluated against much tighter targets. Neither replication nor a backup schedule alone proves the required recovery.
Examination
Four drills, then two situations. Write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
A — configured zone resilience addresses a zone failure. It cannot cover loss of the region containing all those zones. B extends zone protection too far; C ignores zone failover; D confuses zones with regions. Check that every critical dependency is also zone-resilient. A second region helps with regional loss only after the application and data are configured and tested there.
B — test a design that fits the target. A daily local backup is a candidate for the 24-hour data-loss limit, but inspect the actual usable point. Only a timed restore of the service can show whether four hours is achievable. If the requirement includes loss of the whole region, a GRS vault and Cross Region Restore need separate setup and the secondary point may be older than 24 hours. A and D spend on tighter replication without a stated need; C discards catalogue data the business said it needs.
Targets, test evidence and accepted gaps belong together. Name the whole workload and its agreed RPO and RTO. Describe the recovery design and record tests against those numbers. State what can be lost and who accepted it. Tightest possible settings for every workload may waste resources; successful backup jobs alone do not demonstrate recovery.
# campux-retail — BCDR plan (tier summary)
1. Managed Azure SQL orders DB (Tier 1): zone
redundant; failover group for regional recovery;
monitor lag and test checkout.
2. Storefront VMs (Tier 1): Site Recovery to a
supported second region; test application routing.
3. Product catalogue (Tier 2): RPO 24 h, RTO 4 h —
a nightly Azure Backup gives an RPO of 5 minutes.
4. Analytics scratch (Tier 3): no backup of the
derived copy; source protected, rebuild documented.
Line three contradicts its own backup schedule. If the last successful backup was made the previous night, recent changes will not be in that point. The line's 24-hour local target may be reasonable, but the five-minute claim is not. Delete that claim, inspect the actual point age and time a restore against the four-hour target.
Line one gives a managed Azure SQL Database its own zone and regional recovery features; Site Recovery in line two protects the separate VMs. Line four accepts loss of a derived copy only because its source is protected and the rebuild is documented. None of those lines alone proves the full workload target, so the plan still needs an end-to-end test.
Ask what zero means. No system can honestly promise that every component will have zero data loss and zero interruption under every failure. Ask which transactions cannot be reconstructed and how long each service can be unavailable. Bring real outage impact and cost estimates, then let the owner agree on different targets for checkout, catalogue and derived analytics.
Present options with evidence. For each workload, show the proposed local and regional recovery design, its cost, known gaps and the test needed to validate it. A Tier 1 database may need a different cross-region feature from the storefront VMs. State where asynchronous replication can leave recent writes behind.
Record the decision and owners. Leave with agreed RPO/RTO targets, an owner for each dependency, a test schedule and explicit acceptance of any regenerable-data loss. Revise the plan if tests show the proposed design cannot meet those targets.
A strong answer concedes before it argues. The predecessor did real work — backups exist and run, which is more than many shops manage — and opening with that keeps the room on your side. So say it plainly: "We're in a genuinely good starting position; backups are configured and succeeding." Then the pivot, framed as method not blame: a successful backup job proves data went in; only a restore proves it can come out, and no one has tested the coming-out. That gap is a property of the setup, not a fault of the person — the same blameless logic Class Thirty-Six installed: any failure a competent human can leave behind is a missing guardrail, here the missing restore test.
Measure one safe restore. Restore a representative non-production copy to an alternate location and check its data and application behavior. Record the chosen recovery point's age, total elapsed time, cost and any manual steps. This tests a slice of recovery; it does not by itself prove the whole service's RTO.
Expand and repeat. Use the first result to plan a test of critical dependencies and a realistic workload size. Agree on a cadence and test after significant changes. Present this as strengthening the existing backup work, with outcomes compared against the business's targets.
Five things worth carrying out of this part
- Zone resilience covers a zone failure only when critical workload components are configured for it. A region pair alone does not replicate or fail over an application.
- RPO is acceptable data loss; RTO is acceptable time until the service works again. Agree on targets per workload, including dependencies.
- Site Recovery protects supported VMs. A managed database needs its own recovery design. Daily local backup can be a candidate for a one-day RPO, but restore time and secondary-region point age must be checked.
- A BCDR plan names owners, recovery steps, traffic routing, communication, tests and accepted gaps. Regenerable data can be left unbacked only if its source and rebuild path are sound.
- Measure recovery-point age and complete service restoration in drills. Assigning a tier or seeing a successful backup job does not prove the targets are met.
- Microsoft distinguishes zone resilience from region resilience and says services differ in how zone support is enabled. Zone resilience guidance ↩
- Azure SQL Database uses separate features for zone redundancy and regional recovery. Its cross-region replication is asynchronous, so forced failover can lose recent changes. SQL Database reliability ↩
- The tier numbers are fictional business targets, not Azure defaults. For Azure VM backups, Microsoft says the secondary-region recovery point can be up to 36 hours old even when the vault uses GRS and Cross Region Restore. Backup vault and Cross Region Restore guidance ↩
Propose recovery targets with a business owner, then plan tests to check the design.
Play the lesson above to stamp this class.