What changed? — the question that finds most outages
Checkout worked at eleven and fails at eleven-fifteen. Start by checking what is failing, who is affected, and when it began. Then ask what changed near that time: a release, a setting, traffic, a dependency, or an expired certificate. A matching timestamp is a clue, not proof of cause. Some incidents happen without a change in your own system, so keep the customer symptoms in view as you investigate.
Use two views together. Request telemetry shows when errors rose and which operation failed. Change records show what the team or Azure changed near that time. The records narrow your suspect list; they do not identify the cause by themselves. Keep an incident owner and a status channel active while someone checks the evidence and another person prepares a safe mitigation.
Azure's Activity Log records subscription-level control-plane events, such as resource updates and role assignments.1 It is collected by default. It does not replace your deployment pipeline history or application request telemetry. Open the sources that cover the suspected action, and confirm their timestamps before deciding what to reverse.
Reading the ledger — control plane versus data plane
The Activity Log covers management operations on Azure resources, called the control plane. Application Insights request telemetry can show failed checkout requests, but only if the app was instrumented and telemetry is available. The pipeline's release history shows which build was deployed. In this fictional case, the pipeline records a 2:07 release and request telemetry shows failures shortly afterward. Check the Activity Log too for related resource or slot operations. These records answer different questions.
- Activity Log
- Azure's default record of subscription-level management events. It commonly shows resource changes and the caller. It does not contain every application deployment or request. Azure retains the built-in entries for 90 days.
The Activity Log is collected automatically, but its entries typically take 3–20 minutes to become available. A very recent change may not appear yet. The built-in retention is 90 days; export it if your incident review needs a longer record.2 Use the caller and operation fields when present, then verify the exact release and build in the pipeline. A missing Activity Log entry does not rule out a deployment.
Under pressure you do not browse the whole log; you read a fixed shortlist of change sources against the bend time, and stop at the first that lines up. The list below is that shortlist, in the order a Saturday actually asks for it.
- Activity Log, recent window
- Resource-level management changes such as settings, scale operations, and role edits. Remember that new entries can arrive several minutes late.
- Deployment history
- The pipeline's record of releases, commits, approvals, and target slots. Check it directly to identify the build; do not assume every code deployment appears in the Activity Log.
- App telemetry, the bend
- The failed-request chart and one errors-by-operation query fix the minute it broke and which operation broke. This is the when you align every candidate change against.
- Configuration and change history
- Inspect the relevant resource settings or your infrastructure-as-code change for a suspected configuration edit. Check which settings stay with an App Service slot during a swap.
Find when errors rose, then compare that timestamp with the release and infrastructure records. If a candidate lines up, test whether it could explain the affected operation. A nearby change deserves investigation; it does not justify an unexamined reversal.
Mitigate first, root-cause second
If the recent release is a likely cause and checkout is still failing, prepare the quickest safe mitigation. Check whether the previous build is available and whether schema changes, data writes, or slot-specific settings make a rollback unsafe. If it is safe, reverse the release and watch real checkout requests and health checks. The team can investigate root cause in parallel or after service is stable; customers should not wait for a complete explanation when a safe restoration path exists.
Restore safely. Verify. Then finish the explanation.
Several actions can reduce harm, depending on the evidence: reverse an App Service slot swap when the previous build remains healthy, redeploy a compatible earlier release, disable a faulty feature behind a flag, or add capacity if saturation is confirmed. Serving reads from a replica requires an architecture that already supports it and a decision about stale data. Each option has prerequisites and a verification step. Choose the smallest action likely to restore the affected user journey.
Make one coordinated change at a time when possible. Announce the action, who owns it, and what signal will prove it helped. Record the timestamp and result before trying another lever. Some emergency actions are not fully reversible, so assess their data and security effects first and get the incident owner involved.
Play it through
Three minutes. In this fictional case, a release is the leading suspect and the previous build is available. Check rollback safety, choose a mitigation, verify checkout, and record the action. The game pauses when it needs your input.
The levers — and when to reach for each
A mitigation has to match the failure you can observe. Scaling out will not fix a faulty release; rolling back code will not fix a provider dependency outage. Use the evidence and prerequisites below to choose a lever, then confirm that the affected customer journey actually recovers.
| Lever | Reach for it when | What it does, and the catch |
|---|---|---|
| Slot swap back | The previous App Service build is still in the other slot and compatible with current data and settings | Swap the slots again. Check which settings stay with each slot; warm-up and swap can take time. Verify checkout afterward. |
| Pipeline redeploy | A previous compatible release artifact is available and a slot swap is unsuitable | Deploy that exact artifact through the approved pipeline. Duration depends on the app, tests, and release process. |
| Feature flag off | A new code path is the suspect, shipped dark or behind a toggle | Disables the path with no deploy at all. The catch: only works if the change was actually put behind a flag before it shipped. |
| Scale out | Healthy instances are saturated and more instances can serve the same requests | May add capacity after new instances become ready. It increases cost and will not fix faulty code or a shared dependency. |
| Read replica | The app was designed and tested to send eligible reads to a healthy replica | May preserve read-only features. Replica lag can return stale data; do not assume checkout can use it. |
Before acting, identify the owner, prerequisite, expected signal, and fallback. Record the result. A rollback may itself be unsafe after a database migration, and a feature flag may affect users who are already midway through a transaction. If you cannot establish safety quickly, escalate and choose a safer containment path.
One verified swap in a fictional outage
In this fictional Campux Retail case, a 2:07pm release starts breaking checkout. Request telemetry shows errors rising, and the alert reaches on-call at 2:14. The incident owner confirms that checkout is affected, checks the pipeline's release record, and reviews related Azure changes. The release is a strong suspect, but the team checks for incompatible schema or data changes before reversing it.
At 2:33 the owner approves a reverse slot swap. The previous build is still in staging and does not depend on a changed database schema. By 2:37, the swap has completed and checkout tests and live request errors have returned to normal. That is four minutes from the rollback decision and 30 minutes from the first failure, including the seven-minute detection gap. The team keeps the broken build and incident notes for investigation; a code path in the new build expected a configuration value that production did not have. The older build did not use that value.
When the fix does not fix it, and when to stop
If checkout still fails after a reverse swap, the incident is still active. Confirm which build is live and whether the swap completed, check the same customer-path signal again, and tell the incident owner the result. The release may not have been the cause, or another dependency may also be failing. Investigate the new evidence while considering another safe containment step; do not reverse more changes blindly.
If evidence points to an Azure service or region problem, check Azure Service Health and open a support request when appropriate. Continue customer communication and use your own tested fallback, if one exists. Do not wait to check provider health until every local option is exhausted; and do not assume the provider is responsible just because one rollback failed.
The usable habit is: confirm impact, compare telemetry with release and resource changes, select a safe mitigation, announce it, and verify the customer path. If the result differs from your expectation, update the incident record and investigate before the next action.
The engineer who reaches for the calendar first
Checkout is down and a manager asks for an update. You acknowledge the page, confirm the affected user path, and compare the error timeline with the pipeline release record and recent resource changes. The previous build is available, so you check rollback compatibility with the incident owner. After the approved reverse swap, you verify checkout and the error rate and report the result. In this fictional case, the decision-to-recovery interval is four minutes; the full failure lasts 30 minutes. Preserve the release and telemetry for the cause analysis.
Decide after a failed rollback
Use the fictional evidence below. This is a desk exercise; it does not provision Azure resources or measure a live service.
09:00 checkout 500 responses begin 09:02 release deployed 09:10 rollback completes 09:12 errors continue; database connections exhausted
Explain why release timing alone is not proof of cause.
Choose a reversible mitigation to investigate and record what would falsify it.
Write a short update with impact, action and next update time.
On your own: change one fact. Errors begin at 09:03, after the 09:02 release, and stop when the rollback completes at 09:10. What changes in your decision, and what still needs confirming?
Check your reasoning after you finish
On your own: now the timing and the recovery both point at the release, so keeping the rollback in place is reasonable. It is still a strong lead rather than proof; confirm with the change diff and the error detail before closing. The failures begin before this release, and rollback did not restore service. Investigate exhausted connections and their source; reduce or isolate the relevant load if evidence supports it. Keep customer impact separate from the unconfirmed cause.
Examination
Four drills, then two situations. Write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B. The team has checked that rollback is compatible, so it can try the previous build without waiting for a complete root-cause analysis. The incident owner coordinates the action; the team verifies the user path and error rate afterward. If rollback fails, keep the incident open and investigate the next candidate. A release near the failure is evidence, not proof.
B. The Activity Log is collected without a diagnostic setting and records management events such as resource and role changes. Entries may arrive several minutes after the operation. Use pipeline history for the exact application release and Application Insights for request failures. A Log Analytics workspace can hold exported Activity Log entries if that export was configured.
The compatible swap, feature flag, and pre-tested replica route. Each could reduce impact if its stated prerequisite holds; each still needs verification after use. Deleting the resource group risks data and service loss. Simultaneously editing several resources obscures which change helped or hurt. Choose and record a coordinated action, then check the customer path.
# triage runbook — checkout is down
1. Confirm the symptom and scope from telemetry;
note the minute the error rate bent.
2. Check pipeline history for releases and the
Activity Log for related resource changes.
3. Open the suspected deploy's diff and read it
fully; do not touch anything until I understand
exactly why it broke.
4. Tell the incident owner the impact and the
planned mitigation; keep stakeholders updated.
Line three. The runbook rules out all mitigation until someone has read every line and proved a root cause. A checked rollback may restore service sooner, though it still needs a safety check and verification. Lines one and two gather symptoms and candidate changes from the appropriate sources. Line four keeps the incident owner and stakeholders informed while recovery proceeds.
Verify what happened. Check that the reverse swap completed, identify the build actually serving production, and re-test checkout and the failed-request rate. Report the failed mitigation to the incident owner. It weakens the simple bad-release theory but does not prove another cause.
Investigate and contain in parallel. Compare the new evidence with pipeline history, resource changes, dependency health, and application errors. Try another lever only when it matches a likely cause and its risks are understood. A short code or configuration inspection may now be useful.
Check provider health when indicated. If a managed Azure dependency or region shows errors, inspect Service Health and consider a support request and your tested fallback. Keep the incident and customer updates running until the service is actually restored.
Agree that the cause matters. The team must identify and fix it before releasing the new build again. Ask whether the previous build is available and compatible with current data and settings. If so, a reverse swap can reduce customer harm while investigation continues.
State the check and the plan. “We have the earlier build and have checked rollback compatibility. I will reverse the swap with the incident owner's approval, verify checkout and the error rate, and stop if it fails.” This makes the mitigation testable.
Preserve the evidence. Keep the release artifact, pipeline record, telemetry, and incident timeline. Set a follow-up owner for root-cause analysis and a corrected release; do not assume that rollback itself fixed the underlying defect.
The part in five lines
- Confirm what failed, whom it affects, and when it began. Then ask what changed near that time.
- The Activity Log records Azure management events by default; pipeline history records application releases. Neither replaces request telemetry.
- Matching timestamps make a candidate, not a proven cause. Check whether that change explains the customer symptoms.
- Choose a mitigation only after checking prerequisites and risks. Announce the action, then verify the user path and error rate.
- If the first action fails, keep the incident open, report the result, and use the new evidence to select the next step.
- The Activity Log is scoped to the subscription and captures control-plane (management) operations only; data-plane actions — a read from a key vault, a query against a database — are not recorded there and require resource logs with a diagnostic setting. ↩
- Ninety days is the current default retention, and entries typically become available for analysis within three to twenty minutes of the event. Treat the exact figures as movable — Microsoft adjusts them, and your organisation may have exported the log to a workspace with far longer retention. Verify the current numbers on Microsoft Learn before you quote them in an interview. ↩
Triage an incident, mitigate first and fix second, and know when to roll back.
Play the lesson above to stamp this class.