What counts as an incident
Every class before this one assumed the systems were up. This one begins at 2:14 on a Saturday afternoon when they are not. An incident is an unplanned event that disrupts a service or puts its users, data, or business outcome at risk. A warm disk may only need investigation; a warm disk followed by failed checkout requests is an incident. A late report may be minor for one team and critical for another. Agree on the impact that triggers a response before you are on call, so you can act without debating the word during an outage.
- Incident
- An unplanned degradation or outage of a service the business or its customers depend on — measured by impact, not by cause. A failed disk, an expired certificate, and a bad deploy are three different causes of the same thing: something that worked no longer does, and it was not on the calendar.
Unplanned matters, but customer impact still matters during planned maintenance. If checkout is taken down in an agreed window with notice and behaves as expected, follow the maintenance plan. If the work overruns or breaks more than planned, declare and manage an incident. This part helps you decide what happened, how severe it is, and who needs to hear about it.
Those three decisions are the spine of this part. Define the thing (this section), size it (§2), and detect it (§4), with the loop that carries you from the alert to the handover in the middle (§3). None of it is glamorous, and all of it is the difference between an outage that becomes a headline and one that becomes a footnote. The engineer who has done this thinking in advance is the one the room turns to and finds already calm.
How bad is it — the severity ladder
The single worst question to be negotiating at 2:19, with someone senior typing "???" into the channel, is "how bad is this?". So you answer it in advance, once, with a ladder everybody has agreed to: a small fixed set of severity levels, each attached to a concrete response expectation. Severity is not a measure of how upset anyone is; it is a measure of blast radius and business impact, and its only job is to convert a panicked judgement call into a lookup. You read the symptom, you find the row, you get the response — who to page, how fast, and whether anyone is allowed to keep sleeping.
Campux's ladder has four rungs. The top three it settled the week it went live; the fourth it added once it had enough small annoyances to need a rung that means "real, but it can wait." The numbers and thresholds here are Campux's, not a law — a bank and a games studio would draw the lines differently — but the shape is universal: one ladder, few rungs, each with a response the on-call engineer does not have to invent.1
| Level | What it means | Example | Response expected |
|---|---|---|---|
| Sev1 | Revenue or data actively impacted; a core service is dark for everyone | Checkout returning errors in November | Page everyone the runbook names, now; declare an incident with one owner |
| Sev2 | Significant degradation, but a workaround exists or one part still works | Search is down while browsing and checkout hold | Page the on-call engineer; let the rest of the team sleep |
| Sev3 | Annoying and contained; no revenue or data at risk | A back-office report renders the wrong totals | A ticket, fixed in business hours; no page |
| Sev4 | Cosmetic or low impact; noticed but harmless | A footer link points at last year's returns page | Logged to the backlog; scheduled, not paged |
Two habits keep the ladder honest. First, severity can move: you open at your best guess and re-declare the moment scope changes — a Sev2 search outage that turns out to be taking checkout with it is a Sev1, said out loud, and the extra people get paged. Under-calling to avoid waking someone is the expensive mistake, because the cost of a needless page is one grumpy engineer and the cost of a missed Sev1 is the November afternoon. Second, the ladder lives where the paged person can read it at 2am — in the runbook, not in someone's head — because a severity scheme only you understand is not a scheme, it is a bottleneck with your name on it.
The loop you run under pressure
During an outage, it is easy to chase the first possible cause while customers are still getting errors. A short, rehearsed loop gives the team a safer order of work: detect, triage, mitigate, learn. Communicate status and next steps throughout the loop. You can investigate the cause while restoring service; do not delay a safe mitigation just to finish a complete root-cause analysis.
This part owns the first verb and the setup for the second; Parts B and C take the rest at full depth. But you hold the whole shape now, because the order is the discipline. Detect: the machinery notices before a customer does and pages a named owner. Triage: three questions in five minutes — what is broken, for whom, and what changed — producing a severity call and one owner. Mitigate: make it stop hurting by the fastest safe path, which is explicitly not the same as fixing it. Learn: days later, the blameless postmortem that retires the whole class of failure. The figure below is that loop, drawn once against the outage the rest of this class refights.
Verify the impact. Tell people. Restore safely.
Detection — hearing it before a customer does
The impact clock starts when the service breaks, not when someone notices. The interval between impact and detection is time to detect for this incident. MTTD means mean time to detect: the average of that interval across a defined set of incidents. Aim to learn about customer-impacting failures promptly, preferably before customers report them. If a customer reports the problem first, record that detection gap and improve the monitoring afterward.
The Class Twenty-Eight-to-Thirty toolkit can turn a failure into a notification. An availability test is an automated request sent from outside your network to check a user journey such as checkout. In this example, tests run from five locations on a five-minute schedule; the alert rule fires after its configured failure criteria are met. That does not promise a page within five minutes: evaluation and notification add time. Metric alerts watch measures such as failed-request rate and latency against thresholds you set. An action group defines how the alert reaches its receivers. Configure a tested phone, SMS, or other escalation path to the current on-call engineer; an action group does not maintain the rota for you.
Page on symptoms that need a human now, such as failed checkout requests. A CPU threshold can still help diagnose a problem, but a CPU-only page may fire during healthy demand and miss failures caused by something else. Route it to a dashboard or lower-urgency alert unless your service evidence shows that threshold reliably predicts impact. Keep the availability and errors-by-operation views bookmarked so the responder can verify the page quickly. Review false pages and missed incidents; both tell you whether your alert rules need adjustment.
Play it through
Three minutes. In a fictional checkout outage, compare three ways a team might discover the failure. Wire a user-path check, then move a noisy CPU-only page out of the paging route. The clock and customer count are teaching examples, not measured production data. The activity pauses when it needs your input.
What being on call actually asks of you
All of this converges on one arrangement most beginners meet only in production: the on-call rota. Being on call means that for a defined window — a week, usually — you are the human the action group pages, and you have agreed to carry a device that can wake you and to respond within a stated time. It is not a promise to fix everything alone; it is a promise to be the first responder — to acknowledge, to triage, to start the loop, and to escalate when the severity or the scope outgrows you. The person who owns the incident on a Saturday is not the most senior engineer available; it is whoever the pager reached.
A healthy on-call arrangement is a system, not an act of endurance, and the difference is worth naming because it is a thing you should ask about in an interview. It has a written runbook so the paged person is not improvising the severity ladder at 2am. It has a real escalation path — a second name to call when the first cannot answer or the incident is beyond them — so the rota survives one person being asleep or on holiday. And it counts the cost: pages that fire and turn out to be nothing are not free, they are trust and sleep spent, and an on-call rota that burns its people is one that will page an empty chair within two quarters. The senior signal is not the willingness to be woken; it is having built the machinery so that you rarely are, and so that when you are, the page is real.
The week Campux decided what an incident is
Campux Retail is a fictional case. In its backstory, a checkout outage lasted nine hours and a customer reported it first because no useful alert reached an owner. Now the team writes impact-based severity criteria, tests checkout from outside its network, and connects the alert to a tested on-call notification and backup contact. The setup improves the chance that the team finds the next fault promptly; it does not guarantee a fixed detection time.
In the next part of this fictional case, the alert reaches on-call before a customer report. The engineer confirms the impact, communicates it, and begins triage. In a real system, repeat the test regularly: a configured alert is only useful if its receiver actually gets and acknowledges it.
The pager finds a person, not a mailbox
You are on call for the first time, and the arrangement you inherited tells you almost everything about the team. There is a runbook with a severity ladder you can read half-asleep; there is a second name to ring when a Sev1 is beyond you; the alerts that fire are the ones that mean customers are hurting, not the ones that mean a graph twitched. When your pager goes off, it is because something real broke — and because someone built the detection so that you, not a customer, found out first. Ask about all of this in the interview; a team that cannot answer is a team where being on call means being afraid.
Build the incident clock
Use the fictional evidence below. This is a desk exercise; it does not provision Azure resources or measure a live service.
09:00 first failing checkout 09:04 alert delivered 09:07 engineer acknowledges 09:12 customer-impact statement confirmed
Calculate detection and acknowledgement delays separately.
Write the customer-impact statement without inventing a root cause.
Choose one alert change and state how you would check that it catches this failure.
On your own: a new clock. 14:00 first failed sign-in, 14:09 alert delivered, 14:10 acknowledged. Work out both delays, then write the first customer update in two sentences without guessing a cause.
Check your reasoning after you finish
On your own: detection took nine minutes and acknowledgement one. The update says what is failing, for whom, and when the next update comes, and nothing about why. Detection is four minutes from first failure; acknowledgement is three minutes after the alert. State that checkout is failing for the observed callers. Root cause is not established by an alert firing.
Examination
Four drills, then two situations. Write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B. The recent deploy is a useful clue, but timing alone does not prove cause. If the rollback is compatible with current data and state, it is a reversible way to restore service while the team keeps investigating. Tell the incident owner what you are doing, check the error rate after rollback, and change course if it does not recover. Restarting every instance has no evidence behind it here; drafting a postmortem can wait until customers can check out.
B. Severity is blast radius, not the presence of any fault. Search being down is a real, customer-facing degradation — that is why D is wrong; unplanned degradation of a service customers use is an incident even when a workaround exists. But checkout holds and shoppers can still buy, so it is not the November-afternoon emergency that justifies waking everyone the runbook names: over-calling this to Sev1 spends the whole team's trust and sleep on something one rested engineer can handle in daylight-adjacent hours. C under-calls it — a page-nobody Sev3 leaves a broken shopping experience unowned overnight. The point of the ladder is exactly this lookup: read the impact, find the row, get the response, and do not re-argue it at 2am.
The external checkout test, tested on-call route, and failed-request alert. These check customer impact and give someone responsibility for responding. CPU and log warnings can remain useful diagnostic signals, but paging on every occurrence makes urgent alerts harder to recognize. Test the alert rule and the notification route, then review both false pages and missed failures.
# campux on-call & alerting — v1
1. Availability test calls checkout every 5 min from
5 locations; fails in 3 of them -> fires.
2. Alert routes to an action group that pages the
named on-call engineer for the week.
3. Also page on CPU > 80% on any instance, even
when checkout succeeds normally.
4. Escalation: if on-call does not ack in 10 min,
page the secondary; runbook holds the ladder.
Line three. CPU over 80% might mean healthy demand, while checkout can fail for a reason that leaves CPU normal. Here, page on failed checkout or a validated customer-impact measure; retain CPU for diagnosis. The five-location test needs an explicit failure and evaluation rule, so its five-minute cadence does not guarantee a one-minute page. The named receiver and secondary contact are sensible, provided the ten-minute escalation is tested and meets Campux's response target.
Acknowledge first, decide second. The first move is to ack the page — not because you have assessed it, but because acking stops the escalation clock from paging your secondary over something you are already looking at, and it tells the system a human has the ball. Only then do you triage. Ambiguity is not a reason to dismiss and not a reason to panic; it is a reason to gather one or two more facts cheaply before you spend anyone's sleep.
Check the customer impact. A green synthetic test only proves its sampled path succeeded; another operation or group of users may still be failing. Check whether errors are rising, which operation fails, and whether real-user requests are affected. If impact is confirmed or the error rate is growing, declare at the appropriate severity and escalate. If the signal stays small and contained, monitor it against a short, explicit checkpoint rather than dismissing it.
Whatever you decide, write it down and set a checkpoint. Log the time, what you saw, and your call — "2:11, errors flat on one operation, avail green, watching, re-check 2:25." If it was nothing, that line tunes the threshold so this alert is quieter next week; if it grows, you have a timestamped start for the postmortem. The judgement being tested is the refusal to treat ambiguity as a coin-flip: you do not go back to sleep on a hunch, and you do not wake five people on one either — you spend five cheap minutes buying certainty.
Concede the cost before you defend the page. Your colleague is right that waking someone at 3am is expensive, and if you argue otherwise you sound like you enjoy heroics. So agree: a page is a real cost, which is exactly why you page on almost nothing — only the handful of symptoms that mean customers cannot transact. The disagreement is not about whether to protect people's sleep; it is about which failures are allowed to interrupt it.
Compare the response times. If the team only reads email in the morning, a checkout failure overnight can continue for hours before anyone starts recovery. A tested on-call route can shorten that delay, though neither its timing nor revenue impact is guaranteed. Agree on a meaningful failed-request threshold, test the page and escalation, and measure the actual detection and restoration times. Route lower-impact issues to a ticket.
Reframe the ask as tiering, not paging-versus-email. The answer that ends the debate is that you already agree with the principle and have encoded it: the severity ladder is precisely the mechanism that sends Sev3 and Sev4 to the morning queue and reserves the page for Sev1. So the real question is only "is checkout being down a Sev1?" — and if the business will not fund a 3am page to protect its own checkout, that is a decision to make in daylight, in writing, not a default to reach by muting the one alert that pays for itself.
What Part I leaves you holding
- An incident is unplanned degradation of something the business or customers depend on — defined by impact, not cause. Agree the word before the Saturday, not during it.
- Severity is a lookup, not a debate: one ladder, few rungs, each with a response. Campux runs Sev1 to Sev4, and severity can move the moment scope changes.
- The loop is detect, triage, mitigate, learn. Communicate throughout. Verify impact and choose a safe mitigation while investigation continues.
- Detection needs tested checks, meaningful alert thresholds, and a working route to the current on-call person. If a customer reports a failure first, measure and improve the detection gap.
- Page on customer impact. Keep internal metrics for diagnosis or lower-urgency alerts unless they reliably predict impact in your service.
- Treat the specific Sev1–Sev4 numbers, examples, and thresholds here as Campux's, not gospel — the direction is settled, the exact lines move by organisation. Sev1–Sev4 is general incident-management and SRE convention; note that Microsoft's support plans use a separate A/B/C severity for raising cases with Microsoft, which is a different scale for a different purpose. Agree your own ladder and write it in the runbook. ↩
Define severity levels and know how an incident is detected before a customer reports it.
Play the lesson above to stamp this class.