App Insights & alerting

Everything Class Twenty-Eight collected watched from outside: the platform's view of your resources — CPU numbers, blob operations, job executions. What it cannot see is the inside of the application: which HTTP request was slow, which database call it was waiting on, which exception fired on which line. Application Insights is Azure Monitor's instrument for that interior view — wire it into the storefront (for many platforms, without touching code; for the rest, a small OpenTelemetry package and a connection string) and the app begins narrating its own execution into the same workspace family you already query.¹

App Insights

Application performance monitoring from instrumented code. Collected requests, dependency calls, and exceptions can be correlated to show where a slow checkout spent its time. Sampling and instrumentation settings determine which records are available.

Four telemetry types carry most of the value, and they interlock. Requests: every call your app served, with duration and result. Dependencies: every call your app made — SQL, storage, third-party APIs — which is where most slowness actually lives. Exceptions: every error, with stack trace. Traces: the app's own log lines, stitched to the request that produced them. The stitching is the product: one slow checkout request expands into its dependency calls and exceptions as a single correlated story, and the application map draws the whole system — services, dependencies, failure rates on every edge — from that telemetry automatically. Class Twenty-Nine's skills apply unchanged; these are just richer tables.

Custom events and metrics — the telemetry you write

The four automatic types carry the plumbing; the two you add by hand carry the business. A custom event is a named thing that happened in your domain — cart-abandoned, coupon-applied, checkout-completed — recorded in a line of code with a bag of properties attached: which coupon, which store, which basket size. A custom metric is a number you emit per transaction or on a schedule — queue depth, active carts, dollars per order — pre-aggregated at the source so it costs little and charts fast. The distinction that catches people: an event is a record you can slice and count any way you like afterward; a metric is a number already summarised — cheaper to keep, but no longer decomposable into the transactions that made it. Instrument the events your business would ask about in a meeting, because the platform will never know that "coupon applied" matters. That half of the model is yours to name, and naming it well is a quiet part of what a cloud engineer is paid for.

Distributed tracing — one operation across many services

Suppose the storefront calls a pricing service, which calls a database. With compatible distributed-tracing instrumentation, those components propagate trace context, and Application Insights can connect their telemetry through an operation identifier. An end-to-end view may then show where a slow request spent time. Missing instrumentation, broken propagation, or sampling can leave gaps. Check that the services appear in the same trace before drawing a conclusion from the application map.

Live Metrics — watch a deployment

Regular telemetry takes time to ingest and query. Live Metrics provides a near-real-time view of supported request, failure, and performance signals from instrumented instances. During a deployment, watch it alongside your other health checks and deployment gates. A blank or healthy Live Metrics view is not proof of a healthy release: confirm that the new instances are connected, traffic is reaching them, and the expected signals are present before deciding whether to continue or roll back.

Sampling — why the bill and the traces shrink together

Sampling can reduce telemetry volume and cost, but it also means an individual request might not be stored. Some SDKs support adaptive sampling before data leaves the application; ingestion sampling acts after data reaches Azure. Check the settings of your actual instrumentation rather than assuming either is enabled. Aggregated counts may need the telemetry's sampling weight, and an absent trace cannot prove a particular customer request never happened. If you need a complete audit or billing record, design and validate that record separately from sampled monitoring telemetry.

All telemetry so far shares a blind spot: it reports what happened to traffic that arrived. If DNS breaks, the certificate expires, or the front door falls off the internet, the app records nothing wrong — no requests, no errors, a peaceful dashboard over a dead site. The fix is a customer who never sleeps: an availability test, which requests your URL on a schedule from multiple Azure regions around the world and records success, failure, and response time from each.

A standard test requests a public URL and can check its response code, page content, and TLS certificate. A content match helps catch a page that returns HTTP 200 but does not show the expected content. A proactive certificate check can warn before expiry when configured and its alert reaches someone. Microsoft recommends at least five test locations and an alert threshold of at least three failures out of five; for more locations, its guidance is locations minus two.² Multiple locations reduce pages caused by one failing test location, but the tested URL and content must still represent what customers need.

Detection without delivery is a diary. Azure separates the two cleanly: an alert rule is scope + condition (a metric threshold, a failed availability test, or a Class Twenty-Nine log query on a schedule), and an action group is a named, reusable list of destinations — email, SMS, push, a webhook into Teams or Slack, an ITSM ticket, even a Logic App or Azure Function that does something. Rules reference action groups, many-to-one, so "who gets told" is maintained in one place instead of inside forty rules:

# one action group, reused by every alert that pages on-call
az monitor action-group create \
  --name ag-campux-oncall \
  --resource-group rg-campux-platform \
  --short-name oncall \
  --action email oncall-primary oncall@campux.co

Two disciplines make the machinery trustworthy. First, define what each severity means for your team: which conditions page on-call and which wait for business hours. Azure labels alert severity from Sev0 to Sev4, but the label alone does not set the recipient or response time. Second, give each rule an owner and an action group that reaches that owner. A rule can be configured as stateful, so it fires when the condition begins and resolves after the signal recovers; verify that behavior in the rule instead of assuming every alert works that way.

One more piece of machinery earns its line: an alert processing rule can suppress notifications for a scope and a window — the Saturday-night storefront maintenance, the planned failover test — so the work proceeds without paging anyone and the rules themselves stay armed. The alternative, disabling alerts by hand and re-enabling them from memory, is how monitoring acquires permanent holes.

Metric alerts, log alerts, and a threshold that learns

A rule's condition comes in a few shapes. A metric alert evaluates a stored time series such as CPU or request failures at the frequency you configure. A log alert evaluates a KQL result on a schedule; it can ask more specific questions, but ingestion and evaluation can delay the notification, and pricing depends on the rule and data plan. An activity log alert watches events such as resource changes or service-health advisories. Choose the signal that directly answers the operational question, then inspect the configured frequency and cost.

Static thresholds need tuning against normal workload behavior. Dynamic thresholds are supported for many metric signals and log-query results, but they need enough history and have configuration limits. They identify deviations from a learned pattern; they are less suited to slowly worsening conditions, and a new resource may not alert until enough data exists. Start with the customer impact you need to detect, then test both the threshold and the notification path.

Smart detection — the alerts you did not write

Some failures announce themselves without a rule. Smart detection watches an app's telemetry for anomalies you never thought to threshold — a sudden rise in failed requests, a creeping response-time degradation, a spike in exceptions against one dependency — using the app's own recent behaviour as the baseline, and emails when the pattern breaks. Treat it as a smoke detector, not a strategy: it catches the class of problem nobody wrote a rule for, but it does not know your business, cannot be trusted to page on-call, and earns a severity only after the same §4 examination every deliberate alert must pass. It is a useful second net beneath the three chosen alerts — and a poor substitute for choosing them.

Here is where monitoring programs actually die. Not from missing tools — from a pager that cried wolf. The failure sequence is always the same: enthusiasm creates forty alerts; half fire on conditions that need no action; on-call learns, correctly, that most alerts mean nothing; the learning generalises; and eighteen months later a real Sev0 scrolls past unread in a channel nobody has muted only because muting felt like admitting it. The team then buys a new monitoring tool, and the cycle restarts with better dashboards. Alert fatigue is not a personnel problem. It is a signal-to-noise problem, and it is designed in or designed out.

Every alert is a claim on someone's attention.

Actionable

There is something a human should do upon receiving it — and the alert says what, ideally with a runbook line. "FYI, CPU was briefly high" is not an alert; it is a chart that belongs on a dashboard, per §5.

Owned

A named person or rotation answers it. An alert routed to a twelve-person alias is owned by nobody — everyone assumes someone else is looking, and everyone is right.

Rare

It fires only when the condition is real: thresholds tuned to actual behaviour (not defaults), state-based so one incident is one notification, and reviewed on a cadence where anything that fired without requiring action gets retuned or deleted — the same ratchet as Class Twenty-Nine's query pack, run in reverse.

Before a rule pages someone, ask whether the condition requires action now, who owns that response, and what they should do first. Use the team's severity policy to choose the notification route. Review firing history after deployment: a quiet rule that misses real failures also needs repair. The right number of alerts depends on the service and its risks.

What happens to everything that failed the 3am test? It becomes glanceable instead of interruptive — and here too there is a discipline separating useful from decorative. The decorative kind is familiar: a wall of sixteen charts, assembled in week one and understood by nobody by month three, whose actual function is looking impressive behind the team during video calls. The useful kind is built backwards from questions. Before adding any panel, name the question it answers and who asks it: Is checkout healthy right now? (on-call, hourly). Did last night's batch run? (the storefront team, each morning). Is ingestion cost drifting? (the platform lead, monthly). If no one asks the question, the panel does not earn the pixels.

Azure's tool for this is the workbook — an interactive document mixing prose, parameters, and live Class Twenty-Nine queries, closer to a runbook with charts than to a TV wall. The strongest pattern wires the layers of this class together: the alert tells you something is wrong; the workbook it links to shows the affected service's requests, dependencies, and recent deploys side by side — the first three investigation queries pre-run before the engineer has poured coffee. Alerts interrupt; dashboards answer; workbooks investigate. Keep the three jobs separate and each stays good at its own.