Skip to content
CAMPUX Cloud Bootcamp Phase Four · Class Twenty-Eight
Phase Four — Operate, Secure & AI
4 drills · 2 situations · 2 Labs
Aligned to AZ-104
Class Twenty-Eight

Azure Monitor & Log Analytics

The batch job runs at two in the morning with nobody watching. Platform metrics already exist; this class shows how to collect its detailed logs, check the gap, and decide what needs an alert.

Phase Four · Operate, Secure, AI · Class 28 of 42 · 22 min · Assumes Class 10

After this you can send the right logs to a workspace you designed on purpose, and know what each one costs.

New to this? Read Class 10 first. Already done it, or know the material? Carry on.

§1

Metrics and logs — numbers and sentences

The job can fail while nobody is awake. Azure Monitor can collect several kinds of telemetry, but start with two. Platform metrics are numeric measurements such as request count or CPU use; Azure collects supported metrics automatically in its metric store, where Metrics Explorer can chart them. Resource logs are detailed event records such as a failed request. They reach a Log Analytics workspace only after you configure collection for that resource. Neither a metric nor a log sends a person a warning by itself; that requires an alert rule.1

Metric vs log
A metric measures a quantity over time; a log records an event and its fields. Either can support an alert or an investigation, depending on what data exists and the question you ask.

Ask "Is the storefront slow?" and a latency metric may answer quickly. Ask "Which store's upload failed, and what error did it return?" and you need an event record with those fields. First check that the application actually writes the detail you need, then that its collection route exists. Logs cannot explain an error the application never recorded.

§2

Diagnostic settings — route the resource logs you need

Azure collects supported platform metrics automatically into its metric store. Resource logs are different: choose supported categories and create a diagnostic setting to route them to a workspace or another destination. A setting created today does not backfill yesterday's resource logs. The subscription Activity Log is a separate exception: Azure collects management events automatically and retains them for 90 days, even without a workspace setting.2

Figure 1 — Telemetry drawn as plumbing: resources flow through a diagnostic-settings valve into the workspace; metrics bypass it storage account container app job key vault Resources — all talking resource logs diagnostic setting no route for these resource logs by default Log Analytics workspace tables · KQL · retention The reservoir — bills by the litre platform metrics → separate Azure metric store
Figure 1 — The valve between talking and remembering Supported platform metrics go automatically to Azure's metric store, drawn as the separate dashed route. Resource logs need a diagnostic setting to reach this Log Analytics workspace. An empty table can also mean no matching events, the wrong category or table, or ingestion delay.

The setting itself is small — a name, the log categories to route, and the destination workspace:

# open the valve on the storage account's blob service
az monitor diagnostic-settings create \
  --name send-to-campux-logs \
  --resource "$(az storage account show -g rg-campux-platform \
      -n campuxstorage --query id -o tsv)/blobServices/default" \
  --workspace $(az monitor log-analytics workspace show \
      -g rg-campux-platform -n campux-logs --query id -o tsv) \
  --logs '[{"category":"StorageWrite","enabled":true}]'

One storage-specific wrinkle in that command teaches a general lesson: the valve sits on the blobServices/default sub-resource, not the account. A storage account at top level emits only metrics; the read-and-write logs live on the services inside it — blob, file, queue, table — each with its own valve. Point the setting at the account and Azure refuses the category; point it at the service and the logs flow. Most resources are simpler than this, but the habit transfers: check which resource actually emits the category you want before wondering why the workspace is empty.

Opening settings one resource at a time is easy to forget. Azure Policy's DeployIfNotExists effect can deploy diagnostic settings for supported resource types in the policy's scope. It needs a managed identity with permission, and existing resources may need a remediation task. Policy evaluation is not instantaneous. Check compliance and query for fresh rows before claiming coverage; an assignment alone is not proof that every log is arriving.

Play

Play it through

Watch a customer report a missed job before anyone has configured an alert. Then route this job's future resource logs and apply a scoped policy for supported resources. The game stops for your choices and shows why log collection and notification are separate jobs.

§3

Workspace design — one or many

Routed logs can land in a Log Analytics workspace, in typed tables you query with KQL next class. Start by considering a shared workspace for related workloads, then test access, region, retention, and cost requirements before deciding whether to split:

Consider one
A shared workspace makes cross-service investigations easier when relevant records are collected, compatible, and accessible to the investigator. A common destination alone does not guarantee a useful join.
Split for a named requirement
Region and residency, access boundaries, retention, operational ownership, or a billing model that shared attribution cannot meet are reasons to consider more than one workspace. Document the query cost of a split.
Test access explicitly
Resource-context access and Azure RBAC can scope many queries, but workspace permissions and table access matter too. Verify a test identity can read only the intended records before putting sensitive teams together.

The anti-pattern has a recognisable shape: eleven workspaces, one per team, created over three years — and then an incident that spans two of them, and an engineer at 3am writing the same query twice and correlating timestamps by hand in a text editor. Fragmented logging converts every cross-cutting investigation into an archaeology project. One reservoir, well-governed, beats eleven puddles.

§4

Retention and its invoice

For many Log Analytics tables, ingestion volume and retention affect the bill. The charge depends on table plan, data source, and current Azure pricing; the subscription Activity Log and platform metrics follow different rules. Verbose resource-log categories can create significant volume. Retention decides how long a table remains available for normal queries and, where supported, how long it remains in long-term storage. Check the current plan and price of each table before changing its settings.3

You cannot query the logs you never kept.

The discipline is to make both dials deliberate, per table, because the failure modes sit on both sides. Keep everything hot forever and monitoring quietly becomes a top-five line item — the Class Thirty-Two FinOps review will find it. Keep too little and you meet §2's cruelty in a new costume: the breach investigation that needs ninety days of sign-in history in a workspace that kept thirty. Security-relevant tables (sign-ins, audit, firewall) earn long retention; chatty debug categories earn short retention or a closed valve. There is also a middle path between full price and silence: every table carries a plan, and moving a verbose, low-value table from the default Analytics plan to Basic or Auxiliary cuts its ingestion price sharply in exchange for limited querying and short retention. Half of every real workspace cost conversation now happens at the table-plan level, and Class Thirty-Two will lean on it. The question to ask of every table is the same one Class Twelve asked of every storage tier: who needs this data, how old, and how fast? Price the answer, and write it down.

§5

The posture interviews probe

"Tell me about your monitoring" calls for evidence, not a product list. Describe three checks. Coverage: identify the resources and failure modes that matter, configure supported collection, and verify fresh data arrives. Access: keep related records queryable by the people who investigate while honoring region and security requirements. Response: test alerts and triage, then check for gaps before incidents find them. Policy helps with repeatable settings, but it does not prove telemetry or notification by itself.

Notice what the posture is for. Monitoring does not prevent failures — the job will still break some night, disks will still fill, certificates will still expire. What it changes is the shape of the morning after: the difference between "the job failed at 02:04, at the connection to store 23, here is the record" and "the job apparently hasn't run for nine days and nobody can say why." Class One's outage took nine hours partly because diagnosis started from nothing. The posture you build this class is what makes sure no Campux investigation ever starts from nothing again — and being able to say that sentence, with the policy assignment to back it, is worth more in an interview than naming every product in the Azure Monitor family.

Case File · Campux Retail

Eyes on the estate, by law

campux-logs is a shared destination; coverage is checked resource by resource

Campux puts related workload logs in campux-logs so investigators can compare them. The team selects policy definitions for supported storage, key vault, and Container Apps resource types, grants deployment permission, and remediates existing resources. Then it checks diagnostic settings and new records for each service. A policy assignment reduces missed setup; it does not guarantee every resource type, category, or application event is covered.

The first payoff arrives inside a month, and it is not an outage — it is the absence of a mystery. The POS job fails one night (store 23's upload was corrupt), and the on-call engineer reads the execution's failure record over coffee: what failed, when, which store, first try. Total investigation time: four minutes. The Class One version of Campux spent nine hours on its outage largely because minute one started with nothing. This Campux starts every investigation with the tape already rolling — and next class teaches the language that interrogates it.

Fig. 2 · One reservoir, opened by policy
estate resources storage accounts key vaults container apps env supported types in scope Azure Policy DeployIfNotExists creates diagnostic settings automatically campux-logs one workspace central reservoir routes to by policy, not by hand one workspace — incidents cross teams
A shared workspace can simplify cross-service investigation. A scoped DeployIfNotExists assignment can create diagnostic settings for supported resource types after evaluation and remediation. Confirm the policy identity has permission and query each expected table; a diagram is not proof of coverage.
Watch · Microsoft Learn

The official pages, and a CAMPUX overview

The overview for the map; the workspace page before Lab 1
Lab 1 · The empty-workspace surprise

Build the plumbing, and feel the closed valve first

~20 minutes plus ingestion delay · Cloud Shell · Azure charges may apply

Create a fresh workspace and storage account, make a blob request before routing resource logs, then route StorageWrite and make another request. Compare log timestamps. This tests resource logs; the subscription Activity Log is separate. The sign-in based blob commands need a data-plane role, so the role-assignment step requires Owner or User Access Administrator permission at the storage-account scope. Check storage and log charges and use a dedicated lab group.

  1. Create a workspace and a storage account to spy on:

    az group create --name rg-monitor-lab --location eastus
    az monitor log-analytics workspace create \
      --resource-group rg-monitor-lab --workspace-name campux-logs-lab
    STORAGE="campuxmon$(date +%s)$((RANDOM % 10000))"
    az storage account create --name "$STORAGE" \
      --resource-group rg-monitor-lab --sku Standard_LRS
    STG_ID=$(az storage account show -g rg-monitor-lab -n "$STORAGE" --query id -o tsv)
    # Blob data operations with --auth-mode login need a data-plane role.
    ME=$(az ad signed-in-user show --query id -o tsv)
    az role assignment create --assignee-object-id "$ME" \
      --assignee-principal-type User --role "Storage Blob Data Contributor" \
      --scope "$STG_ID"
    What to notice: the workspace is a plain resource — no agents, no magic. It is an empty reservoir until something is routed into it.
  2. Wait for the new role to propagate, then generate a blob request before any diagnostic setting exists. If a 403 appears, wait and retry; an Azure subscription Contributor cannot assign this role without additional RBAC permission.

    az storage container create --name receipts \
      --account-name "$STORAGE" --auth-mode login
    echo "store-23 upload" > r1.txt
    az storage blob upload --container-name receipts --file r1.txt \
      --name r1.txt --account-name "$STORAGE" --auth-mode login
    What to notice: real operations are happening — reads, writes, authentications. The resource is narrating all of it. Nothing is listening.
  3. Now route the blob service's StorageWrite resource-log category to the workspace:

    WS_ID=$(az monitor log-analytics workspace show -g rg-monitor-lab \
      -n campux-logs-lab --query id -o tsv)
    az monitor diagnostic-settings create --name lab-valve \
      --resource "$STG_ID/blobServices/default" --workspace "$WS_ID" \
      --logs '[{"category":"StorageWrite","enabled":true}]'
    sleep 300   # a new setting takes a few minutes to start collecting
    az storage blob upload --container-name receipts --file r1.txt \
      --name r2.txt --account-name "$STORAGE" --auth-mode login
    What to notice: the setting is attached to the blob service and selects its write category. The second upload waits five minutes after routing is enabled. When we ran this in October 2026, an upload made seconds after creating the setting was never logged; one made later was. If Azure rejects a category, list the categories supported by that resource.
  4. Wait ten to fifteen minutes (ingestion is not instant; our run took about fifteen), then query the workspace — first for the pre-valve era, then at all. While you wait, open the in-page query box in Class 29, which holds sample StorageBlobLogs rows, and run StorageBlobLogs | take 5 to see the shape of what you are waiting for:

    az monitor log-analytics query --workspace $(az monitor \
      log-analytics workspace show -g rg-monitor-lab \
      -n campux-logs-lab --query customerId -o tsv) \
      --analytics-query "StorageBlobLogs | project TimeGenerated, OperationName, StatusText | order by TimeGenerated desc" \
      -o table
    The lesson: when the second upload's resource log arrives, compare its timestamp with the first. The later setting does not backfill the earlier blob-service log. Other audit trails, including the subscription Activity Log, are separate. Keep the workspace for Lab 2.
Note · first-time ingestion can take up to 15 minutes — if the query returns nothing, upload another blob, wait, and rerun. The table name for storage is StorageBlobLogs; every resource type writes to its own tables, which is next class's subject.
Lab 2 · Retention & the invoice

Turn the two cost dials with your own hands

~10 minutes · continues Lab 1 · same workspace

Read the workspace's retention, change it, and find where ingestion volume — the thing you will actually be billed for — is measured.

  1. Read the current retention setting:

    az monitor log-analytics workspace show \
      --resource-group rg-monitor-lab \
      --workspace-name campux-logs-lab \
      --query retentionInDays
    What to notice: the default is measured in weeks, not years. If this workspace held sign-in logs and a breach investigation needed day 60, the default would already have deleted it.
  2. Turn the dial — extend interactive retention:

    az monitor log-analytics workspace update \
      --resource-group rg-monitor-lab \
      --workspace-name campux-logs-lab \
      --retention-time 90
    What to notice: this command sets the workspace default retention. Tables can have their own settings, and a change may affect charges depending on plan and included retention. Check current pricing before applying it outside this lab.
  3. Find the meter — query the workspace's own record of what it ingested:

    az monitor log-analytics query --workspace $(az monitor \
      log-analytics workspace show -g rg-monitor-lab \
      -n campux-logs-lab --query customerId -o tsv) \
      --analytics-query "Usage | summarize TotalGB=sum(Quantity)/1000 by DataType | order by TotalGB desc" \
      -o table
    On screen: the Usage table can show reported ingestion by data type after usage records arrive. It is not a complete invoice; use Azure Cost Management and the workspace usage views to confirm billable amounts.
  4. Clean up the lab:

    az group delete --name rg-monitor-lab --yes --no-wait
    The lesson: two dials — what flows in (the §2 valves, measured by the Usage table) and how long it stays (retention, per table in production). Every monitoring bill that ever shocked a manager was one of these two dials left on default. You now know where both of them are.
Think in systems

Zoom out: you can only join what you pooled, and only pool what you plumbed

You can point diagnostics at a workspace. Now reason about what observability does as a system, because a resource born un-watched is invisible exactly until it fails.

Feedback loops

Nothing flows until diagnostics are switched on, so a resource born un-watched stays invisible until it fails silently — and minute one of the incident starts with nothing. What plumbs telemetry automatically, by rule?

Dependencies & coupling

An incident crosses resources, so the investigation depends on their logs living somewhere joinable; one-workspace-per-team fractures exactly what you need together.

The constraint

The value is joinability; the constraint is coverage — a gap in what emits is a blind spot no query can fill later.

Second-order effects

Verbose collection is cheap to enable and expensive to keep; ingestion nobody queries becomes a quiet line item on the Class 32 bill.

What breaks when it scales

Hand-clicking diagnostic settings is fine at ten resources and impossible at a thousand; policy has to open the valves before the estate outgrows the hand.

The engineer who ships is asked "is it logging?" The engineer who gets promoted is asked "and can you join it to everything else during an incident?" — and pooled it into one reservoir.

On the job

Asking the data, not the room

You · Cloud Engineer · "is it just me or is it slow?"

Users report a vague slowness and nobody can point at anything. You are not blind — telemetry is already flowing to Log Analytics because you turned on diagnostics up front. You open the workspace and start asking the data questions instead of asking the room. The difference between a guess and an answer is whether the logs were being collected before you needed them.

Class Twenty-Eight

Examination

Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored; this is between you and the page.

Drill 01Recall · metrics vs logs
"Why did the POS job fail for store 23 last Tuesday at 02:04?" — which telemetry answers this, and why?
Marked

B. The question names a specific event with a cause — that is a sentence, and sentences live in logs. A metric could tell you that something was wrong at 02:04 (executions dropped, errors ticked up), but no number contains "store 23" or "connection timed out"; charts detect, records explain. C is the tempting flattener and it is false — metrics are pre-aggregated numbers and logs are structured events, collected and billed differently, which is why §2's valve applies to one and not the other. D describes the world before centralised logging: attaching a debugger to a 02:00 batch job is what teams without a workspace end up attempting. Detection is metrics; diagnosis is logs. Keep the verbs straight and every tooling choice follows.

Drill 02Recall · the valve
A key vault was created six months ago. No diagnostic setting was ever configured. Today you create one, routing audit logs to the workspace. What can you now query?
Marked

C — and this answer decides audits and breach investigations. Resource logs are a stream, not an archive: with no diagnostic setting, the vault's narration was discarded as it happened, and there is no six-month buffer waiting to be released (A) nor a 30-day one (B). The setting is a valve on a live pipe — it routes what flows from the moment it opens, minutes later at most (D overstates even that delay). The professional consequences are direct: if that vault held production secrets, you now cannot answer "who read them in May?", and no amount of money fixes it retroactively. This is why the case file opens valves by policy at resource creation — the only diagnostic setting that protects you is the one that existed before the question did.

Drill 03Select three
Which three are true of Azure Monitor telemetry collection?
Marked

Options 1, 2 and 4. Supported platform metrics are automatic in the metric store. Resource logs need an explicit route to the workspace, and a scoped policy can help deploy settings for supported types. Merely creating a workspace does not connect every resource. Pricing is not simply "free ingestion, paid queries": it varies by data source, table plan, retention, alert rule, and query type. Check Cost Management for charges, not just the Usage table.

Drill 04Spot the error
This incident postmortem is about to be filed. One line contains the failure that will repeat. Which?
# postmortem: storefront checkout errors, 03:10–04:40
1.  Detection: metric alert on 5xx rate fired at 03:12 —
    on-call was paged within two minutes.
2.  Diagnosis: the app's error logs could not be found in the
    workspace; the app was deployed last quarter and no
    diagnostic setting was ever created for it.
3.  Recovery: rolled back to the previous revision at 04:35;
    errors stopped.
4.  Follow-up: on-call will remember to enable diagnostics
    on new resources going forward.
Marked

Line four. Line two describes the wound accurately — a closed valve discovered mid-incident, ninety diagnostic-free minutes of guessing — but line four is where the postmortem fails, because it prescribes memory as the cure for forgetting. "On-call will remember" is the same mechanism that already failed last quarter, restated as an intention. The Class Twenty-Five instinct applies: a fix that depends on a human remembering is not a fix; it is a scheduled recurrence.

The repair is to assign appropriate policies for the supported resource types, give the policy identity permission, remediate existing resources, and test for fresh records. Add a coverage check so a failed policy deployment is noticed. Metric alerts can detect a 5xx spike when that metric exists, and rolling back an application may be a valid recovery, but neither action replaces collection verification.

Situation 01Write before you reveal
After a painful storefront outage, a director asks, genuinely frustrated: "We pay for Azure Monitor. Why didn't we see this coming?" Answer honestly — including the part where the tool was never the problem.
Separate what monitoring can promise (seeing) from what was actually configured (looking).
Reasoning

Concede the fair core before defending anything. The director's premise — we pay, therefore we should see — is reasonable, and the honest opening is that paying for Azure Monitor buys the capability to see, not the seeing. Then give the diagnosis in the class's terms, without jargon: telemetry in Azure flows only where valves were opened, and the audit that follows this outage will almost certainly find some closed — resources without diagnostic settings, logs kept shorter than the incident's fuse was long, or signals that landed in a workspace no alert and no human ever read. "We had monitoring" usually decomposes into "we had charts nobody was assigned to."

Answer the actual question — "why didn't we see it coming" — with the two honest halves. Half one: some failures genuinely do not announce themselves in advance, and monitoring's real promise for those is a fast, evidence-rich after — minutes to diagnosis instead of the ninety guessing minutes just endured. Half two: many failures do announce themselves — climbing error rates, filling disks, expiring certificates — but only if something is watching the right signal with a threshold and a pager attached, which is alerting, which is configuration, which is work that either was or was not done. The tool detected nothing because nothing was aimed. That sentence stings, and it is the true one.

Convert frustration into the posture, with a bill attached. Offer the §5 three legs as the concrete follow-up: coverage by policy (valves opened automatically, so no resource is ever dark again), one workspace (so the next investigation is one query), and a habit (named owners for alerts and a monthly gaps review — Class Thirty makes this real). Price it honestly: some ingestion cost, a few days of engineering, and a standing hour a week. The director is not really asking about last night; they are asking whether next time will be different. The answer is yes — not because the product changed, but because the posture will exist. Bring the policy assignment to the follow-up meeting, not a slide.

Situation 02Write before you reveal
Three team leads each request their own Log Analytics workspace — "our logs, our budget, our access." The platform lead asks you to settle it. What do you recommend, and what do you concede?
Their three reasons are real needs. Check which ones actually require separate storage.
Reasoning

Take the three reasons seriously — then test each against what workspaces actually separate. "Our access": solved inside one workspace by resource-context RBAC — a team with rights to its resources queries those resources' logs and nobody else's; Class Eight's tooling, not architecture, is the answer to a permissions question. "Our budget": ingestion is attributable — the Usage table and cost views can split the bill by resource and table, so chargeback does not require separate plumbing. "Our logs": this is the one that is usually a feeling rather than a requirement — and the feeling is legitimate, but it prices out badly, as the next paragraph should say out loud.

Name the cost of splitting, with the 3am scene. Incidents do not respect team boundaries: the storefront error that traces to a storage throttle that traces to a batch job touches three teams' resources in one causal chain. In one workspace that is a single query; across three it is an engineer at 3am running the same KQL three times and lining up timestamps by hand. Fragmentation taxes exactly the moments the logs exist for. The §3 rule gives the decision structure: split for data residency, regulatory boundaries, or genuinely separate security domains — and concede honestly that if one of the three teams does have such a requirement (a compliance scope, a sovereignty constraint), that team gets its workspace, because the rule is "named reason", not "never".

Close with the offer that dissolves the politics. Most workspace-sprawl requests are really requests for autonomy dressed as infrastructure — so give the autonomy without the fragmentation: each team gets scoped query access, its own dashboards and alerts over its own resources, and a monthly cost line it can see and control. One reservoir, three sovereign views. If a lead still objects, the closing question usually settles it: "in the next cross-team incident, which of your engineers volunteers to be the one correlating three workspaces by hand?" Silence, in this meeting, is agreement.

Examination record · first attempt
0/4
Class Twenty-Eight · Complete
Retain this much

Five things worth carrying out of this class

  1. Supported platform metrics are collected automatically in the metric store. Resource logs are detailed events that need a configured route to reach Log Analytics. Either can support investigation or alerts.
  2. A new diagnostic setting routes future resource logs; it does not backfill earlier events. The subscription Activity Log is collected separately without this setting.
  3. A scoped DeployIfNotExists policy can help create settings for supported types. Check identity permissions, compliance, remediation, and actual log arrival.
  4. One workspace is the default; split only for a named reason (residency, regulation, security boundary). Scope access with RBAC, attribute cost with the Usage table — correlation is the point of centralising.
  5. The bill has two dials: what flows in (per-GB ingestion, set by the valves) and how long it stays (interactive vs long-term retention, per table). Both left on default is how monitoring becomes a surprise line item — or a missing witness.
Notes
  1. Azure Monitor's family is larger than this class's two shapes — traces, changes, Prometheus metrics in their own workspace type, and more. The metrics/logs split remains the honest backbone: nearly every tool in the family is a producer, store, or consumer of one of the two, and knowing which one you are looking at tells you how it is collected, billed, and queried. Meet the cousins as you need them. ↩
  2. The Activity Log — the subscription's own record of who created, changed, or deleted what — is the one log you get without a valve: ninety days of it, automatically. It answers "who touched this resource?" but nothing about what happens inside resources, which is exactly the gap diagnostic settings exist to fill. Route it to the workspace too; ninety days is shorter than most audits. ↩
  3. Retention specifics move: defaults differ by table type, interactive retention extends to two years for a fee, long-term retention reaches twelve, and the table-level knobs have grown finer-grained over time. Treat the architecture as settled — a hot, queryable window backed by a cheap, deep archive — and check current limits and prices when you set real dials. The direction that never changes: decide per table, on purpose, before the auditor asks. ↩