Skip to content
CAMPUX Cloud Bootcamp Phase Four · Class Twenty-Seven
Phase Four — Operate, Secure & AI
4 drills · 2 situations · Part F of VII
AKS: Zero to Production
Class Twenty-Seven · Part F

Keeping it current, watched, and affordable

A cluster needs a repeatable operating routine: upgrade its Kubernetes version and node image, check that workloads stay ready, and inspect the full Azure bill before cutting capacity.

Phase Four · Operate, Secure, AI · Class 27 · Part F · 16 min · Assumes Class 27 Part B

After this you can set limits and autoscaling, read the cluster's metrics, and preflight an upgrade so you know what it will disrupt before you run it.

New to this? Read Class 27 Part B first. Already done it, or know the material? Carry on.

§1

Two things you upgrade, and why it never ends

Every cluster has two surfaces that age, and confusing them is a common early mistake. The first is the Kubernetes version — the control plane and the kubelet on each node, moving together from 1.29 to 1.30 to 1.31. The second is the node image — the operating system on the node VMs, which needs security patches for the same reason any Linux box does, whether or not Kubernetes itself moves. You can patch the node image on Tuesday without touching the Kubernetes version, and you often should. They are two different clocks, and a mature team watches both.

The reason this is ongoing rather than a one-off is the release train. Kubernetes ships roughly three minor versions a year, and each version is supported for only a window before it falls off the back — AKS will eventually stop giving you security fixes for a version that has aged out, and can force an upgrade to keep the cluster in a supported state.1 So "we upgraded last year" is not a finished task; it is a lap. Plan for upgrading as a standing rhythm, not an event, and the day a version goes end-of-support arrives as a calendar entry you already knew about rather than a surprise.

The two upgrade surfaces
Kubernetes version — the control plane and kubelet, on the ~3-releases-a-year train that ages old versions out of support. Node image — the OS on the node VMs, patched for security independently of the Kubernetes version. Watch both clocks.
§2

Automating the rhythm — channels and maintenance windows

Doing every upgrade by hand does not scale past one cluster, so AKS gives you auto-upgrade channels — one per surface, because the two clocks tick at different rates. The cluster autoupgrade channel governs the Kubernetes version: patch takes the newest patch release inside your current minor version, and stable and rapid carry you across minor versions on different appetites for newness. A separate node OS autoupgrade channel governs the node image: NodeImage takes a freshly patched, AKS-tested VHD — security fixes and bug fixes together — on a weekly cadence. Subscribe both channels, each at the cadence it can tolerate, and the cluster stops being a thing you must remember to feed.

Pair an auto-upgrade channel with a planned maintenance window, such as Sunday 02:00–06:00, so routine disruptive upgrades are scheduled for a known low-traffic period. AKS recommends at least four hours for a Standard-cluster upgrade window. A window is not an absolute promise: urgent or reactive platform maintenance can occur outside it. Schedule a period when engineers can respond, not simply when everyone is asleep. The channel selects eligible updates; the window constrains when planned changes begin. You still need a tested workload and an alert if it stops serving.

The channel picks what. The window picks when.

§3

Surge upgrades — create room before draining a node

An upgrade replaces nodes: each old node is cordoned, its pods are evicted, and a node on the new version takes its place. If the pool is nearly full, removing a node before replacement capacity is ready may leave pods Pending. With max surge, AKS provisions temporary extra nodes before draining old ones. That gives the scheduler somewhere to place evicted pods and reduces the risk of lost capacity. It does not guarantee availability: the surge node must actually provision, the subnet and quota must have room, and applications need enough replicas, readiness checks, and workable PodDisruptionBudgets.

max surge limits the extra nodes, as a count or percentage of the pool. More surge can shorten an upgrade but requires additional VM quota and IP capacity, briefly increases compute cost, and can increase simultaneous disruption. Before raising it, check the node pool's quota and subnet, set an appropriate PodDisruptionBudget, and test the upgrade in non-production. Watch node and pod readiness through the window. A configured surge is a capacity plan, not proof that each app remains available.

Fig. 1 · Surge upgrade — a temporary node joins on the new version before the old one drains
A surge node provides replacement room before an old node is drained, subject to quota, scheduling, and workload readiness. node · A v1.30 · draining node · B v1.30 · serving node · C (surge) v1.31 · new, temporary pods move onto the new node step 1 · add C first room to move before draining
In this simplified example, node C becomes ready before node A drains, giving its pod a place to move. Verify that the extra VM can be created and the pod becomes Ready after rescheduling. A node count alone does not show whether customers can still use the application.
Preflight you can run · read-only

See what an upgrade would touch before you start one

az aks get-upgrades -g "$RG" -n "$CLUSTER" -o table     # versions you may move to
az aks nodepool show -g "$RG" --cluster-name "$CLUSTER" -n nodepool1 \
  --query "{version:orchestratorVersion, surge:upgradeSettings.maxSurge, count:count}" -o table
kubectl get pdb -A                                          # budgets that may block a drain
kubectl get deploy -A -o custom-columns=NAME:.metadata.name,READY:.status.readyReplicas,WANT:.spec.replicas

None of these change anything. Together they answer the preflight questions: which version is next, how much spare room the surge creates, which workloads have a disruption budget, and which have only one replica (those will blink during a drain). Workloads without readiness probes, from Part B, are the ones an upgrade can take down quietly. This part teaches the plan and the preflight; a live upgrade is a separate rehearsal on a cluster you can afford to disturb.

Play

Play it through

Three minutes. Watch a two-node example lose room when one node drains, then add a surge node and maintenance window. The simulation assumes the extra node provisions and the pod becomes ready; a real upgrade needs those checks.

This class has no paid upgrade lab. A meaningful availability test needs enough nodes and traffic to observe pod movement and readiness, which is expensive to create just for a beginner exercise. Use the walkthrough to learn the sequence, then inspect a real staging upgrade with its quota, disruption budgets, events, and readiness measurements when you have a suitable cluster.

§4

Watching it without building the watchtower

You need to see logs, metrics, and dashboards, and the old reflex is to self-host that stack — run your own Prometheus, nurse your own Grafana, keep the log pipeline alive. On AKS you do not have to, and mostly should not. Azure gives you a managed trio that plugs into the cluster and is operated for you, so the observability layer is not one more thing on your pager. You wire it on at cluster create or after, and the pods, nodes, and control plane start reporting into tools you did not have to keep alive.

The division of labour is worth learning. Container Insights can collect container logs and inventory into Azure Monitor and Log Analytics. Azure Monitor managed service for Prometheus collects Prometheus-format metrics when configured. Azure Managed Grafana visualizes connected data sources; it does not replace the metric store. Each service needs setup, access control, retention choices, and a cost check. Start with the signals needed for one useful alert rather than turning on everything without a question to answer.

Table 1 — the managed observability trio: what each one is for
ServiceCollectsYou reach for it when
Azure Monitor Container InsightsLogs and coarse metrics, into Log AnalyticsYou need a pod's logs, or to query across the cluster in KQL
Azure Monitor managed service for PrometheusPrometheus-format time-series metricsYou need workload or cluster metrics and alerts at the configured scrape interval
Azure Managed GrafanaQueries connected data sources; does not store their metricsYou need managed dashboards for the metrics you already collect
§5

Read the whole bill before shrinking a pool

On the AKS Free tier, there is no paid control-plane tier charge, but the cluster still uses billable Azure resources. Node VMs are often a major line; managed disks, load balancers, public IPs, log ingestion, metrics, and network traffic can matter too. Open Cost Management or AKS cost analysis and identify the actual resource groups, services, and trend before changing capacity. Cutting a node does little if the unexpected line is telemetry or storage.

If compute is the problem, compare pod requests with measured use and right-size the pool. The cluster autoscaler adds nodes for unschedulable pods and can remove unneeded nodes within its configured bounds; it does not reduce Deployment replicas just because customer traffic stops. Spot pools may fit jobs that tolerate eviction. A user pool can scale to zero when it has no required pods, but the system pool remains and non-compute costs continue.2 To be ready at 06:00, arrange a tested trigger or schedule that creates the pods and nodes early enough to warm up. A request arriving at 06:00 is not a reliable wake-up plan for a synchronous partner API.

Case File · Campux Retail

The partner cluster learns to patch itself and sleep

a NodeImage channel, an attended weekend window, an alert, and a measured off-hours plan

The partner-integration cluster needs an upgrade routine. Configure the NodeImage channel and an attended weekend maintenance window, then test the gateway in non-production and watch readiness after the change. The partner's weekday traffic makes Sunday a candidate, but confirm the contract, batch jobs, and support coverage first. A Prometheus readiness alert pages if the gateway has too few ready replicas; an urgent Azure maintenance event may still fall outside the planned window.

Cost analysis shows an idle user pool overnight. Before setting its minimum to zero, verify there are no scheduled or always-on pods that must stay there. Agree on the partner's first-call time, then test an off-hours plan that scales the workload down and restores replicas early enough for the autoscaler to provision nodes, pull images, and pass readiness checks. Keep the system pool running, and verify which disk, IP, monitoring, and network charges remain after user nodes go away. If the morning warm-up cannot reliably meet the contract, keep a small ready footprint instead.

On the job

The version end-of-support email arrives

You · Cloud Engineer · the cluster is two minor versions behind

Azure emails that your Kubernetes version goes end-of-support in six weeks. You do not upgrade in a panic on the live cluster. You read the release notes for the versions between here and current, upgrade a non-production cluster first and watch the workloads, then raise max surge on the production pools so the rollout fits the maintenance window, and let the change land inside that window rather than at noon. The upgrade is uneventful precisely because you treated a known deadline as scheduled work, not an emergency — which is the whole difference between operating a cluster and being operated by one.

Guided practice · Supplied training example

Prepare an upgrade window

Use the fictional evidence below. This is a desk exercise; it does not provision Azure resources or measure a live service.

Training cluster: three nodes
Workload: two replicas, readiness check, one allowed disruption
Unknown: surge quota, subnet capacity, deprecated API usage
  1. List the preflight evidence needed for all three unknowns.

  2. Write how you will watch workload readiness during a rehearsal.

  3. Record an abort/escalation condition; do not promise an unsupported control-plane downgrade.

Check your reasoning after you finish

Check the actual upgrade path, quota, network capacity and API compatibility. Rehearse with the workload’s disruption budget and monitor healthy replicas. Recovery needs a supported plan, not an assumed downgrade button.

Class Twenty-Seven · Part F

Examination

Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.

Drill 01Recall
Your node pool is near capacity and an upgrade is about to replace nodes. Which setting asks AKS to create temporary replacement room before draining old nodes?
Marked

B. Max surge asks AKS to create extra nodes during an upgrade before draining old ones. That is the right capacity mechanism, but you still need VM quota, subnet room, healthy replicas, readiness checks, and PodDisruptionBudgets. The autoscaler responds to Pending pods, a maintenance window schedules routine changes, and the NodeImage channel selects OS-image updates. None alone guarantees a service stays available.

Drill 02Recall
A user node pool has no required pods overnight, and the partner API must be ready at 6am. Which plan can reduce overnight node compute cost without assuming the first request wakes the service instantly?
Marked

B. A user pool may be scaled to zero when it has no required pods, reducing its VM compute charge. The system pool and other resources still cost money. Restoring the service needs a trigger or schedule that brings back the workload and nodes early enough for image pulls and readiness checks. A deletes the cluster unnecessarily; C leaves a VM running and may lack enough morning capacity; D confuses eviction-tolerant Spot capacity with a scheduled idle period. If the warm-up misses the partner's target, retain ready capacity instead.

Drill 03Select three
Which three statements about operating an AKS cluster are true?
Marked

Two upgrade surfaces, channel plus window, and Grafana as a visualization layer. Kubernetes versions eventually age out of support, so one upgrade is not a permanent fix. Scaling a user pool to zero reduces that pool's VM charges but does not erase system-node, disk, network, or telemetry charges. Planned maintenance schedules routine updates; urgent platform work can be an exception.

Drill 04Spot the error
A junior drafts this operations plan for the cluster. One line rests on a false assumption that will cost money or an outage. Which?
# ops plan: partner-gw cluster
1.  Subscribe the cluster's node OS to the NodeImage channel
    so it stays patched automatically.
2.  Pin upgrades to a Sunday 02:00-06:00 maintenance window.
3.  Put the always-on gateway pods on a Spot node pool to
    save money, since Spot is just cheaper VMs.
4.  Scale the user pool to zero on the nights it is idle.
Marked

Line three. Spot is not merely "cheaper VMs" — it is Azure's spare capacity, offered at a discount precisely because Azure can reclaim it with little notice when it needs the hardware back. That trade is right for interruptible, restartable work: batch jobs, CI, the pos-batch style of load. It is exactly wrong for an always-on gateway that the partner contract depends on, because the day Azure evicts the node your integration goes down for reasons that look like nothing you did. The discount is real; so is the eviction, and putting availability-critical pods on Spot swaps a predictable bill for an unpredictable outage.

The other lines describe valid tools. NodeImage can deliver patched OS images. Weekly maintenance windows can schedule routine upgrades, subject to urgent exceptions. A user pool can scale to zero when no required pods remain, though restoring it before the next partner call needs a tested plan and other Azure charges remain. Spot's eviction risk is the specific error in line three.

Situation 01Write before you reveal
A colleague says: "Let's just turn on the fastest auto-upgrade channel and forget about upgrades forever — one less thing to manage." What do you tell them, and what would you actually set up instead?
Automating a thing is not the same as being unable to see it. Ask what happens the night an upgrade breaks a workload.
Reasoning

The instinct is right; the "forget forever" is the trap. Automating upgrades is correct — doing them by hand does not scale, and a channel keeps the cluster from drifting into an unsupported state. So concede that first. The error is treating "automated" as "no longer my concern." An upgrade is still a change that lands on a running workload, and the fastest channel maximises how often and how disruptively change arrives, which for a workload someone depends on is the opposite of what you want.

Choose an upgrade path and plan its timing. A patch channel stays within the current Kubernetes minor version, so it still needs a plan for the next minor before support ends. NodeImage addresses OS-image updates separately. Use an appropriate planned maintenance window, but keep support available and allow for urgent platform exceptions. Test new versions in non-production, confirm quota and disruption budgets, and verify application readiness after every rollout.

Then keep a way to notice it went wrong. Wire an alert — managed Prometheus is right there — that pages if the workload's pods stop being ready. A silent auto-upgrade that quietly breaks the app is worse than a manual one you watched, because nobody is looking. "Automate the toil, keep the visibility" is the whole answer: the channel removes the remembering, the window removes the surprise, the alert removes the silence.

Situation 02Write before you reveal
Finance flags that the cluster costs more than expected and asks you to "make Kubernetes cheaper." Walk through how you reason about the bill and what you would change, out loud.
You are not paying for Kubernetes. Find the thing the invoice is actually for before you touch anything.
Reasoning

Start with the invoice. The Free-tier control plane has no paid tier charge, but AKS uses VMs, disks, load balancers, public IPs, monitoring, and network services. Break down the cost by resource and trend before touching a pool. If nodes dominate, reduce wasted VM hours; if ingestion or storage dominates, a node edit will not solve the problem.

Change the measured driver. For idle compute, test whether an off-hours user pool can scale to zero and recover before traffic resumes. For oversized nodes, compare pod requests and actual use before right-sizing or adjusting autoscaler bounds. Put eviction-tolerant batch work on Spot only if retries and deadlines allow it. For logs and metrics, review retention and ingestion settings. Keep a before-and-after bill and a readiness check for every saving.

Close by making it visible, not a one-off. A cluster gets expensive again the moment someone leaves a pool large and nobody notices, so the durable fix is Cost Management watching the spend, where an oversized pool shows up as a line on a report rather than a shock on the invoice. Cheaper is not a single edit; it is right-sizing plus autoscaling plus scale-to-zero, kept honest by a bill someone reads. That is what you tell finance you are setting up, not just what you cut today.

Examination record · first attempt
0/4
Class 27f · Complete
Retain this much

Five things worth carrying out of Part F

  1. Two upgrade surfaces age independently: the Kubernetes version (control plane and kubelet, ~3 releases a year, old ones lose support) and the node image (OS security patches). Upgrading is an ongoing lap, not a one-off.
  2. Cluster and node OS auto-upgrade channels select updates. Planned maintenance windows schedule routine work, but urgent platform maintenance may be an exception; keep response coverage.
  3. Max surge requests temporary nodes before old ones drain. Check quota, subnet capacity, healthy replicas, readiness, and PodDisruptionBudgets; surge alone does not guarantee availability.
  4. Use the managed trio — Container Insights for logs, managed Prometheus for metrics, Managed Grafana for dashboards — instead of self-hosting Prometheus and Grafana.
  5. Read the whole Azure bill: node VMs, disks, load balancers, IPs, telemetry, and network. Scale idle user pools only with a tested plan to restore service before demand.
Notes
  1. The exact support window for a given Kubernetes version, and how aggressively AKS will auto-upgrade a cluster that has aged out, are the kind of detail Microsoft adjusts over time — check the current AKS supported-versions page rather than trusting a number memorised here. Treat the specifics as movable; treat the direction as settled: versions age out, support ends, and a cluster you never upgrade eventually stops getting security fixes. That last part does not change. ↩
  2. A running cluster cannot scale its system node pool to zero. User pools can reach zero when workload scheduling allows it. Restoring replicas or creating Pending pods can trigger autoscaling, but node provisioning and readiness take time; a synchronous API needs a planned warm-up or retained ready capacity. Storage, IP, telemetry, and other charges may continue. AKS also offers stop/start for suitable non-production clusters, with separate operational tradeoffs. ↩