Scripting for cloud engineers

A cloud engineer often needs to answer a repeatable question: "Which disks are not attached to a VM?" or "Which resource groups lack an owner tag?" The Azure portal can answer one item at a time; a short script can collect the answer across a subscription. A script can also change resources, so begin with a report that only reads. This class introduces the Azure CLI in Bash, then shows the same idea in PowerShell and Python.¹

The honest guidance: learn the az CLI cold, because it is the lingua franca of every pipeline you built this phase; be conversant in PowerShell, because half the shops that interview you run on it; and know when a task has outgrown both and deserves Python. What follows — idempotency, loud failure, querying — applies identically to all three, because the discipline is the skill. The syntax is a costume.

Bash, enough to be dangerous

You have been running Bash since Class 7. The terminal warm-up covered storing and printing a value, quiet commands, file blocks and a small function; if any of that still feels shaky, spend ten minutes there first. This class picks up where it stopped.

The az CLI lives inside a shell, and the shell is where beginners cut themselves. Four fundamentals carry most scripts. Variables take no spaces around the = — SUB=$(az account show --query id -o tsv) assigns, while SUB = ... tries to run a program named SUB — and you quote every expansion, "$SUB" and never bare $SUB, so a value carrying a space, or an empty value, cannot silently reshape the command around it. Command substitution, $(...), captures one command's output as text for another; it is the joint that turns a query into an input.

# act on every unattached disk in one resource group — one line at a time, safely
az disk list --resource-group <resource-group> --query "[?diskState=='Unattached'].id" -o tsv |
tr -d '\r' |   # on Windows the CLI ends each line with a carriage return; strip it
while IFS= read -r id; do
  echo "would remove: $id"
done

Loops walk a list, and the shape above is the sturdy one: -o tsv emits one id per line, and while IFS= read -r consumes them a line at a time — safer than for id in $(...), which splits on any whitespace and mangles a value that happens to contain some. (az disk list scopes to one resource group; sweeping a whole subscription means looping az group list first, which §5 puts to work.) Exit codes are the fourth fundamental: every command leaves one in $? — zero for success, non-zero for failure — and that number, not the text on screen, is what set -e, the next script, and every pipeline you built this phase actually read. Master those four and the rest of Bash is lookup; skip them and you will write the two-line horror story in §3 yourself.

Before automating a task, ask: what happens if it runs twice? A scheduler may retry, or someone may rerun a command after a timeout. A read-only report produces another report. A create or delete script needs checks so the second run does not create duplicates or act on an unexpected target.

Idempotency

The property that running a script once or five times produces the same end state. Achieved by describing outcomes rather than actions: "ensure the group exists", never "create the group"; "set the tag", never "append to the list".

You have met this property twice already — it is why the same Bicep file deploys safely a hundred times, and why Terraform plans converge. Scripts have no engine doing that work for them, so you supply it, two ways. First, prefer verbs that are already idempotent: az group create succeeds harmlessly if the group exists with the same settings, and az tag update --operation merge sets a tag to a value rather than piling values up. Second, when the verb is not idempotent, check before acting: query whether the thing exists, and only create it if not. The anti-patterns to hunt in review are the ones that accumulate — names built from $RANDOM or timestamps (each run makes a new resource), appends to files or lists (each run makes them longer), counters incremented (each run drifts further). Drill 04 will hand you one of these and ask you to spot it; production will do the same, less politely.

By default, Bash can continue after a failed command. Imagine a typo in az account set --subscription prod-uction: the selection fails, but a later cleanup command could run against the previously selected subscription. A script that changes resources should stop on failure and verify its subscription before acting.

Stop on unexpected errors

In Bash, set -euo pipefail catches many failed commands, unset variables, and failed pipeline stages. It has exceptions inside conditions and loops, so check expected failures explicitly. In PowerShell, $ErrorActionPreference = 'Stop' turns non-terminating errors into terminating ones; still handle expected outcomes deliberately.

Assert your context

Before anything destructive, prove you are where you think you are: query az account show and compare the subscription id against the one you intend. Wrong context is the classic scripting disaster, and it is one if statement away from impossible.

Exit with meaning

Exit code zero means success — to schedulers, pipelines, and the next script. A script that prints an error but exits zero tells every machine downstream that all is well. Fail with a non-zero code, and let Class Twenty-Two's workflows turn that into a red check somebody sees.

Notice the common thread with the whole phase: machines act on what your code signals, not what it prints. A pipeline cannot read a sad message in a log; it reads exit codes. The three habits above cost five lines total and convert the silent-continuation failure mode — the one that compounds for twenty minutes before anyone notices — into an immediate, visible, diagnosable stop.

Loud is not the same as fragile: retrying the transient

Some service failures are temporary. HTTP 429 means a request was rate-limited; 503 can mean the service is unavailable. If an API identifies a retryable failure, wait as directed by its retry header when one is supplied, then try again with a bounded number of attempts. Do not retry a permission error or malformed command as if time could fix it. The small example below demonstrates bounded attempts for a read, but the Azure CLI command's exit code alone cannot identify which error occurred; production retry logic must inspect the failure and respect service guidance.

# Bash script fragment: retry a read and publish only a complete result
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
for attempt in 1 2 3 4 5; do
  if az disk list --resource-group <resource-group> -o json > "$tmp"; then
    break
  fi
  if [ "$attempt" -eq 5 ]; then
    echo "disk list failed after 5 attempts" >&2
    exit 1
  fi
  echo "attempt $attempt failed; waiting $((attempt * 2))s" >&2
  sleep $((attempt * 2))
done
mv "$tmp" disks.json

The read is safe to repeat, and the last failure exits non-zero instead of masquerading as success. The temporary file prevents a failed attempt from replacing a previously complete disks.json. The example does not distinguish throttling from a bad resource-group name, so use it to learn the loop, not as a production retry policy.

Every az command returns JSON, and buried in that JSON is the one fact you wanted. The --query parameter — JMESPath, a query language for JSON — extracts it without a single line of parsing code. Four shapes cover nearly everything you will do:

# 1 · reach into one object
az account show --query id -o tsv

# 2 · project one field from a list
az disk list --resource-group <resource-group> --query "[].name" -o tsv

# 3 · filter, then project
az disk list --resource-group <resource-group> --query "[?diskState=='Unattached'].name" -o tsv

# 4 · filter, then shape a report
az disk list --resource-group <resource-group> \
  --query "[?diskState=='Unattached'].{disk:name, gb:diskSizeGB, rg:resourceGroup}" \
  -o table

Read them as a grammar: [] walks a list, [?…] filters it with a condition, .field reaches in, and .{new:old} reshapes the survivors into exactly the columns you want. Pair the query with the right output: -o tsv for values a script consumes, -o table for reports a human reads, and JSON — the default — when the next tool wants structure. The fourth example already is half of this class's deliverable: unattached disks, sized and located, in one command. What separates a one-liner from an audit script is only the discipline of §2 and §3 wrapped around it.²

jq, for the JSON that is not Azure's

--query is a passenger on the az command; the moment the JSON comes from somewhere else — a raw ARM REST call, a webhook payload, a saved file, kubectl — it is not there to help. jq is the same idea packaged as a standalone tool: a filter language for JSON arriving on a pipe. The audit filter, written both ways:

# the same question, two tools
az disk list --resource-group <resource-group> --query "[?diskState=='Unattached'].name" -o tsv

az disk list --resource-group <resource-group> -o json |
  jq -r '.[] | select(.diskState=="Unattached") | .name'

Read the jq: .[] streams the array, select(...) keeps the matches, .name reaches in, and -r ("raw") strips the surrounding quotes so the output feeds a script — the -o tsv of the jq world. The grammar rhymes with JMESPath because both answer the same question; the difference is reach. Prefer --query when the data already comes from az: it filters in-process, needs nothing installed, and keeps one tool in the pipe. Reach for jq the moment JSON arrives from anything that is not the CLI — which, once you start calling ARM directly in a page or two, is often.

A useful first automation is an audit-and-report script: inspect resources and print a list for a human to review. This example makes read-only Azure CLI calls; it does not delete resources. Use an account with only the read permissions it needs, because the report may still reveal names and configuration details.

#!/bin/bash
# audit.sh — untagged resource groups + unattached disks
set -euo pipefail

EXPECTED_SUB="<campux-subscription-id>"
ACTUAL_SUB=$(az account show --query id -o tsv)
if [ "$ACTUAL_SUB" != "$EXPECTED_SUB" ]; then
  echo "Wrong subscription: $ACTUAL_SUB" >&2
  exit 1
fi

echo "== Resource groups with no owner tag =="
az group list \
  --query "[?tags.owner == null].name" -o tsv

echo "== Unattached disks (paid for, used by nobody) =="
az group list --query "[].name" -o tsv | tr -d '\r' |
while IFS= read -r rg; do
  az disk list --resource-group "$rg" \
    --query "[?diskState=='Unattached'].{disk:name, gb:diskSizeGB, rg:resourceGroup}" \
    -o table
done

A script is a decision that repeats.

The script checks its selected subscription before querying. The first query finds resource groups without an owner tag; the loop asks for unattached disks in each group and prints a table. Empty output means no matches for that query, not that the subscription has no resources. A disk shown as unattached is a candidate for review, not automatic deletion: backups, replicas, or a temporary detach may explain it. A human should check the disk's purpose and retention needs before any removal.

And because half the shops that will interview you run this same audit in PowerShell, here is the untagged-groups half in that costume — $ErrorActionPreference = 'Stop' standing where set -euo pipefail stood, and a pipeline of objects doing the work of the JMESPath string. The discipline is identical; only the syntax changed its clothes:

# audit.ps1 — the same guardrails, PowerShell costume
$ErrorActionPreference = 'Stop'

$expected = '<campux-subscription-id>'
$actual   = (Get-AzContext).Subscription.Id
if ($actual -ne $expected) {
  throw "Wrong subscription: $actual"
}

"== Resource groups with no owner tag =="
Get-AzResourceGroup |
  Where-Object { -not $_.Tags -or -not $_.Tags.ContainsKey('owner') } |
  ForEach-Object { $_.ResourceGroupName }

Why the PowerShell version needed no query language

Look again at what replaced the JMESPath string: Where-Object and a comparison against a real property. That is the one idea worth taking from PowerShell even if you never write it in anger — its cmdlets return objects, not text, so a resource group arrives with a .Tags hashtable and a .ResourceGroupName you address by name, never a slab of output you slice with awk and pray the columns did not shift. The pipeline is the same three verbs JMESPath gave you, spelled longer: Where-Object filters, Select-Object projects, ForEach-Object acts. When a task has real branching — this tag present, that one absent, a different action for each — objects with named fields age better than a string you re-parse on every run. That is the §1 table's "object-shaped logic" made concrete, and the reason PowerShell shops do not consider themselves second-class.

And when it becomes Python

Past a certain size the script stops being glue and becomes software — it wants functions, tests, a package other scripts import, a maintainer who is not you. That is Python's cue, and it reaches Azure two ways. The high road is the SDK: azure-identity to authenticate, and a management library such as azure-mgmt-compute to work in objects — pagination and retries handled for you.

# audit, the SDK way — the same unattached-disk question
from azure.identity import DefaultAzureCredential
from azure.mgmt.compute import ComputeManagementClient

sub = "<campux-subscription-id>"
client = ComputeManagementClient(DefaultAzureCredential(), sub)

for disk in client.disks.list():
    if disk.disk_state == "Unattached":
        print(disk.name, disk.disk_size_gb, disk.id)

The low road, when no SDK covers what you need, is to call the Azure Resource Manager REST API yourself with requests — the same endpoints Class Sixteen took apart. You still authenticate through the credential; you just carry the token by hand:

# the same call, one level down — raw ARM over HTTPS
import requests
from azure.identity import DefaultAzureCredential

token = DefaultAzureCredential().get_token("https://management.azure.com/.default")
sub = "<campux-subscription-id>"
url = (f"https://management.azure.com/subscriptions/{sub}"
       "/providers/Microsoft.Compute/disks?api-version=2023-04-02")
r = requests.Session()
while url:
    response = r.get(url, headers={"Authorization": f"Bearer {token.token}"}, timeout=30)
    response.raise_for_status()
    page = response.json()
    for d in page.get("value", []):
        if d["properties"]["diskState"] == "Unattached":
            print(d["name"])
    url = page.get("nextLink")

raise_for_status() turns an HTTP error into an exception instead of treating an error response as disk data. The loop follows each nextLink, so it does not quietly stop after the first page. The SDK's .list() handles paging for you. This REST sketch still needs token refresh and a bounded retry policy for a long-running production job; use the SDK when it covers your task.

Secrets: the line that ends up in a log

Notice what none of these scripts contains: a password, a key, a connection string. That is deliberate, and it is the §5 habit most likely to keep your name out of an incident report. DefaultAzureCredential is why — it tries a fixed chain of identity sources and uses the first that answers: environment variables, then a workload or managed identity (the Class Twenty-Three OIDC token in a pipeline), and, near the end of the chain, your az login session when you run it locally. The same line of code authenticates on your laptop and on a runner without a secret written down anywhere, which is the whole point — the environment supplies the identity, so the script never has to. When a script genuinely needs a secret of its own — a third-party API key — it reads it from Key Vault or an environment variable at runtime, never a literal in the source, because source goes to Git and Git remembers forever. That is Class Nine's lesson wearing a shell.

Two quieter traps close the topic. A secret passed as a command-line argument is visible to anyone who can list processes and lands in shell history; pass it by environment variable or on stdin instead. And a secret echoed while debugging, or returned inside an error, lands in the pipeline log where it lives as long as the run does — set sensitive commands to --output none and refuse to print the thing you were told to protect. The credential you never wrote down cannot leak; the one you printed for five minutes already has.