Prove you can lose a region.

When a region goes dark, two questions decide whether you keep your job: how long until we are back (RTO, recovery time objective) and how much data did we lose (RPO, recovery point objective). Everyone quotes targets; almost nobody has measured their real ones, because measuring means actually destroying something and recovering it. This lab makes you do exactly that — with a stateful database, across regions — so the numbers you put in a runbook are ones you have seen with your own eyes.

The tool is Velero: it snapshots your Kubernetes objects and the disks behind your persistent volumes, and writes both to blob storage. A cluster in East US 2 can already read a container that lives in a storage account in East US — blob storage is reachable across regions on its own, no replication required for that part. What geo-replication buys you is different: it keeps the backup store itself alive if East US, where the storage account lives, has a regional outage. For that you need RA-GRS (read-access geo-redundant), not plain GRS — GRS copies your data to the paired region but will not let anything read it until you trigger a manual account failover; RA-GRS exposes a live, read-only copy the whole time.

A backup you have never restored is a rumour, not a recovery.

Create a primary cluster in East US and a DR cluster in the paired region, East US 2. They must match — same VM size, same node count — or a restored persistent volume can land on an incompatible disk tier and fail. Save each as a named kubectl context so you can aim commands at either cluster deliberately.

export MSYS_NO_PATHCONV=1   # Windows/Git Bash: leave resource-id args alone
RG="rg-dr-lab"; PRIMARY="aks-primary"; DR="aks-dr"
az group create -n "$RG" -l eastus

for pair in "$PRIMARY:eastus" "$DR:eastus2"; do
  NAME="${pair%%:*}"; LOC="${pair##*:}"
  az aks create -g "$RG" -n "$NAME" -l "$LOC" \
    --node-count 3 --node-vm-size Standard_D4s_v3 \
    --network-plugin azure --enable-managed-identity \
    --enable-oidc-issuer --zones 1 2 3 --generate-ssh-keys
done

az aks get-credentials -g "$RG" -n "$PRIMARY" --context aks-primary
az aks get-credentials -g "$RG" -n "$DR"      --context aks-dr
kubectl get nodes --context aks-primary
kubectl get nodes --context aks-dr

Velero writes to a blob container. Make the storage account RA-GRS so the backup store survives an outage of its own region with a live, readable secondary copy — plain GRS replicates the bytes but keeps them unreadable until a manual failover. Give Velero a service principal scoped to that storage account only — Storage Blob Data Contributor, not subscription owner. Least privilege is the difference between a leaked backup key and a leaked cloud.

SA="velerodr$RANDOM"; CONTAINER="velero-backups"
SUB_ID=$(az account show --query id -o tsv)

az storage account create -n "$SA" -g "$RG" -l eastus \
  --sku Standard_RAGRS --min-tls-version TLS1_2 --allow-blob-public-access false
az storage container create -n "$CONTAINER" --account-name "$SA"

STORAGE_ID=$(az storage account show -n "$SA" -g "$RG" --query id -o tsv)
SP=$(az ad sp create-for-rbac -n velero-sp \
  --role "Storage Blob Data Contributor" --scopes "$STORAGE_ID")
CLIENT_ID=$(echo "$SP" | jq -r .appId)
CLIENT_SECRET=$(echo "$SP" | jq -r .password)
TENANT_ID=$(echo "$SP" | jq -r .tenant)
echo "storage: $SA  container: $CONTAINER"

Install Velero on the primary as the writer, and on the DR cluster pointed at the same container, then patch that cluster's copy of the location to read-only. This is the correct production shape — only one cluster ever writes backups, so they cannot fight over the store.

cat > /tmp/velero-creds.conf <<EOF
AZURE_SUBSCRIPTION_ID=$SUB_ID
AZURE_TENANT_ID=$TENANT_ID
AZURE_CLIENT_ID=$CLIENT_ID
AZURE_CLIENT_SECRET=$CLIENT_SECRET
AZURE_RESOURCE_GROUP=$RG
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

# PRIMARY — read/write backup location + volume snapshots
velero install --provider azure \
  --plugins velero/velero-plugin-for-azure:v1.10.0 \
  --bucket "$CONTAINER" \
  --secret-file /tmp/velero-creds.conf \
  --backup-location-config resourceGroup=$RG,storageAccount=$SA \
  --snapshot-location-config apiTimeout=5m \
  --kubecontext aks-primary

# DR — same bucket, same credentials, then flip its copy of the location to read-only
velero install --provider azure \
  --plugins velero/velero-plugin-for-azure:v1.10.0 \
  --bucket "$CONTAINER" \
  --secret-file /tmp/velero-creds.conf \
  --backup-location-config resourceGroup=$RG,storageAccount=$SA \
  --use-volume-snapshots=false \
  --kubecontext aks-dr

kubectl patch backupstoragelocation default -n velero --type merge \
  --patch '{"spec":{"accessMode":"ReadOnly"}}' --context aks-dr

Deploy PostgreSQL with a persistent volume and seed it with relational data — two tables joined by a foreign key — so the restore has to bring back consistent state, not just a flat file. Then compute a checksum of the data. That single hash is how you will later prove the recovery was byte-perfect rather than merely "the pod came up".

helm repo add bitnami https://charts.bitnami.com/bitnami && helm repo update
kubectl create namespace production --context aks-primary

helm install pg-prod bitnami/postgresql --kube-context aks-primary -n production \
  --set auth.postgresPassword=DrLabP@ss123 \
  --set primary.persistence.enabled=true \
  --set primary.persistence.size=10Gi \
  --set primary.persistence.storageClass=managed-csi-premium
kubectl rollout status statefulset/pg-prod-postgresql -n production --context aks-primary

kubectl exec -i pg-prod-postgresql-0 -n production --context aks-primary -- \
  psql -U postgres -d postgres <<'EOF'
CREATE TABLE customers (id SERIAL PRIMARY KEY, name TEXT, email TEXT UNIQUE, tier TEXT DEFAULT 'standard');
CREATE TABLE orders   (id SERIAL PRIMARY KEY, customer_id INT REFERENCES customers(id), amount NUMERIC(10,2), status TEXT);
INSERT INTO customers (name,email,tier) VALUES
  ('Alice Chen','alice@acme.com','enterprise'),
  ('Bob Martinez','bob@globex.com','standard'),
  ('Carol Johnson','carol@initech.com','enterprise');
INSERT INTO orders (customer_id,amount,status) VALUES
  (1,15000.00,'completed'),(1,8750.50,'processing'),(2,2340.00,'completed'),(3,99999.99,'completed');
SELECT md5(string_agg(id::text||name||email, ',' ORDER BY id)) AS checksum FROM customers;
EOF

Create an hourly schedule (your RPO target) and a daily one with 30-day retention, then take one manual backup now and inspect it. The most dangerous Velero mistake is a backup that captured your Kubernetes YAML but not the volume snapshot behind it — it restores pods with empty disks and you do not find out until the incident.

velero schedule create hourly-production --schedule="0 * * * *" \
  --include-namespaces production --ttl 48h0m0s --kubecontext aks-primary
velero schedule create daily-production --schedule="0 2 * * *" \
  --include-namespaces production --ttl 720h0m0s --kubecontext aks-primary

velero backup create manual-$(date +%H%M) \
  --include-namespaces production --wait --kubecontext aks-primary
velero backup get --kubecontext aks-primary        # PHASE=Completed, ERRORS=0
velero backup describe <backup-name> --details --kubecontext aks-primary \
  | grep -A3 "Persistent Volumes"                   # must list 1 PV

Now the drill. Simulate the disaster — delete the production namespace on the primary — then restore from the shared backup into the DR cluster, and re-run the checksum. This is the whole point: not "did a backup exist," but "can I stand the data back up somewhere else and prove it is intact."

# 1 · the disaster
kubectl delete namespace production --context aks-primary

# 2 · restore into the DR cluster from the geo-replicated backup
velero restore create dr-restore --from-backup <backup-name> \
  --kubecontext aks-dr --wait
kubectl rollout status statefulset/pg-prod-postgresql -n production --context aks-dr

# 3 · the verdict — same query, compare to the checksum you saved
kubectl exec -i pg-prod-postgresql-0 -n production --context aks-dr -- \
  psql -U postgres -d postgres -tAc \
  "SELECT md5(string_agg(id::text||name||email, ',' ORDER BY id)) FROM customers;"

A DR plan tested once is theatre. Wire the same backup → restore → checksum sequence into a GitHub Actions workflow on a weekly cron, so degradation surfaces in a green-or-red check while it is cheap to fix — not during the outage. Log in over OpenID Connect so no long-lived Azure secret sits in the repo: create a federated credential on an app registration trusting this repo (az ad app federated-credential create), store only its client, tenant, and subscription IDs as secrets, and once — after the checksum in Step 3 — set the repository variable the job checks against: gh variable set BASELINE_CHECKSUM --body "<the md5 you wrote down>". Fail the job if a later drill's checksum drifts from it.

# .github/workflows/dr-drill.yml (essentials)
name: Weekly DR Drill
on:
  schedule: [{ cron: '0 3 * * 0' }]   # Sundays 03:00 UTC
  workflow_dispatch:
permissions:
  id-token: write   # required to fetch the OIDC token, no secret needed
  contents: read
env:
  RG: rg-dr-lab
jobs:
  drill:
    runs-on: ubuntu-latest
    steps:
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Backup, restore to DR, verify checksum
        run: |
          az aks get-credentials -g "$RG" -n aks-primary --context aks-primary --overwrite-existing
          az aks get-credentials -g "$RG" -n aks-dr      --context aks-dr      --overwrite-existing
          B="dr-drill-$(date +%Y%m%d-%H%M)"
          velero backup create "$B" --include-namespaces production --wait --kubecontext aks-primary
          velero restore create --from-backup "$B" --kubecontext aks-dr --wait
          NEW=$(kubectl exec -i pg-prod-postgresql-0 -n production --context aks-dr -- \
            psql -U postgres -tAc "SELECT md5(string_agg(id::text||name||email, ',' ORDER BY id)) FROM customers;")
          [ "$NEW" = "${{ vars.BASELINE_CHECKSUM }}" ] || { echo "::error::checksum drift"; exit 1; }

Two clusters plus RA-GRS storage add up quickly. One resource-group delete removes both clusters, the storage account, and the disks — because both clusters and the storage account share rg-dr-lab, regardless of which region each one lives in. Do it the moment you are done, and clean up the service principal too.

az group delete -n rg-dr-lab --yes --no-wait
az ad sp delete --id "$CLIENT_ID"        # remove the Velero service principal
az group exists -n rg-dr-lab             # -> false once the async delete finishes

You provisioned matched AKS clusters across paired regions, backed up a stateful PostgreSQL workload with Velero to geo-redundant storage, restored it into a different region, and proved data integrity by checksum — then automated the whole drill in GitHub Actions with measured RTO and RPO. That is "designed and validated cross-region Kubernetes disaster recovery with automated drills and documented RTO/RPO" — a staff-level line on a reliability posting, done rather than described. The transferable lesson outlives Velero: recovery is a claim you verify, on a schedule, or it is not real.