Skip to content
CAMPUX Cloud Bootcamp Phase Four · Class Thirty-Nine
Phase Four — Operate, Secure & AI
4 drills · 2 situations · Part VII of IX
Class Thirty-Nine · Pipelines
Class Thirty-Nine · Part VII

Ship it dark, release it slowly

Deploying code and releasing a feature are two different acts with two different risks; this part gives you the strategies that control how new code reaches users and the feature flags that let it arrive switched off — so the two can fail, and be undone, independently.

Phase Four · Operate, Secure, AI · Class 39 · Part G · 18 min · Assumes Class 39 Part F, Class 39 Part B and Class 11 Part B

After this you can choose between rolling, blue-green and canary deployments from health evidence, and explain what running one on a real platform involves.

New to this? Read Class 39 Part F, Class 39 Part B and Class 11 Part B first. Already done them, or know the material? Carry on.

§1

Two different acts, wearing one word

A deploy and a release feel like a single event — you push a button and something new is live — but they are two acts with two clocks and two blast radii, and confusing them is how a Tuesday afternoon becomes an outage. Deploying code is moving a new build onto the machines that run it. Releasing a feature is the moment a user can actually reach the new behaviour. For most of software's history these happened at the same instant, because the only way to expose new behaviour was to put the code that produced it into production — so the deploy was the release, and every deploy was therefore a bet on a large number of untested users hitting new code at once.1

This part is about pulling those two acts apart so they can fail — and be undone — independently. There are two levers. Deployment strategies control how the new code reaches the infrastructure: all at once, a slice at a time, or machine by machine, each buying you a different rollback story for a different price. Feature flags go further and separate the acts entirely: the code ships to production switched off — dark — and the feature is released later, to whom you choose, by flipping a value in a central store, with no deploy at all. Together they turn "release" from a leap you take once, holding your breath, into a dial you can turn up slowly and back down in seconds. That dial is the difference between an engineer a team lets near production and one it does not.

Shipping code is not releasing a feature.

§2

Four ways to let new code reach users

Every deployment strategy is an answer to one question: when the new version is wrong — and one day it will be — how fast, and how cheaply, can you take it back? A strategy that exposes all users at once has the simplest mechanics and the worst answer; a strategy that exposes a slice first has more moving parts and a far better one. There is no free option. You are choosing where to spend complexity so that a bad release costs minutes and a handful of users instead of an hour and all of them. The four below are the ones worth knowing by name, ranked roughly from blunt to surgical.

Table 1 — Four release strategies, and what each buys
StrategyHow it worksRollback storyCost & complexity
Blue-green Two identical production environments — one live (blue), one idle (green). Deploy the new version to green, test it in isolation, then switch all traffic to green at once. Instant and total: switch traffic back to blue, which is untouched and still running the old version. Rollback is a routing change, not a redeploy. High. You pay to run two full environments and must keep data and state compatible across the switch.
Canary Release the new version to a small slice of traffic — say 5% — while everyone else stays on the old one. Watch the signals; expand in increments if healthy, abandon if not. Good and cheap: a bad canary is pulled after harming a fraction of users, before the increment that would have exposed the rest. Medium. Needs traffic splitting and, to be worth anything, real monitoring to decide expand-or-abort.
Rolling Replace instances in batches — a few machines at a time — until the whole fleet runs the new version, keeping enough of the old serving traffic to stay up throughout. Partial and slower: halt the roll on a bad batch, but instances already updated must be rolled back one batch at a time. Low. No second environment; it is the default way most orchestrators replace instances.
Ring-based Concentric rings of users, widening outward: internal staff first, then early adopters, then everyone. Each ring must look healthy before the next opens — Microsoft's own progressive-exposure model. Contained by design: a fault caught in an inner ring never reaches the outer ones, and you halt the advance rather than undo a global release. Medium to high. Needs a way to map users to rings and the patience to let each ring bake.

Blue-green, canary, rolling, and ring releases describe different ways to expose a change. GitHub Actions does not provide native YAML keywords that perform those rollout strategies for every platform. The workflow coordinates them by calling the platform tools: Azure App Service slots can support a blue-green swap; AKS or a traffic manager can support staged traffic; a matrix or sequence of jobs can coordinate rings. The application platform still has to shift traffic and report health.

Fig. 1 · A canary release — a slice first, then a decision
A canary release routes a small slice of traffic to the new version, watches its signals, and either promotes it to full traffic or rolls it back. traffic v1 · current 95% of traffic v2 · canary 5% of traffic watch errors · latency healthy → promote 100% bad → roll back the slice is the whole point — a small, reversible bet
Draw this once by hand. The whole idea is the small red slice: the new version carries only 5% of traffic until the monitor says it is safe, so a bad release harms a fraction of users for minutes, not everyone for an hour. The decision to expand or abort is the strategy — the split is just plumbing.
Play

Play it through

Three minutes. Wire the new build into a slice of real traffic, set how big the slice is and live with the number, then decide how the rest of the traffic gets there. It plays on its own and stops when it needs your hands.

§3

What GitHub Actions coordinates — and what your platform does

A GitHub Actions job can deploy a release, wait for another job, use an environment, and run platform commands. GitHub Actions coordinates these steps; the target platform or traffic layer implements the traffic change and health checks.

For a simple release, deploy once and verify health. For blue-green on App Service, deploy to a staging slot, run checks against it, and swap only after approval. For canary, route a small share of traffic to the new version, observe agreed health measures, then increase traffic or roll back. For rings, define the cohorts and gates explicitly. Do not label a normal deploy as canary unless traffic is actually being split and observed.

Each rollout needs an explicit decision point: route a measured share of traffic, observe agreed health and business signals, then either increase exposure or roll back. GitHub Actions can run the commands and checks in that sequence, but the Azure service or traffic layer performs the traffic change. Store the rollback command and health thresholds with the workflow so the plan is reviewable.

§4

Feature flags: the release switch that isn't a deploy

Deployment strategies still couple the two acts loosely — the feature goes live as the code arrives, just to fewer people at a time. A feature flag severs them completely. The new code ships to production wrapped in a conditional that is switched off by default, so it sits there, deployed but dormant — a dark deployment — reaching no user until you decide otherwise. Releasing the feature is then a separate act: you flip the flag's value, and because the value lives outside the application, nothing rebuilds and nothing redeploys.

Feature flag
A named variable, typically boolean, that gates a block of code at runtime — the block runs only when the flag is on. Because the flag's value is read from an external store rather than compiled in, you can turn a feature on or off without touching, rebuilding, or redeploying the application.

On Azure, the store is Azure App Configuration and its feature management capability, read through a client-side feature manager library. Flags and their current states live centrally; the app asks the feature manager whether a flag is on, and the feature manager answers from App Configuration — optionally through filters that decide per request, so a flag can be on for 5% of users, or only for internal accounts, or only in one region. That single indirection is what buys you four things at once. It enables trunk-based development: unfinished work merges to main behind an off flag instead of rotting on a long-lived branch. It gives you an instant kill switch: a feature misbehaving in production is turned off in seconds, without the rebuild-and-redeploy that an incident least has time for. It supports gradual rollout, widening a flag's audience on a dial. And it makes A/B testing a configuration change: show variant A to half your users, B to the other half, and measure — no deploy per experiment.

The payoff compounds with the strategies above. A canary release controls which servers see the new code; a feature flag controls which users see the new behaviour — and the flag can be flipped off the instant a signal turns bad, faster than any deploy-based rollback could ever reverse it. The two are not rivals; the safest releases use both, so that a bad change can be pulled back at the infrastructure layer or the feature layer, whichever notices first.

§5

Pricing the peak while it is still calm

The value of everything in this part is that it is done before the risk arrives, not during it. A canary you wire up mid-incident is not a canary; it is a panic. A feature flag you add while the site is down is a redeploy you did not have time for. The whole discipline is front-loaded: you spend the calm week building the slice, the monitor, and the switch, so that when the dangerous change ships you already hold the two things an incident cannot manufacture — a small blast radius and a fast undo. Campux is about to learn this on the one date it cannot afford to learn it the hard way.

Case File · Campux Retail

A new checkout, shipped dark, six weeks before November

The storefront's busiest date is coming — and it wants a rewritten checkout

Campux's November traffic peak — the single date the storefront earns its year on — is six weeks out, and the product team wants the rewritten checkout live for it. Shipping a rewritten payment path into the busiest hour of the year is exactly the bet this part exists to unmake. So the reader does not ship it as a release; the reader ships it dark. The new checkout is deployed to production weeks early, behind an Azure App Configuration feature flag that is off for everyone, riding the same gated staging-to-production flow Part F built — proven in place, carrying no traffic, waiting.

Then the release, decoupled from the deploy, happens on the reader's schedule and not the calendar's. A week before the peak, on a calm afternoon, the flag opens the new checkout to 5% of traffic — a canary in users rather than servers — while App Insights watches conversion, errors, and latency. The numbers hold; the slice widens to 20%, then to everyone, each step a value change and not a deploy. And the guarantee that lets the reader sleep is the same value in reverse: if the new checkout stumbles at any point during the peak itself, the flag flips off in seconds and every shopper is back on the proven path, with no redeploy, no rollback pipeline, and no war room. The risky change was priced while nothing was on fire — which is the only time you can afford it.

That is the whole method of this part in one move: the deploy happened weeks early and quietly; the release happened slowly and reversibly; and the two were never the same event. An engineer who can say that sentence in an interview — "I ship the code dark and release the feature on a dial" — is describing the difference between a team that survives its own busiest day and one that gathers afterward to write the postmortem.

On the job

The person who turns the dial

You · Cloud Engineer · owns how change reaches users

When the risky release comes, the team looks to you for the answer to one question: how do we ship this without betting the day on it? You ship the code dark, put it behind a flag, open it to a slice, watch the signals, and widen or kill it on a dial — deploy weeks early, release on your schedule, undo in seconds. That is not a tool you learned; it is the judgement a team trusts with its busiest hour.

Guided practice · Supplied training example

Choose a release from observed health

Use the fictional evidence below. This is a desk exercise; it does not provision Azure resources or measure a live service.

Stable release: 0.2% checkout errors
Candidate, 5% traffic: 8% checkout errors
Shared schema: candidate added a backwards-compatible nullable field
  1. Decide whether to increase candidate traffic.

  2. Name the request sample and observation window needed to trust the comparison.

  3. Write the action and database-compatibility check before rollback.

Check your reasoning after you finish

Hold or revert the candidate based on this adverse signal; do not increase traffic. Check sample sizes and the affected requests. Even a backwards-compatible field requires verification with both releases and the actual data path.

Class 39g

Examination

Four drills, then two situations. They test the distinction this part is built on — deploy is not release — and then whether you can price a strategy: what each rollback story costs, which words are keywords, and where a safeguard is hollow. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.

Drill 01Recall
The rewritten checkout is deployed to production, but no user can reach it because it sits behind a feature flag that is switched off. What has shipping it this way actually bought you?
Marked

B — you have split the deploy from the release. The code is already on the production machines, exercised by the real environment, but reaching no customer; the risky, time-pressured act — putting new behaviour in front of users — is now a value change you make later, to whom you choose, without a rebuild. A misses the entire point: dark code is not wasted, it is de-risked, because the deploy is proven while calm and the release is decoupled from it. C is dangerous nonsense — an off feature is untested in production, not incapable of failing; the moment you flip it on, it can. D invents behaviour that does not exist and would defeat the purpose: the release is a decision, not an automatic side effect of the next deploy.

Drill 02Recall
Can a GitHub Actions workflow alone perform a canary rollout on every deployment platform?
Marked

A. GitHub Actions coordinates jobs, but the Azure service or traffic-management layer must implement the rollout and expose health signals. A job named canary is not a canary release unless the platform actually shifts traffic and the workflow checks results.

Drill 03Select three
Which three of these are true of the release strategies and feature flags as this part defines them?
Marked

Blue-green's price, the empty monitoring window, and the servers-versus-users split. Those three are the load-bearing distinctions of this part. Blue-green buys the best rollback in the table — a routing change back to a machine that never stopped running the old version — and charges you a second production environment plus the discipline of keeping data compatible across the switch. The canary point is the one people get wrong under questioning: increments alone only slows the release down; it is the watching window between increments, and the on: failure hook behind it, that turns a slow rollout into a gate. The servers-versus-users distinction explains why deployment strategies and feature flags can work together.

The rejects invert the part. Rolling has the worst rollback of the three that stay up — you halt the roll, but the batches already replaced come back one batch at a time, which is why the table rates it partial and slower; it is cheap, not safe. And dark code is fully deployed: it is on the production machines, running in the production environment, reaching no user. That is precisely what makes it valuable — the risky, slow act already happened while nothing was on fire.

Drill 04Spot the error
A workflow routes 10% of traffic to a new version, waits ten minutes, then routes the rest. What essential safety step is missing?
# staged rollout
1. Route 10% of traffic to the new version
2. Wait ten minutes
3. Route the remaining traffic
4. Roll back if customers report problems
Marked

B. A pause by itself does not evaluate the application. Check agreed error rate, latency, and business signals before promoting. The target platform must actually route traffic, and the release needs a tested rollback. GitHub Actions coordinates the sequence; it does not create a canary by itself.

Situation 01Write before you reveal
A tech lead has read about blue-green and wants it for the storefront: "Two environments, swap the traffic, roll back instantly. Why would we do anything else?" The storefront is an App Service app with one shared database, and the release you are both worried about is a schema change. What do you advise?
Blue-green really does have the best rollback in the table. Ask what the switch actually switches — and what it does not.
Reasoning

Agree with the appeal first, because the lead is not wrong about the mechanism. Blue-green's rollback is the best of the four: the old version never stopped running, so going back is a routing change and not a redeploy. On Azure you do not even need a second subscription's worth of machinery for it — an App Service deployment slot is the normal way to build it: deploy to staging, warm it, swap. Say that, so the conversation is about fit rather than about who read what.

Then name the thing the swap does not switch: the data. Both environments talk to the same database. If the release changes the schema, the swap moves every user onto code that expects the new shape while the rollback moves them back onto code that expects the old one — and the data has already moved. The instant rollback is instant only for the half of the system that was duplicated. That is not an argument against blue-green; it is the condition the table already prices as "keep data and state compatible across the switch". Meeting it means shipping the schema change and the code change as separate releases, each one backward-compatible with the version beside it — add the column, deploy code that tolerates both shapes, then remove the old path later.

Close with what you would actually do this month. For a schema change, the exposure you want is over users, not machines: ship the new checkout dark behind an App Configuration flag, open it to a small slice while App Insights watches, and widen on evidence. The flag rolls back faster than any swap, and it does not pretend the database came back with it. Blue-green stays on the table for stateless releases where the slot swap really is the whole story. The sentence for the meeting: blue-green gives you an instant rollback of the code, and we are worried about a change to the data — so let us pick the lever that can actually be undone.

Situation 02Write before you reveal
An interviewer says: "What's the difference between a deployment and a release, and how would you ship a risky change to our busiest week?" You have two minutes. What do you say?
The first half is a definition and takes fifteen seconds. The second half is where the answer is won — draw Figure 1 out loud, and say when the work happens.
Reasoning

Answer the definition cleanly and move on, because it is the setup, not the question. "A deployment puts new code on the machines. A release is the moment a user can reach the new behaviour. For most of software's history they were the same event, and that is why every deploy felt like a bet." Fifteen seconds. Do not elaborate; the interviewer is waiting to hear whether you can do anything with the distinction.

Deploying code behind a disabled feature flag separates deployment from release. The new code can run in production before users receive the feature. Later, enable the flag for a small group, watch meaningful metrics, and expand or turn it off based on results. A canary release is a separate control that sends only some traffic to the new code; use platform routing and health checks to implement it.

Close on the timing, which is the part most candidates leave out. "The thing I would insist on is that none of this gets built during the busy week. A canary you wire up mid-incident is not a canary, it's a panic, and a flag you add while the site is down is a redeploy you don't have time for. You spend the calm week buying the two things an incident can't manufacture — a small blast radius and a fast undo." That sentence is the difference between describing tools you have read about and describing a judgement a team can hand its busiest day to.

Examination record · first attempt
0/4
Class 39g · Complete
Retain this much

Five things worth carrying out of this part

  1. Deploying code and releasing a feature are two acts. Pull them apart and each can fail, and be undone, on its own clock — which is the whole subject of this part.
  2. Four strategies, ranked by what the rollback costs. Blue-green: two environments, instant routing rollback, and the bill for the second one. Canary: a slice first, then a decision. Rolling: cheap batches, partial and slower rollback. Ring-based: widening cohorts, each one gated.
  3. GitHub Actions has no built-in traffic-shifting strategy keywords. The workflow coordinates platform features such as App Service deployment slots or AKS traffic routing.
  4. A canary requires gradual traffic exposure, health monitoring between increases, and a tested rollback. A series of deployment jobs alone is not a canary.
  5. A feature flag in Azure App Configuration turns the release into a value change: ship dark, kill in seconds, widen on a dial, run A/B as configuration. A canary picks which servers run the code; a flag picks which users see the behaviour. Build both while nothing is on fire.
Notes
  1. GitHub Actions workflow jobs coordinate deployments. The actual capabilities and rollback mechanisms depend on the target service, such as App Service slots or AKS traffic routing. Check that platform's current documentation before adopting a rollout design.