Skip to content
CAMPUX Cloud Bootcamp Phase Four · Class Thirty-Nine
Phase Four — Operate, Secure & AI
Reading 15 min · Drills 4 · Part VII of IX
Class Thirty-Nine · Pipelines
Class Thirty-Nine · Part VII

Ship it dark, release it slowly

Deploying code and releasing a feature are two different acts with two different risks; this part gives you the strategies that control how new code reaches users and the feature flags that let it arrive switched off — so the two can fail, and be undone, independently.

§1

Two different acts, wearing one word

A deploy and a release feel like a single event — you push a button and something new is live — but they are two acts with two clocks and two blast radii, and confusing them is how a Tuesday afternoon becomes an outage. Deploying code is moving a new build onto the machines that run it. Releasing a feature is the moment a user can actually reach the new behaviour. For most of software's history these happened at the same instant, because the only way to expose new behaviour was to put the code that produced it into production — so the deploy was the release, and every deploy was therefore a bet on a large number of untested users hitting new code at once.1

This part is about pulling those two acts apart so they can fail — and be undone — independently. There are two levers. Deployment strategies control how the new code reaches the infrastructure: all at once, a slice at a time, or machine by machine, each buying you a different rollback story for a different price. Feature flags go further and separate the acts entirely: the code ships to production switched off — dark — and the feature is released later, to whom you choose, by flipping a value in a central store, with no deploy at all. Together they turn "release" from a leap you take once, holding your breath, into a dial you can turn up slowly and back down in seconds. That dial is the difference between an engineer a team lets near production and one it does not.

Shipping code is not releasing a feature.

§2

Four ways to let new code reach users

Every deployment strategy is an answer to one question: when the new version is wrong — and one day it will be — how fast, and how cheaply, can you take it back? A strategy that exposes all users at once has the simplest mechanics and the worst answer; a strategy that exposes a slice first has more moving parts and a far better one. There is no free option. You are choosing where to spend complexity so that a bad release costs minutes and a handful of users instead of an hour and all of them. The four below are the ones worth knowing by name, ranked roughly from blunt to surgical.

Table 1 — Four release strategies, and what each buys
StrategyHow it worksRollback storyCost & complexity
Blue-green Two identical production environments — one live (blue), one idle (green). Deploy the new version to green, test it in isolation, then switch all traffic to green at once. Instant and total: switch traffic back to blue, which is untouched and still running the old version. Rollback is a routing change, not a redeploy. High. You pay to run two full environments and must keep data and state compatible across the switch.
Canary Release the new version to a small slice of traffic — say 5% — while everyone else stays on the old one. Watch the signals; expand in increments if healthy, abandon if not. Good and cheap: a bad canary is pulled after harming a fraction of users, before the increment that would have exposed the rest. Medium. Needs traffic splitting and, to be worth anything, real monitoring to decide expand-or-abort.
Rolling Replace instances in batches — a few machines at a time — until the whole fleet runs the new version, keeping enough of the old serving traffic to stay up throughout. Partial and slower: halt the roll on a bad batch, but instances already updated must be rolled back one batch at a time. Low. No second environment; it is the default way most orchestrators replace instances.
Ring-based Concentric rings of users, widening outward: internal staff first, then early adopters, then everyone. Each ring must look healthy before the next opens — Microsoft's own progressive-exposure model. Contained by design: a fault caught in an inner ring never reaches the outer ones, and you halt the advance rather than undo a global release. Medium to high. Needs a way to map users to rings and the patience to let each ring bake.

Two of these are named strategies you configure; two are patterns you assemble. Blue-green and ring-based are shapes of intent — Azure gives you the pieces (deployment slots, traffic rules, ring definitions) and you compose them. Canary and rolling, as the next section shows, are keywords the Azure DevOps deployment job understands directly. Canary is the one to picture, because it is the one an interviewer will ask you to draw.

Fig. 1 · A canary release — a slice first, then a decision
A canary release routes a small slice of traffic to the new version, watches its signals, and either promotes it to full traffic or rolls it back. traffic v1 · current 95% of traffic v2 · canary 5% of traffic watch errors · latency healthy → promote 100% bad → roll back the slice is the whole point — a small, reversible bet
Draw this once by hand. The whole idea is the small red slice: the new version carries only 5% of traffic until the monitor says it is safe, so a bad release harms a fraction of users for minutes, not everyone for an hour. The decision to expand or abort is the strategy — the split is just plumbing.
Play

Play it through

Three minutes. Wire the new build into a slice of real traffic, set how big the slice is and live with the number, then decide how the rest of the traffic gets there. It plays on its own and stops when it needs your hands.

§3

What the deployment job actually supports

In an Azure DevOps YAML pipeline, you request a strategy inside a deployment job — the special job type from Part F that records deployment history against an environment. The strategy: keyword accepts exactly three values: runOnce, rolling, and canary. Each one runs your steps through a fixed set of lifecycle hooks — preDeploy, deploy, routeTraffic, postRouteTraffic, and an on: failure / on: success pair — so the strategy decides how many times and against how much of the fleet those hooks fire.

runOnce is the plain one: every hook runs once, then success or failure. rolling replaces targets in batches sized by maxParallel — and, importantly, Azure DevOps currently supports the rolling strategy only against virtual-machine resources, not arbitrary services. canary takes an increments: list — for example [10, 20] — and iterates the deploy-and-watch hooks once per increment before promoting to the remainder, most naturally against Kubernetes or AKS. Notice what is not on that list: there is no blueGreen and no ring keyword. Blue-green on Azure is something you build yourself — most commonly with App Service deployment slots, deploying to a staging slot and then swapping it with production. Ring-based release is a practice, not a YAML value: you express it as a sequence of environments or user cohorts, each gated, following Microsoft's safe-deployment guidance. Knowing which of the four is a keyword and which is a pattern you assemble is exactly the distinction that separates someone who has read the docs from someone who has only heard the words.

The lifecycle hooks are where the strategy earns its keep. postRouteTraffic is the interesting one: it is the window where you monitor the version you just exposed — for the defined interval, before the next increment — and your on: failure hook is where the rollback lives. A canary strategy with an empty postRouteTraffic is not a canary; it is a slow big-bang that happens to deploy in increments while watching nothing.

§4

Feature flags: the release switch that isn't a deploy

Deployment strategies still couple the two acts loosely — the feature goes live as the code arrives, just to fewer people at a time. A feature flag severs them completely. The new code ships to production wrapped in a conditional that is switched off by default, so it sits there, deployed but dormant — a dark deployment — reaching no user until you decide otherwise. Releasing the feature is then a separate act: you flip the flag's value, and because the value lives outside the application, nothing rebuilds and nothing redeploys.

Feature flag
A named variable, typically boolean, that gates a block of code at runtime — the block runs only when the flag is on. Because the flag's value is read from an external store rather than compiled in, you can turn a feature on or off without touching, rebuilding, or redeploying the application.

On Azure, the store is Azure App Configuration and its feature management capability, read through a client-side feature manager library. Flags and their current states live centrally; the app asks the feature manager whether a flag is on, and the feature manager answers from App Configuration — optionally through filters that decide per request, so a flag can be on for 5% of users, or only for internal accounts, or only in one region. That single indirection is what buys you four things at once. It enables trunk-based development: unfinished work merges to main behind an off flag instead of rotting on a long-lived branch. It gives you an instant kill switch: a feature misbehaving in production is turned off in seconds, without the rebuild-and-redeploy that an incident least has time for. It supports gradual rollout, widening a flag's audience on a dial. And it makes A/B testing a configuration change: show variant A to half your users, B to the other half, and measure — no deploy per experiment.

The payoff compounds with the strategies above. A canary release controls which servers see the new code; a feature flag controls which users see the new behaviour — and the flag can be flipped off the instant a signal turns bad, faster than any deploy-based rollback could ever reverse it. The two are not rivals; the safest releases use both, so that a bad change can be pulled back at the infrastructure layer or the feature layer, whichever notices first.

§5

Pricing the peak while it is still calm

The value of everything in this part is that it is done before the risk arrives, not during it. A canary you wire up mid-incident is not a canary; it is a panic. A feature flag you add while the site is down is a redeploy you did not have time for. The whole discipline is front-loaded: you spend the calm week building the slice, the monitor, and the switch, so that when the dangerous change ships you already hold the two things an incident cannot manufacture — a small blast radius and a fast undo. Campux is about to learn this on the one date it cannot afford to learn it the hard way.

Case File · Campux Retail

A new checkout, shipped dark, six weeks before November

The storefront's busiest date is coming — and it wants a rewritten checkout

Campux's November traffic peak — the single date the storefront earns its year on — is six weeks out, and the product team wants the rewritten checkout live for it. Shipping a rewritten payment path into the busiest hour of the year is exactly the bet this part exists to unmake. So the reader does not ship it as a release; the reader ships it dark. The new checkout is deployed to production weeks early, behind an Azure App Configuration feature flag that is off for everyone, riding the same gated staging-to-production flow Part F built — proven in place, carrying no traffic, waiting.

Then the release, decoupled from the deploy, happens on the reader's schedule and not the calendar's. A week before the peak, on a calm afternoon, the flag opens the new checkout to 5% of traffic — a canary in users rather than servers — while App Insights watches conversion, errors, and latency. The numbers hold; the slice widens to 20%, then to everyone, each step a value change and not a deploy. And the guarantee that lets the reader sleep is the same value in reverse: if the new checkout stumbles at any point during the peak itself, the flag flips off in seconds and every shopper is back on the proven path, with no redeploy, no rollback pipeline, and no war room. The risky change was priced while nothing was on fire — which is the only time you can afford it.

That is the whole method of this part in one move: the deploy happened weeks early and quietly; the release happened slowly and reversibly; and the two were never the same event. An engineer who can say that sentence in an interview — "I ship the code dark and release the feature on a dial" — is describing the difference between a team that survives its own busiest day and one that gathers afterward to write the postmortem.

On the job

The person who turns the dial

You · Cloud Engineer · owns how change reaches users

When the risky release comes, the team looks to you for the answer to one question: how do we ship this without betting the day on it? You ship the code dark, put it behind a flag, open it to a slice, watch the signals, and widen or kill it on a dial — deploy weeks early, release on your schedule, undo in seconds. That is not a tool you learned; it is the judgement a team trusts with its busiest hour.

Class 39g

Examination

Four drills, then two situations. They test the distinction this part is built on — deploy is not release — and then whether you can price a strategy: what each rollback story costs, which words are keywords, and where a safeguard is hollow. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.

Drill 01Recall
The rewritten checkout is deployed to production, but no user can reach it because it sits behind a feature flag that is switched off. What has shipping it this way actually bought you?
Marked

B — you have split the deploy from the release. The code is already on the production machines, exercised by the real environment, but reaching no customer; the risky, time-pressured act — putting new behaviour in front of users — is now a value change you make later, to whom you choose, without a rebuild. A misses the entire point: dark code is not wasted, it is de-risked, because the deploy is proven while calm and the release is decoupled from it. C is dangerous nonsense — an off feature is untested in production, not incapable of failing; the moment you flip it on, it can. D invents behaviour that does not exist and would defeat the purpose: the release is a decision, not an automatic side effect of the next deploy.

Drill 02Recall
In an Azure DevOps YAML deployment job, which exact set of values does the strategy: keyword accept?
Marked

B — runOnce, rolling, canary. Nothing else. This is the line that separates having read the docs from having heard the words. Blue-green and ring-based are real strategies, but they are not strategy: keywords — you compose them yourself (blue-green most often with App Service deployment slots; rings as a gated sequence of environments or cohorts). So A and C, which offer blueGreen or ring, name things the YAML schema will reject at parse time. D's featureFlag confuses two layers entirely: feature flags live in your application and Azure App Configuration, not in the pipeline's deployment strategy. Reach for a keyword that does not exist in front of an interviewer and the credibility cost is immediate.

Drill 03Select three
Which three of these are true of the release strategies and feature flags as this part defines them?
Marked

Blue-green's price, the empty monitoring window, and the servers-versus-users split. Those three are the load-bearing distinctions of this part. Blue-green buys the best rollback in the table — a routing change back to a machine that never stopped running the old version — and charges you a second production environment plus the discipline of keeping data compatible across the switch. The canary point is the one people get wrong under questioning: increments alone only slows the release down; it is the watching window between increments, and the on: failure hook behind it, that turns a slow rollout into a gate. And the third is the reason the two levers combine rather than compete.

The rejects invert the part. Rolling has the worst rollback of the three that stay up — you halt the roll, but the batches already replaced come back one batch at a time, which is why the table rates it partial and slower; it is cheap, not safe. And dark code is fully deployed: it is on the production machines, running in the production environment, reaching no user. That is precisely what makes it valuable — the risky, slow act already happened while nothing was on fire.

Drill 04Spot the error
An engineer writes the plan for the rewritten checkout before the November peak. One line hollows out the safeguard it sits inside. Which?
# shipping the new checkout before the peak
1.  Deploy it to production six weeks early, behind an
    App Configuration flag that is off for everyone.
2.  Run the deployment job with strategy: canary and
    increments: [10, 20], but leave postRouteTraffic
    empty — the increments are the safety, and someone
    has the dashboards open anyway.
3.  A week before the peak, open the flag to 5% of
    traffic and watch conversion, errors and latency.
4.  If the checkout stumbles during the peak itself,
    flip the flag off rather than redeploy the old build.
Marked

Line two — and the tell is "the increments are the safety". They are not. increments only decides how much of the fleet each pass touches; the hook that makes it a canary is postRouteTraffic, which runs after traffic is routed and monitors the version you just exposed for a defined interval before the next increment. Leave it empty and the pipeline marches 10, then 20, then the remainder, never once pausing to ask whether the thing it is spreading is healthy. That is a big-bang with extra steps, and "someone has the dashboards open" is a human doing a machine's job at the exact moment a human is least reliable. The rollback belongs in the same place: the on: failure hook.

The other three lines are the part at its best. A misreads the whole idea: dark code is deployed and exercised in production, which is what makes the later release cheap. C inverts the canary — a small first slice is the point, and the sample only has to be large enough for the monitor to read a trend. D inverts the flag — flipping a value is seconds, a redeploy under peak load is the thing you built the flag to avoid.

Situation 01Write before you reveal
A tech lead has read about blue-green and wants it for the storefront: "Two environments, swap the traffic, roll back instantly. Why would we do anything else?" The storefront is an App Service app with one shared database, and the release you are both worried about is a schema change. What do you advise?
Blue-green really does have the best rollback in the table. Ask what the switch actually switches — and what it does not.
Reasoning

Agree with the appeal first, because the lead is not wrong about the mechanism. Blue-green's rollback is the best of the four: the old version never stopped running, so going back is a routing change and not a redeploy. On Azure you do not even need a second subscription's worth of machinery for it — an App Service deployment slot is the normal way to build it: deploy to staging, warm it, swap. Say that, so the conversation is about fit rather than about who read what.

Then name the thing the swap does not switch: the data. Both environments talk to the same database. If the release changes the schema, the swap moves every user onto code that expects the new shape while the rollback moves them back onto code that expects the old one — and the data has already moved. The instant rollback is instant only for the half of the system that was duplicated. That is not an argument against blue-green; it is the condition the table already prices as "keep data and state compatible across the switch". Meeting it means shipping the schema change and the code change as separate releases, each one backward-compatible with the version beside it — add the column, deploy code that tolerates both shapes, then remove the old path later.

Close with what you would actually do this month. For a schema change, the exposure you want is over users, not machines: ship the new checkout dark behind an App Configuration flag, open it to a small slice while App Insights watches, and widen on evidence. The flag rolls back faster than any swap, and it does not pretend the database came back with it. Blue-green stays on the table for stateless releases where the slot swap really is the whole story. The sentence for the meeting: blue-green gives you an instant rollback of the code, and we are worried about a change to the data — so let us pick the lever that can actually be undone.

Situation 02Write before you reveal
An interviewer says: "What's the difference between a deployment and a release, and how would you ship a risky change to our busiest week?" You have two minutes. What do you say?
The first half is a definition and takes fifteen seconds. The second half is where the answer is won — draw Figure 1 out loud, and say when the work happens.
Reasoning

Answer the definition cleanly and move on, because it is the setup, not the question. "A deployment puts new code on the machines. A release is the moment a user can reach the new behaviour. For most of software's history they were the same event, and that is why every deploy felt like a bet." Fifteen seconds. Do not elaborate; the interviewer is waiting to hear whether you can do anything with the distinction.

Then answer the real question with two levers and a date. "For the busiest week, I ship the code weeks early and release it later. The deploy goes out on a quiet afternoon behind a feature flag in App Configuration, switched off — it's in production, exercised, proving nothing to any customer. Then the release is its own decision: open the flag to a small slice, five percent, while I watch errors, latency and the business metric that actually matters — conversion, for a checkout. If the numbers hold I widen it in increments; if they don't, I turn the flag off and everyone is back on the proven path in seconds, with no redeploy." Name the infrastructure lever beside it: "On the deploy side the same shape is a canary — strategy: canary with increments and a postRouteTraffic window that watches between them. The canary controls which servers run the code; the flag controls which users see the behaviour. I would use both, because whichever notices first is the one that saves you."

Close on the timing, which is the part most candidates leave out. "The thing I would insist on is that none of this gets built during the busy week. A canary you wire up mid-incident is not a canary, it's a panic, and a flag you add while the site is down is a redeploy you don't have time for. You spend the calm week buying the two things an incident can't manufacture — a small blast radius and a fast undo." That sentence is the difference between describing tools you have read about and describing a judgement a team can hand its busiest day to.

Examination record · first attempt
0/4
Class 39g · Complete
Retain this much

Five things worth carrying out of this part

  1. Deploying code and releasing a feature are two acts. Pull them apart and each can fail, and be undone, on its own clock — which is the whole subject of this part.
  2. Four strategies, ranked by what the rollback costs. Blue-green: two environments, instant routing rollback, and the bill for the second one. Canary: a slice first, then a decision. Rolling: cheap batches, partial and slower rollback. Ring-based: widening cohorts, each one gated.
  3. The strategy: keyword accepts exactly runOnce, rolling, and canary. Rolling is supported against virtual-machine resources only. Blue-green is a pattern you build, most often with App Service deployment slots; ring-based is a practice, not a YAML value.
  4. A canary is not its increments. postRouteTraffic is the window where you monitor what you just exposed, and on: failure is where the rollback lives. Empty that hook and you have a big-bang delivered in instalments.
  5. A feature flag in Azure App Configuration turns the release into a value change: ship dark, kill in seconds, widen on a dial, run A/B as configuration. A canary picks which servers run the code; a flag picks which users see the behaviour. Build both while nothing is on fire.
Notes
  1. "Deploy" and "release" are used loosely across the industry, and plenty of good teams say one when they mean the other — so do not correct a colleague on vocabulary. The distinction earns its keep as a design question, not a definition: for any given change, ask what has to be true before a user can reach it, and whether that thing is a deploy you can only do once or a value you can change at will. Microsoft's own framing of this sits under feature management in Azure App Configuration, which describes feature flags as decoupling feature release from code deployment.