The cost of a defect climbs the later you meet it
A defect is cheapest at the moment it is written and most expensive at the moment a customer finds it, and everything between those two points is a slope you are either sliding down or climbing back up. The developer who catches a broken calculation in a unit test on their own laptop pays a minute and a shrug. The same broken calculation caught in code review costs an hour of two people's attention. Caught by QA a week later, it costs a bug ticket, a context-switch back into code nobody remembers, and a re-test. Caught by a customer in production, it costs all of that plus the money the wrong number moved, the trust it spent, and an incident bridge convened at an hour nobody chose.1 The defect did not get worse. The distance you had to reach back to fix it did.
Shift-left is the whole discipline named after that slope. If the cost of a defect rises the further right it travels — right being later, toward release and the customer — then the move that pays is to drag detection as far left as it will go, toward the commit that introduced it. The pipeline you built across this track is where that move becomes automatic: tests run on every change, before the artifact is ever published, so a defect meets a machine that was watching for it on the same afternoon it was born, not a customer three weeks later. This is not about testing more for its own sake. It is about buying the cheap catch instead of the expensive one, every time, without relying on anyone remembering to.
- Shift-left
- Moving defect detection earlier in the delivery flow — leftward, toward the commit — so faults are caught by automated checks in continuous integration rather than by humans downstream. The name treats the pipeline as a timeline that runs left to right: build, test, publish, deploy. Every step you can make a test fail at is a step the defect never reaches.
Fail in the pipeline, not in production.
Tests as steps, and the results the pipeline keeps
A test in your repository does nothing for you until the pipeline runs it on every change, so the first move is mechanical: add a step that executes the test suite. In a GitHub Actions job that is usually a single script line — dotnet test, npm test, pytest — and it carries a property you get for free that matters more than it looks. A test runner exits non-zero when a test fails, and a pipeline step that exits non-zero fails the job. So the mere act of running the tests, with no extra configuration, already makes the pipeline refuse to continue when they fail. That default is the entire fail-fast behaviour, and §4 is mostly about not throwing it away by accident.
Running tests and recording their results are separate tasks. The test command should return a non-zero exit code when a test fails; that failure stops later steps in the job by default. Save a JUnit, TRX, or other report as a workflow artifact so the team can inspect failures after the run. GitHub Actions does not turn every report into a built-in Tests tab by itself; a formatted report view requires an additional action or an external test system. Preserve the raw report even when using a viewer.
The tests themselves come in tiers, and a healthy pipeline runs more than one. The distinction an interviewer wants you to hold is speed-and-scope against confidence.
| Tier | What it exercises | Speed & where it runs | What a failure tells you |
|---|---|---|---|
| Unit | One function or class in isolation, with its collaborators faked, asserting the logic does what it claims for given inputs. | Milliseconds each; thousands run on the build agent with no external dependency. First tests in the pipeline. | A precise, local fault: this function computes the wrong thing. The stack points near the line that broke. |
| Integration | Several components together across a real boundary — a database, a queue, an HTTP call — checking the seams hold, not the logic within them. | Seconds to minutes; needs services stood up (a container, a test database), so it runs after units pass. | A wiring fault: the pieces are each fine but disagree at the join — a schema, a serialization, a contract mismatch. |
Order them by that table. Units are fast and cheap, so run them first and let them fail the build in seconds; integration tests are slower and need scaffolding, so run them after, once the cheap checks have already caught the obvious. This is shift-left applied inside the pipeline itself: the fastest, most localised catch goes leftmost, so most bad changes die before the expensive tests ever start.
Coverage: an honest number people misread
Once tests run in the pipeline, the next question a manager asks is "how much of the code do they cover?" — and code coverage answers it, with a caveat you must state in the same breath. Your test runner, given a coverage flag, records which lines and branches of the product code were actually executed while the tests ran, and writes it in a standard format — Cobertura or JaCoCo. Upload the report as a workflow artifact or send it to a dedicated reporting service. GitHub Actions does not provide a universal built-in coverage tab; the display depends on the viewer or service you add.
Here is the caveat, and saying it out loud is how you sound like an engineer instead of a dashboard. Coverage measures execution, not assertion. A test can call a function, cover every line inside it, and check nothing about the result — the coverage number climbs and the code is no safer. So eighty per cent coverage does not mean eighty per cent of your logic is verified; it means eighty per cent of your lines were run by something during the tests. Coverage is a useful floor and a dishonest ceiling: a low number reliably tells you a region is untested and dangerous, while a high number tells you almost nothing about whether the tests would notice if the behaviour broke. Read it as "what is definitely untested," never as "what is definitely safe."2
Which is exactly why coverage belongs in a pipeline but not as a target to be chased. Track it so a pull request that drops coverage on a critical module gets a second look; publish it so the trend is visible. But an engineer who treats the percentage as the goal will get the percentage — teams reliably write assertion-free tests to hit a mandated number, and the pipeline goes green over code no one has actually checked. The number is a smoke detector, not a certificate.
The quality gate: where the pipeline refuses
Everything so far produces signals — pass or fail, covered or not. A quality gate is the point in the pipeline where those signals are turned into a verdict that can stop the line. It is not a feature you switch on; it is a policy you assemble from the fail-fast behaviours you already have, placed deliberately before the deploy so that nothing which fails it can reach an environment.
- Quality gate
- A pass/fail checkpoint in the pipeline that must be satisfied before the flow may proceed to the next stage — typically before deployment. It aggregates conditions such as "all tests passed" and "coverage did not fall below the threshold," and when any condition fails, the pipeline stops rather than shipping. A gate is a machine saying no so a tired human does not have to.
The simplest gate you already own: because a failing test exits non-zero and fails the job, and because a failed job halts the stage, an unbroken test suite is a de facto gate on every run. The discipline is not to defeat it — which is precisely what a stray continue-on-error: true on a test step does, quietly turning "the tests must pass" into "the tests may fail and we ship anyway." Half the failures of this section are not missing gates; they are gates disarmed by one hopeful line.
GitHub Actions does not set a universal coverage threshold for you. Make the test or coverage-check command exit non-zero when the agreed threshold is missed, or use a reporting service that returns a failing status. Then require that status in a repository ruleset or branch protection setting. Uploading a coverage report makes results available for inspection; uploading alone does not create a blocking gate. Treat line coverage as evidence about which code ran, not proof that its behaviour was asserted.
The last piece places the gate where it cannot be skipped. A gate that only runs when someone remembers to run the pipeline is optional, and optional gates are decoration. You make it mandatory with a branch policy — build validation on the main branch — so the pipeline, tests and all, must pass before a pull request can merge at all. Now the gate is not a step in a flow a developer might route around; it is a condition on the branch itself. The tests do not ask to run. They must.
Play it through
Three minutes, two things to do. Put a real gate in the gap before deploy, then wire it to fail the build instead of just warning about a bad number. It plays on its own and stops when it needs your hands.
Keeping the gate honest — and fast
A gate that takes too long will be bypassed. Keep fast unit tests on every pull request, then put slower integration or end-to-end suites in deliberate later jobs. GitHub Actions can run independent test shards in parallel with a matrix, but parallel jobs consume runner capacity and may increase usage. Measure elapsed time and cost before increasing the matrix size; fewer minutes on the clock do not always mean lower cost.
The other honesty is ordering, which you have already met: cheap and fast on the left, slow and broad on the right, so most bad changes fail in the first fast slice and never pay for the expensive one. A gate arranged this way gives its verdict on the common case — a genuine bug — in seconds, and reserves the minutes only for changes that earned a deeper look by surviving the quick checks. That is what lets a gate be both strict and tolerable, and a gate that is both is a gate that stays. Campux is about to install exactly this in front of the one release it cannot afford to get wrong.
The rewritten checkout gets a gate before it earns the right to ship dark
Last part, the reader shipped Campux's rewritten checkout dark — deployed to production weeks before November, behind a feature flag switched off for everyone, waiting to be released on a dial. But shipping code dark only removes the risk of it being reached; it does nothing about the risk of it being wrong. A broken checkout deployed dark is still a broken checkout the moment the flag opens. So before that artifact was allowed anywhere near production, it had to pass through a gate the reader built into the pipeline: the unit suite covering the new pricing and tax math runs first and fails the build in seconds if a number is off; an integration suite then stands up a test database and a stub payment provider and checks the seams hold; results publish to the Tests tab so a failure names the exact broken case; and a branch policy on main makes the whole thing mandatory — no change reaches the checkout without passing it.
The payoff arrives on the calm afternoon a week before the peak, when the flag opens the new checkout to its first five per cent. The reader is not hoping the code is correct; the code cleared a gate that would have refused to deploy it otherwise. The strategy from Part G controls who sees the new path; this part is why the path was trustworthy enough to expose at all. Shipping dark bought the reader control over the release. The gate is what made the thing being released worth releasing — and it did so on the one estate, in the one month, where finding a payment bug in front of a customer would have cost Campux its year.
That is the sentence this part is built to earn in an interview: "I don't test to raise a number; I put a gate before the deploy so a defect fails in the pipeline on the afternoon it's written, not in front of a customer three weeks later." An engineer who can say that — and name where the gate sits, what disarms it, and why coverage is a floor and not a certificate — is describing the difference between a pipeline that ships confidence and one that ships hope.
The person who put the gate before the deploy
When a release goes wrong, the first question is always "how did this reach production?" — and the engineer who wired the gate has the answer ready: it couldn't have, if it failed a test. You make the tests run on every change, publish the results where the team reads them, put a mandatory gate before the deploy, and keep it fast enough that no one wants it gone. That is not a task you completed once; it is the standing reason the team trusts the pipeline with its busiest day. The gap was never that you couldn't test — it was proving the code is safe before it ships, and now that proof is a machine, not a promise.
One function, one assertion, one red run
The artifact gate below checks a file's contents. That is a gate, but it is not an application test. Write one real test first, so "unit test" means something you have run. In the practice repository, add two small files:
mkdir -p src
cat > src/price.js <<'EOF'
exports.withTax = (net, rate) => Math.round(net * (1 + rate) * 100) / 100;
EOF
cat > src/price.test.js <<'EOF'
const test = require('node:test');
const assert = require('node:assert');
const { withTax } = require('./price');
test('10.00 plus 20% tax is 12.00', () => {
assert.strictEqual(withTax(10, 0.2), 12);
});
EOF
node --test src/price.test.js
price.js is the application: one function that adds tax. price.test.js calls it with known inputs and states the answer it must give. That is a unit test: one small piece, checked on its own.
Run it locally if you have Node.js; the summary reports pass 1, fail 0, and the command exits with 0. Then add a step to your workflow's job, after checkout: - run: node --test src/price.test.js. Push, and see the job go green.
Predict, then break it: change the expected 12 to 13 and push. The test reports fail 1, the command exits with 1, and that non-zero exit is what turns the job red and stops the jobs that depend on it. That is the whole mechanism of a quality gate.
Put the 12 back. Coverage (§3) would now ask how much of price.js this test touched; for a one-line function the answer is all of it, which says nothing about whether the test is any good.
Carry the same build forward
Use the same disposable repository and artifact from the first workflow lab. Its supplied continuation gives you a failing content check, a skipped downstream job, checksum verification and an optional observable approval pause. Read the account prerequisites before claiming that a gate exists.
Examination
Four drills, then two situations. These test whether you can tell a signal from a gate, and a floor from a certificate — the two distinctions this part turns on. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.
B — the test step runs the tests; the publish step makes their verdict legible. Execution and recording are different jobs. The runner already ran everything and already failed the job on a red test by exiting non-zero — so C is wrong, and dangerously so: believing the publish step is what enforces failure is how someone deletes it to "speed things up" and quietly loses the fail-fast they thought they had. A is backwards; the tests ran at the dotnet test line. D invents auto-retry, which is not what publishing does and would hide exactly the flakiness you want visible. Publishing buys you the Tests tab — which test, what message, how long, and the trend across runs — instead of a wall of console output to scroll.
B — coverage is execution, not assertion. A test can run every line of a function and check nothing about the result; the number climbs and the code is no safer. So 85% means 85% of the lines were touched by some test, not that 85% of the behaviour is verified — C's equation of the two is the exact misread that gets bad code shipped behind a green dashboard. A treats 100% as a mandate, which reliably produces assertion-free tests written to hit the target. D overcorrects into throwing away a genuinely useful signal: a low number reliably flags untested, dangerous regions. Read coverage as a floor that tells you what is definitely untested — never as a ceiling that certifies what is safe.
Fast tests first, gate made mandatory by branch policy, stage parallelised to stay fast. Those three are the whole posture: catch cheaply and early, make the catch un-skippable, and keep it quick enough to survive a deadline. The two rejects are the section's named failure modes. Putting the gate after the deploy is not a gate at all — production already ran the code, so the check became a postmortem; the entire point of Figure 1 is that the red box sits before the deploy. And continueOnError on the test step is the one line that disarms the fail-fast you got for free — it turns "the tests must pass" into "the tests may fail and we ship anyway," which is a gate in name and a hole in fact.
# checkout pipeline — test stage before deploy
1. Run the unit suite first; it fails the build in
seconds if the pricing math is wrong.
2. Run the integration suite next, against a test
database and a stub payment provider.
3. Set continueOnError: true on the test step so a
red build never blocks the team's afternoon.
4. Publish the results and coverage, and require the
pipeline to pass via a branch policy on main.
Line three — and the tell is that it undoes the one thing you got for nothing. A test runner exits non-zero on failure and a failing step fails the job; that default is the gate. continue-on-error: true tells the pipeline to treat the step's failure as a pass, so a red test suite still lets the stage proceed to deploy — the checkout ships broken and the run shows green. Every other line is the plan at its best: fast unit tests first is correct shift-left ordering (A inverts it), a stubbed payment provider is exactly how you integration-test a payment path without charging a real card (B is wrong), and a branch policy makes the gate mandatory, not optional (D inverts its purpose). The dangerous line is the one that looks like a small convenience and is actually the off switch.
Separate the two asks, because one is negotiable and one is not. Lowering a coverage threshold is a floor moving, not a gate opening — it means new code may be less tested than your standard, which is a real cost but a bounded one, and on the eve of a freeze it may be a defensible trade for a small, well-understood change. So you can discuss the threshold: agree it drops only for this change, that the lower number is written down with the reason, and that it snaps back the day the freeze lifts. Whichever mechanism holds your threshold, the drop is a commit somebody can see and revert. That is a floor bent, in writing, with an expiry.
Refuse the second ask, because it is not the same kind of thing. continueOnError on the integration test does not lower a floor; it removes the gate — the pipeline goes green whether or not that test passes, on this release and every release after, until someone remembers to take it back out, which no one will. Two days before the busiest date of the year, on the payment path, is the exact worst moment to make the checkout's integration test advisory. The flaky test is a real problem, but the fix is to fix the flake — quarantine it into a non-blocking lane you actually triage, or stabilise it — not to blind the pipeline to it.
Give them the sentence for the room. "I'll drop the coverage floor for this one PR with an expiry — that's a trade we can price. I won't make the checkout's tests advisory two days before peak; a flaky test gets quarantined and triaged, not switched to always-pass, because 'ignore this test for now' is how the one bug we can't afford ships green." That distinguishes the engineer who protects the gate under pressure from the one who quietly dismantles it to make a deadline.
Name the principle before the mechanics, so they know you understand the why. "The whole idea is shift-left: I want a defect to fail in the pipeline on the afternoon it's written, not in front of a customer three weeks later, because the later you catch it the more it costs." Then walk the shape: a change triggers a build; the fast unit suite runs first and fails the build in seconds if the logic is wrong; the slower integration tests run next against a test database and stubbed dependencies; results and coverage publish so a failure names the exact broken case.
Then put the gate where it belongs and say what would break it. "The gate sits before the deploy — a failing test fails the job, so nothing that fails reaches an environment — and I make it mandatory with a branch policy on main, so tests must pass before the code can even merge. The one thing that would quietly defeat all of that is a continueOnError on the test step, so I watch for exactly that. And I keep the stage parallelised, because a gate that gets slow gets switched off." Naming the failure mode unprompted is what signals you have actually run one of these, not just read about it.
Close on the honest limit, because it is the most senior thing you can say. "I track coverage but I don't chase it — it tells me what's definitely untested, not what's definitely safe, since it measures execution and not assertion. So the gate proves the tests I have all pass; it doesn't prove I've written the right tests. That's a judgement, not a number, and it's the part of the job the pipeline can't do for me." That answer moves you from "person who set up some tests" to "engineer who owns what ships" — and it survives the follow-up because every clause of it is true.
Five things worth carrying out of this part
- A defect gets no worse over time — the distance you must reach back to fix it does. Shift-left is dragging detection toward the commit, so the pipeline catches faults on the afternoon they are born instead of in production three weeks later.
- Running tests and publishing their results are separate acts. The test step fails its job on a non-zero exit code. Saving the report as an artifact makes the details available after the run, but a report artifact alone does not create a built-in Tests tab.
- Coverage measures execution, not assertion. A low number reliably flags what is untested; a high number certifies almost nothing. Read it as a floor, never a ceiling, and never make the percentage the goal.
- A quality gate is a pass/fail checkpoint before merge or deployment. Tests fail the workflow through their exit code; coverage thresholds need an explicit checker; and repository rulesets or branch protection make required checks mandatory.
- A gate survives only if it stays fast. Order cheap fast tests left, slow broad tests right, and parallelise the stage — because a slow gate gets resented, and a resented gate gets switched off, and a gate that is off protects nothing.
- The claim that a defect costs dramatically more the later it is caught is an old one in software, often cited with confident multipliers — ten times more in test, a hundred times more in production. Treat the specific numbers with suspicion; the original studies are dated and their methodology is argued about. The direction, though, is settled and matches anyone's lived experience: a fault is cheaper to fix the closer you are to having written it, and the whole value of a pipeline gate is buying the cheap catch instead of the expensive one. ↩
- This is not a reason to skip coverage — it is a reason to read it correctly. Mutation testing exists precisely to measure what coverage cannot: it changes your code deliberately and checks whether a test notices, which is closer to "are these tests any good." It is worth knowing the concept exists even if your pipeline does not run it, because it names the exact gap between a line being executed and a behaviour being verified. ↩
- A coverage upload is evidence, not enforcement. The workflow needs a command or service that reports failure below the chosen threshold, and GitHub repository rules must require the resulting check for protected branches. Keep the coverage report so reviewers can inspect the detail behind the status. ↩
Run tests in the pipeline and fail the build for the right reasons.
Play the lesson above to stamp this class.