Private endpoints — the Class 14 pattern, reapplied
Class Thirty-Four left one door open on purpose: the Azure OpenAI endpoint is, by default, a public address — identity guards who gets an answer, but the door itself stands on the open internet, taking knocks from every scanner on earth. You closed this exact door in Class Fourteen, for storage and databases, and the moves are identical because the mechanism is: a private endpoint gives the resource a private IP inside your VNet; a private DNS zone (privatelink.openai.azure.com) makes the resource's own hostname resolve to that private IP from inside the network; and the resource's public network access setting turns the public door off entirely. Nothing about the application changes — same URL, same managed identity — but the route now runs through your network, and the public internet gets a closed door instead of a login prompt.
Why bother, when §4 of the last class already required identity? For the same reason Class Thirty-One kept both the vault and the VNet: layers fail independently. Identity protects against the unauthorised; the private path protects against the day identity is misconfigured — and it converts the compliance conversation from an argument into a diagram. When the auditor asks where customer questions about hiking boots travel, "over our network, to a private address, in the region we chose" is an answer with a drawing. The one operational cost is the Class Fifteen lesson returning: when a private-endpoint call fails, check DNS first — a client resolving the public name from the wrong network is the whole outage, every time.
APIM in front — one valve for every caller
Month two's real problem is not the network; it is arithmetic. Success multiplied the callers — the storefront bot, then the internal product-search tool, then somebody's analytics script — and every caller talks to the same deployment, drawing from the same TPM, running the same meter, indistinguishably. When the bill triples, "who tripled it" has no answer, and when one caller misbehaves, the throttling lands on everyone. The fix is a pattern older than AI: put a gateway in front. Azure API Management sits between every caller and the deployment, and becomes the single place where callers are named, limited, and counted.1
- AI gateway
- A managed gateway in front of the model endpoint where each consumer holds its own credential, its own token allowance, and its own line in the logs — the place "who is spending this" gets an answer before the invoice asks it.
The mechanics stack three things you know. Each consuming team gets an APIM subscription key — their own credential, revocable per team, which is Class Nine's "one identity per consumer" applied to consumers you cannot give managed identities to. Policies on the gateway enforce token-based limits per key — APIM's AI-gateway policies understand tokens natively, so the storefront can hold 25K TPM while the analytics script holds 2K and hits its own ceiling without touching production.2 And the backend leg uses APIM's managed identity to call Azure OpenAI — so the teams hold keys to the valve, never to the model, and §1's private path stays the only road in. The same policies can emit per-key token counts to your telemetry, which is §4's raw material, and cache semantically similar answers, which is §3's cheapest lever.
Token economics — how pennies compound
Token prices look like rounding errors — fractions of a cent per thousand tokens — and that is precisely what makes them dangerous: nobody budgets a rounding error. The bill arrives at scale through three multipliers, and you should be able to recite them. First, output costs more than input — typically several times more per token — so verbose answers, not long questions, dominate spend, and "answer in two sentences" in a system prompt is a cost control.3 Second, the model is a multiplier all by itself: frontier and small models differ in price not by percentages but by an order of magnitude or more, so routing traffic that a small model can handle to a large one burns money on every single call. Third — the one that catches everyone — chat history compounds. The model has no memory; every turn resends the entire conversation, so turn ten carries nine turns of baggage plus the system prompt, and the tokens of a session grow roughly with the square of its length. A user who chats twice as long costs four times as much.
Nobody budgets a rounding error.
The levers, in the order to pull them:
| Lever | What it does | What it costs you |
|---|---|---|
| Cap the output | Bound answer length in the request and instruct brevity in the system prompt; output is the expensive direction. | An hour. Some answers get terser. |
| Truncate history | Send the last few turns, or a running summary, instead of the whole session — breaks the quadratic. | A day of engineering. Very long chats lose distant context. |
| Cache repeats | The same forty questions about hiking boots get asked hourly; answer repeats from cache — APIM can do this semantically — and the model never runs. | Configuration plus a staleness policy someone must own. |
| Right-size the model | Route routine questions to a small model, escalate the hard ones; the price gap is roughly thirty-fold. | An evaluation someone must run — quality is now a routing decision. |
| Provisioned throughput | For high, steady volume: buy dedicated capacity by the hour instead of paying per token. | Class 32's clause returns — use-it-or-lose-it. Commit to the measured floor only. |
Watching it — the Class 28–29 toolkit, pointed at tokens
Everything in §3 is guesswork until you can see token flow, and the seeing is machinery you already own. Diagnostic settings on the Azure OpenAI resource flow request logs and metrics to the Class Twenty-Eight workspace — nothing flows until you say so, as ever — and the platform metrics already count processed tokens per deployment. The APIM valve adds the dimension Azure cannot know: which consumer, because its AI-gateway policies emit token counts per subscription key. Between the two, the Class Twenty-Nine drill applies verbatim — summarize sum(tokens) by consumer, bin(TimeGenerated, 1h) is the same shape as every query you wrote there, pointed at a new table.
Then wire the shouts, Class Thirty's rules unchanged: one alert on token throughput anomalies (a consumer at ten times its baseline is either a launch or a leak — both are worth a page), one Class Thirty-Two budget on the resource's cost with a forecasted threshold, each with a named owner and a runbook line. And one dashboard, deliberately small: tokens by consumer, cost by day, 429 rate, cache hit rate. Four tiles answer every question month three will ask — including the question §5 is about to raise, because filtered-request counts land in the same logs.
Content filtering — the infrastructure view
Azure runs a content filter on every deployment — prompts and completions both, screened for categories like hate, violence, sexual content, and self-harm, plus attack patterns like prompt injection — and it is on by default, at default severity thresholds, whether you configure it or not. The infrastructure view, which is your view: the filter is configuration attached to the deployment, not behaviour inside the model. You can build a custom filter policy, tune severity thresholds per category and per direction, and attach it to specific deployments — loosening some thresholds requires an approved use case with Microsoft, which is itself worth knowing: the default posture is deliberately conservative.4
What you own operationally is the filter's failure modes, and they surface as tickets, not incidents. A filtered request returns an error with the filter's annotations — to the application it can look like the model "refusing," and to support it looks like the bot being broken, which is why the application must handle the filtered response as a first-class case and why the annotation belongs in your logs. The classic false positive is domain vocabulary: a retailer of hiking gear sells knives and bear spray, and a customer asking which knife to pack is not a safety incident. When the tickets arrive, the fix is not "turn the filter off" — it is the same discipline as every threshold in this bootcamp: measure the false-positive rate from the §4 logs, adjust the specific category's threshold deliberately, document why, and keep the audit trail. The filter is a control like any other. Controls get tuned; they do not get disabled in a hurry by whoever is on call.
Month two: the bill triples, and the valve goes in
The chatbot works — that is the problem. Month one closes at $410 in tokens; month two at $1,290, and climbing. The Class Thirty-Two budget alert catches it mid-month (the machinery paying for itself), and the §4 telemetry — hastily enabled that afternoon — tells the story in three lines. The storefront bot's sessions run long: customers chat, history compounds, and the average session's token count has doubled since launch. The product team quietly pointed their new search-suggestion feature at the same deployment, unthrottled. And one enthusiastic analyst has a script asking the model to summarise every product review, one review per call, around the clock — nobody's fault; there was nothing to tell anyone no.
The fix is this class in order. APIM goes in front: storefront at 25K TPM, product search at 8K with its own key, the analyst's script at 2K — where it promptly hits its ceiling, and the conversation that follows is about batching, not blame. History truncation caps sessions at the last six turns; answers cap at 300 tokens; the forty perennial boot questions start answering from semantic cache, which alone erases a fifth of the load. Review summarisation moves to the small model — nobody can tell the difference, which is the point. A private endpoint closes the public door while the toolbox is open, per Class Fourteen. Month three closes at $520 — with traffic up 40 percent — and the dashboard's four tiles go in the monthly cost report beside Class Thirty-Two's. The CFO's comment, forwarded without further remark: "why don't the other systems have this."
The official pages, and a CAMPUX overview
AI gateway capabilities in Azure API Management
learn.microsoft.com/azure/api-management/genai-gateway-capabilities
Azure OpenAI content filtering
learn.microsoft.com/azure/foundry-classic/foundry-models/concepts/content-filter
The valve going in — an APIM token-limit policy throttling one key while another sails through — and a filtered request read from the logs will live here. Video to be added.
Compute the compounding by hand, once
The §3 quadratic is the kind of claim engineers nod at and never feel. Feel it once, with a pencil, using round numbers: a 60-token system prompt, 40-token questions, 150-token answers.
Price turn one. Input: system prompt (60) + question (40) = 100 tokens. Output: 150 tokens.
Running total: 100 in, 150 out. Trivial, as advertised.Price turn five. Input: system prompt + all four earlier questions and answers + the new question = 60 + 4×(40+150) + 40 = 860 tokens in; 150 out.
What to notice: the new question is 40 tokens; the baggage is 820. Over 95% of turn five's input is history — billed again, this turn and every turn after.Sum a 20-turn session (a spreadsheet helps): total input ≈ 38,000 tokens against just 800 tokens of actual questions. Now cut history to the last 6 turns and re-sum: input falls to about 21,000 — close to half gone.
What to notice: nothing about the model, the traffic, or the users changed — only what you chose to resend. That is an infrastructure decision, which is why it is your decision.Last step: multiply both totals by a real price. Look up today's per-token price for a small and a frontier model and price the same session on each.
The lesson: the session's cost moves by an order of magnitude on the model choice alone, and by nearly half on the history choice. Those two numbers, remembered, are the entire §3 argument — and a very good interview answer to "how do you control AI costs."
The same discipline as any other service
Two questions land in the same week: "why is the AI service so expensive?" and "is our data leaving the network?" You answer both — a private endpoint keeps traffic off the public internet, and a gateway meters tokens per caller so spend is visible and capped. The AI workload gets the same network and cost discipline as any other service, because that is what it is.
Examination
Four drills, then two situations. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored; this is between you and the page.
B — the whole design goal of the pattern is that the application cannot tell. The private DNS zone answers for the resource's own hostname, returning the private IP to anyone asking from inside the network; the code, the URL, and the managed identity are untouched. That is what makes the change safe to roll out — and it is also the trap when it breaks: A is wrong but describes what people assume, and that assumption sends them editing application config during an outage when the actual failure is Class 15's, every time — a client resolving the public name from a network whose DNS never heard of the private zone. C invents a rule (auth and routing are separate layers), and D confuses a private endpoint with subnet placement; same VNet, or a peered one, is enough. The interview form: "what breaks first when a private endpoint rolls out?" — answer: "DNS, from whoever wasn't consulted."
C — keys to the valve, never to the model. The split is what makes every §2 promise real: because teams hold gateway credentials, a leaked key is revoked per team without touching anyone else; because limits attach to those keys, one runaway script throttles alone; because APIM's managed identity is the only principal on the backend, the §1 private path stays the single road in. A recreates Class 34's anonymous-key problem with more steps — two shared keys cannot name multiple consumers, and revoking one takes down everybody holding it. B is the audit anti-pattern this bootcamp has spent ten classes dismantling. D describes a gateway that governs nothing — if consumers can reach the model directly, the valve is decoration, and the first bypass proves it. The design test to carry: could a consumer skip the gateway? If yes, you built a suggestion, not a control.
Break the quadratic, skip the repeats, right-size the answering. History truncation attacks the biggest multiplier; caching removes whole calls; model routing and output caps shrink what each remaining call costs. The rejects fail differently and both are instructive. The filter reject is doubly wrong: screening is not billed as your tokens, and trading a safety control for pennies is the kind of decision that reads catastrophically in the postmortem where it is eventually discovered. Lowering TPM confuses the ceiling with the meter — TPM caps the rate, so cutting it converts spend into 429s at peak: same users, worse product, and the bill mostly intact because the tokens just queue into the next minute. Every real lever reduces tokens per interaction; the fake ones reduce quality or availability and leave the arithmetic standing.
# design review: AI gateway rollout
1. All consumers call APIM; each team holds its own
subscription key with a per-key token limit.
2. APIM authenticates to Azure OpenAI with its
system-assigned managed identity.
3. To keep a fallback path during migration, the
Azure OpenAI resource keeps public network access
enabled and local key auth on, indefinitely.
4. APIM emits per-key token metrics to the Class 28
workspace; dashboard tiles per consumer.
Line three — the word doing the damage is "indefinitely." A bypass during a dated migration window is engineering; a bypass with no end date is the architecture. Run the tape six months out: every limit, every per-team key, every dashboard tile on lines one, two, and four governs only the traffic that chooses to be governed — and the first team that hits its token limit discovers the ungoverned door in an afternoon, because the public endpoint still answers and key auth still works. From then on the gateway's numbers are fiction: the bill diverges from the dashboard, and the divergence is the bypass traffic. Drill 02's test failed in writing: a consumer could skip the valve, so it is a suggestion.
The fix is one sentence in the review: the fallback gets an expiry date and an owner, and on that date public access closes and local auth flips off — Class 34's switch, this time with the gateway ready to catch everything. The distractors: per-key limits are the design's entire point (A — a shared pool recreates the analyst-script problem), APIM holds managed identities fine (B), and per-consumer metering is cost attribution, not surveillance (D — Class 32 called it accountability). In reviews, hunt the temporary things for missing end dates; "for now" is how permanent architecture gets approved without a diagram.
The trap is treating it as one problem. "The bill tripled" is an aggregate, and aggregates do not have fixes — components do. So the first day is §4, not §3: diagnostic settings on if they are not already, APIM's per-key metrics if the valve exists, and the Class 29 group-and-filter drill against token counts — by consumer, by hour, by session length. Campux's own triple decomposed into three unrelated stories (longer chats, a new unannounced consumer, a well-meaning script), each with a different fix, and throttling blindly on day one would have degraded the storefront to punish the analyst. You cannot know which story yours is without the telemetry, and "we spent Tuesday finding out which" is a sentence leadership accepts when it comes with a chart.
Then pull the invisible levers before the visible ones. Friday is achievable because the biggest levers do not touch what users see: history truncation breaks the quadratic (the desk labs near-halving), semantic caching erases the recurring questions entirely, output caps trim the expensive direction, and unauthorised consumers move behind their own keys and limits. Model routing is next — cheaper by an order of magnitude where quality genuinely holds — but it ships behind an evaluation, not a deadline, because "not visibly worse" is a promise about quality and routing is a quality decision. Say so plainly: four levers by Friday, the fifth the following week with evidence.
Close by making month four boring. The fix is not the numbers coming down; it is the system that notices next time: per-consumer limits so the next surprise script throttles itself politely, the four-tile dashboard in the monthly report, the budget's forecasted alert owned by a name. Leadership asked for a fire put out; deliver the smoke detector with it, and price the whole engagement in Class 1's terms — the telemetry and the valve cost days; the alternative was discovering month three's bill the way you discovered month two's.
Split the ticket before you answer it. Tonight's problem is customer experience: people asking legitimate product questions hit a raw error. That is fixed in the application, without touching the filter at all — handle the filtered response as a first-class case ("I can't help with that one — here's our gear guide, or talk to a human") and route the customer somewhere useful. Ugly errors are an app bug wearing a safety control's name tag. This month's problem is filter fit: the retail domain includes knives, bear spray, and survival gear, and the default thresholds — deliberately conservative, and right to be — were tuned for no domain in particular. Different problem, different clock, different fix.
Then say no to the disable, with the tape run forward. The filter that false-positives on camping knives is the same filter that catches prompt-injection attempts and genuinely harmful requests — and a public-facing bot with no filter is a screenshot generator: the first user who makes it say something vile posts it with Campux's logo in frame, and that incident costs more than every knife sale the error message ever lost. The on-call instinct — "we're retail, not social" — misreads who the filter protects; it is not moderating a community, it is bounding what the company's own voice can be made to say. §5's rule stands: controls get tuned, not disabled in a hurry by whoever is on call — and "in a hurry, by whoever is on call" is precisely how it would happen at 4pm on a Thursday.
The tune, done properly, is a week of unglamorous work. Pull the filter annotations from the §4 logs and measure the false-positive rate by category — it will be concentrated in one or two, at the mild end. Adjust those thresholds specifically, through the custom-filter process, with the approval trail Microsoft requires and a document saying what changed and why. Re-measure, keep both numbers — false positives down, true catches intact — and put them in the monthly report. The closing sentence for the on-call channel: the customer gets an answer tonight, the filter gets a tuning this month, and nobody gets a screenshot.
Five things worth carrying out of this class
- The private endpoint is Class 14 unchanged: private IP, private DNS zone, public access off — the application cannot tell, and when it breaks, it broke in DNS.
- APIM is the valve: per-team subscription keys with token limits on the front, one managed identity on the back, and the design test is "could a consumer skip it?" A gateway with a bypass is a suggestion.
- Three multipliers make token bills: output costs more than input, models differ by an order of magnitude, and resent chat history grows with the square of session length. Nobody budgets a rounding error.
- Pull levers in cost order — cap output, truncate history, cache repeats, then route models behind an evaluation — and watch it all with the 28–29 toolkit: tokens by consumer, cost by day, 429s, cache hits.
- The content filter is deployment configuration, on by default and deliberately conservative: handle filtered responses in the app, tune thresholds from measured false positives with the audit trail — never disable in a hurry.
- APIM is not the only shape of this valve — gateway offerings, both Azure-native and third-party, are multiplying quickly in exactly this niche. The pattern is the durable part: a governed intermediary where consumers are named, limited, and counted, holding credentials to the gateway rather than the model. Learn it as a pattern and the products become interchangeable details. ↩
- The AI-gateway policy names (token limits, token metrics, semantic caching) have already been renamed once as the features matured from preview, and their tier availability shifts. Check current policy names and your APIM tier's support before writing the design document — the capability set is stable; the spelling is not. ↩
- Treat every specific ratio in this section with suspicion and the directions as settled: output has consistently priced above input, small models have consistently priced an order of magnitude or more below frontier ones, and prices overall have fallen for years — which cuts both ways, since falling unit prices are how organisations talk themselves into unmetered usage. The desk lab makes you look up today's numbers precisely because printing them here would date the page by spring. ↩
- Filter categories, severity levels, the annotation format, and which threshold changes require an approved use case all evolve as the product line does — and additional protections (prompt-injection shields, groundedness checks) keep arriving alongside the core categories. The infrastructure posture outlives the feature list: filtering is deployment-attached configuration, conservative by default, tunable with process and an audit trail. ↩