Incident Response / Operational System Design

“Our internal RAG assistant’s answer quality dropped hard over the weekend. Every dashboard is green — no error spike, no latency spike. A team lead noticed Monday morning because people started complaining in Slack. Walk me through it.”

First question: do we have any quality signal at all being tracked continuously — an eval score, a thumbs-up/down rate — or is “everything’s green” purely availability and latency?

“Purely availability and latency. There’s no automated quality signal in production today.”

Then that’s actually the headline finding, not a side note — this system has no way to detect this class of failure automatically, which is why it took a human noticing complaints instead of a page. Before I even look for root cause, I’d want to know: did anything change over the weekend — a deploy, a data refresh, an underlying model version, a vendor-side change?

“Nothing on our side shipped. But it’s possible the embedding model we call through an API got silently updated by the provider.”

That’s exactly the kind of failure that wouldn’t show up in our dashboards at all — a provider-side model swap can shift embedding space just enough that retrieval quality degrades without any error being thrown. I’d pull a sample of this week’s actual queries and run them back through our eval set from before the weekend, comparing retrieval relevance scores, not just spot-checking a few by hand.

“Say that confirms it — the embeddings shifted. What do you do right now, today, before you’ve even fully confirmed it?”

I’d pin the embedding model to an explicit version if the provider supports it, rather than tracking “latest,” and re-embed our corpus against that pinned version as the immediate mitigation. If pinning isn’t possible with this vendor, I’d fail over to a self-hosted or alternate embedding model we control, even if it’s a downgrade in some other dimension, since a known-quantity system beats an unpredictable one.

“How do you confirm it’s actually fixed, not just that you did something reasonable?”

Re-run the same eval set against the pinned or alternate embedding model and confirm the relevance scores are back in range before declaring it resolved — not just “this feels better,” an actual number moving back to where it was.

“And the prevention — what stops this exact thing from happening again silently?”

Two changes: pin every third-party model dependency to an explicit version instead of “latest,” and add a continuous quality signal in production — even something as lightweight as running a small canary eval set against live traffic daily and alerting on a score drop. Right now this system genuinely can’t detect its own most damaging failure mode, and that’s the actual gap, not just this one incident.

“Finance just flagged that our agent platform’s API spend was 40x normal overnight. No user complaints, no downtime. What do you do first?”

(follow-up, immediately) “And to be clear — this is happening right now, still running.”

If it’s still active, step one is stopping the bleeding, not investigating — I’d want an immediate way to identify which agent sessions or API keys are driving the spend spike, even roughly, and kill or rate-limit those specifically rather than pausing the whole platform if I can avoid it.

“You can see it’s concentrated in a handful of sessions from one customer’s workspace. What next?”

I’d hard-stop those specific sessions immediately — kill the process or revoke the session token, whichever is faster — and only then start looking at what those sessions were actually doing. My working hypothesis, given the shape of this, is a tool-call loop: something is causing the agent to call a tool, get a result, and call it again indefinitely without ever reaching a stopping condition.

“Confirmed — it’s a loop. The agent kept calling a search tool, getting an empty result, and retrying with a near-identical query each time. How does something like that ship in the first place?”

Almost certainly because there was no hard iteration cap on the agent loop, and no logic distinguishing “retry with a new approach” from “retry with essentially the same input.” The model itself has no built-in sense of “I’ve tried this already” unless the system explicitly tracks and enforces it.

“What’s the actual guardrail you’d put in, concretely?”

Two layers, not one. First, a hard cap on tool calls per agent turn and per session — a deterministic circuit breaker outside the model’s control, the same way a rules engine sits outside an agent’s reasoning for a high-stakes action. Second, a cost-anomaly monitor that compares live spend against a rolling baseline per session or per customer, and pages someone — or auto-throttles — well before it reaches 40x, not after finance notices the invoice.

“Why not just tell the model in the system prompt not to loop?”

Because a prompt is an instruction, not an enforcement mechanism — under enough conversational pressure or a strange enough input, the model can still end up looping anyway. The same principle as any guardrail: it has to live in code the model doesn’t control, not in a request you’re hoping it follows.

“What do you tell the customer, and what do you tell finance?”

Finance gets a clear timeline and the actual fix, not just an apology — a hard cap and a real-time cost-anomaly alert, so this specific failure mode is now bounded by design, not by luck. The customer gets an honest explanation and, depending on the contract, likely a credit for the anomalous usage, since it was clearly a platform bug and not something they did.

“One of our GPU regions just went unhealthy and traffic failed over automatically. Inference latency for everyone jumped 10x, including customers who were never on the affected region. On-call, go.”

First: is the failover itself still in progress, or did it already complete and we’re now stuck with degraded performance on the remaining regions? That changes whether the priority is “let the failover finish” or “the current steady state is broken.”

“Failover completed two minutes ago. This is the new steady state, and it’s bad.”

Then the working hypothesis is that the remaining regions are now oversubscribed — they absorbed the failed-over traffic without the accelerator capacity to actually serve it at normal latency. I’d confirm that by checking GPU utilization and queue depth on the healthy regions specifically, not just overall latency.

“Confirmed — the healthy regions are pegged near 100% utilization. What do you do?”

Immediate mitigation, in order of speed: first, shed non-critical load if any exists — batch or lower-priority inference traffic gets deprioritized or queued, freeing capacity for latency-sensitive requests. Second, if we have any capacity headroom elsewhere, even a slower or more expensive region, route overflow there rather than leaving everyone on the oversubscribed regions.

“There’s no spare capacity anywhere else right now. Everything’s already near max.”

Then the honest move is graceful degradation over silent failure — if there’s a way to serve a smaller or cheaper model variant for lower-priority traffic while capacity is constrained, that’s better than uniformly slow responses for everyone. I’d also want visibility into whether this is truly a capacity ceiling or whether the failed region is recoverable soon, since “wait it out” might genuinely be the fastest real fix if recovery is imminent.

“Turns out the failed region will be back in about 20 minutes. Does that change your answer?”

It changes the priority, not the actions — I’d still shed non-critical load and consider degrading gracefully for 20 minutes rather than doing nothing and hoping customers don’t notice, but I wouldn’t invest in a heavier structural fix for what’s genuinely a temporary, self-resolving window. I’d confirm the region actually recovers on schedule and traffic rebalances cleanly before declaring this over.

“What’s the actual prevention here — this can’t be the plan every time a region goes down?”

The real gap is that failover moved traffic without checking whether the destination regions actually had headroom to absorb it — that’s a capacity-aware failover problem, not just a health-check problem. I’d push for failover logic that accounts for current utilization on the target regions, not just “is it up,” plus enough standing headroom margin across regions that a single-region failure doesn’t automatically oversubscribe whatever’s left.