The Behavioral Round

“Tell me about a time you caused, or were closely involved in, a production incident. What happened?”

I pushed a config change to raise the batch size on an inference service, expecting better throughput. Instead it pushed memory usage over the limit under peak load and the service started getting OOM-killed intermittently for about forty minutes before I connected the change to the symptom.

“Walk me through what you did once you realized it was your change.”

Rolled it back immediately, confirmed the OOM kills stopped, and then wrote the postmortem myself rather than waiting to be asked. I named the actual gap plainly — I hadn’t load-tested the new batch size against peak, not average, traffic — and the follow-up was adding a load test at realistic peak volume to the deploy checklist, not just “be more careful next time.”

“Tell me about a time you had to work with a research or ML engineer who didn’t think about reliability the way you did.”

A research engineer wanted to ship a new prompt-chaining approach straight to production because it scored better on their offline eval. I pointed out it made three sequential model calls per request instead of one, which would roughly triple our latency and API cost at current traffic — numbers they hadn’t actually looked at, since their eval only measured answer quality.

“How did you get from disagreement to an actual resolution?”

I didn’t frame it as “no” — I ran the latency and cost numbers myself and brought them back with a proposal: ship it behind a flag to 5% of traffic first, watch the actual production latency and cost, and use that data to decide on wider rollout instead of arguing from intuition on either side. It shipped two weeks later, to everyone, once the numbers came back acceptable.

“Tell me about a time you were paged for something and had to decide, in the moment, whether it was actually serious enough to wake someone else up.”

I got paged for elevated error rates on an agent service around 2 a.m. My first move was checking whether the error rate was actually affecting real user traffic or just a batch job that runs overnight and is expected to have some retryable failures — it turned out to be the batch job, already self-recovering.

“So you didn’t escalate. What would have made you escalate instead?”

If the errors had been on the customer-facing path, or if the batch job’s self-recovery hadn’t kicked in within its normal window, I’d have paged the service owner without hesitating, even at that hour — the cost of waking someone up unnecessarily is a bad night’s sleep, the cost of not escalating a real customer-facing issue is much higher, and that asymmetry is what I default to when I’m unsure.

“Tell me about a time you pushed back on shipping something because it would burn too much of your error budget, even though the team wanted to ship.”

Product wanted to launch a new agent feature during a week our latency SLO was already close to breach from an unrelated infra migration. I flagged that shipping on top of an already-strained budget meant we’d likely breach the SLO within days, and that our own error-budget policy said a breach should pause non-critical launches until we recovered.

“Product pushed back — how did you handle that?”

I didn’t just cite the policy and stop there — I proposed a one-week delay tied to a specific, measurable recovery condition, rather than an open-ended “not now.” That gave product a concrete date instead of an indefinite blocker, and made it clear the policy wasn’t me being cautious for its own sake, it was the thing the whole org had already agreed to.