The AI Product Sense Round

“Our internal docs search works fine — it’s keyword search, people find what they need. But leadership wants to add an AI chat assistant to it because ‘everyone’s doing AI search now.’ You’re the PM. Go.”

Before I design anything — what’s actually broken about the current search, from the user’s side? “People find what they need” sounds like the feature is already working, so I want to know what problem leadership thinks this solves.

“Nothing measurable. It’s mostly a competitive-optics thing — they saw a competitor announce AI search.”

Then I’d say that directly, not just design around it: shipping an AI feature with no user problem behind it risks real cost — hallucinated answers, added latency, added maintenance — for zero measurable benefit. If there is a real gap, though, keyword search does have a known weak spot: it fails on questions phrased differently than the doc’s wording. I’d want to check support tickets or search logs for “no results” or reformulated-query patterns before assuming that’s actually happening here.

“Say the logs do show that — about 15% of searches get reformulated two or three times before someone finds the doc, or gives up.”

That’s a real, measurable problem — semantic mismatch between how people ask and how docs are written — and it’s exactly what a retrieval-augmented system is good at, more than a full open-ended chat assistant is. I’d scope this narrower than “add an AI chat assistant”: start with better retrieval — embeddings-based semantic search layered on top of keyword search — before adding a generative answer on top.

“Why not just go straight to the full chat assistant leadership actually asked for?”

Because a generative answer on top of docs introduces a new failure mode that doesn’t exist today: a wrong, confidently-stated answer, versus today’s worst case, which is just no results. I’d ship the retrieval improvement first, measure whether the reformulation rate actually drops, and only add a generated answer — with visible source citations and an easy way to jump to the original doc — once retrieval is solid enough that the generated layer isn’t compounding a search problem with a trust problem.

“How do you know if this was worth it?”

Primary metric: the reformulation/no-result rate, the number we already know is broken — it should drop meaningfully. Guardrail metric, once the generated layer ships: a spot-checked or eval-set-measured hallucination rate on real queries, because a lower reformulation rate achieved by giving people a confident wrong answer instead of no answer isn’t actually a win.

“You’re a PM on an AI image-generation product with about a million weekly active users. You have three engineers for the next quarter. How would you double usage?”

Before picking a lever — is “usage” sessions, images generated, or unique weekly users? Those point at different problems, and doubling any one of them with three engineers in a quarter means picking one lever hard, not spreading thin across several.

“Let’s say weekly active users specifically — growing the user base, not just engagement from existing ones.”

Then I’d want to know where users are dropping off today — is it acquisition (people never trying it), activation (they try it once and don’t come back), or retention (they used it for a while and stopped)? With three engineers, I can’t attack all three, so I’d want funnel data before committing.

“Activation is the weak point — about 60% of new users generate one image and never return.”

That’s a strong, specific signal to work from. My hypothesis: the first generation isn’t landing well enough to create a reason to come back — either the output quality on a first, unguided prompt is weak, or the person doesn’t know how to get a better result and assumes the product can’t do what they wanted. I’d want to see a sample of first-session prompts and outputs before picking a fix.

“Say that’s roughly right — first-time prompts tend to be vague, and the output often doesn’t match what the person actually wanted.”

With three engineers, I’d put most of the effort into a guided first-generation flow — a short, structured prompt-building step for new users instead of a blank text box, so the first result is more likely to land. I’d pair that with a lightweight “not what you wanted? try this” refinement affordance right after the first image, since a second successful attempt in the same session is a much cheaper way to save a user than hoping they come back tomorrow.

“What do you explicitly decide not to do, given only three engineers?”

I’d explicitly not touch acquisition channels or retention/re-engagement campaigns this quarter — both are real levers, but splitting three engineers across all three funnel stages means shipping nothing well. I’d also skip any model-quality fine-tuning work, since the data points at a UX and expectation-setting problem in the first session, not a raw output-quality problem.

“How do you know in a quarter whether this actually worked, versus just feeling productive?”

Primary metric: second-session return rate for new users specifically — did the guided flow and refinement affordance actually convert more first-time users into a second session. I’d also track first-session generation count as a leading indicator, since it should move faster than the return-rate metric and tell me within weeks whether I’m on the right track, not just at the end of the quarter.

“We shipped an AI assistant inside our banking app that answers questions like ‘what’s my balance’ or ‘did my paycheck deposit yet.’ A user reports the assistant confidently told them their balance was $400 higher than it actually was. What do you do?”

First: is this assistant generating the answer from a model’s general reasoning, or is it grounded in a live pull from the actual account data? That changes whether this is a hallucination in the classic sense or a data-freshness or retrieval bug.

“It’s meant to be grounded — it’s supposed to query the account balance API and report the real number. But it phrased the answer in natural language, and this time the number was wrong.”

For a financial balance, that’s about as high-stakes as this gets — a wrong number stated with full confidence can cause real harm, like someone overdrafting because they trusted the assistant. Immediate step: I’d want to know how many other users got a wrong balance in the same window, so I know if this is one bad response or a systemic issue with the retrieval or API integration.

“Turns out about 2% of balance queries in the past 24 hours returned a stale or incorrect number, tied to a caching bug.”

That’s a real incident, not a one-off. Immediate mitigation: I’d push to disable natural-language balance answers right now and fall back to a direct link to the actual account page, until the caching bug is fixed and re-verified — for something this sensitive, a slower but definitely-correct experience beats a fast but sometimes-wrong one.

“Longer term, how do you redesign this so it doesn’t happen again, or so it’s less damaging when it does?”

A few layers. First, for account-specific numeric facts like balance, I’d always show the raw number pulled directly from the API alongside — or instead of — a generated sentence, so there’s a source the user can trust independent of the model’s phrasing. Second, I’d add a confidence or freshness signal — “as of 2 minutes ago” — so staleness is visible instead of silent. Third, for anything above a set risk threshold — balance, transfers, anything with a dollar amount — I’d bias toward showing the number, not narrating it, since narration is exactly where a caching or formatting bug can silently introduce an error the raw display wouldn’t.

“What hallucination or error rate would you consider acceptable for a feature like this going forward, and how would you enforce it?”

For anything touching an actual account number, I’d target as close to zero as realistically achievable — this isn’t a case for “mostly right is fine,” because the cost of one wrong answer is asymmetric with the cost of the assistant being slightly less convenient. Concretely, I’d set an alerting threshold — even 0.1% of balance queries disagreeing with a direct API check — that pages someone automatically, rather than waiting for a user complaint to be the detection mechanism.