Harvey Applied AI Engineer interview questions

Models are trained to be helpful, which quietly trains them to never refuse. Getting honest abstention back is a system problem, not a prompt tweak. Here is the stack that actually moves the refuse-when-unsure rate.

When a drug name or a Hindi word becomes nine subword tokens, you pay in cost, context, and accuracy. Adding tokens is easy and adding them well is not. Here is the tradeoff.

A model that tops MTEB can still flop on your jargon-heavy corpus, and most teams jump straight to the expensive fix. There is a ladder, and the cheap rungs are the ones people skip. Here is the order to climb it.

A 90% eval score and angry users at the same time means your eval set doesn't look like reality. The fix is to make evaluation track production, not the other way around. Here is how.

An LLM can cite a source that does not say what the answer claims. The signal is checking claim-by-claim entailment against the cited text, not just that a citation marker exists, and knowing what to do when grounding fails.

Legal review is high-stakes and precision-critical: a missed clause or hallucinated citation has real consequences, and lawyers have been sanctioned for fabricated cites. The signal is grounding in the actual documents, citing exact passages, and lawyer-in-the-loop. Here is the design.

Editing a prompt is a code change with no compiler and no unit test by default, so quality regressions ship invisibly. Treating prompts and models as testable artifacts is what separates a toy from a product. Here is the harness.

An enterprise RAG that retrieves across everyone's documents is a breach waiting to happen. The fix lives in the retrieval layer, not the prompt. Here is the design that survives a security review.

Almost everyone can quote softmax(QKᵀ/√d_k)V. The interviewer wants the variance argument and the exact training failure the scale prevents. Here is the answer that separates memorization from understanding.

The single most-asked applied GenAI question, and the one most candidates answer as a definition dump. The interviewer wants a decision framework with a default and the conditions that flip it. Here is the call that scores.

The modal Applied AI design round. Anyone can draw embed-retrieve-generate. The signal is in chunking, hybrid retrieval, the rerank/latency tradeoff, and how you prove it works. Here is the structure that scores.

A favorite practical screen at the labs, because it tests concurrency, time handling, and judgment in 20 lines. The trap is the background thread that wastes CPU. Here is the lazy-refill version interviewers want, plus the follow-ups.

MoE is why some frontier models have huge parameter counts but serve cheaply. The signal is the sparse-activation idea, the routing mechanism, and the operational cost (memory, load balancing) that the FLOP savings quietly hide.

These are the knobs every LLM product tunes, and the interviewer wants to know what each does to the probability distribution and which setting fits which task. Including why factual and creative tasks want opposite settings.

Agents are over-applied. The strong answer resists the hype: most tasks want a single structured call, and agents earn their cost only under specific conditions. Here is when to reach for one and how to stop it from compounding errors.

The two knobs that make or break a RAG system, and the ones candidates hand-wave. The signal is tuning chunking against recall and knowing exactly what dense retrieval misses that BM25 catches. Here is the reasoning, not the rules of thumb.

A classic that tests whether you reach past sorting for the right structure. The signal is the heap solution (O(n log k)), the bucket-sort O(n) trick, and how it changes for an unbounded stream. Here is the answer.

Applied AI rarely waits for certainty, and companies (Amazon's Bias for Action, NVIDIA's speed) test whether you can act decisively without freezing or being reckless. The signal is a reversible, well-reasoned call with a plan to validate. Here is the arc that lands.

Alignment is now table stakes even outside 'safety' roles. The interviewer wants the three-stage RLHF pipeline and a crisp account of why DPO dropped the reward model entirely. Here is the answer with the mechanism, not the buzzwords.

Tokenization looks like plumbing until it explains your bill, your latency, and why the model fumbles numbers and rare languages. The signal is knowing how subword tokenizers work and the consequences that fall out of them.

Enterprise RAG fails on isolation, not retrieval quality. The signal is enforcing tenant and document-level permissions server-side, at retrieval time, so the model can never surface data a user cannot see. Here is the design that survives a security review.

The canonical grid DP where each cell depends on three neighbors at once. The signal interviewers want is the min-of-three recurrence, and the reason it has to be min. Here is the answer.

Applied AI is half translation. Interviewers ask this to see whether you can drop the jargon, frame tradeoffs in business terms, and earn trust with people who control the roadmap. Here is how to show that skill in the answer itself.

Evaluation is the hardest, most underrated part of shipping LLMs. The signal is knowing why public benchmarks mislead, why LLM-as-judge is biased, and how to build a task-specific eval you actually trust.