Scale AI Applied AI Engineer interview questions

Few-shot accuracy can swing on example order alone. If your prompt is fragile to things that shouldn't matter, the fix is structural, not lucky example-hunting. Here is what actually reduces variance.

CoT is not a universal upgrade. On some tasks it does nothing or hurts, and the reason tells you what to try instead. Here is the escalation ladder past 'let's think step by step'.

A model that scores 92% on your eval can still ship a product users hate, because the model is one component in a system. Teams that only do model evals get blindsided. Here is the distinction that matters.

Push PPO hard enough and true quality peaks then falls while the reward keeps climbing. The signal is knowing the Gold-vs-proxy gap, the KL budget that bounds it, and how you actually measure when to stop.

A reward model is only as good as its labels, and human preference labels are noisy and inconsistent. The signal is measuring inter-annotator agreement and the concrete moves that lift label quality.

Pairwise asks which is better, pointwise asks how good. The signal is knowing which is more reliable, how Elo/Bradley-Terry aggregates comparisons, and where each method quietly misleads.

If a benchmark leaked into pretraining, the score measures memorization, not ability. The signal is the detection methods (n-gram overlap, canaries, perturbation tests) and why fresh held-out evals matter.

A reward model turns pairwise preferences into a scalar signal RLHF can optimize. The signal is the Bradley-Terry loss, the base-model and head choices, and how you validate it before trusting it.

Everyone budgets image tokens. Almost nobody budgets image latency. Images are a prefill problem, prompt caching stops paying for itself, and the cheapest answer is often not to call a VLM at all.

A VLM that reads an invoice wrong and a VLM that reads it right and reasons wrong produce the same end-to-end score, and they need opposite fixes. Here is how to decompose a multimodal eval, build the set from real traffic, and probe for the hallucinations that accuracy never catches.

Final-answer accuracy hides how an agent got there. The signal is trajectory-level metrics: tool-selection and argument accuracy, step efficiency, and matching against reference paths, plus when exact-match is the wrong yardstick.

Agents that drive a browser or screen fail in ways chat agents never do: stale DOM, dynamic pages, wrong clicks. The signal is the grounding and reliability techniques that turn a flaky demo into something usable.

A VLM that reads one page perfectly can still fail an 80-page contract, and the reason is arithmetic before it is model quality. The signal is picking the right architecture and knowing what per-page processing silently loses. Here is the answer.

A staple ML-coding screen. Writing the loop is easy; the signal is k-means++ init, a real convergence criterion, and knowing the failure modes. Here is the tight implementation plus the follow-ups interviewers always ask.

Comparing every pair of a million documents is a trillion comparisons. MinHash estimates Jaccard similarity from a tiny signature, and LSH turns dedup into a near-linear scan. Here is the implementation.

The classic recursive-descent exercise with an applied-AI twist: the JSON your model streams is truncated mid-token for the entire generation. The signal is a clean strict parser plus a small repair layer, not a second parser. Here is the implementation.

The most job-shaped coding screen in Applied AI: fan out N LLM calls without melting the rate limit or losing the batch to one bad request. The signal is in the retry policy, not the async syntax. Here is the version that passes.

Every LLM product streams, and almost nobody can write the server. The interesting part is not the async generator, it is what happens when the user hits stop, the provider dies at token 90, or moderation flags text you have already put on the screen.

Small labeled sets are the normal starting condition, not an excuse. The strong answer is a ladder of techniques that squeeze signal from unlabeled data, pretrained models, and the labeling budget. Here it is.

Most real datasets have wrong labels, and they cap your accuracy invisibly. Cleaning all of it by hand doesn't scale. Here is how to find the bad labels and train robustly around them.

Labeled data is the fuel for ML, and a labeling platform lives or dies on quality control. The signal is the workflow plus the quality math: consensus, gold honeypots, inter-annotator agreement, and active learning to spend the budget where it counts.

Batch scoring a billion items is a throughput-and-cost problem, the opposite of low-latency serving. Every choice that hurts latency helps you here. Here is how to design for dollars-per-million-documents.

Every team eval-ing prompts in ad-hoc notebooks is how an org ships regressions and argues about vibes. A shared eval platform makes quality measurable and comparable. Here is what it has to provide.

RLHF and evals are only as good as the preference data behind them, and that data is generated by humans whose quality varies wildly. The platform that produces trustworthy labels is itself a serious system. Here is its design.

Models are only as good as their labels, and noisy annotation silently caps performance. The signal interviewers want is a measured quality process, not a louder collection effort. Here is the answer.

You cannot put a 2 TB dataset in git, and copying it per experiment bankrupts you. The signal is knowing how content-addressed versioning works and when file-level (DVC) versus branch-level (lakeFS) fits. Here is the decision.

Production usage can become your best source of training data, or a self-reinforcing trap. The signal is knowing how to capture clean feedback and how to break the loops that quietly corrupt the model.

Most AI pilots fail not on the model but on scoping: vague success criteria, the wrong use case, or data that was never ready. For forward-deployed and applied roles, running a pilot well is the job. Here is the playbook.

Scope creep quietly kills deployments and burns trust on both sides. Interviewers want to see you protect the timeline without becoming the team that says no to everything. Here is the move.

A failed prod deployment is a stress test of judgment under pressure. Interviewers want to see stabilize-first instincts, clean communication, and a root-cause fix that prevents a repeat. Here is the arc.

Executives buy outcomes and care about risk, not your architecture. Interviewers want to see you lead with the decision, quantify in their terms, and surface risk honestly. Here is the pattern.

When everything is urgent, the skill is a defensible triage and honest communication to whoever loses. Interviewers want a framework and the courage to say no clearly. Here is the move.

Vague asks are the norm in forward-deployed work. Interviewers want to see structured discovery that finds the real problem and a measurable first win, not a rush to build. Here is the method.

Almost everyone can quote softmax(QKᵀ/√d_k)V. The interviewer wants the variance argument and the exact training failure the scale prevents. Here is the answer that separates memorization from understanding.

The single most-asked applied GenAI question, and the one most candidates answer as a definition dump. The interviewer wants a decision framework with a default and the conditions that flip it. Here is the call that scores.

The modal Applied AI design round. Anyone can draw embed-retrieve-generate. The signal is in chunking, hybrid retrieval, the rerank/latency tradeoff, and how you prove it works. Here is the structure that scores.

MoE is why some frontier models have huge parameter counts but serve cheaply. The signal is the sparse-activation idea, the routing mechanism, and the operational cost (memory, load balancing) that the FLOP savings quietly hide.

These are the knobs every LLM product tunes, and the interviewer wants to know what each does to the probability distribution and which setting fits which task. Including why factual and creative tasks want opposite settings.

Agents are over-applied. The strong answer resists the hype: most tasks want a single structured call, and agents earn their cost only under specific conditions. Here is when to reach for one and how to stop it from compounding errors.

The two knobs that make or break a RAG system, and the ones candidates hand-wave. The signal is tuning chunking against recall and knowing exactly what dense retrieval misses that BM25 catches. Here is the reasoning, not the rules of thumb.

Applied AI rarely waits for certainty, and companies (Amazon's Bias for Action, NVIDIA's speed) test whether you can act decisively without freezing or being reckless. The signal is a reversible, well-reasoned call with a plan to validate. Here is the arc that lands.