evaluation

Every customer asks how to stop the model making things up. The weak answer is 'better prompts.' The signal is knowing why models hallucinate and layering grounding, abstention, and measurement into a system you can defend.

Perplexity is the classic LM metric. The signal is knowing it is just exponentiated cross-entropy, what it intuitively measures, and why a low perplexity does not mean a good product.

The classic automatic metrics for generated text. Interviewers want you to match each to its task and name the blind spot they all share. The candidates who pass say why none of them is the final word on quality.

Everyone quotes benchmark scores; few read them honestly. The signal is naming what each probes and why leaderboard numbers run ahead of real-world ability. The follow-up the interviewer is holding back is how you would detect contamination.

LLM-as-judge is how you grade open-ended generation at scale, and G-Eval formalized it. The signal is naming the biases that make raw judge scores untrustworthy and showing exactly how you harden them.

Offline metrics gate releases on a fixed dataset, online metrics show what users actually did, and offline wins routinely fail to hold. Why you need both.

'Eval is the new system design' for LLM apps, and starting from zero data is the real test. The signal is bootstrapping a golden set, picking task-specific metrics, and running evals continuously as a regression gate.

Preventing hallucinations is one job; catching the ones that slip through at runtime is another. The signal is the detection toolkit, why each signal is imperfect, and how you combine them into an action.

Hand-tuned prompts break on the next model upgrade and don't compose across pipeline stages. Frameworks like DSPy compile prompts against a metric instead. The signal is treating prompting as an optimized program. Here is the answer.

A model advertising a million-token window may not use it. The signal is knowing needle-in-a-haystack, where it breaks, and how to test real multi-fact reasoning across the full context. Here is the answer.

Alignment that adds refusals and politeness often quietly degrades reasoning and coding. The alignment tax is real and measurable. Here is how to keep the safety gains without paying for them in capability.

Optimize any proxy hard enough and the model finds the exploit, not the goal. Sycophancy, padding, and fake citations are reward hacking. Here is why it happens and the controls that actually hold.

Few-shot accuracy can swing on example order alone. If your prompt is fragile to things that shouldn't matter, the fix is structural, not lucky example-hunting. Here is what actually reduces variance.

CoT is not a universal upgrade. On some tasks it does nothing or hurts, and the reason tells you what to try instead. Here is the escalation ladder past 'let's think step by step'.

A model that scores 92% on your eval can still ship a product users hate, because the model is one component in a system. Teams that only do model evals get blindsided. Here is the distinction that matters.

For open-ended generation there's no ground-truth string to match, so accuracy is meaningless. The field uses a layered mix of automatic, model-based, and human metrics. Here is how to assemble a credible eval.

Pairwise asks which is better, pointwise asks how good. The signal is knowing which is more reliable, how Elo/Bradley-Terry aggregates comparisons, and where each method quietly misleads.

If a benchmark leaked into pretraining, the score measures memorization, not ability. The signal is the detection methods (n-gram overlap, canaries, perturbation tests) and why fresh held-out evals matter.

The same model can score ten points apart on MMLU depending on prompt format and scoring method. The signal is knowing the knobs (log-prob vs generation, few-shot, normalization) that make evals reproducible.

A VLM that reads an invoice wrong and a VLM that reads it right and reasons wrong produce the same end-to-end score, and they need opposite fixes. Here is how to decompose a multimodal eval, build the set from real traffic, and probe for the hallucinations that accuracy never catches.

Every team ships English first, then finds that the retriever, the prompt, the latency budget and the cost model each degrade differently in the next five languages. The applied playbook looks nothing like the model-level answer, and the thing that sinks launches is the eval set nobody built.

The modal Applied AI design round. Anyone can draw embed-retrieve-generate. The signal is in chunking, hybrid retrieval, the rerank/latency tradeoff, and how you prove it works. Here is the structure that scores.

Agents are over-applied. The strong answer resists the hype: most tasks want a single structured call, and agents earn their cost only under specific conditions. Here is when to reach for one and how to stop it from compounding errors.

Most RAG systems ship with no real eval, which is why most RAG systems quietly degrade. The signal is decomposing evaluation into retrieval and generation, and measuring faithfulness separately from relevance. Here is the framework that makes RAG improvements measurable.

Reflection is the pattern where an agent critiques and revises its own work. The signal is knowing when it genuinely helps (external feedback grounds the critique) versus when self-critique alone is theater.

A model that tops MTEB can still flop on your jargon-heavy corpus, and most teams jump straight to the expensive fix. There is a ladder, and the cheap rungs are the ones people skip. Here is the order to climb it.

When the right answer isn't even in the top-50, the generator can't save you. Low recall has a short list of usual suspects. Here is the order to check them so you fix the cause, not a symptom.

More chunks means higher recall but more noise, more cost, and more lost-in-the-middle. The right k is an empirical tradeoff, not a default of 5. Here is how to find it.

When a chain of LLM calls and tools produces a wrong final answer, 'the model was bad' is not a diagnosis. You need to see every step. Here is the tracing layer that turns a black box into something debuggable.

A 90% eval score and angry users at the same time means your eval set doesn't look like reality. The fix is to make evaluation track production, not the other way around. Here is how.

The most role-defining judgment call in Applied AI: a shiny benchmark number against the operational price of a system nobody can debug at 2am. Most candidates ship on the number and lose the loop there. Here is the answer that passes.

A wrong split quietly inflates every metric you'll ever report, and the bugs are subtle: leaked groups, shuffled time, preprocessing fit on everything. Here is the split done right.

A sudden 22-point AUC jump is a gift and a warning. Junior candidates celebrate; strong ones get suspicious and know exactly which leakage checks to run before anything ships.

The interviewer is testing whether you pick metrics for the problem or recite definitions. The real signal is knowing when accuracy and even AUC lie, and tying each metric to a decision.

Imbalance shows up in fraud, churn, and abuse, and the naive answer (oversample, done) leaks data and inflates offline metrics. The signal is preferring cost-sensitive learning, fixing the metric, and resampling correctly. Here is the full toolkit with tradeoffs.

A point-estimate win on a metric is not a real win. The signal is testing significance with the right paired method on a clean held-out set, weighing effect size, and confirming online. Here is how to compare models without fooling yourself.

Ranking is not classification, so accuracy is the wrong tool. The signal is matching the rank-aware metric to the task: one right answer versus many, binary versus graded relevance. Here is the answer.

Regression needs different metrics than classification, and each tells a different story. The signal is knowing what RMSE vs MAE vs R² vs MAPE actually measure and their failure modes around outliers, scale, and zeros.

A 92% aggregate accuracy can hide 60% on the segment that matters most. Average metrics are exactly where these failures hide. Here is how to surface them and the menu of fixes that actually map to the cause.

If labels only exist for the cases you already acted on, the model learns a distorted world: great offline, blind to everyone you never saw. Worse, its own decisions pick the next labels. Here is how to spot it and counter it.

The built-in importance scores from XGBoost can rank a random ID above a vital feature, and stakeholders will make decisions on that bar chart. Here is why default importance lies and what a careful answer reports instead.

Accuracy ties are common, and accuracy is rarely the deciding factor anyway. The real tiebreakers are the ones a junior candidate forgets to mention. Here is the checklist a staff engineer runs before shipping either one.

'The model got worse' has three very different causes and three different fixes. Retraining a model that's actually broken by a pipeline bug just bakes in garbage. Here is the triage order.

Most real datasets have wrong labels, and they cap your accuracy invisibly. Cleaning all of it by hand doesn't scale. Here is how to find the bad labels and train robustly around them.

For search and recommendation, A/B tests can be slow and noisy because they compare different users. Interleaving compares two rankers within the same user's results and detects winners with far less traffic. Here is how.

The modal ML system design round. The structure interviewers reward is the funnel: candidate generation then ranking then re-ranking, with the right model at each stage and an honest plan for cold start, freshness, and feedback loops. Here is that structure.

A Devin-style agent that turns an issue into a merged PR is the hardest agent to make reliable, because every step can fail and errors compound. Here is the architecture and the loop that keeps it honest.

Every team eval-ing prompts in ad-hoc notebooks is how an org ships regressions and argues about vibes. A shared eval platform makes quality measurable and comparable. Here is what it has to provide.

An AI support agent handling millions of chats will sometimes be wrong, rude, or unhelpful, and you cannot read them all. Monitoring quality at that scale is its own design problem with its own failure modes.

Separate offline gates (AUC, calibration, NDCG) from the online verdict (A/B with revenue and advertiser guardrails), plus counterfactual replay between.

Summarizing a 200-page contract is not one LLM call: it is chunking, hierarchical reduction, and a faithfulness check so you never invent facts. Learn the map-reduce pattern, when long-context beats it, and how to evaluate summaries at scale.

A writing assistant has to feel instant while a cursor moves through a live document. Learn how to split fast deterministic checks from LLM rewrites, stream suggestions, and keep edits stable so the text does not flicker as the user types.

Auto-resolving support tickets means grounding answers in your knowledge base, safely calling real APIs (refunds, account changes), and knowing when to hand off to a human. Learn the agent loop, the guardrails, and how to measure resolution without breaking trust.

Offering RAG as a product means handling messy ingestion, multi-tenant isolation, and per-customer index freshness, all behind a simple API. Learn the ingestion and query planes, how to keep tenants isolated, and how to give customers eval and observability.

Taking a cloud-API POC into a bank or defense network kills your model provider, your judge API, and your vendor dashboards all at once. Learn the signed-bundle supply chain, local evaluation loop, and zero-egress observability that make an air-gapped LLM stack actually operable.

You cannot improve an LLM app you cannot see into, and LLM observability is not service monitoring. The signal is tracing multi-step chains, capturing inputs/outputs/tokens/cost, and online quality signals, not just latency and errors.

LLMOps is not MLOps with bigger models. The signal is the genuinely new surfaces (prompts as deployable artifacts, eval-driven development, often no training step) and treating prompts with the same version discipline as code.

Editing a prompt is a code change with no compiler and no unit test by default, so quality regressions ship invisibly. Treating prompts and models as testable artifacts is what separates a toy from a product. Here is the harness.

FP8 serving is fast and usually fine, until it silently degrades on the one workload your benchmark did not cover. The signal is knowing what FP8 breaks and how to validate it. Here is the recipe.

Prompt injection has no single fix, and 'sanitize the input' fails the round. The signal is defense in depth: privilege boundaries, treating retrieved content as data not instructions, and a human gate on irreversible actions. Here is the layered answer.

Fairness questions trip up engineers who treat it as a vibe. The signal is knowing the formal fairness metrics conflict mathematically, that you must pick one deliberately for the context, and where in the pipeline bias enters. Here is the rigorous, honest answer.

A classic ML-security question that catches people who only know clean-data accuracy. The signal is understanding why models are brittle, the realistic threat model, and that no single defense holds. Here is the honest answer.

Red teaming is how you find an AI system's failures before users or attackers do. The signal is a structured program covering the real attack surface (jailbreaks, harmful content, leakage, bias) and feeding every finding back into defenses.