calibration

Every customer asks how to stop the model making things up. The weak answer is 'better prompts.' The signal is knowing why models hallucinate and layering grounding, abstention, and measurement into a system you can defend.

LLM-as-judge is how you grade open-ended generation at scale, and G-Eval formalized it. The signal is naming the biases that make raw judge scores untrustworthy and showing exactly how you harden them.

Models are trained to be helpful, which quietly trains them to never refuse. Getting honest abstention back is a system problem, not a prompt tweak. Here is the stack that actually moves the refuse-when-unsure rate.

Push PPO hard enough and true quality peaks then falls while the reward keeps climbing. The signal is knowing the Gold-vs-proxy gap, the KL budget that bounds it, and how you actually measure when to stop.

Pairwise asks which is better, pointwise asks how good. The signal is knowing which is more reliable, how Elo/Bradley-Terry aggregates comparisons, and where each method quietly misleads.

A reward model turns pairwise preferences into a scalar signal RLHF can optimize. The signal is the Bradley-Terry loss, the base-model and head choices, and how you validate it before trusting it.

A model can rank perfectly and still emit meaningless probabilities. The signal is knowing exactly when calibration matters, how to measure it, and the post-hoc fixes. Here is the answer most candidates miss.

Two cheap regularizers that fix overconfident classifiers. The signal is knowing that one softens the target and the other softens the input, and exactly why softer signals calibrate the model. Here is the answer.

Wrap any model and get prediction sets with a provable coverage rate, no distributional assumptions required. The signal is the calibration-set plus nonconformity-score mechanism and exactly what the guarantee does and does not promise. Here is the answer.

A model can rank perfectly and still lie about its probabilities; its 0.9 may be right 70% of the time. The signal is knowing the three fixes, which data size each needs, and how to measure the gap. Here is the answer.

Defaulting to 0.5 is leaving money or safety on the table. The right cutoff comes from the cost of each error and the operating constraint, not the model. Here is how to set it deliberately.

Accuracy ties are common, and accuracy is rarely the deciding factor anyway. The real tiebreakers are the ones a junior candidate forgets to mention. Here is the checklist a staff engineer runs before shipping either one.

There is no single fairness number, and a famous result proves you cannot satisfy the main three together. The signal is defining each metric precisely and explaining the impossibility, not picking one blindly. Here is the answer.

Ads ranking is where calibrated probabilities meet hard latency and money. The signal is knowing that CTR must be calibrated (not just ranked), the feature and serving design, and the auction context. Here is the design that goes beyond 'train a classifier.'

Sales has finite rep-hours, so the model's job is to point them at the leads worth a call. The signal interviewers want is value-weighted prioritization, careful conversion labels, and proof of revenue uplift over a holdout.

This is the question to design the scoreboard, not the player. A strong answer separates offline gates (AUC, calibration, NDCG) from the online verdict (A/B with revenue, user, and advertiser guardrails), and adds counterfactual replay so you can trust a model before it ever serves a real auction.

You cannot wait for ground truth to tell you the model broke. Prediction-side monitoring catches failures in minutes, not weeks, if you know which output signals actually move first.

Quantization is the first lever for fitting and speeding up models, and the interviewer wants more than 'use fewer bits.' The signal is knowing what precision buys you, why outliers break naive quantization, and how you prove quality held. Here is that answer.

Naive 4-bit quantization can tank quality, and the reflex to give up and serve fp16 leaves a big speedup on the table. The accuracy is usually recoverable. Here is the ladder of fixes.

FP8 serving is fast and usually fine, until it silently degrades on the one workload your benchmark did not cover. The signal is knowing what FP8 breaks and how to validate it. Here is the recipe.