Mistral Applied AI Engineer interview questions

Alignment is now table stakes even outside 'safety' roles. The interviewer wants the three-stage RLHF pipeline and a crisp account of why DPO dropped the reward model entirely. Here is the answer with the mechanism, not the buzzwords.

Tokenization looks like plumbing until it explains your bill, your latency, and why the model fumbles numbers and rare languages. The signal is knowing how subword tokenizers work and the consequences that fall out of them.

MoE is why some frontier models have huge parameter counts but serve cheaply. The signal is the sparse-activation idea, the routing mechanism, and the operational cost (memory, load balancing) that the FLOP savings quietly hide.

Attention is permutation-invariant, so without position information a transformer cannot tell word order. The signal is knowing why, and why the field moved from absolute sinusoidal encodings to relative ones like RoPE and ALiBi that extrapolate to longer contexts.

Perplexity is the classic LM metric. The signal is knowing it is just exponentiated cross-entropy, what it intuitively measures, and why a low perplexity does not mean a good product.

PEFT is how everyone fine-tunes large models now. The signal is the low-rank insight behind LoRA, why it cuts memory so much, and what 4-bit QLoRA adds. Here is the answer that goes past 'it's efficient fine-tuning.'

RL alignment moved from PPO to leaner methods, and DeepSeek-R1 made GRPO famous. The signal is knowing what the value/critic model does in PPO and how GRPO replaces it. Here is the mechanism, not the acronyms.

Fine-tune a model on your domain and it can forget how to do everything else. The signal is explaining why shared weights cause it and naming the concrete mitigations that all reduce to one principle.

Fine-tuning is mostly a data problem, not a hyperparameter one. The signal is naming what makes a good set (quality, diversity, format, dedup) and arguing why a few thousand clean examples beat a million noisy ones.

Context windows are not arbitrary numbers; they fall out of attention's quadratic cost, KV-cache memory, and how the model was trained. The signal is naming all three causes and matching each to the technique that extends it.

Full attention is O(n^2), so long context needs cheaper attention. The signal is naming the families (sparse, sliding-window, linear), what each gives up, and why FlashAttention is not one of them.

This is an architecture and business decision, not a religious one. The signal is weighing capability, cost, privacy, control, and operational burden, then making a call instead of declaring 'open is always better.'

Modern LLMs dropped learned absolute positions for RoPE and ALiBi to handle long context. The signal is knowing both encode RELATIVE position, and exactly how that enables length extrapolation past the trained window.

MQA and GQA shrink the KV cache, the thing that bottlenecks LLM serving. The signal is knowing they share key/value heads across query heads to cut memory and bandwidth, with GQA as the quality-preserving middle ground. Here is the answer.

DPO simplified RLHF, and a family of variants now trade off what data and reference model they need. The signal is knowing what each variant removes or changes (reference model, paired data, separate SFT stage). Here is the answer.

Fine-tuning a 65B model used to mean a node of A100s. QLoRA collapses it onto one card with three specific tricks. The signal is knowing what gets quantized, what stays trainable, and why quality barely moves. Here is the answer.

Combine several fine-tunes into one model by doing arithmetic on their weights, with no retraining and no data. The signal is knowing why it works and which method to reach for when merges interfere.

Both are conditional compute, but one scales parameters and the other scales depth. The signal is knowing which axis each routes along and why that changes the FLOP story. Here is the sharp version.

Distillation buys you most of a frontier model's quality at a fraction of the serving cost. The signal is naming the three LLM-specific variants and knowing which one works against a closed API. Here is the answer.

Alignment that adds refusals and politeness often quietly degrades reasoning and coding. The alignment tax is real and measurable. Here is how to keep the safety gains without paying for them in capability.

Neural text degeneration is a known failure of likelihood-maximizing decoding, not a broken model. The fixes are specific decode-time knobs. Here is which lever to pull and what each one costs.

A few thousand carefully chosen examples can beat a million scraped ones. The signal is knowing which filters matter, how you measure example quality, and why diversity beats raw volume.

GRPO drops the value model and estimates advantage from a group of samples per prompt. The signal is the exact mechanics, why the group baseline works, and the normalization traps that quietly bias training.

DPO is simple and stable, but it can push down the probability of chosen responses, overfit the preference margin, and amplify length bias. The signal is naming these and the variants that fix each.

Replace a learned reward model with a checker that returns right-or-wrong, and reward hacking mostly disappears. The signal is why verifiable rewards beat learned ones for math and code, and where they break.

The ratio of web, code, books, and math in pretraining quietly decides downstream skills. The signal is knowing how mixtures are chosen, why upsampling helps, and how methods like DoReMi automate it.

MQA and GQA shrink the KV cache by sharing key/value heads. MLA takes a different route: compress K and V into a low-rank latent and cache that. The signal is knowing it is a cache trick, not a head-sharing trick, and why it keeps quality.

Drop the oldest tokens to bound the KV cache and the model collapses. Keep just the first few tokens and it generates indefinitely. The signal is understanding why those first tokens act as an attention sink.

A model trained at 4K tokens can be stretched to 128K without full retraining. The signal is knowing why naive extrapolation fails and how PI, NTK-aware scaling, and YaRN each rescale RoPE frequencies differently.

Both squeeze an LLM to 4-bit weights, but they decide what to protect very differently. The signal is GPTQ's error-correcting solve versus AWQ's activation-aware scaling, and the calibration each needs.

MoE saves compute but is awkward to serve: experts must be sharded, tokens routed across devices, and batches balanced. The signal is the all-to-all communication and the load-imbalance problem, not the training story.

BPE is the tokenizer behind GPT-style models, and implementing it proves you understand how subword vocabularies are built, not just that they exist. The signal is the iterative merge of the most frequent adjacent pair.

Every chat model's 'temperature' and 'top_p' knobs are a few lines of logit surgery. Implementing them shows you understand decoding, not just call it. Here is the from-scratch version.

RoPE is why modern LLMs extrapolate to longer contexts, and it's a rotation, not an addition. Coding it shows you actually understand how position enters attention. Here is the implementation.

Taking a cloud-API POC into a bank or defense network kills your model provider, your judge API, and your vendor dashboards all at once. Learn the signed-bundle supply chain, local evaluation loop, and zero-egress observability that make an air-gapped LLM stack actually operable.

Quantization is the first lever for fitting and speeding up models, and the interviewer wants more than 'use fewer bits.' The signal is knowing what precision buys you, why outliers break naive quantization, and how you prove quality held. Here is that answer.

Distillation buys you a small fast model that keeps most of a big model's quality. The signal is explaining why soft targets transfer more than hard labels, and knowing when to distill versus quantize versus prune. Here is the model-compression answer.

At billion-parameter scale, training can be cruising and then the loss jumps and never comes back, burning a fortune in compute. The causes and the playbook are well-known to the few who've done it. Here it is.

MoE buys you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle introduce failure modes dense models never see. Here is the training playbook.

FP8 promises to roughly double training throughput over BF16, but with only a handful of mantissa bits the numerics are unforgiving. Per-tensor scaling and a selective recipe are what make it converge.

Almost everyone can quote softmax(QKᵀ/√d_k)V. The interviewer wants the variance argument and the exact training failure the scale prevents. Here is the answer that separates memorization from understanding.

The single most-asked applied GenAI question, and the one most candidates answer as a definition dump. The interviewer wants a decision framework with a default and the conditions that flip it. Here is the call that scores.

The modal Applied AI design round. Anyone can draw embed-retrieve-generate. The signal is in chunking, hybrid retrieval, the rerank/latency tradeoff, and how you prove it works. Here is the structure that scores.

These are the knobs every LLM product tunes, and the interviewer wants to know what each does to the probability distribution and which setting fits which task. Including why factual and creative tasks want opposite settings.

Agents are over-applied. The strong answer resists the hype: most tasks want a single structured call, and agents earn their cost only under specific conditions. Here is when to reach for one and how to stop it from compounding errors.

The two knobs that make or break a RAG system, and the ones candidates hand-wave. The signal is tuning chunking against recall and knowing exactly what dense retrieval misses that BM25 catches. Here is the reasoning, not the rules of thumb.

Applied AI rarely waits for certainty, and companies (Amazon's Bias for Action, NVIDIA's speed) test whether you can act decisively without freezing or being reckless. The signal is a reversible, well-reasoned call with a plan to validate. Here is the arc that lands.

Enterprise RAG fails on isolation, not retrieval quality. The signal is enforcing tenant and document-level permissions server-side, at retrieval time, so the model can never surface data a user cannot see. Here is the design that survives a security review.