Cohere Applied AI Engineer interview questions

The single most-asked applied GenAI question, and the one most candidates answer as a definition dump. The interviewer wants a decision framework with a default and the conditions that flip it. Here is the call that scores.

Alignment is now table stakes even outside 'safety' roles. The interviewer wants the three-stage RLHF pipeline and a crisp account of why DPO dropped the reward model entirely. Here is the answer with the mechanism, not the buzzwords.

A clean fundamentals question that catches people who only know GPT. The signal is matching the architecture to the task by its attention pattern, and explaining why the field consolidated on decoder-only even for non-generation tasks. Here is that answer.

Every customer asks how to stop the model making things up. The weak answer is 'better prompts.' The signal is knowing why models hallucinate and layering grounding, abstention, and measurement into a system you can defend.

Tokenization looks like plumbing until it explains your bill, your latency, and why the model fumbles numbers and rare languages. The signal is knowing how subword tokenizers work and the consequences that fall out of them.

These are the knobs every LLM product tunes, and the interviewer wants to know what each does to the probability distribution and which setting fits which task. Including why factual and creative tasks want opposite settings.

Embeddings are the substrate of modern NLP and retrieval. This checks whether you understand the distributional idea behind them, not just 'words become vectors.' The signal is how training on context produces geometry that encodes meaning.

Perplexity is the classic LM metric. The signal is knowing it is just exponentiated cross-entropy, what it intuitively measures, and why a low perplexity does not mean a good product.

PEFT is how everyone fine-tunes large models now. The signal is the low-rank insight behind LoRA, why it cuts memory so much, and what 4-bit QLoRA adds. Here is the answer that goes past 'it's efficient fine-tuning.'

The difference between a demo prompt and a production prompt is structure, versioning, and defense. The signal is treating the system prompt as a contract and templates as parameterized, testable artifacts. Here is the answer.

Complex tasks rarely fit one prompt, and free-text LLM output rarely plugs into code cleanly. The signal is decomposing into chained steps and parsing each output. Here is the answer that connects prompting to real pipelines.

Instruction tuning is what turns a raw next-token predictor into something that follows instructions. The signal is placing it correctly in the pretrain to SFT to alignment pipeline and knowing exactly what it does and does not fix.

Fine-tune a model on your domain and it can forget how to do everything else. The signal is explaining why shared weights cause it and naming the concrete mitigations that all reduce to one principle.

PEFT is a family, not just LoRA, and interviewers probe whether you know how each one injects trainable parameters. The signal is where each method adds capacity and the latency and quality tradeoffs that follow.

The classic automatic metrics for generated text. Interviewers want you to match each to its task and name the blind spot they all share. The candidates who pass say why none of them is the final word on quality.

Fine-tuning is mostly a data problem, not a hyperparameter one. The signal is naming what makes a good set (quality, diversity, format, dedup) and arguing why a few thousand clean examples beat a million noisy ones.

LLM-as-judge is how you grade open-ended generation at scale, and G-Eval formalized it. The signal is naming the biases that make raw judge scores untrustworthy and showing exactly how you harden them.

Cross-attention is how a model attends from one sequence into another, the mechanism behind encoder-decoder models and multimodal conditioning. The signal is knowing exactly where the queries vs keys/values come from.

This is an architecture and business decision, not a religious one. The signal is weighing capability, cost, privacy, control, and operational burden, then making a call instead of declaring 'open is always better.'

RLAIF and Constitutional AI swap human preference labels for AI-generated ones to scale alignment. The signal is naming exactly what they substitute for human feedback, and the consistency-versus-bias tradeoff that swap creates.

DPO simplified RLHF, and a family of variants now trade off what data and reference model they need. The signal is knowing what each variant removes or changes (reference model, paired data, separate SFT stage). Here is the answer.

How you decode from the model's probabilities shapes the output as much as the model. The signal is knowing the deterministic vs sampling methods and when each fits (factual vs creative). Here is the answer.

BERT and GPT split on one decision made before a single weight is trained: what does each token get to see? That choice cascades into attention, use case, and why one family now dominates. Here is the answer.

Multilingual models are uneven across languages, and most candidates only name the obvious cause. The differentiator is the tokenizer: the same sentence can cost a non-English user several times the tokens. Here is the answer.

A transformer emits one vector per token, but retrieval needs one vector per text. The signal is knowing the pooling options and the catch that almost everyone misses: the model has to be trained for whichever one you pick.

RAG and fine-tuning get framed as either/or. RAFT trains the model to be good at RAG itself. The tell is what goes in the training set: relevant docs mixed with distractors. Here is the answer.

Hand-tuned prompts break on the next model upgrade and don't compose across pipeline stages. Frameworks like DSPy compile prompts against a metric instead. The signal is treating prompting as an optimized program. Here is the answer.

The senior signal is matching the method to what is actually missing instead of defaulting to your favorite. Knowledge, behavior, or actions each point at a different tool. Here is the decision framework that scores highest.

One trained embedding you can truncate to any length and still use. The signal is the nested-prefix training objective and the coarse-to-fine retrieval win it unlocks at scale. Here is the answer.

Token count is latency and dollars, not just style. 'Be concise' barely works, and max_tokens just truncates mid-sentence. Here is how to actually shape length without amputating answers.

Alignment that adds refusals and politeness often quietly degrades reasoning and coding. The alignment tax is real and measurable. Here is how to keep the safety gains without paying for them in capability.

Optimize any proxy hard enough and the model finds the exploit, not the goal. Sycophancy, padding, and fake citations are reward hacking. Here is why it happens and the controls that actually hold.

Neural text degeneration is a known failure of likelihood-maximizing decoding, not a broken model. The fixes are specific decode-time knobs. Here is which lever to pull and what each one costs.

Generation has to stop somewhere, and the wrong stop rule either truncates answers or burns tokens on trailing garbage. Here is how the EOS token, stop strings, and max-token caps actually interact.

When a drug name or a Hindi word becomes nine subword tokens, you pay in cost, context, and accuracy. Adding tokens is easy and adding them well is not. Here is the tradeoff.

A few thousand carefully chosen examples can beat a million scraped ones. The signal is knowing which filters matter, how you measure example quality, and why diversity beats raw volume.

DPO is simple and stable, but it can push down the probability of chosen responses, overfit the preference margin, and amplify length bias. The signal is naming these and the variants that fix each.

A reward model is only as good as its labels, and human preference labels are noisy and inconsistent. The signal is measuring inter-annotator agreement and the concrete moves that lift label quality.

Pairwise asks which is better, pointwise asks how good. The signal is knowing which is more reliable, how Elo/Bradley-Terry aggregates comparisons, and where each method quietly misleads.

A reward model turns pairwise preferences into a scalar signal RLHF can optimize. The signal is the Bradley-Terry loss, the base-model and head choices, and how you validate it before trusting it.

RLAIF replaces expensive human labels with an LLM's preferences, which scales but inherits the labeler model's biases. The signal is how AI feedback is collected and where it quietly fails.

Modern embedding models want a task instruction prepended to the text, and getting the prefix wrong quietly wrecks retrieval. The signal is why instructions help and the asymmetric query/passage convention.

Every team ships English first, then finds that the retriever, the prompt, the latency budget and the cost model each degrade differently in the next five languages. The applied playbook looks nothing like the model-level answer, and the thing that sinks launches is the eval set nobody built.

The modal Applied AI design round. Anyone can draw embed-retrieve-generate. The signal is in chunking, hybrid retrieval, the rerank/latency tradeoff, and how you prove it works. Here is the structure that scores.

The two knobs that make or break a RAG system, and the ones candidates hand-wave. The signal is tuning chunking against recall and knowing exactly what dense retrieval misses that BM25 catches. Here is the reasoning, not the rules of thumb.

Most RAG systems ship with no real eval, which is why most RAG systems quietly degrade. The signal is decomposing evaluation into retrieval and generation, and measuring faithfulness separately from relevance. Here is the framework that makes RAG improvements measurable.

Enterprise RAG fails on isolation, not retrieval quality. The signal is enforcing tenant and document-level permissions server-side, at retrieval time, so the model can never surface data a user cannot see. Here is the design that survives a security review.

Every RAG system rests on a vector index, and the interviewer wants to know you understand the recall/latency/memory tradeoff, not just 'use a vector DB.' The signal is why exact search does not scale and how HNSW and IVF make different bets.

Retrieval quality is capped by the query, and raw user queries are often bad for search. The signal is knowing the techniques that rewrite the query before retrieval and which failure each fixes.

Vector RAG quietly fails on two query shapes: multi-hop chains and global 'what are the themes' questions. The signal is knowing exactly what the graph buys you and the construction bill it charges.

Always retrieving is wasteful and sometimes harmful; never retrieving hallucinates. Self-RAG makes retrieval a decision the model controls, then critiques what it got back. The signal is the retrieve-on-demand plus self-critique loop.

Chunking quietly sets the ceiling on RAG quality, and 'split every 500 tokens' leaves recall on the table. The signal is knowing the strategy families and the parent-child trick that decouples retrieval size from context size.

Citations turn a RAG answer from 'trust me' into something verifiable, and they are a real engineering problem. The signal is grounding each claim to a specific passage and verifying the attribution, not appending a bag of links at the end.

Real corpora contradict themselves (old vs new policy, different teams), and naive RAG silently picks one or blends them into a wrong answer. The signal is detecting conflict and resolving it by recency, authority, and transparency.

The embedding model is the foundation of retrieval, and swapping it is deceptively dangerous. The signal is choosing on domain-relevant retrieval quality (not leaderboard) and knowing that a new model means re-embedding everything. Here is the answer.

Dense vs sparse is the core retrieval tradeoff, and 'just use embeddings' silently loses every exact-match query. The signal is knowing what each representation captures and why production blends them. Here is the answer.

Garbage parsing means garbage retrieval, and naive PDF text extraction destroys tables and reading order. The signal is fixing the parser, not the retriever. Here is the answer.

Real documents carry images, charts, and tables, not just text, and text-only RAG silently drops them. The signal is knowing the two retrieval approaches and why the generator must change too. Here is the answer.

RAG and agents get expensive and slow fast: retrieval plus reranking plus big-model calls, multiplied across agent steps. The signal is naming the dominant cost first, then the levers that actually move it. Here is the playbook.

Reranking is the cheapest large win in RAG quality: retrieve broadly, then reorder precisely. The signal is the bi-encoder-versus-cross-encoder distinction and the retrieve-then-rerank two-stage design. Here is the answer.

Pure vector search misses exact terms; pure keyword search misses meaning. The signal is knowing why each fails alone and how to merge two rankings on incomparable scales without hand-tuning weights. Here is the answer.

Single-shot retrieval fails on questions that chain facts ('who directed the highest-grossing film of 2019?'). The signal is decomposing or iterating retrieval instead of retrieving once, and guarding against errors compounding across hops. Here is the answer.

A RAG system is only as current as its index, and stale or duplicated content silently degrades answers. The signal is an incremental pipeline with deletes, updates, and recency signals, not a one-time bulk load. Here is the answer.

Late interaction gets cross-encoder-like quality with near-bi-encoder scalability. The signal is the per-token embeddings plus MaxSim matching that buys you the middle ground, and knowing what it costs. Here is the answer.

Forcing every query through the same pipeline burns latency, money, and precision. A router decides what each query actually needs. The signal is naming the four decisions a router makes and how you keep a misroute from sinking the answer.

A short question and the passage that answers it live in different neighborhoods of embedding space. HyDE closes that gap by embedding a fake answer instead of the question. The signal is knowing why a wrong hypothetical still retrieves the right docs.

Small chunks match precisely but starve the LLM of context; big chunks have context but match badly. Parent-child retrieval refuses to pick. The signal is decoupling what you match on from what you feed the model.

A chunk embedded in isolation forgets which document, section, and entity it came from, so it retrieves poorly. Contextual retrieval writes that context back in before indexing. The signal is fixing retrieval at index time, not query time.

Fixed-size chunking splits a coherent idea mid-thought; semantic chunking splits on meaning. The signal is naming the methods (structure-aware, embedding-similarity, LLM-based) and the tradeoffs that decide which one to ship.

Standard RAG retrieves once and generates; agentic RAG turns retrieval into an iterative, reasoning-driven loop. The signal is the agent deciding whether, what, and when to retrieve, then re-retrieving until the evidence holds.

Standard RAG trusts whatever it retrieves, even when it's irrelevant. Corrective RAG grades retrieval quality and acts on it. The signal is the grade-then-correct loop: re-retrieve, fall back to web search, or discard.

RAG failures are usually retrieval failures, yet most teams only measure the final answer. The signal is scoring retrieval on its own (recall@k, precision@k, MRR/nDCG) to isolate exactly where the system breaks.

Late chunking flips the usual order of operations to fix the lost-context problem without a single extra LLM call. The whole signal is which step happens first. Here is the answer.

A model that tops MTEB can still flop on your jargon-heavy corpus, and most teams jump straight to the expensive fix. There is a ladder, and the cheap rungs are the ones people skip. Here is the order to climb it.

When the right answer isn't even in the top-50, the generator can't save you. Low recall has a short list of usual suspects. Here is the order to check them so you fix the cause, not a symptom.

More chunks means higher recall but more noise, more cost, and more lost-in-the-middle. The right k is an empirical tradeoff, not a default of 5. Here is how to find it.

Late interaction stores one vector per token, so a corpus that fit in a few GB as single vectors can balloon 100x. The signal is knowing the compression and indexing tricks (centroids, residuals, PLAID) that make multi-vector retrieval shippable.

A single embedding can't represent a question with three independent parts. The signal is knowing when to split a query, how to retrieve and recombine per sub-query, and the latency and drift costs that make decomposition a net loss on simple questions.

HyDE generates a fake answer to embed, which helps on vague queries but actively hurts on factual or out-of-domain ones. The signal is naming exactly when the hypothetical document misleads retrieval and which multi-draft and hybrid variants recover.

An answer can be correct and still unfaithful, stating things the retrieved context never said. The signal is scoring faithfulness against the context, not against truth, and knowing why LLM-judge faithfulness scores drift.

Cranking up top-k raises recall but drowns the answer in noise, and a tight reranker raises precision but can drop the one chunk you needed. The signal is knowing which metric caps the system and how to move both with a retrieve-wide-then-rerank shape.

Hybrid retrieval only beats either method if the fusion is tuned. The signal is knowing why you cannot just add BM25 and cosine scores, how RRF's k constant behaves, and when a learned weight beats rank fusion.

An off-the-shelf cross-encoder is general; your domain has jargon and relevance rules it never saw. The signal is knowing when fine-tuning pays off, how to mine hard negatives, and how to avoid training a reranker that just memorizes your retriever's mistakes.

A 0.87 cosine score on a chunk that answers nothing is not a bug in your vector database. It is the embedding model telling you exactly what it was trained to tell you. Here is how to read that signal instead of thresholding it away.

Beam search is the decoding algorithm everyone references and few can implement correctly. The signal is keeping k hypotheses by cumulative log-prob, summing logs (not multiplying probs), and handling completed sequences and length. Here is a correct implementation and the tradeoffs.

The core operation under every embedding/RAG retrieval, asked as a coding exercise. The signal is vectorizing the similarity, normalizing correctly, and using a partial selection (argpartition) instead of a full sort. Here is the efficient implementation and the scaling follow-up.

BPE is the tokenizer behind GPT-style models, and implementing it proves you understand how subword vocabularies are built, not just that they exist. The signal is the iterative merge of the most frequent adjacent pair.

Every chat model's 'temperature' and 'top_p' knobs are a few lines of logit surgery. Implementing them shows you understand decoding, not just call it. Here is the from-scratch version.

Before transformers, this was the language model, and it is still the cleanest way to show you understand what perplexity actually measures. Smoothing is the part people get wrong. Here is the implementation.

Everyone can recite the chunking tradeoff. Far fewer can write the splitter, keep the offsets a citation needs, and answer the question that kills the fixed splitter: what happens to a fact that straddles a boundary.

Metric learning lives or dies on the pairs you feed it. Random negatives teach almost nothing, and the margin and mining strategy decide whether the embeddings are any good. Here is how the choices interact.

As soon as a company uses LLMs in more than one place, it needs a gateway. The signal is the cross-cutting concerns (cost, reliability, observability, governance) that a gateway centralizes, not just 'proxy the API.' Here is the design.

Offering RAG as a product means handling messy ingestion, multi-tenant isolation, and per-customer index freshness, all behind a simple API. Learn the ingestion and query planes, how to keep tenants isolated, and how to give customers eval and observability.

Taking a cloud-API POC into a bank or defense network kills your model provider, your judge API, and your vendor dashboards all at once. Learn the signed-bundle supply chain, local evaluation loop, and zero-egress observability that make an air-gapped LLM stack actually operable.

Routing each request to the right model is one of the biggest LLM cost/latency levers in production. The signal is matching query difficulty to model capability and knowing when the cascade pattern beats a classifier.

Eyeballing a few outputs does not scale, and a prompt tweak that fixes one case quietly breaks ten. A real LLM eval pipeline is a versioned dataset, layered scorers, and a CI gate. Here is the architecture.

FP8 serving is fast and usually fine, until it silently degrades on the one workload your benchmark did not cover. The signal is knowing what FP8 breaks and how to validate it. Here is the recipe.

RLHF with PPO is not one model training, it is four models in the loop at once, three of them on the GPU during every step. The memory math and the generation bottleneck are what trip people up.

Almost everyone can quote softmax(QKᵀ/√d_k)V. The interviewer wants the variance argument and the exact training failure the scale prevents. Here is the answer that separates memorization from understanding.

The trap is jumping straight to 'add more GPUs.' The signal is profiling first, then applying the cheap, no-retrain levers in the right order. Here is the diagnosis-then-optimize playbook for p99.

MoE is why some frontier models have huge parameter counts but serve cheaply. The signal is the sparse-activation idea, the routing mechanism, and the operational cost (memory, load balancing) that the FLOP savings quietly hide.

Agents are over-applied. The strong answer resists the hype: most tasks want a single structured call, and agents earn their cost only under specific conditions. Here is when to reach for one and how to stop it from compounding errors.

A sudden 22-point AUC jump is a gift and a warning. Junior candidates celebrate; strong ones get suspicious and know exactly which leakage checks to run before anything ships.

At thousand-GPU scale, hardware failure is the norm, not the exception, and a run that cannot resume wastes weeks. The signal is checkpointing strategy, deterministic resume, and minimizing lost work. Here is the fault-tolerant design.

Evaluation is the hardest, most underrated part of shipping LLMs. The signal is knowing why public benchmarks mislead, why LLM-as-judge is biased, and how to build a task-specific eval you actually trust.

Offline metrics gate releases; online metrics tell you what actually happened with users. The signal is knowing why offline wins routinely fail to hold online, and how the two feed each other.