vision language models

VLMs like GPT-4V and LLaVA bolt a vision encoder onto a language model through a projector. The signal is the image-tokens-as-prefix design, the alignment training, and the resolution and hallucination failure modes.

Everyone budgets image tokens. Almost nobody budgets image latency. Images are a prefill problem, prompt caching stops paying for itself, and the cheapest answer is often not to call a VLM at all.

A VLM that reads an invoice wrong and a VLM that reads it right and reasons wrong produce the same end-to-end score, and they need opposite fixes. Here is how to decompose a multimodal eval, build the set from real traffic, and probe for the hallucinations that accuracy never catches.

Multimodal is now table stakes, and this checks whether you understand how images and text reach a shared model. The signal is CLIP's contrastive alignment and how modern VLMs feed image features into an LLM's token space.