Classic NLP: Bag-of-Words, TF-IDF, and Word2Vec

TL;DR: Bag-of-words and TF-IDF represent a document as a sparse vector over the vocabulary: counts, weighted so common words count less and distinctive words count more. They are fast, interpretable, and meaning-blind. Word2Vec and GloVe learn dense vectors where context predicts the word, so "king" and "queen" land close even though they never share a count. The sparse methods did not die; they are the lexical leg of hybrid retrieval and the baseline you should beat before reaching for anything neural.

Sparse representations: counting words

Bag-of-words drops word order and represents a document as a vector of counts over a fixed vocabulary. "the cat sat" and "sat the cat" produce the identical vector. To recover a little order you add n-grams: treat "machine learning" as one feature so adjacent-word phrases survive, at the cost of a much larger, sparser vocabulary.

Raw counts overweight common words, so TF-IDF rescales each term by two factors. Term frequency (how often the term appears in this document) is multiplied by inverse document frequency (how rare the term is across the whole corpus). A word like "the" appears everywhere, so its IDF is near zero and it gets crushed; a word like "thrombosis" appears in few documents, so it gets boosted. The result is a sparse vector where the heavy dimensions are the document's distinctive vocabulary.

Worked example: a 10,000-document corpus, the word "neural" appears in 100 of them. IDF is log(10000 / 100) = log(100) ≈ 4.6. In a document where "neural" appears 5 times out of 500 words, TF is 0.01, so the TF-IDF weight is about 0.046. The word "data," appearing in 8,000 documents, has IDF log(10000 / 8000) ≈ 0.22, so even at the same count it barely registers. That is the whole mechanism: distinctiveness, not frequency, drives the weight.

Dense representations: learning meaning

The sparse vectors have a fatal blind spot. "car" and "automobile" are orthogonal dimensions with zero similarity, because the representation only knows exact tokens, not meaning. Word2Vec fixes this by training a small network to predict context. The CBOW variant predicts a center word from its surrounding window; the skip-gram variant predicts the surrounding words from the center one (skip-gram is stronger on rare words, CBOW is faster). After training, each word is a dense vector, typically 100 to 300 dimensions, and words used in similar contexts end up near each other. The famous property is that vector arithmetic carries relational meaning: king minus man plus woman lands near queen.

GloVe reaches a similar place from a different angle: it factorizes a global word co-occurrence matrix rather than sliding a prediction window. Both produce one fixed vector per word type, which is their shared limit: "bank" gets a single vector that blends river and money. Contextual embeddings from transformers later fixed that by giving each word occurrence its own vector.

Why this still matters

Sparse methods are not nostalgia. TF-IDF and its retrieval cousin BM25 run in milliseconds, need no GPU, are fully interpretable, and stay strong when matching is genuinely lexical: product codes, names, error strings, acronyms, exact phrases. Dense embeddings shine at paraphrase and concept matching but happily miss an exact SKU that a keyword index nails. That is why production retrieval is usually hybrid: a BM25 lexical score fused with a dense vector score, getting both exact-match precision and semantic recall. The classic methods are also the baseline that keeps you honest. A logistic regression on TF-IDF features is the number a fancier model has to beat before it earns its serving cost.

Why interviewers probe this

This screens for whether you reach for the cheapest thing that works or default to embeddings reflexively. The strong-answer move is to state the precise gap dense vectors close (synonymy and paraphrase) and the precise gap they open (exact lexical matching, plus cost and opacity), then propose hybrid retrieval rather than picking a side. The held-back follow-up is "your semantic search misses a part number a user typed verbatim, why, and what do you add?" The answer is BM25 in the mix, because the dense model embedded meaning and a part number has none.

Common misconceptions

• "Embeddings made TF-IDF obsolete." BM25-style lexical scoring still beats dense retrieval on exact-match and rare-term queries, which is why hybrid search exists.
• "Word2Vec understands context." It assigns one vector per word type, so "bank" is a single blended vector; only contextual transformer embeddings resolve word sense.
• "TF-IDF measures importance." It measures distinctiveness within a corpus; a critical word that appears in every document gets near-zero weight.
• "Bag-of-words keeps some order." It keeps none; you only recover local order by explicitly adding n-gram features.

Key takeaways

• BoW and TF-IDF are sparse, fast, interpretable, and blind to meaning and order.
• TF-IDF weights terms by distinctiveness (TF times IDF), so ubiquitous words contribute almost nothing.
• Word2Vec (CBOW / skip-gram) and GloVe learn dense vectors where similar-context words sit close, but assign one vector per word type.
• Sparse methods survive as cheap baselines and as the lexical half of hybrid retrieval.