TL;DR: Embed text queries and product images into one shared vector space (a CLIP-style dual encoder), index the catalog with ANN (HNSW), and serve query-to-item retrieval by nearest neighbors. Union with lexical search for exact attributes (brand, SKU, size), rerank for business signals (popularity, margin, in-stock), and design for catalog churn and cold start. Gate offline on labeled relevance, but trust online engagement as the real objective.
How to approach it. Clarify the query modes (text-to-image, image-to-image, text-and-image), the catalog scale, and the update rate. Then anchor on the shared embedding space, because that is the enabling idea, and lay out the retrieve-then-rank funnel with the practical concerns: hybrid lexical, freshness, cold start.
A strong answer. Shared embedding space. A CLIP-style model is trained so an image and its describing text land near each other in one vector space. That lets a text query retrieve images and the reverse, by cosine similarity, with a single nearest-neighbor lookup. Precompute one embedding per product (from its image, optionally fused with title and attributes) offline and index it.
Retrieve then rank.
- Embedding retrieval handles semantic and visual similarity: "red floral summer dress" matches images that never use those exact words.
- Lexical and attribute search handles what embeddings blur: exact brand, model number, size, price filters. Hybrid is non-negotiable in e-commerce, because shoppers mix "looks like this" with hard constraints in the same query.
- Reranking folds in business signals embeddings ignore: in-stock, popularity, margin, personalization.
Scale and operations. Catalogs are large and churn constantly (new products, price and stock changes), so design for incremental indexing: embed new items on ingest, keep volatile metadata (stock, price) in a fast store queried at rank time, and refresh embeddings periodically. Shard the ANN index for catalog size. Serve the query encoder behind a batched GPU endpoint, since image queries are heavier than text.
Cold start for new products: use the content embedding immediately, so a new item is searchable before it has any interaction data, then let engagement signals enrich its ranking over time.
Evaluation. Offline relevance (labeled query-to-product judgments, nDCG) to gate releases, plus online engagement (click-through, add-to-cart, conversion) as the truth. Offline relevance does not capture commercial intent, so a model that wins on nDCG can still lose on conversion.
Key takeaways
- One shared CLIP-style space turns cross-modal search into a single nearest-neighbor lookup, no bridging two indexes.
- Always union embedding retrieval with a lexical path, or exact brand, SKU, and size queries fail.
- Keep volatile fields (price, stock) out of the embedding and apply them at rank time from a fast store.
- Gate on offline nDCG, but treat online conversion as the real objective.
What interviewers probe next.
- "Why a shared space, not separate text and image indexes?" It enables cross-modal queries (text finds images) with one lookup instead of stitching two systems together.
- "Encode a product, image only or fused?" Often fuse the image embedding with title and attribute text for richer matching. Pure image misses textual specifics like material or brand.
- "Filtering (price, in-stock) with ANN?" Pre-filter the vector query on metadata, or post-filter with over-fetch. Keep volatile attributes in a fast store applied at rank time, not baked into the embedding.
- "How do you keep it fresh?" Stream catalog updates: re-embed on content change, update metadata continuously, compact the index periodically.
Common mistakes.
- Pure embedding search with no lexical or attribute path, failing exact brand, size, and SKU queries.
- Baking volatile fields (price, stock) into embeddings instead of filtering at rank time.
- Ignoring catalog churn and cold start, so new products are invisible.
- Judging by offline similarity only, missing that conversion is the real objective.