Semantic Search

What Is Semantic Search?

Semantic search is a retrieval method that matches queries to documents by meaning instead of shared keywords, typically by converting both into vector embeddings and ranking documents by how close their vectors sit to the query's. A search for "reset my password" can surface a document titled "account credential recovery" even though the two share no words. The approach reached mainstream scale in 2019, when Google deployed BERT in Search to interpret query meaning and said the model would improve understanding of one in 10 English searches in the U.S. [1]

Key Takeaways

• Matching happens in embedding space. Query and documents become vectors, and relevance is measured by geometric similarity, usually cosine similarity between them.
• It excels exactly where keyword search fails: synonyms, paraphrases, cross-lingual matches, and questions phrased nothing like the answering text.
• It fails where keyword search excels: exact identifiers, product codes, error strings, and rare names, which is why production systems run hybrid search rather than choosing one.
• Semantic search is the retrieval engine inside most RAG systems, and increasingly a tool that agents call directly when hunting through codebases and document stores.

How It Works

The foundation is an embedding model, a neural network trained so that texts with similar meaning map to nearby points in a high-dimensional space, commonly several hundred to a few thousand dimensions. At index time, every document, or more often every chunk of every document, is run through this model and its vector embedding is stored. At query time the same model embeds the query, and the system finds the stored vectors closest to it, scoring closeness with cosine similarity or dot product. Because "credential recovery" and "password reset" occur in similar contexts across the training data, their vectors land near each other, and that proximity is what the search exploits.

Scanning millions of vectors exhaustively per query is too slow, so a vector database or search engine builds an approximate nearest neighbor index, using structures like HNSW graphs, that trades a sliver of recall for orders-of-magnitude speed. Real deployments then add layers around the core: metadata filters to scope results by tenant, date, or permissions; hybrid scoring that fuses vector similarity with a keyword algorithm like BM25; and often a reranking stage where a cross-encoder re-scores the top candidates by reading query and document together, which is more accurate than comparing precomputed vectors but too costly to run over the whole corpus. The size of that cost gap is why precomputed embeddings won: the 2019 Sentence-BERT paper by Reimers and Gurevych reported that finding the most similar pair in 10,000 sentences took about 65 hours with BERT cross-encoding versus roughly 5 seconds with embedding comparison, at similar accuracy [2].

The quality ceiling is set by the embedding model and the chunking. An embedding model poorly matched to the domain, say general web text embeddings applied to legal contracts or source code, retrieves plausible-looking but subtly wrong passages. Chunks that split concepts mid-thought embed as noise. Most semantic search tuning is really tuning these two inputs.

Example

An engineering team runs semantic search over their internal wiki of 12,000 pages for an on-call assistant. During an incident, an engineer types "payments stuck after deploy." Keyword search returns pages containing the word "payments," mostly product specs. The semantic index instead surfaces a postmortem titled "transaction queue backlog following release rollout," which never uses the words "stuck" or "deploy" but describes the same failure. The team's setup embeds each wiki section separately, filters by the team's namespace, and fuses vector scores with BM25 so that exact error codes like "ERR_TXN_5012" still match literally. That last part earned its keep the week keyword-less pure vector search kept missing pages that contained the exact error string being searched.

What People Get Wrong

The common misconception is that semantic search understands your query the way a model reading it would. It does not reason; it measures geometric proximity between compressed representations. Negation is the classic casualty: "databases that don't support transactions" embeds close to "databases that support transactions," so results can be precisely wrong. Constraints, negations, and multi-condition logic need query rewriting, filters, or a reranking model that actually reads the text. Similarity is correlation with meaning, not comprehension of it.

FAQ

What is the difference between semantic search and keyword search? Keyword search matches and scores literal terms, with algorithms like BM25 weighting how rare and frequent each term is. Semantic search compares meaning through embeddings, so wording can differ completely. Keyword wins on exact strings and identifiers, semantic wins on paraphrase and intent, and hybrid search exists because real query traffic contains both.

Is semantic search the same as vector search? Nearly. Vector search names the mechanical operation, nearest-neighbor lookup over embeddings, regardless of what the vectors represent. Semantic search names the goal, retrieval by meaning, and text embeddings plus vector search is how it is achieved in practice.

Why does my RAG system retrieve irrelevant chunks even with semantic search? Usual suspects, in order: chunking that severs context, an embedding model mismatched to the domain or to the query style, missing metadata filters letting other tenants' or eras' documents leak in, and no reranker cleaning up the top of the list. Measure retrieval quality directly with a labeled query set before blaming the generator.

Sources

1. Google, The Keyword blog. "BERT in Search improving one in 10 U.S. English queries." https://blog.google/products/search/search-language-understanding-bert/. Accessed August 2026.
2. Reimers & Gurevych. "Sentence-BERT similarity search speedup versus BERT cross-encoding." https://arxiv.org/abs/1908.10084. Accessed August 2026.