Text embeddings are a fundamental concept in modern natural language processing (NLP), converting text into numerical vectors that capture its meaning. This allows software to compare language based on concepts, not just keywords. This guide offers a complete technical overview, with text embeddings explained through step-by-step examples, common applications, and key limitations.

Key takeaways

  • Text embeddings convert text into numerical vectors, creating a high-dimensional "meaning space" where semantically similar text is placed closer together.
  • Modern embeddings are contextual, meaning a word's vector changes based on the sentence it's in (On the Sentence Embeddings from Pre-trained Language Models). They are created through a multi-step process of tokenization, neural network processing, and pooling.
  • Embedding models are trained to group similar texts and separate dissimilar ones. This is often achieved through contrastive training, which teaches the model what "similar" means by pushing related text vectors closer and unrelated ones farther apart.
  • Cosine similarity is a common metric for comparing embedding vectors by measuring the angle between them. However, model quality depends on the task (e.g., search, classification) and should be checked on benchmarks like MTEB (MTEB: Massive Text Embedding Benchmark).
  • Vectors from different embedding models generally cannot be compared directly. Their vector spaces are incompatible due to differences in training data, architecture, and dimensionality (MTEB: Massive Text Embedding Benchmark).

What are text embeddings?

Text embeddings convert text into numerical vectors that capture its semantic meaning. A specialized neural network processes a string of text and outputs a fixed-length list of numbers. This process maps text into a high-dimensional "meaning space" where sentences with similar meanings have vectors that are close to each other. This allows algorithms to compare and reason about text based on its content, not just keywords.

An analogy for understanding embeddings

Imagine a library where books are organized by their content, not alphabetically. All books about space travel are on one shelf, and books about gardening are in a distant corner. Text embeddings do something similar for sentences and paragraphs.

In this "meaning map," called a vector space, each piece of text is a single point. The coordinates of that point are its embedding vector—a list of numbers. The distance between points tells you how related their meanings are. Texts that are close together are semantically similar.

For example, an embedding model might place "I like cats" and "I do not like cats" very close together because most of the words are the same. The model sees statistical similarity in the words but may not grasp the complete reversal of meaning caused by "not." Embeddings approximate meaning based on patterns in their training data; they are not a source of truth or logical reasoning.

How are text embeddings generated? A step-by-step guide

Generating a high-quality sentence embedding from raw text involves several steps, from breaking the text down to processing it through a sophisticated neural network. Modern text embeddings are contextual, meaning a word's representation depends on the words around it (On the Sentence Embeddings from Pre-trained Language Models). The process typically involves a deep neural network, such as a Transformer model, which has been specifically trained to produce useful embeddings.

  1. Tokenization: The input text is broken into smaller units called tokens. These can be words, parts of words (subwords), or characters. For example, the sentence "Embeddings are useful" might become the tokens ["em", "##bed", "##dings", "are", "useful"].

  2. Contextual Representation: The tokens are fed into a neural network like BERT. The model's self-attention mechanism allows it to weigh the importance of different words when interpreting any given word in the sentence. This creates a contextual vector for each token, capturing its meaning within that specific sentence.

  3. Pooling: The model now has a list of contextual token vectors. To create a single vector for the entire sentence, a pooling strategy is applied. A common method is mean pooling, where the model averages the vectors of all tokens in the sentence. The Sentence-BERT model, for instance, adds a pooling operation to the outputs of models like BERT to produce a single, fixed-size sentence embedding (Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks).

  4. Normalization (Optional): The final vector is often normalized to a unit length of 1. This step can simplify distance calculations, making metrics like cosine similarity and dot product produce equivalent rankings.

This process works because the models are fine-tuned on massive datasets with a specific goal: to create a vector space that reflects semantic relationships. Models like Sentence-BERT use siamese and triplet network structures during training (Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks). This process, known as contrastive training, adjusts the model’s parameters to pull vectors for similar sentences (e.g., a question and its answer) closer together while pushing vectors for unrelated sentences farther apart.

A worked example: Calculating sentence similarity

To make the concept of vector space and similarity concrete, let's compare three sentences using simplified 3-dimensional embedding vectors. In practice, embedding dimensionality is a model-design choice, and vectors can have hundreds of dimensions (MTEB: Massive Text Embedding Benchmark).

  • Sentence A: "The cat sat on the mat" -> Vector A = [0.9, 0.4, 0.2]
  • Sentence B: "A feline was on the rug" -> Vector B = [0.8, 0.5, 0.3]
  • Sentence C: "The dog played in the park" -> Vector C = [-0.1, 0.2, -0.9]

We can see that the numbers in Vector A and Vector B are quite similar, while Vector C's numbers are very different. To quantify this, we use a metric called cosine similarity.

What is cosine similarity?

Cosine similarity measures the cosine of the angle between two vectors. A score of 1 means the vectors point in the same direction (highly similar), 0 means they are orthogonal (unrelated), and -1 means they point in opposite directions (dissimilar). It is widely used for comparing sentence embeddings (Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks).

The formula is the dot product of the vectors divided by the product of their magnitudes (lengths).

Similarity(A, B) = (A · B) / (||A|| * ||B||)

Calculation: A vs. B

  1. Dot Product (A · B): (0.9 * 0.8) + (0.4 * 0.5) + (0.2 * 0.3) = 0.72 + 0.20 + 0.06 = 0.98
  2. Magnitudes (||A|| and ||B||): ||A|| = √(0.9² + 0.4² + 0.2²) = √(0.81 + 0.16 + 0.04) = √1.01 ≈ 1.005 ||B|| = √(0.8² + 0.5² + 0.3²) = √(0.64 + 0.25 + 0.09) = √0.98 ≈ 0.990
  3. Cosine Similarity: Similarity(A, B) ≈ 0.98 / (1.005 * 0.990) ≈ 0.985

The score of 0.985 is very close to 1, confirming that Sentence A and Sentence B are highly similar.

Calculation: A vs. C

  1. Dot Product (A · C): (0.9 * -0.1) + (0.4 * 0.2) + (0.2 * -0.9) = -0.09 + 0.08 - 0.18 = -0.19
  2. Magnitudes (||A|| and ||C||): ||A|| ≈ 1.005 (from before) ||C|| = √((-0.1)² + 0.2² + (-0.9)²) = √(0.01 + 0.04 + 0.81) = √0.86 ≈ 0.927
  3. Cosine Similarity: Similarity(A, C) ≈ -0.19 / (1.005 * 0.927) ≈ -0.204

The negative score confirms our intuition: the sentence about the cat and the sentence about the dog are semantically dissimilar.

How do you choose a similarity metric?

Cosine similarity and dot product are two common metrics for comparing embeddings, but they answer slightly different questions about vector relationships.

  • Cosine similarity measures the angle between two vectors, ignoring their magnitude (length). It answers the question: "How similar are the directions of these two vectors?" Its values range from -1 to 1. This is useful when the direction of the vector holds the most semantic information.

  • Dot product considers both direction and magnitude. It answers the question: "How much do these vectors point in the same direction, scaled by their lengths?" Its value range is unbounded.

When embedding vectors are normalized to have a length of 1, the dot product and cosine similarity become equivalent and produce the same ranking of results (On the Sentence Embeddings from Pre-trained Language Models). While cosine similarity is common, research has shown that with some contextual embedding models, vectors occupy a narrow cone in the vector space (anisotropy), which can make cosine similarity less meaningful (Problems with Cosine as a Measure of Embedding Similarity for Contextual Embeddings).

Common applications of text embeddings

Embeddings enable a wide range of applications that rely on understanding text meaning.

  • Semantic Search: Instead of matching keywords, semantic search retrieves documents based on conceptual relevance. A user searching for "ways to improve engine performance" might find documents that mention "fuel injection tuning" or "turbocharger upgrades," even if the original search terms are absent (MTEB: Massive Text Embedding Benchmark).
  • Clustering: Documents can be grouped into thematic clusters by embedding them and then applying clustering algorithms to the resulting vectors. This is useful for topic discovery and organizing large document collections (MTEB: Massive Text Embedding Benchmark).
  • Semantic Sentence Matching: Given two sentences, embeddings can produce a similarity score indicating how closely they are related in meaning. This is a foundational task for question answering and duplicate detection (Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks).
  • Multilingual Retrieval: Using a single multilingual model, you can embed texts from different languages into a shared vector space. This allows for cross-lingual information retrieval, where a query in one language can find relevant documents in another (MTEB Multilingual Leaderboard)).
Feature Word Embeddings (e.g., Word2Vec) Sentence/Text Embeddings (e.g., Sentence-BERT)
Unit An individual word or token. An entire sentence, paragraph, or short passage.
Context Often non-contextual or limited. The word "bank" has one vector. Fully contextual. "River bank" and "investment bank" produce different vectors (Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks).
Output One vector for each word in the vocabulary. A single vector for the entire input text.
Use Case Analyzing word relationships; component in older NLP models. Semantic search, clustering, classification, retrieval-augmented generation (RAG).

Document embeddings can be a single vector for an entire document. However, for tasks like search and retrieval, this is often ineffective. A long document's meaning is too broad to be captured in one vector, and important details get diluted. The common practice is to split long documents into smaller, overlapping chunks (like paragraphs) and create a separate sentence/text embedding for each chunk.

When should you not use text embeddings?

Embeddings are a powerful tool for semantic tasks, but they are not a one-size-fits-all solution. Using them in the wrong context can lead to poor results.

  • For fact-checking or truth verification. Embeddings represent statistical similarity, not factual correctness. A model may find "The sky is green" and "The sky is blue" to be highly similar because they share structure and concepts.
  • When you need perfect keyword matching. If your application must find exact terms, like a legal clause, a product SKU, or a specific person's name, traditional keyword search (or a hybrid approach) is more reliable. Semantic similarity can sometimes match a query like "CEO" to "founder," which may not be what the user needs.
  • For tasks requiring precise logical reasoning. As noted earlier, embeddings can miss the meaning of negation. They also struggle with quantitative comparisons ("more than 500" vs. "less than 500") and other precise logical constraints.
  • Without task-specific evaluation. A model that performs well on a public leaderboard may not work well on your specific domain's data. The Massive Text Embedding Benchmark (MTEB) found that no single model is the best across all tasks like retrieval, clustering, or classification (MTEB: Massive Text Embedding Benchmark).
  • When comparing outputs from different models. Vectors from different models are not interoperable. You cannot calculate a meaningful similarity score between a vector from model A and a vector from model B, as they exist in completely different, incompatible vector spaces (MTEB: Massive Text Embedding Benchmark).

What to learn next

Sources and further reading