Sentence-BERT

Base stats

Compare with

Parameters: 47
Data: 55
Compute: 55
Context: 25
Generality: 35
Transparency: 30
Total: 247

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

It is less accurate than a cross-encoder, which reads both texts together.
It inherits BERT's window: long documents must be split into chunks.
Its notion of similarity depends on the fine-tuning data and does not transfer equally to every domain.
A single similarity score does not distinguish paraphrase, entailment and mere topical affinity.

Habitat

It was born by fine-tuning BERT on a million sentence pairs from SNLI and MultiNLI, labeled by people as entailment, contradiction or neutral. Today it lives in nearly every semantic search engine, in RAG retrievers and in vector databases, usually through the sentence-transformers library.

Moves

Abilities

Siamese network

Two copies of BERT with the same weights process each sentence separately and are trained by comparing their vectors.

Mean pooling

Averaging the output vectors of all tokens yields a single fixed-size vector per sentence.

Bi-encoder

Each text is encoded once and stored; comparing it with another costs only a dot product.

Anatomy

Sentence-BERT (SBERT) is BERT fine-tuned so that a whole sentence becomes a single vector, allowing two sentences to be compared with a simple cosine similarity. Nils Reimers and Iryna Gurevych introduced it in 2019 (Reimers & Gurevych, 2019), and their library, sentence-transformers, became one of the most widely used tools for producing text embeddings (Alammar & Grootendorst, 2024, pp. 62, 293).

The problem: BERT compares well, but at a price

BERT works as a cross-encoder: to measure how similar two sentences are, you feed them in together, separated by a [SEP] token. It is accurate but does not scale: finding the most similar pair among 10,000 sentences takes about 50 million passes through the network, some 65 hours on a V100 GPU (Reimers & Gurevych, 2019, p. 3982; Alammar & Grootendorst, 2024, p. 294). The obvious alternative, passing each sentence alone and averaging its vectors or taking the [CLS] vector, yields poor sentence embeddings: on seven semantic similarity tasks both do worse than simply averaging GloVe vectors, and [CLS] is the worst of all (Reimers & Gurevych, 2019, pp. 3982, 3985).

How it works

SBERT adds a pooling layer (by default, the mean of the output vectors) and fine-tunes BERT inside a siamese network: two copies with the same weights process each sentence separately and produce vectors u and v (Reimers & Gurevych, 2019, p. 3984; Alammar & Grootendorst, 2024, pp. 294–295). During fine-tuning, a classifier receives (u, v, |u − v|) and must decide whether the pair is entailment, contradiction or neutral, using the natural language inference data of SNLI (570,000 pairs) and MultiNLI (430,000) (Reimers & Gurevych, 2019, pp. 3984–3985). One epoch is enough, under 20 minutes of compute (2019, pp. 3984–3985). The decisive component turned out to be the difference |u − v|, which pushes similar pairs together and dissimilar ones apart (2019, pp. 3988–3989). Afterwards the classifier is discarded: in use, only the vectors and the cosine remain. It is a case of contrastive learning: the model learns what makes two texts alike by seeing similar and dissimilar pairs (Alammar & Grootendorst, 2024, pp. 292–295).

What changed

The result is a bi-encoder: each text is encoded once and stored. Finding the most similar pair among 10,000 sentences drops from 65 hours to about 5 seconds, plus 0.01 seconds of cosines; with suitable indexes, searching millions of questions takes milliseconds (Reimers & Gurevych, 2019, p. 3983). On semantic similarity, SBERT also beat the best sentence embedding models of the day, InferSent and the Universal Sentence Encoder (2019, pp. 3983, 3985).

This architecture is the basis of dense semantic search: the query and the documents are projected into the same space and the nearest neighbors are returned (Alammar & Grootendorst, 2024, pp. 226–228), stored in a vector database. It is the typical retriever of Retrieval-augmented generation (RAG).

Limits

The bi-encoder is fast but less accurate than the cross-encoder, which reads both texts at once; that is why search systems often chain them: SBERT retrieves candidates and a cross-encoder reranks them (Alammar & Grootendorst, 2024, pp. 226, 244, 295). It inherits BERT's window, so long documents must be split (2024, p. 235). And the similarity it learns depends on its data: fine-tuned on NLI, it did worse than the Universal Sentence Encoder on SICK-R and on TREC question classification, domains closer to the latter's training data (Reimers & Gurevych, 2019, pp. 3985, 3988).

Professor Robledom's note

SBERT learned its geometry from inferential relations: fine-tuned to tell entailment, contradiction and neutrality apart, it ends up producing a space where closeness measures "semantic similarity" (Reimers & Gurevych, 2019, p. 3984). But entailment is asymmetric ("John runs" entails "someone runs," not the other way around), while the cosine is symmetric: cos(u, v) = cos(v, u). Something is lost in translating inferential relations into distances. For an inferentialist such as Brandom (1994), the content of a sentence is its place in a network of entailments and incompatibilities; an embedding space keeps, at most, a metric shadow of that network.

Psychology had already debated whether resemblance is a distance. Tversky (1977) showed that human similarity judgments violate the metric axioms: they are asymmetric and depend on the context of comparison. Gärdenfors (2000), by contrast, defended geometric conceptual spaces as a level of representation in its own right, between the symbolic and the connectionist. SBERT is, in a way, an industrial-scale experiment on that bet.

For discussion: what distance should there be between "the cat is on the mat" and "the cat is not on the mat," nearly identical in topic and contradictory in logic? Does "semantic similarity" name one relation or several (paraphrase, entailment, topical affinity), and which of them does the cosine measure?

Sources

Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §

Brandom, R. B. (1994). Making it explicit: Reasoning, representing, and discursive commitment. Harvard University Press. §

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423 §

Gärdenfors, P. (2000). Conceptual spaces: The geometry of thought. MIT Press. https://doi.org/10.7551/mitpress/2076.001.0001 §

Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982–3992). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1410 §

Tversky, A. (1977). Features of similarity. Psychological Review, 84(4), 327–352. https://doi.org/10.1037/0033-295X.84.4.327 §