ElemedexA pocket encyclopedia of AI architectures
ENES

Glossary (TM) › Architectural parts

TM079

Bi-encoders and cross-encoders

Also: reranker, siamese encoder, dual encoder

In Spanish: Bicodificadores y codificadores cruzados

Two ways of comparing texts with an encoder: separately, one vector per text (bi-encoder), or reading them together to score the pair (cross-encoder).

There are two ways of using an encoder such as BERT to measure how related two texts are, be they a query and a document or two sentences. The cross-encoder receives them together, separated by a [SEP] token (a special token), so that attention crosses from one text to the other, and it returns a score; it produces no reusable embeddings (Alammar & Grootendorst, 2024, p. 294). The bi-encoder processes each text separately with the same network (two copies with shared weights, a siamese architecture) and compares them afterwards, for instance by cosine similarity (2024, p. 295). This is the solution adopted by Sentence-BERT.

The costs are asymmetric. With a cross-encoder, finding the most similar pair among 10,000 sentences takes n(n − 1)/2 = 49,995,000 passes through the network, about 65 hours on a V100 GPU; with a bi-encoder one computes 10,000 vectors (≈ 5 seconds) and then cosines (≈ 0.01 seconds) (Reimers & Gurevych, 2019, pp. 3982–3983). But the cross-encoder is more accurate: when rating the similarity of arguments on an unseen topic, SBERT scored about 7 correlation points below BERT, which can compare both sentences word by word, whereas SBERT must place each one in the space without seeing the other (2019, pp. 3986–3987).

That is why search engines chain them: a bi-encoder (or a keyword search) shortlists a hundred or a thousand candidates and a cross-encoder reorders them (reranking) (Alammar & Grootendorst, 2024, pp. 242, 244). On the multilingual MIRACL benchmark, that step raises nDCG@10 from 36.5 to 62.8 (2024, p. 243). This is the standard pipeline of semantic search and of Retrieval-augmented generation (RAG).

The difference has a philosophical flavor. The bi-encoder assumes that the meaning of each text can be fixed in advance, independently of whatever it will be compared with: a point in a space, in atomist fashion. The cross-encoder treats relevance as a relation that can only be assessed by looking at both terms at once. Can relevance (or entailment) be factored into two separate representations and a distance metric?

Sources

  • Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
  • Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach (arXiv:1907.11692). arXiv. https://doi.org/10.48550/arXiv.1907.11692 §
  • Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982–3992). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1410 §