An automatic metric that scores a translation by how many word sequences it shares with one or more human reference translations; for two decades, the yardstick of machine translation.
Papineni et al. (2002), at IBM, presented BLEU as an understudy to human
judges of machine translation: a translation is better the closer it is to a professional one.
How it is computed. One counts the candidate's n-grams (sequences of 1 to 4 words) that
appear in the references, with a cap: each counts at most as many times as it occurs in a
reference. Without this clipping, the the the the the the the would have perfect precision
against The cat is on the mat; with it, 2/7. The geometric mean of the four precisions pₙ
is multiplied by a brevity penalty, because leaving things out also avoids errors:
BLEU = BP · exp(Σ ¼ · log pₙ). Not even a human translator reaches 1: in the original paper,
one scored 0.35 against four references. This was the yardstick for the Transformer:
28.4 points (out of 100) on English–German (Vaswani et al., 2017, sec. 6.1).
Limits. BLEU looks at form, not meaning (form vs meaning): a correct paraphrase in other
words loses points. Callison-Burch et al. (2006) showed that one sentence admitted
at least 40,320 permutations with a similar score, and that BLEU underrated a rule-based
system that judges rated highly. When Google optimized BLEU with reinforcement learning,
the score rose from 40.35 to 41.16 but human evaluation did not improve (4.46 versus 4.44)
(Wu et al., 2016, sec. 8.6): Goodhart's law in miniature. Reviewing 284 correlations, Reiter (2018)
concludes that BLEU is fit for comparing translation systems, not for judging individual
texts or other tasks. And it misses coherence, fluency and correctness
(Alammar & Grootendorst, 2024, p. 374).
In Wittgenstein's vocabulary, BLEU would be a symptom of good translation, not a
criterion: it correlates with it as the barometer does with rain (Wittgenstein, 1953/2009, §354).
Why does a surface symptom work well when averaged over thousands of sentences and poorly
on each one?
Sources
- Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
- Callison-Burch, C., Osborne, M., & Koehn, P. (2006). Re-evaluating the role of BLEU in machine translation research. In D. McCarthy & S. Wintner (Eds.), 11th Conference of the European Chapter of the Association for Computational Linguistics (pp. 249–256). Association for Computational Linguistics. https://aclanthology.org/E06-1032/ §
- Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In P. Isabelle, E. Charniak, & D. Lin (Eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073135 §
- Reiter, E. (2018). A structured review of the validity of BLEU. Computational Linguistics, 44(3), 393–401. https://doi.org/10.1162/coli_a_00322 §
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html §
- Wittgenstein, L. (2009). Philosophical investigations (G. E. M. Anscombe, P. M. S. Hacker, & J. Schulte, Trans.; P. M. S. Hacker & J. Schulte, Eds.; Rev. 4th ed.). Wiley-Blackwell. (Original work published 1953) §
- Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., . . . Dean, J. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation (arXiv:1609.08144). arXiv. https://doi.org/10.48550/arXiv.1609.08144 §
