The collection of texts a model is trained on; for LLMs, hundreds of billions or trillions of tokens drawn mostly from the web.
In linguistics, a corpus is a sample of texts chosen by criteria: the Brown Corpus
(Francis & Kučera, 1964/1979) gathered about a million words in 500 samples from fifteen genres. The training
corpora of LLMs are more like a harvest: gather everything available, then filter.
Some milestones. BooksCorpus: some 7,000 unpublished books, used by GPT-1 and,
together with Wikipedia, by BERT (800 million and 2,500 million words, respectively)
(Radford et al., 2018, sec. 4.1; Devlin et al., 2019, p. 4175). WebText: for GPT-2, the text behind
outbound links from Reddit that had received at least 3 karma, an indirect human filter;
about 8 million documents and 40 GB, with Wikipedia removed (Radford et al., 2019, sec. 2.1).
Common Crawl: a public web archive, from which GPT-3 took 45 TB of compressed text
that shrank to 570 GB after filtering (Brown et al., 2020, sec. 2.2). C4: Common Crawl cleaned with
heuristics (only lines ending in punctuation; out with any page containing a word from a list
of “bad words” or a curly bracket {; only text classified as English), about 750 GB
(Raffel et al., 2020, pp. 6–7). The Pile: 825 GiB of English in 22 subsets, many from academic or
professional sources (Gao et al., 2020).
Tiny example of what mixing implies. GPT-3 did not read its sources in proportion to their
size: filtered Common Crawl (410 billion tokens) made up 60% of the mix and Wikipedia (3
billion) 3%; but Wikipedia was gone through 3.4 times and Common Crawl less than once
(Brown et al., 2020, sec. 2.2). Building a corpus means making editorial decisions.
Those decisions matter. Filters define what counts as “quality text” and which voices are left
out; a huge corpus is not thereby a diverse one (Bender et al., 2021). Scale also makes it hard to know
what is inside, including the questions of benchmarks (see
data contamination); hence the proposals to document every dataset with a datasheet
(Gebru et al., 2021).
Sources
- Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? 🦜. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). Association for Computing Machinery. https://doi.org/10.1145/3442188.3445922 §
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., . . . Amodei, D. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems (Vol. 33, pp. 1877–1901). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html §
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423 §
- Francis, W. N., & Kučera, H. (1979). Manual of information to accompany A Standard Corpus of Present-Day Edited American English, for use with digital computers (Rev. and amplified ed.). Department of Linguistics, Brown University. http://korpus.uib.no/icame/manuals/BROWN/INDEX.HTM (Original work published 1964) §
- Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., & Leahy, C. (2020). The Pile: An 800GB dataset of diverse text for language modeling (arXiv:2101.00027). arXiv. https://doi.org/10.48550/arXiv.2101.00027 §
- Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., III, & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92. https://doi.org/10.1145/3458723 §
- Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training [Technical report]. OpenAI. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf §
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners [Technical report]. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf §
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140), 1–67. https://jmlr.org/papers/v21/20-074.html §
