Glossary (TM) › Architectural parts
A model's ability to take in or produce more than one type of data (text, images, audio, actions) and to relate them to one another.
Each type of data is a "modality": text, image, audio, video, a robot's actions. A multimodal
model handles more than one; it may accept a modality as input without being able to
generate it (Alammar & Grootendorst, 2024, p. 259).
The common recipe: everything becomes vectors. A Transformer does not know what
an image is; it knows how to process sequences of vectors. Multimodality largely consists in
turning each modality into "tokens" that live in a shared space:
- Vision Transformer (ViT) cuts an image into 16×16-pixel patches and treats them as words
(Alammar & Grootendorst, 2024, pp. 261–262).
- CLIP trains a text encoder and an image encoder together so that a photo and its
caption land close to each other in the same space (contrastive learning)
(Alammar & Grootendorst, 2024, pp. 265–267).
- Models that "see" and chat connect a frozen image encoder to a language model through a
bridge that turns the visual input into a kind of soft prompt
(Alammar & Grootendorst, 2024, pp. 273–277). GPT-4 accepts images as input (OpenAI, 2023).
- Gato serializes text, images and joystick actions into a single sequence
(Reed et al., 2022).
- Latent diffusion generates images conditioned on text through
cross-attention (Rombach et al., 2022) (Diffusion model).
The transformer's authors already announced in 2017 that they meant to extend it to inputs
and outputs other than text (Vaswani et al., 2017, sec. 7).
A philosophical angle. Multimodality is often presented as an answer to the
symbol grounding problem problem: if a model learns from text alone, its symbols never touch the
world (Harnad, 1990; Bender & Koller, 2020); if it also sees images, do they? There are reasons
for caution. An image is still data, not an interaction with things, and pairing photos with
captions teaches correlations between two forms of representation. It remains open whether
that suffices for reference, or whether action, situated perception or social practice are
needed.
Sources
- Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
- Bender, E. M., & Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 5185–5198). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.463 §
- Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations (ICLR 2021). https://openreview.net/forum?id=YicbFdNTTy §
- Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1–3), 335–346. https://doi.org/10.1016/0167-2789(90)90087-6 §
- OpenAI. (2023). GPT-4 technical report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774 §
- Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (pp. 8748–8763). PMLR. https://proceedings.mlr.press/v139/radford21a.html §
- Reed, S., Zolna, K., Parisotto, E., Gómez Colmenarejo, S., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., & de Freitas, N. (2022). A generalist agent. Transactions on Machine Learning Research. https://openreview.net/forum?id=1ikK0kHjvj §
- Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 10674–10685). IEEE. https://doi.org/10.1109/CVPR52688.2022.01042 §
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html §
