When no classes are given, clustering looks for those that fall naturally together (Witten et al., 2017, p. 45). The best-known algorithm is k-means (MacQueen, 1967): choose k initial centers, assign each point to the nearest center, move each center to the mean of its points, and repeat until nothing changes (Witten et al., 2017, p. 142). There are hierarchical variants, which produce trees of groups within groups, and probabilistic ones, in which each item belongs to each group with some probability.
Today clustering is done mostly in embedding spaces: each document is turned into a vector and nearby vectors are grouped. That is how the recurring topics in thousands of support tickets can be found without having defined them beforehand (Alammar & Grootendorst, 2024, p. 27).
k-means requires knowing in advance how many groups there are, and Witten et al. (2017) point out the irony: often nothing is known about that, and finding out is the whole point of clustering; the very existence of the clusters is what one is trying to discover (2017, p. 142). Unlike classification, the result cannot be graded against a correct answer; its success is usually judged by its usefulness to whoever uses it (Witten et al., 2017, p. 45).
The underlying question is philosophical. A clustering depends on which features are measured and which distance is used: with other features, the same data group differently. Are there natural kinds waiting to be discovered, or is similarity imposed by the observer? As Buckner (2024) recalls, citing Goodman, any two things resemble each other in infinitely many ways (2024, p. 201); a clustering algorithm is one way of fixing which ones count.