By the chain rule, the probability of a sentence is the product of the probability of each word given everything before it. The Markov assumption truncates that history: P(wₜ | w₁ … wₜ₋₁) ≈ P(wₜ | wₜ₋ₖ … wₜ₋₁). With k = 1 one gets a first-order model, which predicts each symbol from the previous one; an n-gram model is a Markov model of order n − 1 (Witten et al., 2017, pp. 403–404). Without the truncation there would be nothing to count: almost no complete history ever recurs in a corpus.
Tiny example. A bigram model gives dog after the the same probability in "the dog barks" as in "I bought a new leash for the dog": leash does not count, only the. Shannon (1948) generated his approximations to English this way and called such sources "discrete Markoff processes" (1948, secs. 3–4).
This is what Chomsky (1957) attacked. Constructions like if… then or either… or can be nested inside one another, and each opening must be closed in mirror order, at an arbitrary distance. A finite-state device cannot keep that count, so English is not a finite-state language (1957, ch. 3). The objection bears on idealized competence: in corpora of seven European languages, Karlsson (2007) found no more than three levels of clausal center embedding in writing, and almost none in speech.
With neural networks, the assumption moved. A recurrent network is Markovian with respect to its hidden state, which sums up the past in a fixed-size vector, and a Transformer is Markovian with respect to its context window, a "state" of thousands of tokens. The question is no longer whether the past matters, but how much of it fits into the present. In reinforcement learning the same idea rules: a state is "Markov" if it contains everything about the past that makes a difference for the future (Sutton & Barto, 2018, ch. 3).