CS 8803 Sequence Prediction

Course materials for CS 8803 Sequence Prediction, Spring 2026 at Georgia Tech.

View project on GitHub

Lecture 8: Foundations of Sequence Models: From Markov to Transformers

1. Introduction

In the previous lecture, we established that the fundamental limit of predictability for a sequence is its Entropy Rate $H(\mathcal{X})$ (Lecture 7, Section 5). We also showed that the cost of prediction under log-loss is exactly the KL divergence from the true distribution, and that Bayesian updating is the minimax-optimal strategy for sequential prediction (Lecture 7, Section 3).

However, calculating the true Entropy Rate requires conditioning on an infinitely long history, which is computationally intractable. To build practical sequence models, we must find ways to approximate this conditioning. Today, we will explore the evolution of sequence models—starting with the foundational Markov assumptions, moving to their deep learning generalizations in Recurrent Neural Networks (RNNs), and finally arriving at the Transformer architecture. We will also examine how we scale these models and align them with human preferences.

The Unifying Thread: Throughout this lecture, the training objective for every model we discuss is the same: minimize the negative log-likelihood (i.e., the log-loss from Lectures 1 and 7). This is not a coincidence—as we established in Lecture 7, minimizing log-loss is simultaneously optimal for compression, prediction, and gambling. Every practical sequence model is an approximation to the universal predictor, trading off expressiveness against computational tractability.

2. The Foundations: Markov Chains and HMMs

Markov Chains

To build computable models, we apply the Markov Assumption: the future depends only on the current state. For an $N$-gram model (an order-$(N-1)$ Markov Chain), inference is linear time $\mathcal{O}(n)$.

A first-order Markov Chain over a state space $\mathcal{X}$ is parameterized by:

  • An initial state distribution $\pi_i = P(X_1 = i)$.
  • A transition matrix $A_{ij} = P(X_t = j \mid X_{t-1} = i)$.

The Markov Bottleneck: To capture longer-range dependencies, we can increase the order to $N-1$, conditioning on the last $N-1$ tokens. But this requires specifying a conditional distribution for each of the $\lvert \mathcal{X} \rvert^{N-1}$ possible conditioning contexts — an exponential explosion in parameters. For a vocabulary of size 50,000 and a 5-gram model, this means $50{,}000^4 \approx 6 \times 10^{18}$ entries in the transition table. This is the fundamental tension: longer context gives better approximations to the Entropy Rate, but the parameter count grows exponentially.

While computationally efficient, pure Markov Chains are often too rigid. To capture more complex dynamics without exploding the state space, we can introduce latent variables.

Hidden Markov Models (HMMs)

What if the true state generating the data is not directly visible? We introduce a latent (hidden) sequence of states $z_1, \dots, z_n \in \mathcal{Z}$ that forms a Markov Chain. The actual observations $x_1, \dots, x_n \in \mathcal{X}$ depend only on the current hidden state $z_t$.

An HMM is parameterized by three components:

  • A transition matrix $A_{ij} = P(z_{t+1} = j \mid z_t = i)$ (dynamics of the hidden states).
  • An emission matrix $B_{ij} = P(x_t = j \mid z_t = i)$ (how hidden states generate observations).
  • An initial state distribution $\pi_i = P(z_1 = i)$.

Inference: Given a trained HMM, the two fundamental inference tasks are:

  • Filtering/Smoothing (Forward-Backward algorithm): Compute the posterior marginals $\gamma_t(i) = P(z_t = i \mid x_{1:n})$ over the hidden states.
  • Decoding (Viterbi algorithm): Find the single most likely hidden state sequence $\arg\max_{z_{1:n}} P(z_{1:n} \mid x_{1:n})$ via dynamic programming.

Learning (Baum-Welch / EM): To learn the parameters $A$, $B$, $\pi$ without knowing the hidden states, we use the Expectation-Maximization (EM) algorithm:

  • E-Step (Expectation): Compute the posterior probabilities of the hidden states given the full sequence of observations using the Forward-Backward algorithm. We compute forward probabilities $\alpha_t(i) = P(x_{1:t}, z_t=i)$ and backward probabilities $\beta_t(i) = P(x_{t+1:n} \mid z_t=i)$. These are combined to find the marginals $\gamma_t(i)$ and pairwise marginals $\xi_t(i, j) = P(z_t=i, z_{t+1}=j \mid x_{1:n})$.
  • M-Step (Maximization): Use these expected counts to re-estimate $A$, $B$, and $\pi$.

The Catch: The E-step requires sequential backwards induction over the data sequence. You cannot compute $\beta_t$ until you have computed $\beta_{t+1}$. This makes it difficult to fully parallelize.

Summary of Markov/HMM Approaches:

  • Positives: They act as finite state machines with a clear probabilistic interpretation. Inference and computation are linear in the sequence length $\mathcal{O}(n)$.
  • Negatives: Training algorithms (like EM) have sequential bottlenecks. Furthermore, they suffer from the Markov Bottleneck described above—scaling the context window causes an exponential explosion in parameters, making long-range dependencies impossible to capture effectively.

3. Recurrent Neural Networks (RNNs)

Recurrent Neural Networks can be viewed as the “deep learning generalization” of Hidden Markov Models. Instead of maintaining a discrete probability distribution over hidden states, an RNN maintains a continuous, dense hidden state vector $h_t \in \mathbb{R}^d$.

Tokenization and Embeddings

Before processing text, we must convert strings into numbers. This is done via Tokenization (e.g., Byte-Pair Encoding or WordPiece). The text is chunked into frequent sub-words, and each sub-word is mapped to an integer ID from a vocabulary $V$.

Each token ID is used to look up a row from an embedding matrix $E \in \mathbb{R}^{\lvert V \rvert \times d}$, mapping the discrete token into a continuous $d$-dimensional vector (where $d \ll \lvert V \rvert$, e.g. $d = 4096$ vs. $\lvert V \rvert = 50{,}000$). This continuous vector $x_t \in \mathbb{R}^d$ is what is fed into the network. The embedding matrix is learned jointly with the rest of the model.

The RNN Forward Pass and Objective

At each time step $t$, the RNN observes the current input $x_t$ and the previous hidden state $h_{t-1}$ to update the hidden state. In the simplest (Elman) RNN, this takes the form:

\[h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)\]

More generally, we can write this as $h_t = f_\theta(h_{t-1}, x_t)$ for a learned function $f_\theta$. A secondary function (usually a linear layer followed by a softmax) outputs a probability distribution $y_t$ over the vocabulary for the next token:

\[y_t = \text{softmax}(W_o h_t + b_o)\]

The Training Objective: To train the RNN, we use the standard Next-Token Prediction loss, which is the cross-entropy (negative log-likelihood) between the predicted distribution $y_t$ and the actual next token $x_{t+1}$:

\[\mathcal{L}_{\text{RNN}} = -\frac{1}{n} \sum_{t=1}^n \log y_t[x_{t+1}]\]

LSTMs: A Special Case

Standard RNNs struggle to remember information over long sequences. The Long Short-Term Memory (LSTM) network is a special case of an RNN that introduces a “cell state” and gating mechanisms (input, forget, and output gates) to explicitly control what information is remembered or forgotten, mitigating some of the memory decay.

Summary of RNNs:

  • Positives: Like HMMs, inference requires linear computation $\mathcal{O}(n)$ in sequence length. They can theoretically process sequences of arbitrary length using a fixed number of parameters.
  • Negatives:
    • Unparallelizable Training: The hidden state $h_t$ strictly depends on $h_{t-1}$. We must process the sequence sequentially, making training on GPUs highly inefficient.
    • Optimization Instability: Backpropagating through time requires computing gradients that involve the product $\prod_{k=s}^{t} \frac{\partial h_{k}}{\partial h_{k-1}}$. If the spectral norm of each Jacobian is consistently greater than 1, this product explodes; if less than 1, it vanishes. Either way, learning long-range dependencies becomes extremely difficult.
    • Drift: During generation, sequences can “forget” their initial context and “go off track” because the entire history is compressed into a single vector $h_t$.

4. The Transformer Architecture

To escape the sequential bottleneck, the Transformer architecture (introduced by Vaswani et al. in 2017) abandons recurrence entirely.

Positional Encoding

A critical observation: the self-attention operation is permutation-equivariant — if we shuffle the input tokens, the output is shuffled in exactly the same way. Unlike RNNs, there is no inherent notion of token order. To give the model a sense of position, we add a positional encoding $P \in \mathbb{R}^{n \times d}$ to the token embeddings before they enter the Transformer:

\[X = E_{\text{tokens}} + P\]

The original Transformer used fixed sinusoidal encodings $P_{t,2k} = \sin(t / 10000^{2k/d})$, $P_{t,2k+1} = \cos(t / 10000^{2k/d})$, which encode position as a superposition of frequencies. Modern LLMs typically use Rotary Position Embeddings (RoPE), which encode relative positions directly into the Query-Key dot products.

Key Idea: Self-Attention

Instead of compressing the past into a single hidden state, the Transformer maintains the entire sequence of past token representations and selectively “attends” to them dynamically.

Given a sequence of token representations $X \in \mathbb{R}^{n \times d}$, self-attention computes Queries ($Q$), Keys ($K$), and Values ($V$):

\[Q = X W_Q, \quad K = X W_K, \quad V = X W_V\]

The Causal Mask: For an autoregressive sequence model, we cannot allow token $t$ to “look ahead” at token $t+1$ during training. We enforce this “causal” property by adding a mask matrix $M$ where $M_{ij} = -\infty$ for $j > i$, and $0$ otherwise. When passed through the softmax function, the $-\infty$ values become exactly $0$, ensuring that future tokens receive zero attention weight.

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^\top}{\sqrt{d_k}} + M\right) V\]

Multi-Head Attention

Rather than performing a single attention computation with $d$-dimensional keys, queries, and values, the Transformer uses multi-head attention. It runs $H$ independent attention heads in parallel, each with its own learned projections of dimension $d_k = d/H$, and concatenates the results:

\[\text{MultiHead}(X) = \text{Concat}(\text{head}_1, \dots, \text{head}_H) W_O\]

where $\text{head}i = \text{Attention}(X W{Q_i}, X W_{K_i}, X W_{V_i})$. Different heads can specialize in different types of patterns (e.g., one head might attend to syntactic structure, another to semantic similarity, another to recent context).

The Transformer Block

A full Transformer layer combines multi-head self-attention with a Feed-Forward Network (FFN), connected by residual connections and layer normalization:

\[Z = \text{LayerNorm}(X + \text{MultiHead}(X))\] \[\text{Output} = \text{LayerNorm}(Z + \text{FFN}(Z))\]

A complete Transformer stacks $L$ such blocks. The FFN is typically a two-layer network with a hidden dimension $4d$ and a nonlinearity (ReLU or SwiGLU).

  • Time Complexity: For a sequence of length $n$, model dimension $d$, and $L$ layers, the computational complexity is $\mathcal{O}(L(n^2 d + n d^2))$. The $n^2 d$ term comes from computing the $n \times n$ attention matrix; the $n d^2$ term comes from the FFN and projection layers.

Pre-Training Objective

The foundation of modern LLMs is pre-training via autoregressive language modeling. The mathematical objective is identical to the RNN—we minimize the negative log-likelihood of the sequence:

\[\mathcal{L}_{\text{pre}} = -\sum_{t=1}^n \log P_\theta(x_t \mid x_{<t})\]

  • Why training is fast: During training, all target tokens are known in advance (this is called teacher forcing). The causal mask ensures that each token’s prediction depends only on previous tokens, but all these masked predictions can be computed simultaneously via a single batched matrix multiplication. This completely bypasses the sequential bottleneck of RNNs. Note that inference (generation) is still autoregressive — we must generate one token at a time, since each new token depends on the previous output. Modern systems cache the Key and Value matrices from previous steps (the KV-cache) to avoid redundant computation during generation.

  • Connection to Entropy Rate: Recall from Lecture 7 that the Entropy Rate $H(\mathcal{X}) = \lim_{n \to \infty} H(X_n \mid X_{1:n-1})$. A Transformer with context length $n$ approximates this by conditioning on $X_{<t}$ for each $t$ — the longer the context, the closer the model can approach the true Entropy Rate.

5. Post-Training: Alignment and RL

Next-token prediction creates a model that simulates internet text, which isn’t always useful. To get transformers to produce “desirable” responses, we use alignment techniques.

Supervised Fine-Tuning (SFT)

We start by applying the exact same cross-entropy objective, but restrict the dataset to high-quality instructions and desired responses.

Reinforcement Learning (RL) and Reward Models

To further align the model with nuanced human preferences, we train a Reward Model $r_\phi(x,y)$. We collect a dataset $\mathcal{D} = {(x, y_w, y_l)}$ where humans rank a winning response ($y_w$) over a losing response ($y_l$) for a prompt $x$. The reward model is trained using a pairwise ranking loss (based on the Bradley-Terry model):

\[\mathcal{L}_R = - \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} [\log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))]\]

Given this trained reward model, the standard RL objective (often optimized via PPO) aims to maximize the expected reward while penalizing the KL divergence from the original SFT policy $\pi_{\text{ref}}$ (to prevent reward hacking):

\[\max_\pi \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot|x)} [r_\phi(x,y)] - \beta \mathbb{D}_{\text{KL}}(\pi(\cdot|x) \parallel \pi_{\text{ref}}(\cdot|x))\]

Direct Preference Optimization (DPO)

Recent advances like DPO (Rafailov et al., 2023) bypass the need to explicitly train a separate reward model. The key insight is that the KL-constrained RL objective above has a closed-form optimal policy:

\[\pi^\star(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left(\frac{r(x,y)}{\beta}\right)\]

where $Z(x)$ is a normalizing constant. By inverting this relationship to express the reward in terms of the policy, and substituting back into the Bradley-Terry preference loss, DPO derives a single objective that optimizes the language model $\pi_\theta$ directly on the preference data without ever training a reward model:

\[\mathcal{L}_{\text{DPO}} = - \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]\]

6. Scaling Laws

Why has the Transformer architecture dominated? It scales predictably. Empirical research has established that the cross-entropy loss $L$ of a language model follows a predictable power-law formulation with respect to the number of parameters $N$ and the number of training tokens $D$:

\[L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}\]

Where $E$ is the irreducible entropy of the text (the Entropy Rate!), and $A, B, \alpha, \beta$ are empirically fitted constants.

The History of Scaling Laws:

  • Kaplan et al. (2020) at OpenAI first established these power-law relationships. Their fitted exponents suggested that model size $N$ mattered much more than dataset size $D$, leading to the recommendation: when scaling compute, grow the model much faster than the dataset. This philosophy produced GPT-3 (175B parameters, 300B tokens).
  • Hoffmann et al. (2022) (“Chinchilla”) revisited these experiments with more careful controls and reached a different conclusion: the original analysis had confounded learning rate schedules with model size. Their revised exponents showed that $N$ and $D$ should scale roughly equally.

Optimal Compute Allocation (Chinchilla): If you have a fixed compute budget $C$ (measured in FLOPs, where $C \approx 6ND$), how should you allocate it? By minimizing the loss formula subject to the compute constraint, the Chinchilla scaling laws reveal that the optimal parameter count $N_{\text{opt}}$ and dataset size $D_{\text{opt}}$ scale equally:

\[N_{\text{opt}} \propto C^{a}, \quad D_{\text{opt}} \propto C^{b} \quad \text{where } a \approx b \approx 0.5\]

The takeaway: If you double your compute budget, you should train a model that is roughly $\sqrt{2}$ times larger on $\sqrt{2}$ times as much data. The Chinchilla paper demonstrated this by training a 70B parameter model on 1.4T tokens, which outperformed the 280B parameter Gopher model trained on only 300B tokens.

7. Computational Limits of Transformers

Despite their success, Transformers have strict computational limitations due to their architecture.

A Transformer with a fixed number of layers $L$ is fundamentally a constant-depth, parallel computation graph.

  • Hard-Attention Transformers (where attention weights are 0 or 1) are limited to the complexity class $AC^0$ (Hahn, 2020), making them theoretically unable to recognize even simple properties like the parity (even/odd number of 1s) of a binary sequence.
  • Soft-Attention Transformers reach the class $TC^0$ (Merrill & Sabharwal, 2023), because softmax allows for counting and recognizing hierarchical structures like $k$-Dyck languages. However, they still cannot recognize parity robustly.

Because they process inputs in parallel with constant depth, Transformers fundamentally struggle with problems that require strict, linear-time sequential reasoning.

Chain-of-Thought as a Workaround: A striking practical observation is that prompting LLMs to “think step by step” (Chain-of-Thought prompting) dramatically improves performance on reasoning tasks. Why? Each generated token effectively adds one more layer of sequential computation. By producing $k$ intermediate reasoning tokens, the model transforms itself from a depth-$L$ circuit into a depth-$(L \cdot k)$ circuit, partially circumventing the $TC^0$ barrier. This is why modern “reasoning” models (like OpenAI’s o1/o3 and DeepSeek-R1) are trained to produce long chains of thought — they are trading inference-time compute for increased effective depth.

8. Calibration and Hallucination

A persistent issue with LLMs is “hallucination”—stating false facts confidently. A fascinating 2024 paper, Calibrated Language Models Must Hallucinate (Kalai & Vempala), proved mathematically that this is not a bug, but a statistical necessity of the pre-training objective.

The Definition of Calibration

Recall from Lecture 4 that a forecaster is calibrated if its predicted probabilities match the actual empirical frequencies of the events it predicts. We showed there (using Blackwell Approachability) that calibrated forecasting is always achievable, even against an adversary.

A language model is calibrated if its predicted probabilities match the actual empirical frequencies in the world. If a calibrated model outputs a statement with 70% confidence, that statement should be true exactly 70% of the time.

The Hallucination Lower Bound

The authors prove that for any set of “arbitrary facts” (facts that cannot be logically deduced, like a specific person’s birthday), the probability of generating a hallucination $g(H)$ is bounded from below:

\[g(H) \ge p(U) - \text{calibration error} - \text{concentration term}\]

The dominant term here is $p(U)$, which represents the “missing mass”—the true probability of facts that the model never saw during training.

The Good-Turing Connection

How do we estimate this missing mass? In classical statistics, the Good-Turing estimator predicts the probability of encountering a completely unseen event based on the frequency of events seen exactly once (called “monofacts” or hapax legomena):

\[\hat{p}(U) \approx \frac{N_1}{n}\]

Where $N_1$ is the number of facts seen exactly once, and $n$ is the total number of facts in the training data.

The Result: If an arbitrary fact appears only once in a massive pre-training dataset, a perfectly calibrated model cannot be 100% certain it is true. Statistically, that single occurrence could be a typo or noise. To remain calibrated, the model is mathematically forced to hedge its bets by assigning non-zero probability mass to alternative, false versions of that fact (hallucinations).

Therefore, a statistically calibrated language model must inherently hallucinate at a rate bounded by the Good-Turing estimate ($N_1/n$) of the sparse data in its training set. Scaling up data does not eliminate this; it simply shifts the boundary of what is considered a “rare” fact.