CS 8803 Sequence Prediction

Course materials for CS 8803 Sequence Prediction, Spring 2026 at Georgia Tech.

View project on GitHub

Lecture 6: Kolmogorov Complexity, Universal Priors, and Solomonoff Induction

Guest Lecturer: Abhishek Shetty Date: February 27, 2026

1. Recap and the Universality Ladder

In previous lectures, we developed a toolkit for sequential prediction and decision-making:

  • Lectures 1–2: The halving algorithm and weighted majority — optimal prediction with expert advice, achieving regret $\mathcal{O}(\ln N)$ for $N$ experts.
  • Lecture 3: Exponential weights, game-theoretic foundations, minimax regret.
  • Lecture 4: Information theory — entropy, KL divergence, mutual information, and their role in compression and prediction.
  • Lecture 5: Kelly betting and Cover’s universal portfolio — universal strategies that compete with the best constant-rebalanced portfolio without knowing the data-generating process.

Each lecture introduced a “universal” method that works without assumptions. Today we reach the top of the ladder: a universal method for all prediction tasks.

Level Method Competes With
1 Halving / Weighted Majority Best of $N$ fixed experts
2 Cover’s Universal Portfolio Best constant-rebalanced portfolio
3 Solomonoff Induction (Today) All computable probability measures

Recall Cover’s universal portfolio mixes over all constant-rebalanced portfolios with a prior. Today, Solomonoff induction mixes over all computable probability measures with a universal prior derived from Kolmogorov complexity.

2. Kolmogorov Complexity

2.1 Motivation: What is the “Simplest” Explanation?

Consider two binary strings of length 1000:

  • $x_1 = 010101010101\dots$ (repeating 01)
  • $x_2 = 101100011101\dots$ (random-looking)

Intuitively, $x_1$ is “simpler” than $x_2$. We can describe $x_1$ with a short program (“print 01 five hundred times”), but any description of $x_2$ seems to require spelling out most of its bits. Kolmogorov complexity formalizes this intuition by defining complexity as the length of the shortest program that generates the string.

2.2 Formal Definition

Fix a universal Turing machine $U$.

Definition (Kolmogorov Complexity). The Kolmogorov complexity of a string $x \in \{0, 1\}^\star$ (with respect to $U$) is:

\[K_U(x) = \min \{ \lvert p \rvert : U(p) = x \}\]

the length of the shortest program $p$ that causes $U$ to output $x$ and halt.

Theorem (Invariance Theorem; Solomonoff 1964; Kolmogorov 1965; Chaitin 1966). For any two universal Turing machines $U_1, U_2$, there exists a constant $c$ (depending only on $U_1, U_2$) such that for all $x$:

\[\lvert K_{U_1}(x) - K_{U_2}(x) \rvert \le c\]

Proof sketch. Since $U_1$ is universal, it can simulate $U_2$ via a fixed-length interpreter program of length $c_{12}$. Given any program $p$ for $U_2$, we can construct a program $p’$ for $U_1$ of length $\lvert p \rvert + c_{12}$ that first runs the interpreter, then feeds $p$ to the simulated $U_2$. Thus $K_{U_1}(x) \le K_{U_2}(x) + c_{12}$. By symmetry, the reverse also holds. $\square$

Thanks to the Invariance Theorem, we drop the subscript and write $K(x)$, understanding that it is defined up to an additive constant.

2.3 Basic Properties

(i) $K(x) \le \lvert x \rvert + c$ for all $x$ (the “print” program).

(ii) $K(n) \le \log_2 n + c$ for natural numbers $n$.

(iii) $K(xy) \le K(x) + K(y) + c$ (sub-additivity, up to a constant).

(iv) For any $n$, there exists $x \in \{0, 1\}^n$ with $K(x) \ge n$ (counting argument: there are $2^n$ strings but fewer than $2^n$ programs of length $< n$).

2.4 Incomputability

Theorem. $K(x)$ is not a computable function.

Proof via Berry Paradox. Suppose $K$ is computable. Consider a program that enumerates strings $x$ by increasing length and computes $K(x)$, outputting the first $x$ with $K(x) > c$ for a sufficiently large constant $c$. This program has fixed length $\mathcal{O}(\log c)$, so for large enough $c$ we get a string $x$ with $K(x) > c$ produced by a program of length $\mathcal{O}(\log c) < c$ — a contradiction. $\square$

Proof via Halting Problem reduction. If $K$ were computable, we could decide the halting problem: given $(M, x)$, compute $K(M, x, \text{“halts”})$ vs. $K(M, x, \text{“loops”})$. The correct one has lower complexity. This contradicts the undecidability of the halting problem. $\square$

Incomputability of $K$ is a fundamental barrier. It motivates computable approximations: Minimum Description Length (MDL), BIC/AIC, and Lempel-Ziv compression.

2.5 Conditional Kolmogorov Complexity

The conditional Kolmogorov complexity of $x$ given $y$ is:

\[K(x \mid y) = \min \{ \lvert p \rvert : U(p, y) = x \}\]

Theorem (Symmetry of Information).

\[K(x, y) = K(x) + K(y \mid x) + \mathcal{O}(\log K(x, y))\]

This is the algorithmic analogue of the information theory chain rule $H(X, Y) = H(X) + H(Y \mid X)$.

3. Occam’s Razor and Minimum Description Length (MDL)

3.1 Occam’s Razor: From Heuristic to Theorem

Occam’s Razor is typically stated as a heuristic: “prefer simpler explanations.” Kolmogorov complexity turns this into a mathematical result.

Under the universal prior (defined in Section 4), simpler hypotheses are automatically given higher prior probability. If hypothesis $H_1$ is $c$ bits simpler than $H_2$ (i.e., $K(H_1) = K(H_2) - c$), then:

\[\frac{M(H_1)}{M(H_2)} \ge 2^{c - \mathcal{O}(1)}\]

Occam’s Razor is not an assumption — it is a consequence of the universal prior.

3.2 Minimum Description Length (MDL)

The MDL principle operationalizes Kolmogorov complexity into practical model selection.

Definition (Two-Part MDL). Given data $x$ and a model class $\mathcal{M} = \{M_1, M_2, \dots\}$, the two-part MDL selects:

\[\hat{M} = \arg \min_{M \in \mathcal{M}} \Big[ \underbrace{L(M)}_{\text{model code}} + \underbrace{L(x \mid M)}_{\text{data given model}} \Big]\]

where $L(M)$ is the description length of $M$ and $L(x \mid M) = -\log_2 P(x \mid M)$.

Example (Polynomial Regression). Fitting a degree-$d$ polynomial to $n$ data points:

  • $L(M_d) \approx (d+1) \cdot b$ bits (encoding $d+1$ coefficients at $b$ bits each).
  • $L(x \mid M_d) = -\sum_{i=1}^{n} \log_2 P(x_i \mid M_d)$ (residual coding cost).

Low-degree polynomials have smaller $L(M_d)$ but may have larger $L(x \mid M_d)$. MDL finds the optimal tradeoff, automatically balancing fit vs. complexity.

3.3 Refined MDL: Normalized Maximum Likelihood (NML)

Modern MDL replaces the two-part code with the NML distribution.

Definition (NML Distribution). For a model class $\mathcal{M}$ with parameter space $\Theta$:

\[P_{\text{NML}}(x) = \frac{P(x \mid \hat{\theta}(x))}{\sum_{y \in \mathcal{X}^n} P(y \mid \hat{\theta}(y))}\]

where $\hat{\theta}(x)$ is the MLE for data $x$. The denominator is the parametric complexity or Shtarkov sum.

Proposition. The NML distribution is the unique minimax optimal solution:

\[P_{\text{NML}} = \arg \min_Q \max_x \log \frac{P(x \mid \hat{\theta}(x))}{Q(x)}\]

The minimax formulation connects directly to our game-theoretic approach from Lectures 2–3. NML is the minimax strategy for the log-loss prediction game.

4. The Universal Prior

4.1 Setup: Programs as Hypotheses

Let $U$ be a universal prefix-free Turing machine. The set of all programs that halt and output finite binary strings forms our hypothesis class. The key property of a prefix-free machine is that no valid halting program is a prefix of another — this ensures the set of programs forms a prefix-free code.

Definition (Solomonoff’s Universal Prior / Algorithmic Probability). For any string $x \in \{0, 1\}^\star$, the algorithmic probability (or universal prior) is:

\[M(x) = \sum_{p : U(p)=x} 2^{-\lvert p \rvert}\]

That is, $M(x)$ is the probability that $U$ outputs $x$ when fed a uniformly random infinite binary input.

This is a well-defined probability distribution (or more precisely, a semimeasure) because we use a prefix-free machine, ensuring by Kraft’s inequality that $\sum_p 2^{-\lvert p \rvert} \le 1$.

4.2 The Coding Theorem

Theorem (Coding Theorem; Levin 1974).

\[\log_2 \frac{1}{M(x)} = K(x) \pm \mathcal{O}(1)\]

Equivalently, $M(x) = 2^{-K(x) \pm \mathcal{O}(1)}$.

Proof sketch.

  • Upper bound: The shortest program $p^\star$ for $x$ contributes $2^{-\lvert p^\star \rvert} = 2^{-K(x)}$ to $M(x)$, so $M(x) \ge 2^{-K(x)}$.
  • Lower bound: By Kraft’s inequality, $M(x) = \sum_{p:U(p)=x} 2^{-\lvert p \rvert} \le 1$, and each program contributes at most $2^{-K(x)}$ (otherwise it would be shorter than the shortest). The total number of programs for $x$ is bounded, giving $M(x) \le c \cdot 2^{-K(x)}$ for a constant $c$. $\square$

This bridges complexity and probability: the universal prior assigns probability $\approx 2^{-K(x)}$ to each string, automatically favoring simple (low-complexity) strings.

4.3 Dominance Property

Theorem (Dominance). For any computable probability measure $\mu$ on $\{0, 1\}^\star$, there exists a constant $c_\mu$ such that for all $x$:

\[M(x) \ge c_\mu \cdot \mu(x)\]

Proof sketch. Since $\mu$ is computable, there exists a program $p_\mu$ of length $K(\mu)$ that computes $\mu$. Using $p_\mu$, we can sample from $\mu$, and $p_\mu$ followed by a sample contributes to $M(x)$. We get $c_\mu = 2^{-K(\mu) - \mathcal{O}(1)}$. $\square$

$M$ dominates every computable measure. It is a “mixture over all computable hypotheses,” weighted by their complexity. This is analogous to how Cover’s universal portfolio mixes over all constant-rebalanced portfolios.

5. Solomonoff Induction

5.1 The Prediction Setup

Given a sequence $x_1, x_2, \dots$ generated by an unknown computable measure $\mu$, the Solomonoff predictor uses Bayesian updating with $M$ as the prior:

\[M(x_{n+1} = 1 \mid x_1 \dots x_n) = \frac{M(x_1 \dots x_{n} 1)}{M(x_1 \dots x_n)}\]

This is simply Bayesian updating with $M$ as the prior.

5.2 The Main Convergence Theorem

Theorem (Solomonoff 1964, 1978). Let $\mu$ be any computable probability measure on $\{0, 1\}^\infty$. Then:

\[\sum_{n=1}^\infty \mathbb{E}_\mu \left[ D_{\text{KL}}\left( \mu(\cdot \mid x_{<n}) \;\lVert\; M(\cdot \mid x_{<n}) \right) \right] \le K(\mu) \cdot \ln 2 < \infty\]

What this means:

  1. The total expected KL divergence between Solomonoff’s predictions and the true distribution is finite, bounded by $K(\mu) \ln 2$.
  2. Since the sum converges, $D_{\text{KL}}(\mu \lVert M \mid x_{<n}) \to 0$ almost surely ($\mu$-a.s.).
  3. The convergence rate is $\mathcal{O}(K(\mu) / n)$ on average.
  4. Solomonoff induction learns to predict as well as the true $\mu$, regardless of what $\mu$ is, as long as it is computable.

Compare to Expert Regret: With $N$ experts and the weighted majority algorithm, the total regret is $\le \ln N$. Here, the “experts” are all computable measures, and the total “regret” (KL divergence) is bounded by $K(\mu) \ln 2$. The complexity $K(\mu)$ plays the role of $\ln N$.

5.3 Proof of the Convergence Theorem

Proof. The proof has four key steps.

Step 1: KL divergence chain rule. The one-step divergences telescope into the total divergence:

\[\sum_{n=1}^{N} \mathbb{E}_\mu \left[ D_{\text{KL}}(\mu(\cdot \mid x_{<n}) \lVert M(\cdot \mid x_{<n})) \right] = \mathbb{E}_\mu \left[ \sum_{n=1}^{N} \log \frac{\mu(x_n \mid x_{<n})}{M(x_n \mid x_{<n})} \right] = \mathbb{E}_\mu \left[ \log \frac{\mu(x_{1:N})}{M(x_{1:N})} \right]\]

The right-hand side is $D_{\text{KL}}(\mu(\cdot) \lVert M(\cdot))$ on $\{0,1\}^N$.

Step 2: Apply the Dominance Theorem. By Theorem 4.3, $M(x) \ge 2^{-K(\mu)} \cdot \mu(x)$ for all $x$. Therefore:

\[\log \frac{\mu(x_{1:N})}{M(x_{1:N})} \le \log \frac{\mu(x_{1:N})}{2^{-K(\mu)} \cdot \mu(x_{1:N})} = K(\mu)\]

Step 3: Take expectation. Since the bound holds for every sequence:

\[\mathbb{E}_\mu \left[ \log \frac{\mu(x_{1:N})}{M(x_{1:N})} \right] \le K(\mu)\]

Step 4: Take $N \to \infty$. By the monotone convergence theorem (each term is non-negative):

\[\sum_{n=1}^{\infty} \mathbb{E}_\mu \left[ D_{\text{KL}}(\mu(\cdot \mid x_{<n}) \lVert M(\cdot \mid x_{<n})) \right] \le K(\mu) \cdot \ln 2\]

The $\ln 2$ factor converts from $\log_2$ to natural logarithm, since $D_{\text{KL}}$ is conventionally measured in nats. $\square$

6. Large Deviations and Universal Coding

Large deviations theory provides the probabilistic foundation underlying universal prediction and coding.

6.1 Cramer’s Theorem

Theorem (Cramer’s Theorem). Let $X_1, X_2, \dots$ be i.i.d. with distribution $P$ on $\mathbb{R}$, and let $\bar{X}n = \frac{1}{n}\sum{i=1}^{n} X_i$. For any closed set $F$:

\[\limsup_{n \to \infty} \frac{1}{n} \log P(\bar{X}_n \in F) \le -\inf_{x \in F} \Lambda^\star(x)\]

where $\Lambda^\star(x) = \sup_{\lambda \in \mathbb{R}} \{\lambda x - \Lambda(\lambda)\}$ is the rate function (Fenchel-Legendre transform of the log-moment generating function $\Lambda(\lambda) = \log \mathbb{E}[e^{\lambda X}]$).

6.2 Sanov’s Theorem

Theorem (Sanov’s Theorem). Let $X_1, \dots, X_n \sim P$ i.i.d. on a finite alphabet $\mathcal{A}$, and let $\hat{P}_n$ be the empirical distribution. For any set $E$ of distributions:

\[P^n(\hat{P}_n \in E) \approx \exp\left(-n \inf_{Q \in E} D_{\text{KL}}(Q \lVert P) \right)\]

Sanov’s theorem reveals that $D_{\text{KL}}(Q \lVert P)$ is the natural “cost” of observing empirical distribution $Q$ when the true distribution is $P$. This connects directly to:

  • Why log-loss is the natural loss function for prediction.
  • Why MDL works: the “redundancy” $\log P(x \mid \hat{\theta}) / Q(x)$ is governed by KL divergence.
  • Why Solomonoff converges: the dominance inequality bounds the total divergence.

6.3 Solomonoff Convergence as a Large Deviations Result

The bound $\sum_n \mathbb{E}\mu[D{\text{KL}}(\mu_n \lVert M_n)] \le K(\mu) \ln 2$ can be interpreted through the lens of large deviations:

  • The “atypical event” is Solomonoff predicting differently from $\mu$.
  • The rate function is governed by $K(\mu)$, the complexity of the true environment.
  • The convergence $D_{\text{KL}} \to 0$ is exponentially fast in terms of the cumulative cost.

6.4 The AEP and Lempel-Ziv

Theorem (Asymptotic Equipartition Property). For an ergodic source $\mu$:

\[-\frac{1}{n}\log \mu(X_1, \dots, X_n) \xrightarrow{\text{a.s.}} H(\mu)\]

where $H(\mu)$ is the entropy rate. This is a direct application of the law of large numbers, and large deviations gives the rate of convergence.

Theorem (Lempel-Ziv Convergence). For any ergodic source, the Lempel-Ziv compression rate converges to the entropy rate:

\[\frac{\lvert LZ(x_1 \dots x_n) \rvert}{n} \xrightarrow{\text{a.s.}} H(\mu)\]

This provides a computable bridge between Kolmogorov complexity (incomputable) and Shannon entropy (computable). LZ compression is a practical algorithm that asymptotically achieves the information-theoretic limit.

While Solomonoff induction is optimal for prediction, it is incomputable. Levin’s Universal Search provides a computable approach to universal problem-solving.

7.2 The Dovetailing Idea

Given a problem with a verifiable solution (i.e., a solution checker that runs in time $t(n)$):

  1. Enumerate all programs $p_1, p_2, \dots$ in order of increasing length.
  2. Run all programs in parallel using dovetailing: at step $N$, allocate $2^{-\lvert p_i \rvert}$ fraction of computational time to program $p_i$.
  3. When any program halts with a verified solution, output it.

7.3 Kt Complexity

Definition (Kt Complexity; Levin 1973).

\[Kt(x) = \min_{p : U(p)=x} \{ \lvert p \rvert + \log_2 t(p) \}\]

where $t(p)$ is the running time of program $p$. This penalizes both program length and computation time, capturing the intuition that a useful explanation should be both simple and fast to compute.

7.4 Optimality Theorem

Theorem (Levin’s Optimality). Levin’s Universal Search finds a solution in time:

\[\mathcal{O}\left(2^{Kt(\text{solution})} \cdot t_{\text{verify}}\right)\]

where $t_{\text{verify}}$ is the verification time. This is optimal up to a multiplicative constant in the exponent: no algorithm can solve all problems faster by more than a constant factor in the exponent.

Proof sketch. Suppose the optimal program $p^\star$ runs in time $t^\star$. Levin Search allocates fraction $2^{-\lvert p^\star \rvert}$ of time to $p^\star$, so $p^\star$ effectively runs at speed $2^{-\lvert p^\star \rvert}$. It will finish after total time $t^\star / 2^{-\lvert p^\star \rvert} = t^\star \cdot 2^{\lvert p^\star \rvert}$. Since $Kt = \lvert p^\star \rvert + \log_2 t^\star$, the total time is $2^{\lvert p^\star \rvert} \cdot t^\star = 2^{\lvert p^\star \rvert + \log_2 t^\star} = 2^{Kt}$. $\square$

7.5 The Speed Prior (Schmidhuber)

Schmidhuber’s Speed Prior weights hypotheses not just by program length but by computation time:

\[S(x) \propto \sum_{p : U(p)=x} 2^{-\lvert p \rvert} \cdot t(p)^{-1}\]

This favors hypotheses that are both simple and fast to compute.

7.6 Modern Connections: One-Way Functions

Recent work by Liu and Pass (2020) establishes a deep connection between time-bounded Kolmogorov complexity (Kt complexity) and the existence of one-way functions (OWFs) in cryptography:

\[\text{OWFs exist} \iff Kt \text{ complexity is hard on average}\]

This bridges algorithmic information theory and computational complexity in a surprising way.

8. Summary and Looking Ahead

Key Formulas

Concept Formula Significance
Kolmogorov Complexity $K(x) = \min\{\lvert p \rvert : U(p) = x\}$ Ultimate measure of complexity
Universal Prior $M(x) = \sum_{p:U(p)=x} 2^{-\lvert p \rvert}$ Prior over all computable hypotheses
Coding Theorem $\log(1/M(x)) = K(x) \pm O(1)$ Bridges complexity and probability
Dominance $M(x) \ge 2^{-K(\mu)}\mu(x)$ $M$ dominates all computable $\mu$
Solomonoff Bound $\sum_n \mathbb{E}[D_{\text{KL}}] \le K(\mu) \ln 2$ Universal prediction convergence
Kt Complexity $Kt(x) = \min\{\lvert p \rvert + \log t(p)\}$ Time-bounded complexity

The core ideas bridge Prediction, Compression, and Learning:

  • Log-loss = code length
  • Bayes = mixture
  • MDL = model selection

In the next lecture, we will move from the idealized universal predictor to practical sequence models like Markov Chains, Hidden Markov Models (HMMs), and Recurrent Neural Networks (RNNs). These act as computable approximations to Solomonoff induction.

Exercises

  1. Counting Argument. Prove that for any $n$, at least half of all strings in $\{0,1\}^n$ have $K(x) \ge n - 1$.

  2. MDL in Practice. Consider a dataset generated by a degree-3 polynomial with Gaussian noise. Derive the MDL criterion and show that it correctly selects degree 3 as $n \to \infty$.

  3. Solomonoff vs. Bayes. Show that if $\mu$ is computable but the prior $\pi$ assigns zero probability to $\mu$, then Bayesian updating with $\pi$ never converges to $\mu$, while Solomonoff induction does.

  4. Levin Search Simulation. Implement a simplified version of Levin’s Universal Search for solving SAT instances. Compare with brute-force search for small instances.

  5. Incomputability. Show that the function $f(n) = \max\{K(x) : x \in \{0,1\}^n\}$ is not computable, but $f(n) = n$ for all $n$.

References

  1. M. Li and P. Vitanyi, An Introduction to Kolmogorov Complexity and Its Applications, 4th ed., Springer, 2019.
  2. R. J. Solomonoff, “A formal theory of inductive inference,” Information and Control, 7(1):1–22, 1964.
  3. A. N. Kolmogorov, “Three approaches to the quantitative definition of information,” Problems of Information Transmission, 1(1):1–7, 1965.
  4. L. Levin, “Universal sequential search problems,” Problems of Information Transmission, 9(3):265–266, 1973.
  5. M. Hutter, Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability, Springer, 2005.
  6. P. Grunwald, The Minimum Description Length Principle, MIT Press, 2007.
  7. T. Cover and J. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
  8. J. Rissanen, “Modeling by shortest data description,” Automatica, 14(5):465–471, 1978.
  9. Y. Liu and R. Pass, “On one-way functions and Kolmogorov complexity,” STOC, 2020.
  10. J. Schmidhuber, “The Speed Prior: A new simplicity measure yielding near-optimal computable predictions,” COLT, 2002.
  11. A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, 1998.