Sequences Are Trajectories

Dive into Deep Learning · §15.7

Sequences are trajectories
text generation is a degenerate MDP · one factorization identity · what collapses, what survives · the contract the Language Models part inherits

The Dictionary

language modeling reinforcement learning
prompt x start state s_0 \sim \mu_0
token y_t action a_t
prefix (x, y_{<t}) state s_t
response y trajectory \tau
EOS terminal state
next-token softmax the policy \pi_\theta
generate rollout

Transitions are concatenation: deterministic, known, probability one. All randomness is the policy’s; the reward is terminal.

One Identity

\nabla_\theta \log \pi_\theta(y \mid x) = \sum_{t=1}^{T} \nabla_\theta \log \pi_\theta(y_t \mid x, y_{<t})

  • chain rule, no transition factors to drop: three-line proof
  • terminal reward \Rightarrow one response-level weight r(x, y) - b(x) on every token’s score (not the prefix-dependent A(s_t, a_t))
  • one action per episode: the contextual bandit of :numref:sec_qlearning, until multi-turn closes the loop

What Collapses, What Survives

Everything that leaned on the kernel or on intermediate reward collapsed; the policy-optimization spine survived whole.

The Group Is the Baseline

Sample K responses per prompt, standardize within the group, one step on the log-probs. Same-group centering is biased: its expectation is (K-1)/K of the gradient (leave-one-out is exact). Prediction: at K = 1 the shrinkage reaches zero and the update vanishes identically.

K =  1: success of a sampled response 0.062
K =  2: success of a sampled response 0.500
K =  4: success of a sampled response 0.500
K =  8: success of a sampled response 0.500
K = 32: success of a sampled response 0.500

0.062 is the reference, untouched: self-inclusion bias at its maximum, not “no relative information”. “The group mean replaced the critic”, as a measurement.

The Loophole

A sloppy grader greps for the answer; a hedge that lists every number passes everything. :eqref:eq_kl_optimum prices it: exploit pays iff $< $ reward gap / reference log-odds = 1/4.

beta = 0.0: sloppy 1.00, gold 0.50, hedge 0.50
beta = 0.1: sloppy 0.99, gold 0.50, hedge 0.49
beta = 0.2: sloppy 0.60, gold 0.48, hedge 0.12
beta = 0.3: sloppy 0.42, gold 0.39, hedge 0.04
beta = 0.5: sloppy 0.25, gold 0.24, hedge 0.01

Hacked at \beta = 0; stops paying past 0.2; every row lands close to the tilted optimum. \beta is an exchange rate, not a safety valve.

GRPO, Assembled from Owned Parts

  • group-standardized advantage: :numref:sec_baselines’s :eqref:eq_pg_normalized, group mean as baseline (biased by self-inclusion; LOO is the exact variant), no value network
  • a few epochs per group: token-level clipped ratios :eqref:eq_ppo_clip
  • \beta\, D_{\textrm{KL}} to the frozen reference as a separate estimator: :eqref:eq_kl_objective
  • both KLs at once: clip against the previous iterate, penalty against the frozen reference
  • the toy simplifies: one step per group (nothing clipped), KL folded into the reward, one-token responses

Read :eqref:eq_kl_optimum backwards and preferences fit the policy directly: DPO :cite:Rafailov.Sharma.Mitchell.ea.2023.

The Contract

Reward is learned (Bradley-Terry, :numref:sec_regularized) or checked (a verifier: RLVR :cite:DeepSeekAI.2025). Either way it is an estimate, and an optimizer finds an estimate’s errors: \hat{Q} in :numref:sec_offline, r_\phi in :numref:sec_regularized, the grader here.

:numref:tab_rl_notation_contract: x, y, \tau, \mu_0, \pi_{\textrm{ref}}, \hat{G}_t, A, K, \rho_t, \epsilon, \beta, \delta_t, w: inherited verbatim by the Language Models part.

Where To Go Next

  • search: MCTS is a policy-improvement operator, distilled back into the network; AlphaZero \to MuZero \to world models; the same pattern as test-time reasoning
  • continuous control: same objectives, gains from normalization and scale
  • omitted with pointers: multi-agent, distributional, meta, hierarchical, POMDPs
  • next read: the RLHF book :cite:Lambert.2026; you own every estimator in it

Recap

  • prompt = start state, token = action, prefix = state, response = trajectory; transitions are concatenation
  • one gradient, two views; terminal reward puts one response-level weight r - b(x) on every token
  • K = 1’s update is identically zero (self-inclusion bias at its maximum), measured; the hedge gets hacked, measured; $= $ gap / log-odds prices the exploit
  • what survives is the policy-optimization spine, and the Language Models part inherits it verbatim