Temporal Differences, Q-Learning and Exploration

Dive into Deep Learning · §14.4

Temporal differences, Q-learning and exploration
one sampled branch instead of the sum · the TD error · what actually converges · the price of exploration

The Sampled Backup

Value iteration needs P inside one expectation. Replace it with the transition you just observed:

\delta = r + \gamma \max_{a'} Q(s', a') - Q(s, a), \qquad Q(s, a) \leftarrow Q(s, a) + \alpha\, \delta

  • \delta is the temporal-difference error: reality’s one-step report minus the table’s claim.
  • Bootstrap masked by terminated, never truncated.

:numref:fig_rl_backups, panel (d): one blue branch instead of the sum, the max at the next state kept.

What Actually Converges

The sampled least-squares objective is not the right justification:

L(Q) = E_\mu \big[ (Q - TQ)^2 \big] + \gamma^2\, E_\mu \big[ \mathrm{Var}_{s'} ( \max_{a'} Q(s', a') ) \big]

Q^* zeroes the first term; the variance term moves the argmin (double sampling). Deterministic transitions kill it; ice does not.

The update is what deserves trust: E[\delta \mid s, a] = (TQ)(s, a) - Q(s, a), zero exactly at Q^*. A stochastic approximation of value iteration, convergent under Robbins-Monro steps.

The Update in Code

def epsilon_greedy(q, epsilon, rng):
    """Explore with probability epsilon, else act greedily on the values q."""
    if rng.random() < epsilon:
        return int(rng.integers(len(q)))
    # Random tie-breaking is load-bearing: np.argmax would always return
    # action 0 on a zero-initialized table, and an agent that only ever
    # proposes *left* on this lake never finds the goal.
    return int(rng.choice(np.flatnonzero(q == q.max())))
def q_learning(seed, Q, visits, env, num_episodes,
               alpha=lambda n: 1 / (1 + 0.1 * n)):
    """Tabular Q-learning; updates Q in place, yields each episode's return."""
    rng = np.random.default_rng(seed)
    epsilon = linear_schedule(1.0, 0.05, num_episodes // 2)
    env.reset(seed=seed)
    for episode in range(num_episodes):
        s, done, ret = env.reset()[0], False, 0.0
        while not done:
            a = epsilon_greedy(Q[s], epsilon(episode), rng)
            s_next, r, terminated, truncated, _ = env.step(a)
            visits[s, a] += 1
            delta = r + gamma * (1 - terminated) * Q[s_next].max() - Q[s, a]
            Q[s, a] += alpha(visits[s, a]) * delta
            s, done, ret = s_next, terminated or truncated, ret + r
        yield ret

Graded Against Dynamic Programming

The check no agent in the wild can run: we kept the solved MDP.

max_s |V_Qhat(s) - V*(s)| per seed: [0.019 0.012 0.021 0.008 0.006]
success rate: learned greedy 71.2% to 74.1% over 5 seeds; pi* 73.6%
pi* forced to explore at epsilon = 0.05: 54.1%
median environment steps: 95569

The table is within hundredths of V^*; the greedy policy within noise of \pi^*. The training curve plateaus at the behavior’s ceiling: \pi^* itself, taxed at \epsilon = 0.05, scores 54\%.

Step Sizes: the Ticket Is Not the Race

Q*(s0, <) = 0.180
alpha = 0.9 (constant): final estimates [0.056 0.221 0.206 0.265 0.236]
alpha =  1/(1 + 0.1 n): final estimates [0.19  0.186 0.195 0.184 0.18 ]
alpha =      1/(1 + n): final estimates [0.003 0.024 0.008 0.009 0.002]
  • constant 0.9: a noise ball that never shrinks (0.06 to 0.27)
  • 1/(1 + 0.1 n): converged, leaning slightly high
  • 1/(1 + n): passes Robbins-Monro, strands all five seeds below 0.025

Exploration, Priced: Regret

A bandit is an MDP with one state. Regret charges each pull the gap to the best arm.

Greedy 824 · fixed \epsilon 117 (a linear tax) · annealed 69 · UCB 37 · Thompson 32.

Optimism Pays, and Remember the Sign

a_t = \mathrm{argmax}_a \big[ \hat{\mu}(a) + \kappa \sqrt{\log t / n(a)} \big]

Per-arm, self-extinguishing exploration: logarithmic regret where any fixed \epsilon is linear (proved at \kappa = \sqrt 2; play each arm once first). Thompson: sample a Beta posterior, play the argmax.

The sign. Online exploration adds a count-shrinking confidence radius; offline pessimism (:numref:sec_offline) subtracts one. Optimism is safe only where it gets tested.

Which Policy Is Being Learned

  • The \max_{a'} ignores the action the behavior took: off-policy. Learn about the greedy policy from data collected by any policy.
  • One symbol away: SARSA bootstraps on the action taken, learning the behavior’s value, \epsilon floor and all.
  • The \max also leans high: four of five final estimates sat above the true 0.180, none below. Maximization bias, repaired in :numref:sec_dqn.

Recap

  • TD error :eqref:eq_td_error: the one-step residual; reused by every algorithm ahead.
  • Correctness lives at the fixed point of the update, not the argmin of the sampled objective (double sampling).
  • Robbins-Monro is necessary for the guarantee; budgets decide between schedules that both pass.
  • Self-correction: overvalued actions summon the data that convicts them; severed offline.
  • Regret: greedy is a lottery, fixed \epsilon a linear tax, optimism self-extinguishing. The confidence radius flips sign in :numref:sec_offline.
  • Off-policy by one \max; maximization bias by the same \max.