class ReplayBuffer:
"""A ring of transitions in preallocated numpy; sample() returns a Batch."""
def __init__(self, capacity, obs_dim):
self.obs = np.zeros((capacity, obs_dim), np.float32)
self.act = np.zeros(capacity, np.int64)
self.rew = np.zeros(capacity, np.float32)
self.next_obs = np.zeros((capacity, obs_dim), np.float32)
self.term = np.zeros(capacity, np.float32)
self.capacity, self.size, self.ptr = capacity, 0, 0
def add(self, obs, act, rew, next_obs, term):
i = self.ptr
self.obs[i], self.act[i], self.rew[i] = obs, act, rew
self.next_obs[i], self.term[i] = next_obs, term
self.ptr, self.size = (i + 1) % self.capacity, min(self.size + 1,
self.capacity)
def __len__(self):
return self.size
def sample(self, batch_size, rng):
i = rng.integers(self.size, size=batch_size)
return d2l.Batch(self.obs[i], self.act[i], self.rew[i],
self.next_obs[i], self.term[i],
np.array([batch_size]))The Deadly Triad
Function approximation + bootstrapping + off-policy data :cite:Sutton.Barto.2018,vanHasselt.Doron.Strub.ea.2018. Every region is an algorithm already taught:
Baird’s Counterexample, Live
Seven states, every true value 0, and w = 0 can say so. Expected updates, no noise, uniform (off-policy) weighting:
sup norm of w after 0, 500, 1000 sweeps: 10, 77, 335
the value the weights claim for state 7: 677, true value 0Exponential divergence, and no learning rate fixes it: a property of the composed operator :cite:Baird.1995, not of sampling.
The Two Repairs
One Boolean, Two Worlds
With the copy: every seed climbs into the hundreds, the strongest stretches near the ceiling, nothing settles. Without: collapse to a pole that falls immediately, values past 10^8, every seed, both tabs. The greedy policy collects data that confirms its own collapse.
Converging Values, Churning Policy
continuing-task ceiling: 100; the no-target arm ends at 2e+08, 2e+08, 2e+08The trace converges to roughly the right number; nothing diverges (one probe state: a sentinel, not a certificate). What churns is the greedy policy read off those values: greedy evaluations span 90 to 500 across tab-seeds, some tying the untaxed ceiling, some caught mid-stumble, on value changes too small to see. The update bootstraps through the time limit, so the objective it defines is the continuing one: no policy is worth more than 1/(1-\gamma) = 100 from the start. A seed settling above the line claims what cannot be earned, and across tabs and seeds, some do.
The Max Leans High
single estimator, E[max of the estimates]: 1.031
select with one, evaluate with the other: -0.006One unit of bias from noise and a \max :cite:Thrun.Schwartz.1993; the double estimator removes it :cite:vanHasselt.2010. The hard \max is the \beta \to 0 corner of :numref:sec_regularized’s soft backup.
Double DQN in Three Lines
Select with the online network, evaluate with the frozen one :cite:Hasselt.Guez.Silver.2016:
@nnx.jit
def q_step_double(qnet, target, opt, obs, act, rew, next_obs, term):
"""q_step with selection split from evaluation, eq_double_dqn."""
sel = qnet(next_obs).argmax(-1)
fit_q(qnet, opt, obs, act, rew + gamma * (1 - term) * jnp.take_along_axis(
target(next_obs), sel[:, None], -1).squeeze(-1))
DQN: final value estimate at s0, per seed: [100.3 157. 94.4]
Double DQN: final value estimate at s0, per seed: [ 97.1 109.2 92.7]Most tab-seeds end lower, typically by a few points: two actions is where the \max has least room to flatter itself, which is panel (b)’s prediction. At Atari’s eighteen actions the same three lines change scores decisively.