Dynamic Programming

Dive into Deep Learning · §14.2

Dynamic programming
value functions · the Bellman equations · a contraction · why the optimal path is not the shortest

Two Value Functions and Their Gap

• V^\pi(s): expected discounted return, following \pi from s.
• Q^\pi(s, a): same, but the first action is pinned to a.
• Linked by averaging: V^\pi(s) = \sum_a \pi(a \mid s)\, Q^\pi(s, a).

The gap is the advantage (:eqref:eq_advantage):

A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s), \qquad E_{a \sim \pi(s)}[A^\pi] = 0

A^\pi(s,a) > 0 means “do a more often”. Improvement lives here.

The Bellman Equations

One step now, value thereafter (the Markov assumption at work):

V^\pi(s) = \sum_{a} \pi(a \mid s) \Big[ r(s, a) + \gamma \sum_{s'} P(s' \mid s, a)\, V^\pi(s') \Big]

For the optimal policy the average over actions becomes a max (:eqref:eq_bellman_optimality):

V^*(s) = \max_{a} \Big[ r(s, a) + \gamma \sum_{s'} P(s' \mid s, a)\, V^*(s') \Big]

“The remainder of an optimal trajectory is also optimal.”

Backup Diagrams

Every algorithm in these two chapters walks one step down this tree; the sampled backup (right) is :numref:sec_qlearning in one picture.

Why It Converges

Proposition. \|TV - TV'\|_\infty \leq \gamma\, \|V - V'\|_\infty: the Bellman operator is a \gamma-contraction.

• unique fixed point V^*
• \|V_k - V^*\|_\infty \leq \gamma^k \|V_0 - V^*\|_\infty from any start
• stopping certificate: \|V_k - V^*\|_\infty \leq \frac{\gamma}{1-\gamma} \|V_k - V_{k-1}\|_\infty
• at \gamma = 1, all guarantees void

Value Iteration

def value_iteration(mdp, num_iters):
    """Sweep V <- max_a backup(V); return the whole history of iterates."""
    V, history = np.zeros(mdp.num_states), []
    for _ in range(num_iters):
        V = mdp.backup(V).max(axis=1)
        history.append(V)
    return np.array(history)

gap = np.abs(np.diff(history, axis=0)).max(axis=1)
true_err = np.abs(history[1:] - V_star).max(axis=1)
certified = gamma / (1 - gamma) * gap
assert (true_err <= certified + 1e-12).all()
for name, e in [('sweep-to-sweep change', gap),
                ('certified error bound', certified),
                ('distance to V*', true_err)]:
    print(f'{name} first below 1e-6 at sweep {np.argmax(e <= 1e-6) + 2}')
k_cert = np.argmax(certified <= 1e-6) + 2

sweep-to-sweep change first below 1e-6 at sweep 128
certified error bound first below 1e-6 at sweep 164
distance to V* first below 1e-6 at sweep 158

Naive test at 128, certificate at 164, truth at 158: a guarantee costs a handful of sweeps.

Policy Iteration and GPI

Evaluate the policy, act greedily on its values, repeat. Improvement provably never hurts.

ok = [(mdp.backup(V).argmax(axis=1) == pi_star).all() for V in history]
print(f'policy iteration: {num_outer} rounds of evaluate-then-improve')
print(f'value iteration: certified at sweep {k_cert}')
print(f'its greedy policy already equals pi* from sweep '
      f'{np.argmax(ok) + 1} on')

policy iteration: 2 rounds of evaluate-then-improve
value iteration: certified at sweep 164
its greedy policy already equals pi* from sweep 14 on

Not the Shortest Path

The payoff experiment: \pi^* against the calm-ice shortest path, 2000 episodes each, on slippery ice.

shortest = np.array([DOWN, RIGHT, DOWN, LEFT,
                     DOWN, LEFT, DOWN, LEFT,
                     RIGHT, DOWN, DOWN, LEFT,
                     LEFT, RIGHT, RIGHT, LEFT])   # optimal on calm ice, by hand
env.reset(seed=0)
for name, p in [('slip-aware optimum', pi_star),
                ('shortest-path policy', shortest)]:
    success = evaluate(env, lambda s, _: int(p[s]), num_episodes=2000)
    print(f'{name}: reaches the goal in {success:.1%} of 2000 episodes')

slip-aware optimum: reaches the goal in 73.6% of 2000 episodes
shortest-path policy: reaches the goal in 4.7% of 2000 episodes

Sixteen times the success rate. The optimum points away from the goal at four cells and commands into walls so that only harmless slips remain: not cleverness, just :eqref:eq_optimal_policy.

Recap

• V^\pi, Q^\pi, advantage A^\pi = Q^\pi - V^\pi: defined once, used for two chapters.
• Bellman: expectation form for a policy, optimality form :eqref:eq_bellman_optimality for the best one.
• The operator contracts at rate \gamma: unique V^*, geometric convergence, checkable certificate.
• Value iteration, policy evaluation, policy iteration: one proof, three algorithms.
• Generalized policy iteration is the skeleton of everything ahead.
• On ice, the optimal policy is not the shortest path, and we measured the difference: 0.74 vs 0.05.
• This was the book’s model-based corner; from :numref:sec_qlearning on, only samples.