7/13 Deep Q-Networks

From Finite to Infinite State Space

Finite State Space: Matrix Model

Recall that in $Q$-learning, the model we train is an approximate optimal action value function $$ \mathscr S\times\mathscr A\xrightarrow Q\mathbf R. $$ In case we have $|\mathscr S|=N<\infty$ and $|\mathscr A|=M<\infty$, the model $Q$ is in fact an $N\times M$ matrix. This is why we called $Q$-learning a tabular RL method.

Infinite State Space: DNN Model

In case the state space is an infinite subset $\mathscr S\subseteq\mathbf R^d$, but we still have finite action space $|\mathscr A|=M<\infty$, we can rewrite the action-value function model as a function $$ \mathscr S\xrightarrow {Q_\theta}\mathbf R^M. $$ We will make the model a DNN. That's why it's called a Deep $Q$-Network (DQN).

Loss: MSE in Bellman Optimality

Recall that given a sequence of states, actions, and rewards $s, a, r, s'$, the local action value estimate at $(s,a)$ we used in $Q$-learning is $r+\gamma V(s')$ where we have $$ V(s'):=\max{Q(s',a'):a'\in\mathscr A}. $$

We use a similar update in deep $Q$-learning. However, as $Q_\theta$ is a model with continuous state space, we cannot assume $Q_\theta(s)=\boldsymbol0$ for all terminal states, so we'll need to adjust to that.

First of all, let $$ V_\theta(s):=\max{Q(s,a):a\in\mathscr A} $$ denote the approximate value function defined by the DQN. Then if at state $s$ we chose action $a$, then got reward $r$, and got in next state $s'$, while $o=0$ if the episode ended (and thus $s'$ is the initial state of a new episode) and $o=1$ otherwise: $$ \ell(s,a,r,o,s';\theta)=SE(Q_\theta(s,a),r + \gamma o V_\theta(s')). $$

Stability Considerations

No Gradients through $V_\theta$

We discussed previously that the value function value $V_\theta(s')$ at the next state is viewed as more precise than the action-value function value $Q_\theta(s,a)$. Therefore, when we calculate the gradient $\nabla_\theta\ell$ which we feed the optimizer, we only compute gradients with respect to $Q_\theta(s,a)$, while we view $V_\theta$ as frozen.

Value Function by Target Network

In fact, we stabilize the targets further by calculating the approximate values at the next state by a target network $Q_{\theta'}$, the parameters of which follow those of the trained network $Q_\theta$ with some lag:

In the original DQN paper [1], the target network parameters $\theta'$ are updated to match the trained network parameters $\theta'$ with a fixed frequency.
One can get a softer such stabilizer by making an Exponential Moving Average (EMA) update $$ \theta'\leftarrow(1-\beta)\theta'+\beta\theta $$ each step, with new hyperparameter the EMA weight $\beta\in[0,1]$.

Offline RL via a Replay Buffer

As we aren't directly training a policy model, we can more easily learn from trajectories gotten via an older version of the model. Therefore, we can keep a replay buffer of past trajectories, and in each training step:

Update the replay buffer by making one environment step via an $\epsilon$-greedy step following the training network $Q_\theta$.
Update the training network parameters $\theta$ by the MSE loss following a sample from the replay buffer.

We usually make the replay buffer have a finite capacity $C$, a new hyperparameter. We keep the most recent $C$ steps.

We also initialize the replay buffer by a number of warmup steps, which are taken following a uniform policy.

References

[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). doi pdf