6/30 Q-Learning

Today, we will discuss $Q$-learning, an algorithm that we can use to learn policies that achieve high return on finite MDPs with no other information than interaction with the environment.

Q-Learning

This algorithm assumes that the MDP is finite with bounded reward, and discount $\gamma<1$..

This is an iterative algorithm. Let $Q_n(s, a)$ denote the approximate action-value function at step $n$. The we get the approximate value function

$$ V_n(s):=\max_{a\in\mathscr A(s)}Q_n(s, a). $$ We make infinite steps by restarting the MDP once we reach a terminal state. We also have learning rates $\alpha_n\in[0,1)$.

Suppose that at step $n$: in state $s_n\in\mathscr S$, we choose $a_n\in\mathscr A(s_n)$, get reward $r_n$ and get in state $s_{n+1}\in\mathscr S$. Then we let

\[ Q_{n+1}(s',a')=\begin{cases} (1-\alpha_n)Q_n(s,a) + \alpha_n(r_n + \gamma V_n(s,a)) & \text{if }s'=s\text{ and }a'=a\text{ and} \\ Q_n(s',a') & \text{otherwise.} \end{cases} \]

Theorem [1] Suppose that the following conditions hold:

The initial values $Q_0(s,a)$ are arbitrary, besides having $Q_0(s,a)=0$ for terminal states $s$.
The rewards are bounded: $|r_n|\le R$ for some $R\ge0$.
For $1\le i$, let $n^i(s,a)$ denote the index $n$ of the $i$-th time such that $s_n=s$ and $a_n=a$. Then for all $s\in\mathscr S$ and $a\in\mathscr A(s)$, we have $$ \sum_{i=1}^\infty\alpha_{n^i(s,a)}=\infty\text{ and } \sum_{i=1}^\infty\alpha_{n^i(s,a)}^2<\infty. $$

Then for all $s\in\mathscr S$ and $a\in\mathscr A(s)$, we have $$ Q_n(s,a)\to q_*(s,a)\text{ as }n\to\infty $$ with probability 1.

Q-Learning in Practice

So, we have a theorem that says if we follow Q-learning, then eventually, we get a nice policy. But how can we make this quicker?

Greedy Policy for Evaluation

When we want to evaluate our progress, that is get as great discounted returns as possible, we choose as policy the deterministic policy that maximizes the approximate action-state values: $$ \pi_\text{greedy}(s)=\mathop{\mathrm{argmax}}_{a\in\mathscr A(s)}Q_n(s,a). $$

$\epsilon$-Greedy Policy for Training

Note that for training, the theorem requires us to use policies that don't rule out any action entirely. Still, we would like to visit more promising states more often. A crude way to balance this is to use an $\epsilon$-greedy policy: This is the stochastic policy such that:

with probability $\epsilon$, we choose among the actions $\mathscr A(s)$ uniformly, and
otherwise, we use the greedy policy.

In other words, we have:

\[ \pi_\text{$\epsilon$-greedy}(a|s)=\begin{cases} 1-\epsilon + \frac{\epsilon}{|\mathscr A(s)|} & a=\pi_\text{greedy}(s) \\ \frac{\epsilon}{|\mathscr A(s)|} & \text{otherwise}. \end{cases} \]

It is common pratice to use a hyperparameter schedule for $\epsilon$: we start with $\epsilon=1$ as at first we have no knowledge of the environment and we decrease it as training progresses. In today's lab, we use a static schedule: with a fixed number of training steps, we linearly decrease $\epsilon$ from 1 to 0.

Optimistic Initialization

Note that we are free to pick the values $Q_0(s, a)$ for non-terminal states $s\in\mathscr S$. As the greedy and $\epsilon$-greedy policies will favor actions with large values of $Q_n$, the larger the initial values, the more the agent will explore as less visited values get less updates. Setting the initial values high is called optimistic initialization: the agent will think it is a good idea to try unexplored actions.

Validation in Reinforcement Learning

Note that in RL, there is no distinct validation set. On the other hand, a new dataset is generated in each episode. So we validate a policy by running a number of episodes. We'll see in the lab that this can be done in parallel to save time.

References

[1] Christopher J.C.H. Watkins and Peter Dayan: $Q$-Learning, 1992. Machine Learning, Volume 8, pp. 279-292. link