Skip to content

6/29 Introduction to Reinforcement Learning II

Policy

The agent can be formalized as a mapping from states to actions: how should the agent react if the system is in a given state. It is usually denoted by \(\pi\).

  1. The policy can be deterministic. That is, we have a mapping \(\mathscr S\xrightarrow\pi\mathscr A\): given a state \(s\in\mathscr S\), the policy will always take the same action \(\pi(s)\in\mathscr A(s)\).
  2. The policy can be stochastic. That is, we have a mapping \(\mathscr S\xrightarrow\pi\mathscr P(\mathscr A)\): given a state \(s\in\mathscr S\), the probability that the agent will take action \(a\in\mathscr A(s)\) is \(\pi(a|s)\).

Given a state random variable \(S_t\), for example the initial state random variable at time step \(t=0\), we get an agent action random variable \(A_t\).

Partially Observable Markov Decision Processes (POMDP)

Many times, it is impossible to obtain the actual, full state of a system. We can only gather partial information, summarized as an observation. Therefore, in this case, the policy is a function of observations and not states.

Transitions

If the system is in state \(s\in\mathscr S\) and the agent takes action \(a\in\mathscr A(s)\), then the agent receives a reward \(r\in\mathbf R\) while the system transitions to another state \(s'\in\mathscr S\). Just like a policy, this can also be either:

  1. Deterministic: there is a fixed mapping \(\mathscr S\times\mathscr A\xrightarrow t\mathbf R\times\mathscr S\) from state-action pairs \((s,a)\) to reward-next state pairs \((r,s')\).
  2. Stochastic: the system is governed by a mapping \(\mathscr S\times\mathscr A\xrightarrow{p(r,s'|s,a)}\mathscr P(\mathbf R\times\mathscr S)\) from state-action pairs to distributions of reward-next state pairs.

For a terminal state \(s_T\), we let \(t(s_T,*)=(0, s_T)\).

Given a state-action random variable pair \((S_t, A_t)\), we get a reward-next state random variable pair \((R_{t+1}, S_{t+1})\).

Trajectories

Thus, the MDP and the policy give rise to a sequence or trajectory $$ S_0,A_0,R_1,S_1,A_1,\dotsc $$

If the chain continues until a terminal state \(S_T\) is reached, we call it an episode.

Returns

At each time step \(t\), we consider the return $$ G_t=\sum_{k=0}^\infty\gamma^kR_{t+k+1}. $$ Here, the value \(\gamma\in(0, 1]\) is the discount, a hyperparameter. Values smaller than 1 give the agent incentive to obtain rewards as soon as it can. A common baseline is \(\gamma=0.99\).

Value- and Action-Value Functions

For a random variable \(X\), we denote by \(\mathbf E_\pi X\) the expected value of \(X\) provided policy \(\pi\) is followed.

Two very important functions using this are:

  1. The Value Function under \(\pi\): Given a state \(s\in\mathscr S\), this is $$ v_\pi(s):=\mathbf E_\pi(G_t|S_t=s), $$ the expected return of trajectories starting with \(s\), following \(\pi\).
  2. The Action-Value Function under \(\pi\): If furthermore we fix an action \(a\in\mathscr A(s)\), this is $$ q_\pi(s,a):=\mathbf E_\pi(G_t|S_t=s,A_t=a), $$ the expected return of trajectories starting with \(s\) and \(a\), following \(\pi\).

Optimal Policy

Take two policies \(\pi\) and \(\pi'\). Then we write \(\pi\ge\pi'\) if we have $$ v_\pi(s)\ge v_{\pi'}(s)\text{ for all }s\in\mathscr S. $$ Note that this relation equips the set of policies with a partially ordered set (poset) structure. A maximum element of this set is called an optimal policy \(\pi^*\). That is, we need \(\pi^*\ge\pi\) for all policies \(\pi\).

At this point, you may wonder if optimal policies exist. In Reinforement Learning, there are constructive proofs for specific cases:

  • We say that the MDP is finite if the state and action spaces are finite: \(|\mathscr S|,|\mathscr A|<\infty\).
  • We say that the MDP has deterministic reward if the transition function can be written in the form $$ \mathscr S\times\mathscr A\xrightarrow{t_S\times r}\mathscr P(\mathscr S)\times\mathbf R, $$ that is, given a state-action pair \((s,a)\in\mathscr S\times\mathscr A\), we always get the same reward \(r(s)\), even though the next state can be stochastic, given by the distribution \(t_S(s,a)\in\mathscr P(\mathscr S)\).
  • We say that the MDP has bounded reward, if there exists a constant \(R\in\mathbf R_{\ge0}\), such that the absoluate value of all rewards we can get is smaller than \(R\).

Using the Policy Iteration Algorithm, one can find an optimal policy [1, Algorithm 3.1], in case the MDP is finite with deterministic, bounded reward, and discount \(\gamma<1\).. We will not discuss this algorithm in detail.

It is important to note that the Policy Iteration Algorithm requires knowledge of the transition and reward functions \(t_S, r\). Note that this is unrealistic, as for example, when you start playing a computer game, you don't know what next states are going to be.

Therefore, from a practical point of view, the \(Q\)-learning algorithm that we will cover on next Tuesday, is more realistic, as it does not require knowledge of the environment, just interaction results from it. Just like how you would learn to play a game by playing with it.

References

[1] Lodewijk Kallenberg: Markov Decision Processes. Last accessed on March 6, 2026. link