Skip to content

6/25 Introduction to Reinforcement Learning I

Today, we'll get acquainted with a new task type: Reinforcement Learning (RL). Here, we're teaching a model to make decisions based on a reward signal. The entity that makes decisions, or acts, is usually called an agent.

Note that this is closer to how living organisms learn: (usually) we are not shown the exact thing we need to do, but we get feedback to actions and we adapt accordingly.

This is a huge area of research. A nice, detailed reference textbook for non-deep reinforcement learning is Sutton--Barto: Reinforcement Learning: An Introduction [1]. It has numerous exercises to help you learn. In particular, implementing the toy environments they describe is pretty good coding practice.

Markov Decision Processes

We formalize many such problems as a Markov Decision Process (MDP). This is given by the following:

State Space

The state space \(\mathscr S\) collects the possible states of the system.

  1. It can be discrete, for example the set of possible positions in a board game.
  2. It can be continuous, for example the position, velocity and acceleration of all players and the ball in a simulated ball game.

Markov Property

The Markov Property is the assumption that the system is entirely described by its state. This is why I mentioned a simulated ball game: what I gave as state does not describe eg. the weather during the game.

Initial State

We let \(S_0\) denote the random variable with value the initial state of the system. That is for each state \(s\in\mathscr S\), we are given a probability \(p_0(s):=\mathbf P(S_0=s)\) that the system starts in state \(s\).

Terminal State

Oftentimes there are special states: terminal states. If the system enters such a state, then the process ends.

Action Space

When in state \(s\in\mathscr S\), the agent can take one of the actions in the action space \(\mathscr A(s)\). This can also be discrete or continuous. For convenience, usually we take a total action space \(\mathscr A=\cup_{s\in\mathscr S}\mathscr A(s)\) and make actions \(a\in\mathscr A-\mathscr A(s)\) invalid (illegal) when in state \(s\) by either

  1. stopping the process at once or
  2. making the agent inert, if that makes sense in the environment.

For terminal states \(s_T\), we let \(\mathscr A(s_T)=\{*\}\).

References

[1] Richard S. Sutton and Andrew G. Barto: Reinforcement Learning: An Introduction, second edition, 2018. The MIT Press, Cambridge, Massachusetts, London, England. link