Skip to content

6/22 Stochastic Gradient Descent

with Replacement (Random Sampling)

As the MNIST train set has 60 000 and the test set 10 000 entries, we randomly split off 10 000 entries from the train set for the validation set.

That is, if we perform gradient descent with this training set, in each step, we have to calculate the gradient \(\nabla_\theta\ell(\mathbf x, y; \theta)\) at 50 000 dataset entries \((\mathbf x, y)\).

Now, recall that we view the train set as a collection of random samples from the joint distribution \((X, Y)\). So, we can try out taking training steps on small samples from the train set. If you know bootstrapping, this is like that: we approximate sampling from the distribution by sampling from the dataset.

It turns out this:

  1. Works!
  2. It not just works, it works better: the noise we get from the random samples helps the optimization process escape local minima.

The size of the small samples we take is the minibatch size, yet another hyperparameter. It is also called batch size, but I recommend the former as

  1. minibatch only refers to the batch of entries you use in one training step, while
  2. batch can mean batches of entries for other means.

Beware: we need to take a different small sample in each training step. Training on the same small sample causes the model to overfit.

without Replacement (Random Reshuffling)

It turns out that the Stochastic Gradient Descent method actually used in most cases is another version: here, we repeat the following:

  1. Shuffle the train set.
  2. Perform gradient descent on successive subsets of minibatch size. The size of the last minibatch can be smaller.

One such outer loop is called an epoch. That is, during an epoch we take \(\left\lceil\frac{\text{train set size}}{\text{minibatch size}}\right\rceil\) train steps.

Many times, training length is measured in epochs. Note that during an epoch, multiple gradient descent steps are taken, each one changing the parameters.

One may wonder if random reshuffling really works better than random sampling. A theoretical paper on this matter is [1]

Pseudorandom Generators

With SGD, random choices become an essential part of the training algorithm. So let's discuss stochasticity in Machine Learning.

Machine Learning experiments have two opposing requirements:

  1. they should be reproducible, and
  2. they should use random sampling.

To reconcile these two, we use pseudorandom generators:

  1. These are stateful, deterministic objects, that is if you know the present state, then you can precisely determine the future outputs.

  2. On the other hand, the outputs approximate probabilistic distributions to the extent that Machine Learning experiments require.

Warning

The starting state of a pseudorandom generator is determined by the seed. Usually, it is an integer. Never forget to

  1. set the seed and
  2. record it

at the start of an experiment. Otherwise, it may get arbitrarily difficult to reproduce results or errors.

For maximum safety, I recommend also recording:

  1. the exact code you got the experiment with, for example with a git commit ID1 and
  2. the versions of the modules you used, which in turn can be gotten using a mamba list or pip list terminal command.

Note

This and other requirements for secure applications such as in cryptography are checked by statistical tests. For example, see here: BSI evaluation criteria for pseudorandom number generators

References

[1] Pierfrancesco Beneventano: On the Trajectories of SGD Without Replacement. arXiv:2312.16143

  1. git is a version control system you can use to track changes in a codebase such as the course repository. After each unit of recorded change -- a commit --, it generates a unique ID that can be used to revert the codebase to the state at the commit. ↩