6/23 Repeated Experiments

Reporting Results of Randomized Experiments

Yesterday, we introduced Stochastic Gradient Descent. This brought randomness into our experiments: starting with different random seeds, the minibatches will be different, which will in turn change how the parameters are updated.

Therefore, given the results of one experiment, one may ask: am I seeing the result of a super lucky draw, or can I expect about the same performance if I rerun the experiment with a different random seed? To decide questions as such, we need to run experiments multiple times with different random seeds and summarize the results.

Estimating the Mean via the Law of Large Numbers

For $i=0,\dots$, let $X_i$ denote a performance metric of an experiment, say the validation accuracy of a logistic regression model. Suppose that these variables are identically and independently distributed (i.i.d.), with mean $\mu$ and finite variance $\sigma^2$.

Then by the Law of Large Numbers, the sample mean $\bar X_n:=\frac{X_1+\dotsb+X_n}{n}$ converges almost surely to $\mu$ as $n\to\infty$. That is, the average performance $\bar X_n$ out of $n$ independent experiments is a better and better estimate of the true expected performance $\mu$ as we increase the number $n$ of experiments.

But how close can these approximations get as a function of $n$?

Getting Confidence Intervals

Student $t$-Distributions

For this, we make the additional assumption that the random variables $X_i$ have distribution $\mathscr N(\mu, \sigma)$, that is normal distribution with mean $\mu$ and standard deviation (std) $\sigma$. Let $$ S_n^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X_n)^2 $$ denote the so-called sample variance. Then the expression $$ T_n:=\frac{\bar X_n-\mu}{\sqrt{S^2_n / n}} $$ has Student $t$-distribution with $n-1$ degrees of freedom. We get a relation between the precision of the estimate and a confidence in said precision: $$ \mathbf P(|\bar X_n-\mu|<c\sqrt{S^2_n / n}) =\mathbf P(|T_n|<c)=\alpha. $$ As the middle term is a value of the cumulative distribution function of a known distribution, it can be approximated numerically. For example, setting $\alpha=95\%$, and in case $n=10$, we get $c\approx2.262$.

That is: if we repeated a lot of times the experiment of running $n$ experiments, then $\alpha$ proportion of the time, we would have $|\bar X_n-\mu|<cS_n/\sqrt n$. The interval $$ (\bar X_n-cS_n/\sqrt n,\bar X_n+cS_n/\sqrt n) $$ is called a confidence interval. It is customary to report such a result as $\bar X_n\pm cS_n/\sqrt n$ (and declare somewhere that confidence intervals are with level $\alpha$).

A Central Limit Theorem

Note that above, we made the assumption that the variables $X_i$ one by one have a normal distribution. On the other hand, by the Lindenberg--Lévy Central Limit Theorem, the sequence of random variables $\sqrt n(\bar X_n-\mu)$ converges to the normal distribution $\mathscr N(0,\sigma^2)$ in distribution, that is for each $x\in\mathbf R$ we have $$ \mathbf P(\sqrt n(\bar X_n-\mu) < x) \xrightarrow{n\to\infty} \mathbf P(\mathscr N(0, \sigma^2) < x). $$

If you want to write confidence intervals for normal distributions, you can use the so-called 68-95-99.7 rule: we have $$ \mathbf P(|\mathscr N(\mu, \sigma^2) - \mu| < \sigma) \approx 68.27% $$ $$ \mathbf P(|\mathscr N(\mu, \sigma^2) - \mu| < 2\sigma) \approx 95.45% $$ $$ \mathbf P(|\mathscr N(\mu, \sigma^2) - \mu| < 3\sigma) \approx 99.73%. $$

Writing an Ensemble Training Loop

An ensemble is a collection of models trained on the same task. These have a lot of uses such as:

Performance metrics with confidence intervals, as we just saw.
Aggregating predicions of an ensemble usually gives a better result than the individual models; we can see this in Homeworks 1 and 3.
We can train the models in an ensemble with different hyperparameters, thus seeing which is the better; this we shall see tomorrow.

When we write the ensemble training loop, we shall focus on making it efficient: instead of training the ensemble models one by one, we will vectorize the process. This brings a vast speed increase on a GPU and it is also beneficial when training with CPU thanks to SIMD vectorization.

On Arrays and Tensors

As they will be used more and more as we progress, let's discuss the general dimensional analogue of vectors and matrices we'll be working with: arrays and tensors.

The basic object we're working with in Deep Learning is the array. This is a multidimensional grid of values of the same data type. Note that by dimension of an array, we mean the dimension of the grid.

That is, one possible way to represent an array is as a function $$ [n_0]\times\dotsb\times[n_{r-1}]\xrightarrow aK $$ where

for a nonnegative integer $n$, we let $[n]=\{0,\dotsc,n-1\}$ and
$K$ is a set of numbers,
1. either Mathematical, such as the integers $\mathbf Z$ or the reals $\mathbf R$
2. or Computer Scientifical, such as 64-bit integers int64 or 32-bit floating point numbers float32.

For a collection of indices $(i_0,\dotsc,i_{r-1})\in[n_0]\times\dotsb\times[n_{r-1}]$, we denote $a((i_0,\dotsc,i_{r-1}))$ by $a[i_0,\dotsc,i_{r-1}]$.

The shape of the array is the tuple $(n_0,\dotsc,n_{r-1})$.

For example:

A vector is a 1-dimensional array of shape $(d,)$. The dimension of the vector is the single value in the shape.
A matrix is a 2-dimensional array of shape $(m, n)$. Usually, the first shape value is the number of rows and the second shape value is the number of columns.
A collection of matrices of the same shape is a 3-dimensional array.

Now popular deep learning libraries such as tensorflow or torch, our library of preference, refer to these objects as tensors instead of arrays. This is not 100% correct, as Mathematically, a tensor is a multilinear transformation. It can only be described as a multidimensional grid of scalars once you select a basis in each vector space involved.

In applications, usually the vector spaces are the standard ones $K^d$ with the standard basises, this is how you can get around this distinction.