6/24 Early Stopping and Grid Search

Broadcasting

To write an effective ensemble training loop, we will make use of broadcasting. Broadcasting means that when you perform a binary operation among tensors, you can leave out some dimensions from one or the other and the operation will be conducted as if the tensors were constant along the given dimensions.

Take tensors $a$ and $b$, of respective shapes $(m_1,\dotsc,m_k)$ and $(n_1,\dotsc,n_l)$. Up to swapping the two tensors, we can assume $k\le l$.

If $k<l$, then we change the shape of $a$ by filling it in with 1's from the left until $l=k$.
For any $1\le i\le k$:
1. If $m_i=n_i$, then the operation is performed normally.
2. If a size $m_i$ or $n_i$ is 1, then the tensor is viewed as constant along that dimension.
3. If the sizes $m_i$ and $n_i$ are different and neither are 1, then the two tensors are not broadcastable, the operation is forbidden.

Let's see if the following operations are legal and what is their result:

[1, 2, 3] + [1].
[1, 2, 3] + [[1]].
[[1, 2, 3]] * [2, 3]
[[1, 2, 3]] * [1, 2, 3].
[1, 2, 3] == 2.
[1, 2, 3] - [[1, 2, 3], [4, 5, 6]].

For more information, see here: NumPy Manual: Broadcasting

Broadcasting Rules in Matrix Multiplication

Let $a$ and $b$ be tensors of shapes $(m_1,\dotsc,m_k)$ and $(n_1,\dotsc,n_l)$, respectively. Special broadcasting rules apply for matrix multiplication $a\cdot b$: we need

The batch shapes $(m_1,\dotsc,m_{k-2})$ and $(n_1,\dotsc,n_{l-2})$ to be broadcastable, and
The shapes $(m_{k-1},m_k)$ and $(n_{l-1},n_l)$ allow matrix multiplication, that is $m_k=n_{l-1}$.

Let $(r_1,\dotsc,r_{l-2})$ denote the broadcasted batch shapes. Then the matrix product $a\cdot b$ will have

shape $(r_1,\dotsc,r_{l-2},m_{k-1},n_l)$ and
for all index $(l-2)$-tuples $(i_1,\dotsc,i_{l-2})\in[r_1]\times\dotsb\times[r_{l-2}]$, we have $(a\cdot b)_{i_1,\dotsc,i_{l-2}} = a_{i_1,\dotsc,i_{l-2}}\cdot b_{i_1,\dotsc,i_{l-2}}$.

Use Case: Getting Logits from Ensembles of Weight Matrices and Bias Vectors

In today's notebook, we will train an ensemble of $e$ logistic regression models on the MNIST dataset.

Weight Matrix Ensemble

Instead of a weight matrix, we'll have a weight matrix ensemble or weight tensor $W$ of shape $(e,d,c)$ where:

$(e,)$ is the ensemble shape.
$d$ is the feature dimension. In the preprocessed MNIST dataset, this is $28\cdot28=784$, plus 1 if you are adding a column of 1s.
$c$ is the number of labels. In MNIST, this is 10.

To apply the corresponding ensemble of linear transformations to a feature matrix $X$ of shape (n, d), we can make use of broadcasting: the matrix product $X\cdot W$ will have shape (e, n, c), where for each $0\le i<e$: the matrix $(X\cdot W)_i$ will be the result of the matrix product $X\cdot W_i$.

Bias Vector Ensemble

If we do not add a column of 1's to the feature matrices, the we need to make use of a bias vector ensemble. This will be a tensor $\mathbf b$ of shape (e, c).

To add this to the matrix product $X\cdot W$ of shape (e, n, c), using broadcasting rules, we can reshape $\mathbf b$ to have shape (e, 1, c), and use broadcasted addition. Alternatively, you can make the shape of the tensor $\mathbf b$ (e, 1, c) already, so that you don't have to reshape before each addition.

Ensemble Dataloader

Given a tensor $X$ of shape $(n_1,\dotsc,n_l)$ and an index tensor $I$ of shape $(m_1,\dotsc,m_k)$ where all entries of $I$ are in the interval $[-n_1,n_1)$, the tensor $X[I]$ will have shape $(m_1,\dotsc,m_k,n_2,\dotsc,n_l)$, where for $(i_1,\dotsc,i_k)\in[m_1]\times\dotsb\times[m_k]$, we have $X[I][i_1,\dotsc,i_k]=X[I[i_1,\dotsc,i_k]]$.

That is, then tensor $X[I]$ is the composite $$ [m_1]\times\dotsb\times[m_k]\times[n_2]\times\dotsb\times[n_l] $$ $$ \xrightarrow{(i_1,\dotsc,i_k,j_2,\dotsc,j_l) \mapsto(I[i_1,\dotsc,i_k],j_2,\dotsc,j_l)} [-n_1,n_1)\times[n_2]\times\dotsb\times[n_l]\xrightarrow{X}K. $$ For further details, you can think of the first map as the Cartesian product of the maps $$ [m_1]\times\dotsb\times[m_k]\xrightarrow I[-n_1,n_1) $$ $$ \text{ and } [n_2]\times\dotsb\times[n_l]\xrightarrow{\mathrm{id}\text{ (identity)}} [n_2]\times\dotsb\times[n_l]. $$

Here, we want a dataloader that outputs minibatches of shape (ensemble_size, minibatch_size,). Therefore, we want not one vector of shuffled dataset indices, but a tensor $I$ of shape (e, n) such that for each index $0\le i<e$, the vector $I_i$ is an independent shuffle of the index vector $[n]$. Thus, we can't directly use one permutation. We turn to another method:

Generate a tensor $U$ of shape (e, n) of values drawn from $\mathscr U([0, 1))$, the uniform distribution on the interval $[0, 1)$.
We can let the index tensor $I$ be the argsort tensor along the last dimension, that is one such that for each ensemble index $0\le i<i$, we have $$ U_{i, I_{i, 0}}\le U_{i, I_{i, 1}} \le\dotsb\le U_{i, I_{i, n-1}}. $$

Selecting the Best Model

In a training procedure, performance can deteriorate after having reached a best score. A common reason for this is overfitting, which we'll investigate later. For this, it is best not to keep the model at hand at the end of the training, but the one with the best validation result.

Here's how to keep the model parameters with the best seen validation for each ensemble member: Let $E$ denote the set of ensemble indices. That is, if ensemble_shape=(n_1,...,n_l) then $E=[n_1]\times\dotsb\times[n_l]$. Moreover, let us assume that the greater the metric score, the better it views the model performance. Otherwise, please adjust accordingly.

We can initialize the following:
1. A collection of best scores $(s'_e:e\in E)$, initialized to a value at most the worst possible evaluation score. For example, if the metric is accuracy, these are the nonpositive numbers.
2. A collection of best parameters $(\theta'_e:e\in E)$, initialized arbitrarily.
At each validation:
1. We get a collection of new scores $(s_e:e\in E)$ and current parameters $(\theta_e:e\in E)$.
2. We want to update the best scores and parameters where $s_e$ is better than $s'_e$.
3. To account for floating point errors, we only think of $s_e$ as better than $s'_e$, if it is better by some improvement threshold $\delta>0$: $s_e > s'_e + \delta$: $$ s'_e\leftarrow s_e\text{ and }\theta'_e\leftarrow\theta_e \text{ for }e\in E\text{ such that }s_e > s'_e + \delta $$

Early Stopping

Keeping track of when we got an improved model gives rise to a new stopping condition: we can stop training if we got no improved model for a given number $C$ of training steps.

To this end, we can change the best model selector above as follows:

At initialization, also create a counter $c$ for training steps without improvement. Start this at 0.
At validation:
1. If we got improvement for some $e\in E$, then reset $c$ to 0.
2. Otherwise:
  1. Add to $c$ the number of training steps since the last validation.
  2. If $c\ge C$, then stop training.

Masking

For updating best scores and best parameters, we can use masking, wherein we index into a tensor with a Boolean tensor.

Let $[n_1]\times\dotsb\times[n_k]\xrightarrow tK$ be an arbitrary tensor and $[n_1]\times\dotsb\times[n_l]\xrightarrow b\{\mathtt{True}, \mathtt{False}\}$ a Boolean tensor for some $l\le k$. Let $\mathscr I=(I_j:0\le j<r)$ denote the collection of tuples $I_j=(i_1^{(j)},\dotsc,i_l^{(j)})\in[n_1]\times\dotsb\times[n_l]$ such that $b[i_1^{(j)},\dotsc,i_l^{(j)}]$. Then $a[b]$ is the composite $$ [r]\times[n_{l+1}]\times\dotsb\times[n_k] \xrightarrow{(j,i_{l+1},\dotsc,i_k)\mapsto(i_1^{(j)},\dotsc,i_l^{(j)},i_{l+1},\dotsc,i_k)} [n_1]\times\dotsb\times[n_k]\xrightarrow aK. $$ That is, in its first dimension, $a[b]$ lists the subtensors that $b$ selects.

In torch, we can get $r$ as b.sum() and $\mathscr I$ as b.nonzero().

Info

Follow this link for more information on masking:
https://numpy.org/doc/stable/user/basics.indexing.html#boolean-array-indexing

Grid Search

A simple hyperparameter search option that we'll use today is Grid Search. In it:

We select a finite number of choices we want to try out for a number of hyperparameters and
We evaluate each combination of the choices.

For example, in the lab today, we'll try out:

the learning rates $\{10^i:i=-2, -1.5,\dotsc,1\}$ and
the minibatch sizes $\{2^i:i=3,4,\dotsc,10\}$.

That is, we'll try out $7\cdot8=56$ hyperparameter configurations.