Homework 1

It is general experience in Machine Learning that training multiple models and aggregating their predictions (eg. with averaging) oftentimes yields better performance than that of the individual models. We can already witness this effect. As least squares is a deterministic algorithm, what we can vary is what subset of the training set we fit a linear regression model to. This way, we can form an ensemble, that is a collection of models.

Recall that, with the 85%-15% train-test split of Abalone, as in class, the train and test sets have respectively $|\mathscr D_\mathrm{train}|=3550$ and $|\mathscr D_\mathrm{test}|=627$ entries.
To get a baseline for comparison, as we did in Notebook 6/16, record the Test MSE $\ell^\mathrm{single}$ of a single linear regression model, trained with least squares on the full train set $\mathscr D_\mathrm{train}$.
Now, we'll create ensembles of linear regression models trained on random subsets of $\mathscr D_\mathrm{train}$ of 3000 entries. Create a list to collect the Test MSE of the ensembles of various number of models.
Loop over the number $m$ of models you train, from 2 to 100:
1. Take $m$ random subsets $\mathscr D_0,\dotsc,\mathscr D_{m-1}\subseteq\mathscr D_\mathrm{train}$ of the train set of 3000 entries. You can use the same method as in Notebook 6/16: you are basically taking train splits of the train split.
2. On each random subset $\mathscr D_i$, fit a linear regression model, thus getting weight vectors $\mathbf w_i:0\le i<m$.
3. For each $0\le i<m$, we get predicted test targets $\mathbf z_\mathrm{test}^{(i)}=X_\mathrm{test}\mathbf w_i$.
4. We can form ensemble predictions as the average of the individual predictions: $$ \mathbf z_\mathrm{test}^\mathrm{ensemble}=\frac{1}{m}\sum_{i=0}^{m-1}\mathbf z_\mathrm{test}^{(i)}. $$ Calculate the MSE $\ell_m^\mathrm{ensemble}:=\frac{1}{|I_\mathrm{test}|}|\mathbf z_\mathrm{test}^\mathrm{ensemble}-\mathbf y_\mathrm{test}|^2$ using the ensemble predictions and record it in the ensemble MSE list.
Tip

Using vectorization (to be covered soon), you can solve the $m$ least squares problems at once. That is, you do not have to loop over $0\le i<m$ in your code. This is not required for this homework, but it will be good practice and it can speed up the computation greatly.
Compare the two results:
1. Use plt.plot to plot the numbers $m$ of subsets against the ensemble test MSE $\ell_m^\mathrm{ensemble}$.
2. Using plt.hlines, draw a horizontal line with y-value the test MSE $\ell^\mathrm{single}$ of a single linear regression trained on the full train set.
3. Show the plot.