6/15 Basics of Supervised Learning

Course Logistics: General Information

Instructor

Office Hours

By appointment, online or in person. Write or tell me if you want to set up a meeting.

Class Website

You can access class content on the class website, and the class repository.

To gain access to the latter, please write me an email with your codeberg account; if you don't have one, you can create one on the Codeberg website.

Info

I am not hosting the course content on GitHub as I try to avoid giving away my data, especially to big companies such as Microsoft. Follow this link to read more on the issue: Software Freedom Conservancy: Give Up GitHub!

Although all necessary class material will be posted on the website, there are at least 4 good reasons to use the class repository:

I will send out announcements, for example when I publish new homework assignments, on the repository.
I will upload homework solutions, not to be made public, to the repository.
Using issues is a good way to discuss course content.
It is easiest to download the latest versions of the lab notebooks by a git pull.

Classes: Lectures, Labs at Home

Classes will be lectures, where I discuss both the theoretical and the practical side of Deep Learning. The coding portion is taught in lab notebooks. You are very welcome to work in groups! Lab notebooks without and with solutions will be posted on the class repository and the class website.

You can always ask questions on the notebooks, both in class and at other times!

Work Environment: Local or Online

You can either work on the lab notebooks and the homeworks in a local environment or online. For setting up a local environment, see the Installation guide. After installation, you can view the notebooks on Jupyter Lab, which you can start in a browser tab by typing

jupyter lab

in a terminal, or a coding IDE (Integrated Development Environment) such as VS Code. Note that the latter may not work in some configurations; in that case, try Jupyter Lab.

If you want to run your code or a notebook online, see Google Colab

Warning

Make sure to download the notebooks and open them from Jupyter Lab/VS Code/Google Colab/something else you use. That is, if you click a notebook, and you see code text in your browser instead of a save file popup, make sure to save the raw file to the file name with ipynb extension seen on the notebook page. If you have any trouble opening a notebook, feel free to write a Codeberg issue, write me an email, or ask me in person.

Highly Recommended Python Exercises

Learning Exercises on Exercism

To make sure that you know the basic concepts of Python, it is highly recommended to solve at least the learning exercises on Exercism: Python Key Concepts on Exercism.

Practice Exercises on Genepy

For more and less wordy practice exercises, I highly recommend taking a look at Genepy

Coursework: Homework, Written Presentation, Oral Presentation

Your total score will be made up of three portions: homework, written presentation, oral presentation. You can see all rules in the Syllabus. I shall discuss homeworks on Tuesday and final project presentations on Wednesday.

Using AI Assistants

In classwork such as a homework or the presentation, if you use an AI assistant, please include in your submission the model or models you used (eg. ChatGPT 5.5, Gemini 3.5 Flash).

Also, be very careful when making use of the output of AI assistants:

They are prone to hallucinate. Therefore, make sure to verify any fact they claim via trustworthy sources, such as academic publications or coding documentation. Now that tool use is getting integrated into AI assistants, you can ask them to cite their sources.
They may misunderstand homework instructions. Therefore, if you use their code, you have to make sure that the code is doing what it is supposed to. To get more success, I recommend giving as much context, eg. in code comments, as possible. Please keep the comments, either by you or the AI assistant, in your homework submissions.

Info

I recommend using AI assistants the privacy policy of which clearly states that your conversations will not be shared with anyone, in any form.

There are providers specializing in such service such as duck.ai or Proton Lumo. By the way, they respectively have DuckDuckGo: a private search engine, and Proton Mail, a private email provider, both of which I'm a happy everyday user of.

As an alternative, certain subscriptions of mainstream providers may offer privacy. For example, if you use ChatGPT with a Team subscription, OpenAI models will not be trained on your usage data: Enterprise privacy at OpenAI. 2 people can make a team, you can ask a friend. This way, you can access state of the art models, while not giving away your usage data to big companies to make profit of.

Supervised Learning: Basic Notions

For most of the course, we shall be dealing with supervised learning: We are given two sets $\mathscr X, \mathscr Y$ and a sequence of pairs $\mathscr D=((x_i, y_i)\in\mathscr X\times\mathscr Y: i\in I)$. We want to predict the values $y_i$ from the $x_i$.

We usually refer to $\mathscr D$ as the dataset. Note that it's not actually a subset of $\mathscr X\times\mathscr Y$ as you may have $(x_i,y_i)=(x_{i'},y_{i'})$ for some $i\ne i'$.

Input Features

The set $\mathscr X$ is called the (input) feature space. In our first examples, we will have $\mathscr X\subseteq\mathbf R^d$ for some $d$. The entries $x\in\mathscr X$ are (input) feature vectors. The components $x_i$ are called features.

Note

We will see later that the power of Deep Neural Network comes from the ability to learn progressively more and more refined internal (hidden) representations of the data. These internal representations are also vectors, their components are also called features. This is why it will be important later on to express whether a feature is an input or a hidden feature.

Targets

The set $\mathscr Y$ is called the target space. The entries $y\in\mathscr Y$ are targets. Usually, we have one of the following two options:

Regression: We have $\mathscr Y\subseteq\mathbf R^k$ for some $k$. In this case, we say that we are dealing with a regression task. For example, in the Abalone dataset that we'll use for introduction, we are trying to predict the age of abalone (a type of mollusc) from physical measurements.
Classification: We have $|\mathscr Y|<\infty$. Then we are dealing with a classification task. The entries of $\mathscr Y$ are called classes or labels. For example, in the MNIST dataset that's going to often recur during the course, we are trying to tell a written digit from its pixels as an image.

Datasets as Samples from a Distribution

In the language of Probability Theory, the sets $\mathscr X$ and $\mathscr Y$ are the sets of outcomes of random variables $X$ and $Y$, and the dataset $\mathscr D$ is a collection of samples from the joint distribution $(X, Y)$ of $X$ and $Y$.

Prediction

By predicting the $y_i$ from the $x_i$, we can mean at least the following two options:

Predict target values: Given $x_i$, one can try to directly approximate the corresponding $y_i$. That is, we want to create a function $\mathscr X\xrightarrow f\mathscr Y$ such that $f(x_i)$ is close to $y_i$ for all $i\in I$. This is the approach we'll take for regression.
Predict target marginal distributions: Given $x_i$, one can also try to approximate the marginal distribution $Y|X=x_i$. This is what we'll do in the classification case. That is, for each $x\in\mathscr X$ and $y\in\mathscr Y$, we'll try to approximate the probability $\mathbf P(Y=y|X=x)$. Or put in a fancier way, we'll want to create a function $\mathscr X\xrightarrow f\mathscr P(\mathscr Y)$ where $\mathscr P(\mathscr Y)$ denotes the set of probability distributions on the set $\mathscr Y$.

Parametric Models

In machine learning, the function $\mathscr X\xrightarrow f\mathscr Y\text{ or }\mathscr P(\mathscr Y)$ will be a function with many parameters $\theta$. Therefore, we'll usually write $f(x)=f(x;\theta)\text{ or }f_\theta(x)$.

For example, in our first experiment, we'll let $f_\theta$ be an affine transformation: $$ f_\theta(\mathbf x)=\mathbf x^TW+\mathbf b^T\text{ where }W\in\mathbf R^{d\times k}\text{ and }\mathbf b\in\mathbf R^k. $$ We call $W$ the weight matrix and $\mathbf b$ the bias vector.