Homework 10

Recall that in Notebook 0409, we trained a Categorical Deep Set model on the binary classification task of the subset of the dair-ai/emotion dataset on tweets labeled as either sadness or joy. We even trained a model that used 1-dimensional word vectors succesfully, which we then were able to interpret: we could see that the values of the 1-dimensional word vectors correlate with how joyful (positive) or sad (negative) they are.

In this homework, we shall train a Categorical Deep Set on the full dataset with 6 labels sadness, joy, love, anger, fear, and surprise. We will be able to adapt our interpretation methods to this setting.

  1. Load and split the dataset. You can do this similar to Notebook 0409, just don't filter it based on the labels.
    1. You can find the list of labels, in the same order as the IDs used in the "label" keys of the dataset entries, as follows:
      1. Take the features attribute of the dataset. This is a dictionary.
      2. Take the value of the above dictionary at the "label" key.
      3. Take the names attribute of the above object.
    2. Using enumerate, print the label ID-label pairs used in the full dataset.
  2. Preprocess the dataset. You can do this similar to Notebook 0409, just use the 64-bit integer datatype to store labels instead of the 32-bit floats.
    1. To save time when you are experimenting, I suggest saving the preprocessed dataset splits.
    2. You should also get the word ID to word assignments either in the form of a list or a dictionary. Note that this is also done in Notebook 0409.
  3. Similar to Notebook 0409, train a categorical deep set model on the dataset. It should use 10-dimensional word vectors and no hidden dimensions. Note that in this case, we have 6 classes. Therefore, you need to use non-binary accuracy and cross-entropy for metric and loss functions, respectively. These are implemented in Notebook 0321.
  4. Note that we used population-based training, thus we have multiple models at our disposal. Let's interpret the best one.
    1. At the "validation metric" key of the log dictionary output by the training function, you can find evaluation metrics of the population members along training. It should have shape (evaluations, population_num). In particular, its last row contains validation metrics for the model in its present state.
    2. Based on the validation metrics, get the ID of the current best population member.
    3. Calling the state_dict method of the model, you can get a dictionary with values the following parameter tensors:
      1. Word vectors, of shape (population_num, vocabulary_size, embedding_dim),
      2. Linear classifier bias, of shape (population_num, label_num), and
      3. Linear classifier weight, of shape (population_num, embedding_dim, label_num).
    4. Taking the subtensors of the above at the best model ID along the leftmost dimension, we get the following:
      1. A word vector matrix \(E\),
      2. A classifier bias vector \(\mathbf b\), and
      3. A classifier weight matrix \(W\).

  5. Via the tensors you got, given a document \(D=(i_1,\dotsc,i_k)\) where the \(i_j\) are word indices, the class logits are gotten as

    \[ \mathbf b + \frac{1}{k}\left(\sum_{j=1}^kE_{i_j}\right)W =\mathbf b+\sum_{j=1}^k\frac{1}{k}E_{i_j}W. \]

    1. Note that on the right, we see that the class logits are a sum of the following:

      1. The bias vector \(\mathbf b\), and
      2. The logit contributions of individual words \(\frac{1}{k}E_{i_j}W\).

    2. This opens up an interpretation opportunity!

      1. The product \(EW\) of shape (vocabulary_size, label_num) gives logit contributions for each word - label pair.

      2. Loop over the label indices \(j=0,\dotsc,\mathtt{label\_num}-1\), and for each \(j\):

        1. Print the corresponding label name.
        2. Get the indices of the ten largest values of the \(j\)-th column of \(EW\).
        3. Print the words corresponding to these 10 indices.