Skip to content

7/27 BERT Models for Language Modelling and Supervised Tasks

Rotary Position Embedding

Another idea for relative positional encoding is to make a multiplicative change

\[ \mathop{\mathrm{softmax}}\left(\frac{QRK^T}{\sqrt d}\right)V \]

using some 4-tensor \(R\in\mathbf R^{s\times s\times d\times d}\). In Rotary Position Embedding (RoPE) [4], they use the fixed tensors

\[ R_{i,j}=\begin{pmatrix} \cos (j-i)\theta_1 & -\sin(j-i)\theta_1 & 0 & 0 & \cdots & 0 & 0 \\ \sin (j-i)\theta_1 & \cos (j-i) \theta_1 & 0 & 0 & \cdots & 0 & 0 \\ 0 & 0 & \cos (j-i)\theta_2 & -\sin(j-i)\theta_2 & \cdots & 0 & 0 \\ 0 & 0 & \sin (j-i)\theta_2 & \cos (j-i) \theta_2 & \cdots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & 0 & 0 & \cdots & \cos (j-i)\theta_{d/2} & -\sin(j-i)\theta_{d/2} \\ 0 & 0 & 0 & 0 & \cdots & \sin (j-i)\theta_{d/2} & \cos (j-i) \theta_{d/2} \end{pmatrix} \]

where

\[ \theta_k=10000^{-\frac{2(k-i)}{d}}\text{ for }k=1,\dotsc,\frac{d}{2} \]

(we assume \(d\) is even).

For an efficient implementation, that makes use of the sparsity of the tensor \(R\), see [1, Subsubsection 3.4.2].

Special Tokens

We discussed before that to make minibatches of sequences of varying length, we need to make use of padding tokens. In decoded sequences, they are usually denoted by [PAD]. We also discussed the unknown [UNK] token.

We also add a special token [BOS] in the beginning of a sequence. This is used to collect sequence-level information in supervised models and acts as a start signal for generative models. In Bidirectional Encoder Representations from Transformers (BERT) [2] models, to be introduced shortly, this is denoted by [CLS]. In generative models, we also use an end of sequence [EOS] token.

Subsequence (eg. in a question-answer pair) end is signalled by [SEP].

Introduction to pretraining

Last week, we introduced the transformer architecture. We discussed how the success of the transformer architecture stems on its robustness: you can train deep transformer models without much hyperparameter tuning, and they generalize well, that is they are resistant to overfitting.

Said generalization capability can be benefitted from by pretraining: In this procedure, before training our model on the task at hand, we imbue it with generic linguistic knowledge.

Pretraining BERT models

Today, we will discuss pretraining BERT models in detail. Tomorrow, we shall discuss pretraining of generative models.

First of all, pretraining is an unsupervised task: we are only given a sample from one distribution, and we want to model that distribution itself.

In the case of pretraining NLP models, the sample we have at hand is a big corpus of text. This can be obtained for example from Wikipedia or Common Crawl, or less transparent means.

Pretraining Subword Tokens

Let's first discuss training contextual token embeddings for the subword tokens.

In a training set, let \(X\) denote the tensor of shape \((b, s)\) of token ids we sample from the corpus, where \(b\) is a single batch dimension and \(s\) is the sequence dimension (in particular, we hide ensemble shape).

Masked Language Modelling (MLM)

The most popular subword pretraining method for BERT models is Masked Language Modelling (MLM).

  1. We randomly mask out some non-special tokens of \(X\). That is, we randomly sample a Boolean tensor \(B\) of shape \((b, s)\) such that if \(B[i, j]\) then \(X[i, j]\) is not a special token such as [CLS] or [PAD]. To sample \(B\), we use the Bernoulli distribution, that is each non-special token has the same probability of getting masked out. This is a hyperparameter, that is usually set to 10%-15%.
  2. We replace \(X[B]\) with a special token [MASK]. We want to train the model to learn the distribution of possible ways to fill in the masked tokens and get a correct sentence.
  3. To this end, we input the modified \(X\) to our model. We get an output tensor \(Z\) of shape \((b, s, d)\). We multiply this with a trainable matrix of shape \((d, v)\) where \(v\) is the vocabulary size.
  4. We get a tensor \(Y\) of shape \((b, s, v)\). If \(B[i, j]\), then \(Y[i, j]\) is understood as the collections of logits describing which token can be put in the given masked position with what probability.
  5. We get a loss value by calculating the cross-entropy of these probabilities with the token id that was masked out.

Pretraining the Sequence Token [CLS]

Now, we shall discuss methods to pretrain the contextual embedding of the token [CLS] sequences start with. This is supposed to represent sequence-level information. As sequences are often sentences, these embeddings are often referred to as sentence embeddings.

Next Sentence Prediction (NSP)

In the BERT paper [2], NSP is used to pretrain the contextual embedding of the sequence token [CLS]. This is also used to train contextual embeddings for yet another special token: [SEP] indicates separation in a sequence, for example between a question and an answer.

For the task, for each sequence in a minibatch, we select two sentences, A and B. We sample them in such a way that B follows A 50% of the time. The model receives sequences of the form [CLS] A [SEP] B, the output is transformed with a trainable matrix of shape \((d, 1)\), and the model is trained on the binary classification task to decide if B follows A.

References

[1] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo and Yunfeng Liu: RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing, vol. 568, pp. , 2024. doi: 10.1016/j.neucom.2023.127063 link

[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. link