7/21 Tokenization and Embeddings

Today, we'll take our first dip in Natural Language Processing (NLP).

The IMDB Dataset

The IMDB Dataset [1] has

1. movie review texts as inputs and
2. whether the review is positive or negative as labels.

That is, our input space \(\mathscr X\) is that of texts. We need to preprocess this so that we can feed it to a model.

We'll do this in a three-stage process. First, we'll split the text into atomic constituents, called tokens. Then, we'll assign each unique token a trainable vector. Finally, we process the vector sequences we get.

Today, for our first models, we disregard sequence order. That is, our inputs will be sets.

Sequences as Input

We denote by \(\mathscr X_0\) an elementwise feature space, which can be either:

1. a discrete set: \(\mathscr X_0\cong[n]\), sometimes called a vocabulary, or
2. a subset of a Euclidean space: \(\mathscr X_0\subseteq\mathbf R^d\).

Then the feature space is the set \(L\mathscr X_0\) of finite sequences of entries in \(\mathscr X_0\). We denote by \(L_m\mathscr X_0\subseteq L\mathscr X_0\) the subset of finite sequences of \(m\) elements.

The subset \(L_m\mathscr X_0\) has an action by the permutation group \(S_m\).

Text to Vectors: Tokenization

Corpus, Texts, Tokens

Let us first fix notation:

1. We are given a corpus, that is a sequence of texts (documents) \(D=(d_1,\dotsc,d_n)\). In the present case, the texts are the reviews.

2. We decompose each text \(t_i\) as a sequence of tokens (terms) \(d=(t_1^i,\dotsc,t_{l_i}^i)\).

   1. We are free to decide what we term a token:

      1. They can be words. This is what we will do today.
      2. They can be \(n\)-grams, that is sequences of consecutive words of a given number \(n\) (note that the \(n\) in a gram is different from the \(n\) for dataset length). For example a bigram is a pair of consecutive words.

      3. If we can keep track of sequence order, we can use subword tokenization methods. There are data-driven methods that, given a text corpus, try to split it up to subwords such that

         1. The number of distinct subwords, the vocabulary size, is given as a hyperparameter, and
         2. The distribution of subword counts is as balanced as possible.

         If interested, see [2] for a summary of tokenization algorithms.

   2. Note that the texts have varying numbers of tokens. This is a technical issue we will need to be mindful of later.

Vocabulary

The vocabulary is the set

$$ V=\{t_i^j:1\le i\le n,\,1\le j\le l_i\} $$ of all tokens appearing in the corpus. For \(t\in V\) and \(1\le i\le n\), we will write \(t\in d_i\) if there exists \(1\le j\le l_i\) such that \(t_j^i=t\).

Token IDs

We can give the vocabulary an ordering: $$ V=\{t_1,\dotsc,t_v\} $$

The actual process of tokenization is the assignment of each token \(t_i\) its index \(i\in[v]\). Thus, we can transform a document \(d\) to a sequence \(\mathbf t\in L[v]\) of token indices.

Tokenization is a preprocessing operation. That is, from now on, we will usually assume it done.

Token Embeddings

Now that we transformed each token \(t\in V\) to a token index \(i\in[v]\), we can assign it a trainable token embedding vector \(\mathbf e_i\in\mathbf R^d\). We can aggregate these into the token embedding parameter matrix \(E=\mathbf R^{v\times d}\).

This transforms a tokenized sequence \(\mathbf t\in L_s[v]\) into a sequence of vectors \(X=E[\mathbf t]\in\mathbf R^{s\times d}\).

Pooling

Our permutation-equivariant model outputs a sequence of \(k\)-dimensional vectors. To get sequence-level output, we need to aggregate this with a pooling layer. Just like with CNN's, we usually use the mean or max operations, componentwise.

Therefore, we can transform a sequence of token embedding vectors \(X\in\mathbf R^{s\times d}\) to a sequence of document vectors \(\mathrm{pool}(X)\in\mathbf R^d\). We can then postprocess this using a model expecting dense feature vectors such as logistic regression or MLP.

References

[2] Hugging Face, Summary of the tokenizers. link Last opened on July 21, 2026.

Dataset References

[1] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts: Learning Word Vectors for Sentiment Analysis, 2011. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL 2011), pages 142--150. link