7/16 Convolutional Neural Networks II

Yesterday, we noted that an affine map

\[ \mathbf R^{s\times d}\xrightarrow{A(X) = L(X)+B }\mathbf R^{s\times d'} \]

is equivariant under cyclical shifts of the matrix rows if and only if the following conditions hold:

1. The matrix \(B\) is constant along rows.
2. There exist matrices \(W_1,\dotsc,W_s\in\mathbf R^{d\times d'}\) such that for \(X\in\mathbf R^{s\times d}\), we have

\[ L(X)_i=\sum_{k=1}^sX_{(i + k\mod s)}W_k. \]

We can similarly form equivariant affine maps \(\mathbf R^{s_1\times\dotsb\times s_r\times d}\xrightarrow A\mathbf R^{s_1\times\dotsb\times s_r\times d'}\).

Locality and Kernels

Inspired by cats, we want our model to only consider local connections. This means to fix a kernel, that is selecting a collection of sequence index displacements that will get nonzero weights and disabling modular equivalence of indices.

To help understand convolution kernels, let's write up the exact formula for \(r=1\):

1. (Hyper)parameters:

   1. Sequence size \(s\)
   2. Kernel size \(k\)
   3. Dilation \(l\)
   4. Padding \(p_1,p_2\)
   5. Stride \(q\)

2. Displacements: \(I=\{0, l, \dotsc, l(k-1)\}\).

3. Base indices: \(J=\{-p_1,-p_1 + q,\dotsc,q(s'-1)\rfloor\}\) where \(s'=\lfloor\frac{s-1+p_2}{q}\rfloor\).
4. Then the matrices \(\{W_i\in\mathbf R^{d\times d'}:i=0,\dotsc,k-1\}\) determine a map \(\mathbf R^{s\times d}\xrightarrow L\mathbf R^{s'\times d'}\) where

\[ L(X)_j=\sum\{X_{li+j}W_i:i=0,\dotsc,k-1, 0\le li+j<m\}. \]

Pooling

Local data can be aggregated by pooling operations. These have similar formulas as the kernel-based convolutions above, but:

1. instead of summation, we usually use mean or maximum and
2. the linear maps \(W_i\) are identities.