7/15 Convolutional Neural Networks I
History
Convolutional Neural Networks (CNNs) are among the oldest studied ANNs.
In the 50s and 60s, Hubel and Wiesel found out that cat visual cortices contain neurons responding to small regions of the visual fields. These partial informations are then aggregated by other neurons. [1]
1979: Based on this idea, Kunihiko Fukushima constructed a proto-CNN called neocogitron [2]
1989: Using backpropagation, LeCun et al trained LeNet, a CNN for ZIP code recognition. It was successfully used by the UPS. [3]
2004: Oh and Jung wrote a GPU implementation for MLPs, achieving a speedup by the factor of 20. [4]
2011-12: Cireşan et al wrote a GPU implementation for CNNs, achieving a speedup by the factor of 60. [5] With this, they created a CNN ensemble [6], that won a German traffic sign recognition competition [7], surpassing the accuracy of human annotators.
2012: Krizhevsky et al created AlexNet [8], a similar GPU-based CNN [9], that won the 2012 ImageNet competition by a large margin. This event is viewed as a milestone for Deep Learning dominance.
Convolutions are the Cyclic Shift-Equivariant Maps
Spatial and Feature Dimensions
From now on, the input space \(\mathscr X\) will have extra, geometric structure. Right now, we'll have \(\mathscr X\subseteq\mathbf R^{s_1\times\dotsb\times s_r\times d}\). That is, a single input entry \(X\in\mathscr X\) will have shape \((s_1,\dotsc,s_r,d)\) for some \(r\in\mathbf Z_{>0}\). We'll call \(s_1,\dotsc,s_r\) the spatial dimensions, and \(d\) still the feature dimensions.
We let \(\mathbf s=[s_1]\times\dotsb\times[s_r]\). By default, we represent an entry \(X\in\mathbf R^{\mathbf S\times d}\) as a rank-\((r+1)\) tensor. We denote by \(\texttt{flatten}(X)\) the \((s_1\cdot\ldots\cdot s_r\cdot d)\)-dimensional vector we get by flattening its entries. We denote the reverse operation by \(\mathtt{unflatten}\).
For example:
- With \(r=1\), we're dealing with finite sequences of vector data, eg. sound.
- With \(r=2\), we're dealing with 2D data, eg. images. It can be grayscale with \(d=1\) or, for example, RGB-encoded with \(d=3\).
- With \(r=3\), we can process eg. 3D data or 2D videos.
Translation Action
We are interested in creating models \(\mathscr X\to\mathscr P(\mathscr Y)\) that are equivariant/invariant with respect to the translation, or more precisely, cyclical shift, action of \(G=\mathbf Z^{\times r}\) on \(\mathscr X\): for \(g=(k_1,\dotsc,k_r)\) and \(X\in\mathscr X\), the image \(Xg\) is defined by
where overline denotes the modular operation: \(\overline{i_1+k_1}=i_1+k_1\mod s_1,\dotsc,\overline{i_r+k_r}=i_r+k_r\mod s_r\).
Translation-Equivariant Affine Maps
A direct approach to getting an MLP $$ \mathbf R^{\mathbf s\times d} \xrightarrow{A^{(0)}}\mathbf R^{\mathbf s\times n_1}\xrightarrow a \mathbf R^{\mathbf s\times n_1}\xrightarrow{A^{(1)}}\dotsb \dotsb\xrightarrow a\mathbf R^{\mathbf s\times n_{L-1}} \xrightarrow{A^{(L)}}\mathbf R^{\mathbf s\times k} $$ be equivariant with respect to the translation action by \(G\) is to require that the affine maps \(\mathbf R^{\mathbf s\times n_l}\xrightarrow{A^{(l)}}\mathbf R^{\mathbf s\times n_{l+1}}\) are so.
Theorem. An affine map
is equivariant under cyclical shifts of the matrix rows if and only if the following conditions hold:
- The matrix \(B\) is constant along columns.
- There exist matrices \(W_1,\dotsc,W_s\in\mathbf R^{d\times d'}\) such that for \(X\in\mathbf R^{s\times d}\), we have
We can similarly form equivariant affine maps \(\mathbf R^{s_1\times\dotsb\times s_r\times d}\xrightarrow A\mathbf R^{s_1\times\dotsb\times s_r\times d'}\).
References
[1] David Hunter Hubel and Torsten Nils Wiesel: Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology, 1959, vol 148 (3), pp. 574--591. doi: https://doi.org/10.1113/jphysiol.1959.sp006308
[2] Kunihiko Fukushima: Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 1980, vol. 36, pp. 193--202. doi: 10.1007/BF00344251, link
[3] Yann LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel: Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1989, vol. 1 (4), pp. 541--551. doi: 10.1162/neco.1989.1.4.541 link
[4] Kyoung-Su Oh, Keechul Jung: GPU implementation of neural networks. Pattern Recognition, 2004, vol. 37 (6), pp. 1311--1314 doi: 10.1016/j.patcog.2004.01.013 link
[5] Dan C. Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella and Jürgen Schmidhuber: Flexible, High Performance Convolutional Neural Networks for Image Classification. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence (IJCAI), 2011, vol. 2, pp. 1237--1242. link
[6] Dan Cireşan, Ueli Meier, Jonathan Masci and Jürgen Schmidhuber: Multi-column deep neural network for traffic sign classification. Neural Networks, 2012, vol. 32, pp. 333--338. doi: 10.1016/j.neunet.2012.02.023 link
[8] Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton: ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 (NeurIPS 2012). link
Dataset References
[7] Johannes Stallkamp, Marc Schlipsing, Jan Salmen and Christian Igel: The German Traffic Sign Recognition Benchmark: A multi-class classification competition. The 2011 International Joint Conference on Neural Networks. doi: 10.1109/IJCNN.2011.6033395 link
[9] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei: * ImageNet Large Scale Visual Recognition Challenge*. International Journal of Computer Vision, 2015, vol. 115, pp. 211--252. doi: 10.1007/s11263-015-0816-y dataset link paper link