FoNet: A Memory-Efficient Fourier-Based Orthogonal Network for Object Recognition

CVPR 2020

Authors: Feng Wei, Uyen Trang Nguyen, Hui Jiang Description: The memory consumption of most Convolutional Neural Network (CNN) architectures grows rapidly with the increasing depth of the network, which is a major constraint for efficient network training and inference on modern GPUs with limited memory. Several studies show that the feature maps (as generated after the convolutional layers) are the big bottleneck in this memory problem. Often, these feature maps mimic natural photographs in the sense that their energy is concentrated in the spectral domain. In this paper, we propose a \underline{F}ourier-based \underline{O}rthogonal \underline{Net}work (FoNet) that incorporates orthogonal representations and performs both the convolution and the activation operations in the spectral domain to achieve memory reduction. The performance of our FoNet is evaluated on four standard object recognition benchmarks (i.e., MNIST, CIFAR-10, SVHN, and ImageNet), and compared with four state-of-the-art implementations (i.e., LeNet, AlexNet, VGG, and DenseNet). Encouragingly, FoNet is able to reduce memory consumption by about 60\% without significant loss of performance for all tested network architectures.