Time-aware Large Kernel Convolutions (ICML 2020)

ICML 2020

Time-aware Large Kernel Convolutions Vasileios Lioutas and Yuhong Guo ICML 2020 Abstract: To date, most state-of-the-art sequence modelling architectures use attention to build generative models for language based tasks. Some of these models use all the available sequence tokens to generate an attention distribution which results in time complexity of O(n^2). Alternatively, they utilize depthwise convolutions with softmax normalized kernels of size k acting as a limited-window self-attention, resulting in time complexity of O(k⋅n). In this paper, we introduce Time-aware Large Kernel (TaLK) Convolutions, a novel adaptive convolution operation that learns to predict the size of a summation kernel instead of using the fixed-sized kernel matrix. This method yields a time complexity of O(n), effectively making the sequence encoding process linear to the number of tokens. We evaluate the proposed method on large-scale standard machine translation, abstractive summarization and language modelling datasets and show that TaLK Convolutions constitute an efficient improvement over other attention/convolution based approaches. For more information, please visit: project page: https://github.com/lioutasb/TaLKConvolutions arXiv: https://arxiv.org/abs/2002.03184