FNet: Mixing Tokens with Fourier Transforms โ€“ Paper Explained

FNet: Mixing Tokens with Fourier Transforms โ€“ Paper Explained

May 29, 2021
|
68 views
Details
Ms. Coffee Bean explains and visualizes the paper "FNet: Mixing Tokens with Fourier Transforms" and what it all has to do with ๐Ÿฐ. Get some coffee and join Ms. Coffee Bean for this video! Discussed paper: ๐Ÿ“„ Lee-Thorp, J., Ainslie, J., Eckstein, I., & Ontanon, S. (2021). FNet: Mixing Tokens with Fourier Transforms. https://arxiv.org/pdf/2105.03824.pdf Referenced videos: ๐Ÿ“บ Ms. Coffee Bean explains the Transformer: https://youtu.be/FWFA4DGuzSc ๐Ÿ“บ ViLBERT (Vision and Language with Transformers): https://youtu.be/dd7nE4nbxN0 ๐Ÿ“บ ViT: An image is worth 16x16 words: https://youtu.be/DVoHvmww2lQ ๐Ÿ“บ Multimodality: https://youtu.be/P23EWdiPWDw Outline: * 00:00 Why attention is โ€œmehโ€ * 02:46 How does FNet work? * 05:28 How does Fourier help? * 08:15 FNet performance Other references: ๐Ÿ“„ Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., ... & Dosovitskiy, A. (2021). MLP-Mixer: An all-MLP Architecture for Vision. arXiv preprint arXiv:2105.01601. https://arxiv.org/abs/2105.01601 ๐Ÿ“„ Liu, Hanxiao; Dai, Zihang; So, David R.; Le, Quoc V. โ€œPay Attention to MLPsโ€ (2021) https://arxiv.org/abs/2105.08050 Music ๐ŸŽต : Cuttin It Close by DJ Freedem -------------------------------------------------- ๐Ÿ”— Links: YouTube: https://www.youtube.com/AICoffeeBreak Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #researchโ€‹

00:00 Why attention is โ€œmehโ€ 02:46 How does FNet work? 05:28 How does Fourier help? 08:15 FNet performance
Comments
loading...