Self-Attention with Relative Position Representations – Paper explained

Self-Attention with Relative Position Representations – Paper explained

Aug 04, 2021
|
57 views
Details
We help you wrap your head around relative positional embeddings as they were first introduced in the β€œSelf-Attention with Relative Position Representations” paper. Related videos: πŸ“Ί Positional embeddings explained: https://youtu.be/1biZfFLPRSY πŸ“Ί Concatenated, learned positional encodings: https://youtu.be/M2ToEXF6Olw πŸ“Ί Transformer explained: https://youtu.be/FWFA4DGuzSc Papers πŸ“„: Shaw, Peter, Jakob Uszkoreit, and Ashish Vaswani. "Self-Attention with Relative Position Representations." In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464-468. 2018. https://arxiv.org/pdf/1803.02155.pdf Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in neural information processing systems, pp. 5998-6008. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Outline: 00:00 Relative positional representations 02:15 How do they work? 07:59 Benefits of relative vs. absolute positional encodings Music 🎡 : Holi Day Riddim - Konrad OldMoney β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€ πŸ”₯ Optionally, pay us a coffee to help with our Coffee Bean production! β˜• Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€ πŸ”— Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​

00:00 Relative positional representations 02:15 How do they work? 07:59 Benefits of relative vs. absolute positional encodings
Comments
loading...