Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.

Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.

Jul 13, 2021
|
45 views
Details
What are positional embeddings / encodings? ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ NEW (channel update): 🔥 Optionally, pay us a coffee to boost our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 📺 Transformer explained: https://youtu.be/FWFA4DGuzSc ► Outline: 00:00 What are positional embeddings? 03:39 Requirements for positional embeddings 04:23 Sines, cosines explained: The original solution from the “Attention is all you need” paper Paper 📄 Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in neural information processing systems, pp. 5998-6008. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Music 🎵 : Discovery Hit by Kevin MacLeod is licensed under a Creative Commons Attribution 4.0 licence. https://creativecommons.org/licenses/by/4.0/ Source: http://incompetech.com/music/royalty-free/index.html?isrc=USUAN1300023 Artist: http://incompetech.com/ --------------------------- 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​

00:00 What are positional embeddings? 03:39 Requirements for positional embeddings 04:23 Sines, cosines explained: The original solution from the “Attention is all you need” paper
Comments
loading...