Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.

Jul 13, 2021

|

46 views

AI Coffee Break with Letitia

Details

What are positional embeddings / encodings? ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ NEW (channel update): 🔥 Optionally, pay us a coffee to boost our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 📺 Transformer explained: https://youtu.be/FWFA4DGuzSc ► Outline: 00:00 What are positional embeddings? 03:39 Requirements for positional embeddings 04:23 Sines, cosines explained: The original solution from the “Attention is all you need” paper Paper 📄 Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in neural information processing systems, pp. 5998-6008. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Music 🎵 : Discovery Hit by Kevin MacLeod is licensed under a Creative Commons Attribution 4.0 licence. https://creativecommons.org/licenses/by/4.0/ Source: http://incompetech.com/music/royalty-free/index.html?isrc=USUAN1300023 Artist: http://incompetech.com/ --------------------------- 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research

00:00 What are positional embeddings? 03:39 Requirements for positional embeddings 04:23 Sines, cosines explained: The original solution from the “Attention is all you need” paper

Category: Research Talk

Comments

loading...

Reactions (0) | Note

📝 No reactions yet

Be the first one to share your thoughts!

Reactions(0)

Note

loading...

Recommended

ICAPS 2012: Session IIb on "Admissible Heuristics in Optimal Planning"

ICAPS | Aug 18, 2014

ICAPS 2012: Session IIa on "Special Track on Continuous Planning"

ICAPS | Aug 18, 2014

ICAPS 2012: Session IVa "Planning and Scheduling in the Real World"

ICAPS | Aug 18, 2014

ICAPS 2012: Session IVb on "Learning & Local Search in Planning"

ICAPS | Aug 18, 2014

ICAPS 2012: Session V with "Best Papers"

ICAPS | Aug 18, 2014