Adding vs. concatenating positional embeddings & Learned positional encodings

Jul 25, 2021

|

36 views

AI Coffee Break with Letitia

Details

When to add and when to concatenate positional embeddings? What are arguments for learning positional encodings? When to hand-craft them? Ms. Coffee Bean’s answers these questions in this video. ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔥 Optionally, help us boost our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 📺 Positional embeddings explained: https://youtu.be/1biZfFLPRSY 📺 Fourier Transform instead of attention: https://youtu.be/j7pWPdGEfMA 📺 Transformer explained: https://youtu.be/FWFA4DGuzSc Outline: 00:00 Concatenated vs. added positional embeddings 04:49 Learned positional embeddings 06:48 Ms. Coffee Bean deepest insight ever Papers 📄: Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in neural information processing systems, pp. 5998-6008. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Wang, Yu-An, and Yun-Nung Chen. "What do position embeddings learn? an empirical study of pre-trained language model positional encoding." arXiv preprint arXiv:2010.04903 (2020). https://arxiv.org/pdf/2010.04903.pdf Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020). https://arxiv.org/abs/2010.11929 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research

00:00 Concatenated vs. added positional embeddings 04:49 Learned positional embeddings 06:48 Ms. Coffee Bean deepest insight ever

Category: Research Talk

Comments

loading...

Reactions (0) | Note

📝 No reactions yet

Be the first one to share your thoughts!

Reactions(0)

Note

loading...

Recommended

ICAPS 2012: Session Ia on "Temporal Planning & Scheduling"

ICAPS | Aug 18, 2014

ICAPS 2012: Session Ib on "Combining Heuristics and Search for Optimal Planning"

ICAPS | Aug 18, 2014

ICAPS 2012: Session IIb on "Admissible Heuristics in Optimal Planning"

ICAPS | Aug 18, 2014

ICAPS 2012: Session IIa on "Special Track on Continuous Planning"

ICAPS | Aug 18, 2014

ICAPS 2012: Session IVa "Planning and Scheduling in the Real World"

ICAPS | Aug 18, 2014