What is tokenization? How does it work? Tokenizers explained.

What is tokenization? How does it work? Tokenizers explained.

Mar 04, 2022
|
28 views
Details
What is tokenization? Ms. Coffee Bean explains tokenization (highly relevant for transformers nowadays) and highlights the importance of flexible tokenization. Enjoy the animations! πŸ‘‰ Our Veritasium Contest submission: https://youtu.be/a6sKYiGMuXc Watch and like it to support us! only the top 100 most viewed videos qualify for the contest. πŸ“Ί The Charformer: https://youtu.be/debgj24BAZE Thanks to our Patrons who support us in Tier 2, 3, 4: πŸ™ donor, Dres. Trost GbR β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€ πŸ”₯ Optionally, pay us a coffee to help with our Coffee Bean production! β˜• β–Ί Patreon: https://www.patreon.com/AICoffeeBreak β–Ί Ko-fi: https://ko-fi.com/aicoffeebreak β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€ Outline: 00:00 Warning 00:27 What is tokenization? 01:47 How does it work? 02:34 The problems 03:31 Β―\_(ツ)_/Β― ------------------------------------- πŸ”— Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​

00:00 Warning 00:27 What is tokenization? 01:47 How does it work? 02:34 The problems 03:31 Β―\_(ツ)_/Β―
Comments
loading...