BERTology meets Biology | Solving biological problems with Transformers

BERTology meets Biology | Solving biological problems with Transformers

Jan 01, 2022
|
27 views
Details
Transformers are meanwhile famous in natural language processing. New research applies the BERT Transformer architecture on amino-acid sequences and discovers biological properties of proteins! 🧬 Because why not? 😊 "BERTology Meets Biology" paper explained! β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€ πŸ”₯ Optionally, pay us a coffee to boost our Coffee Bean production! β˜• Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€β–€ Transformer explained by Ms. Coffee Bean: https://youtu.be/FWFA4DGuzSc What is probing? Ms. Coffee Bean explained: https://youtu.be/fL22NAtMNYo Outline: * 00:00 Solving biology with the Transformer * 01:08 Language BERT recap * 02:37 Biology Bootcamp * 05:43 BERT on biology πŸ“„ Paper: Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. "BERTology Meets Biology: Interpreting Attention in Protein Language Models." arXiv preprint arXiv:2006.15222 (2020). https://arxiv.org/pdf/2006.15222.pdf πŸ’»Code: https://github.com/salesforce/provis More code providing state of the art pre-trained models for proteins: https://github.com/agemagician/ProtTrans Scientific advisor for this video: Nils Trost Hungry for more? πŸ“š Check out @Yannic Kilcher's video on this: https://youtu.be/q6Kyvy1zLwQ πŸ”— Links: YouTube: https://www.youtube.com/AICoffeeBreak Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ #AICoffeeBreak #MsCoffeeBean #BERT #attention #Biology #MachineLearning #AI #research #Bioinformatics

00:00 Solving biology with the Transformer 01:08 Language BERT recap 02:37 Biology Bootcamp 05:43 BERT on biology
Comments
loading...