TOD-BERT: Pre-trained Transformers for Task-Oriented Dialogue Systems | Research Paper Walkthrough
#bert #transformer #chatbot This paper from EMNLP2020 proposes a pre-training objective on existing BERT base architecture for task oriented dialogue systems/chatbots. Task-Oriented Chatbots are restricted in the variety of tasks that it can help a user with. They can help one accomplish clearly defined tasks like checking one’s account balance, making a reservation or finding the right recipe. Authors use 9 different human-human multi-turn dialogue datasets to pre-train their model with the modified objective function. Their approach outperforms existing approaches such as DialoGPT, GPT2, BERT, etc in this domain, authors verify it using downstream tasks like Intent Detection, Dialogue state tracking, Dialogue state prediction, Response selection. ⏩ Support by subscribing to the channel to not miss out on any video that i upload next - ⏩ Abstract: The underlying difference of linguistic patterns between general text and task-oriented dialogue makes existing pre-trained language models less useful in practice. In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling. We propose a contrastive objective function to simulate the response selection task. Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task-oriented dialogue applications, including intention recognition, dialogue state tracking, dialogue act prediction, and response selection. We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue. ⏩ OUTLINE: 0:00 - Background and Abstract 2:34 - Dataset for Task-oriented Dialogue systems 3:31 - TOD-BERT Overview and Data preparation 5:29 - Masked Language Modeling 6:38 - Response Contrastive Loss 11:44 - Downstream Tasks 13:56 - Sentence Representation comparison ⏩ Title: TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue ⏩ Link: ⏩ Author: Chien-Sheng Wu, Steven Hoi, Richard Socher, Caiming Xiong ⏩ Organisation: Salesforce Research ⏩ IMPORTANT LINKS BERT usecases in NLP - Contrastive Loss - Siamese Neural Networks - ******************************************** ⏩ Youtube - ⏩ Blog - ⏩ LinkedIn - ⏩ Medium - ⏩ GitHub - ********************************************* Tools I use for making videos :) ⏩ iPad - ⏩ Apple Pencil - ⏩ GoodNotes - ⏩ Microphone - About Me: I am Prakhar Mishra and this channel is my passion project. I am currently pursuing my MS (by research) in Data Science. I have an industry work-ex of 3 years in the field of Data Science and Machine Learning with a particular focus in Natural Langauge Processing (NLP). #techviz #datascienceguy #EMNLP2020 #researchpaper #transformers