Automatic Title Generation for Text with Transformer Language Model (Research Paper Walkthrough)
May 01, 2021
Subscribe and Support!
#gpt2 #transformers #naturallanguageprocessing This paper presents a novel approach for title/headline generation based on pre-trained transformer language model GPT-2. A representative and an interesting title is invariably the most important aspect of any document. The title is the first, and sometimes the only part of an article or the document that the potential readers will see. While the title has to be catchy to grab the readers attention and entice them to read the full article, it should also accurately portray the content of the document so that readers are not misled by a catchy but not-quite-accurate title. ⏩ Abstract: In this paper, we propose a novel approach to Automatic Title Generation for a given text using a pre-trained Transformer Language Model GPT-2. The model proposes an unique approach of generating a pool of candidate titles and selecting an appropriate title among them which is then refined or de-noised to get the final title. The approach consists of a pipeline of three modules namely Generation, Selection and Refinement followed by a Scoring function. The Generation and Refinement modules are based on GPT-2, while the Selection module has a heuristic based approach. The model is able to generate accurate titles in spite of having a smaller corpus of relevant training data due to the fact that the natural language generation capabilities come from the pre-training while the model has to primarily learn task and corpus specific nuances. Additionally, Selection and Refinement modules ensure that the titles are representative of the given text and are semantically and syntactically accurate. We train our model for research paper abstracts from arXiv and evaluate it on three different test sets. Our pipeline shows promising results when evaluated on ROUGE and BLEU metrics against the test sets. In addition, we also perform human evaluation for validating the results generated by our proposed approach. Note: The title of this paper was generated automatically by our proposed algorithm from the abstract. Please feel free to share out the content and subscribe to my channel :) ⏩ Subscribe -
⏩ OUTLINE: 0:00 - Intro 01:19 - Generation Module 02:47 - Top-k Top-p Sampling Strategy 04:54 - Selection Module 06:13 - Refinement Module 07:30 - Title Generation Pipeline flow with example 10:02 - Relevancy Score 11:34 - Sample Results ⏩ Paper Title: Automatic Title Generation for Text with Pre-trained Transformer Language Model ⏩ Paper:
⏩ Author: Prakhar Mishra; Chaitali Diwan; Srinath Srinivasa; G. Srinivasaraghavan ⏩ Organisation: IIIT Bangalore, India ⏩ IMPORTANT LINKS Full Playlist on BERT usecases in NLP:
Full Playlist on Text Data Augmentation Techniques:
Full Playlist on Text Summarization:
Full Playlist on Machine Learning with Graphs:
Full Playlist on Evaluating NLG Systems:
Full Playlist on Query Expansion for Information Retrieval using NLP:
Full Playlist on Text Generation Evaluation Techniques:
********************************************* ⏩ Youtube -
⏩ Blog -
⏩ LinkedIn -
⏩ Medium -
⏩ GitHub -
⏩ Twitter -
********************************************* Please feel free to share out the content and subscribe to my channel :) ⏩ Subscribe -
Tools I use for making videos :) ⏩ iPad -
⏩ Apple Pencil -
⏩ GoodNotes -
#techviz #datascienceguy #research #finetuning #titlegeneration
Top-k Top-p Sampling Strategy
Title Generation Pipeline flow with example
Category: Research Paper
📝 No reactions yet
Be the first one to share your thoughts!
ICAPS 2012: Session IIIa on "Robot Path Planning"
| Aug 18, 2014
ICAPS 2012: DC Posters & System Demos
| Aug 18, 2014
ICAPS 2012: Session VIb on "Planning and Scheduling for Shipping and Transportation"
| Aug 18, 2014
ICAPS 2012: Awards & Best Dissertation Presentations
| Aug 19, 2014
ICAPS 2014: Miquel Ramírez on "Directed Fixed-Point Regression-Based Planning..."
| Sep 22, 2014
Crossminds Chrome Extension
Watch videos on arXiv
Subscribe to Our Newsletter
AI Startup Jobs
Terms & Conditions
© Provided by CrossCircles Inc.