Data BAD | What Will it Take to Fix Benchmarking for NLU?

Nov 12, 2021

|

22 views

AI Coffee Break with Letitia

Details

The Coffee Bean explains and comments the sobering take of the paper "What Will it Take to Fix Benchmarking in Natural Language Understanding?" See more videos from Ms. Coffee Bean about natural language understanding: 📺 The road to NLU: https://youtube.com/playlist?list=PLpZBeKTZRGPMjF-Ob-NYjaTtewbMNXKcU ► Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏 donor, Dres. Trost GbR, Yannik Schneider Paper: 📜 Bowman, Samuel R., and George E. Dahl. "What Will it Take to Fix Benchmarking in Natural Language Understanding?." arXiv preprint arXiv:2104.02145 (2021). https://arxiv.org/abs/2104.02145 🔗 SuperGLUE: https://super.gluebenchmark.com/tasks 🔗 WiC: The Word-in-Context Dataset (English): https://pilehvar.github.io/wic/ Outline: 00:00 NLU Benchmarking – Motivation 01:04 How to measure NLU advances? 02:31 Why is NLU benchmarking broken? 04:43 What are the fixes? ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter: https://twitter.com/AICoffeeBreak Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research

Outline: 00:00 NLU Benchmarking – Motivation 01:04 How to measure NLU advances? 02:31 Why is NLU benchmarking broken? 04:43 What are the fixes?

Category: Research Paper

Category: Misc.

Comments

loading...

Reactions (0) | Note

📝 No reactions yet

Be the first one to share your thoughts!

Reactions(0)

Note

loading...

Recommended

SREcon15 - Case Study: Adopting SRE Principles at StackOverflow

USENIX | Apr 9, 2015

SREcon15 - Monitoring without Infrastructure @ Airbnb

USENIX | Apr 9, 2015

SREcon15 - Scaling Networks through Software

USENIX | Apr 9, 2015

SREcon15 - SRE Hiring

USENIX | Apr 9, 2015

SREcon15 - Incident Analysis

USENIX | Apr 13, 2015