Know What You Don't Know: Unanswerable Questions for SQuAD
Pranav Rajpurkar,Robin Jia,Percy Liang +2 more
- 11 Jun 2018
- Vol. 2, pp 784-789
2.3K
TL;DR: SQuADRUn as discussed by the authors is a new dataset that combines the existing Stanford Question Answering Dataset with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.
read more
Abstract: Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuADRUn, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuADRUn, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuADRUn is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQuADRUn. We release SQuADRUn to the community as the successor to SQuAD.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Posted Content
TernaryBERT: Distillation-aware Ultra-low Bit BERT.
TL;DR: TernaryBERT as discussed by the authors uses both approximation-based and loss-aware ternarization methods and empirically investigates the ternarisation granularity of different parts of BERT.
9
Intelligent Voice Instructor-Assistant System for Collaborative and Interactive Classes
Matthew Baker,Xiaohui Hu,Gennaro De Luca,Yinong Chen +3 more
- 24 Mar 2021
TL;DR: A prototype of an intelligent voice instructor-assistant system for supporting large classes, in which AmazonWeb Services, Alexa Voice Services, and self-developed services are used.
Comprehensive Multi-Dataset Evaluation of Reading Comprehension
Dheeru Dua,Ananth Gottumukkala,Alon Talmor,Sameer Singh,Matt Gardner +4 more
- 01 Nov 2019
TL;DR: An evaluation server, ORB, is presented, that reports performance on seven diverse reading comprehension datasets, encouraging and facilitating testing a single model’s capability in understanding a wide variety of reading phenomena.
•Proceedings Article
Decomposing Complex Questions Makes Multi-Hop QA Easier and More Interpretable
Ruiliu Fu,Han Wang,Xuejun Zhang,Jun Zhou,Yonghong Yan +4 more
- 26 Oct 2021
TL;DR: This paper proposed Relation Extractor-Reader and Comparator (RERC), a three-stage framework based on complex question decomposition, which achieved the state-of-the-art performance on the 2WikiMultiHopQA dataset.
9
A Fine-grained Interpretability Evaluation Benchmark for Neural NLP
Lijie Wang,Yaozong Shen,Shuyuan Peng,Shuai Zhang,Xinyan Xiao,Hao Líu,Hongxuan Tang,Ying Chen,Hua Wang,Haifeng Wang +9 more
- 01 Jan 2022
TL;DR: A novel benchmark for evaluating the interpretability of neural NLP models and saliency methods. It includes three tasks, English and Chinese data, token-level rationales, and a new metric for uniform evaluation.
References
Deep contextualized word representations
Matthew E. Peters,Mark Neumann,Mohit Iyyer,Matt Gardner,Christopher Clark,Kenton Lee,Luke Zettlemoyer +6 more
- 15 Feb 2018
TL;DR: This paper introduced a new type of deep contextualized word representation that models both complex characteristics of word use (e.g., syntax and semantics), and how these uses vary across linguistic contexts (i.e., to model polysemy).
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Pranav Rajpurkar,Jian Zhang,Konstantin Lopyrev,Percy Liang +3 more
- 16 Jun 2016
TL;DR: The Stanford Question Answering Dataset (SQuAD) as mentioned in this paper is a reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage.
6.3K
•Posted Content
SQuAD: 100,000+ Questions for Machine Comprehension of Text
TL;DR: The Stanford Question Answering Dataset (SQuAD) as mentioned in this paper is a reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage.
5.8K
A large annotated corpus for learning natural language inference
Samuel R. Bowman,Gabor Angeli,Christopher Potts,Christopher D. Manning +3 more
- 21 Aug 2015
TL;DR: The Stanford Natural Language Inference (SNLI) corpus as discussed by the authors is a large-scale collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning.
•Proceedings Article
Teaching machines to read and comprehend
Karl Moritz Hermann,Tomáš Kočiský,Edward Grefenstette,Lasse Espeholt,Will Kay,Mustafa Suleyman,Phil Blunsom +6 more
- 07 Dec 2015
TL;DR: A new methodology is defined that resolves this bottleneck and provides large scale supervised reading comprehension data that allows a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure to be developed.