Open AccessPosted Content
Learning from Task Descriptions
TL;DR: The authors proposed a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area, and instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks.
read more
Abstract: Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task. Moreover, the dataset's structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Posted Content
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions
Swaroop Mishra,Daniel Khashabi,Chitta Baral,Hannaneh Hajishirzi +3 more
- 18 Apr 2021
TL;DR: NATURAL INSTRUCTIONS as discussed by the authors ) is a dataset of instructions and task-specific input/output data, consisting of 61 distinct language instructions and about 600k task instances, and is used to evaluate existing state-of-the-art language models in addressing new tasks by few-shot prompting of GPT3 and fine-tuning BART.
40
•Posted Content
Towards General Purpose Vision Systems.
TL;DR: The authors proposed a task-agnostic vision-language system that accepts an image and a natural language task description and outputs bounding boxes, confidences, and text for a wide range of vision tasks such as classification, localization, question answering, captioning, and more.
6
•Posted Content
FLEX: Unifying Evaluation for Few-Shot NLP
TL;DR: The authors presented FLEX, the first benchmark, public leaderboard, and framework that provides unified, comprehensive measurement for few-shot NLP techniques, including meta-learning and prompt-based approaches.
5
•Posted Content
Cross-Policy Compliance Detection via Question Answering
TL;DR: The authors decompose policy compliance detection into question answering, where questions check whether the conditions stated in the policy apply to the scenario, and an expression tree combines the answers to obtain the label.
References
•Posted Content
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
TL;DR: A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
81.7K
•Posted Content
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TL;DR: This systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks and achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.
Multitask Learning
Rich Caruana
- 01 Jul 1997
TL;DR: Multi-task Learning (MTL) as mentioned in this paper is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias.
A unified architecture for natural language processing: deep neural networks with multitask learning
Ronan Collobert,Jason Weston +1 more
- 05 Jul 2008
TL;DR: This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
•Posted Content
SQuAD: 100,000+ Questions for Machine Comprehension of Text
TL;DR: The Stanford Question Answering Dataset (SQuAD) as mentioned in this paper is a reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage.
5.8K