It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

TL;DR: The SACA corpus incorporates linguistic semantic information into discourse relation recognition and includes 9546 adversative complex sentences annotated with internal semantic elements. The corpus follows the Penn Discourse Treebank (PDTB) annotation scheme, except for sense classification, which is based on the Chinese Discourse Treebank (CDTB).

TL;DR: This work proposes a novel Explainable Active Learning framework (XAL) for low-resource text classification, which aims to encourage classifiers to justify their inferences and delve into unlabeled data for which they cannot provide reasonable explanations.

TL;DR: The evaluation indicates that the best LLMs, despite not being trained on TS, perform comparably with state-of-the-art TS baselines, and certain LLMs demonstrate a greater range and diversity of edit operations.

TL;DR: This paper introduces an approach for distilling small models that generalize to new intent classes and domains using only a handful of labeled examples, and conducts experiments on public intent classification benchmarks, confirming the generalization ability of the small distilled models while having lower computational costs.

TL;DR: A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

TL;DR: It is found that BERT was significantly undertrained, and can match or exceed the performance of every model published after it, and the best model achieves state-of-the-art results on GLUE, RACE and SQuAD.

TL;DR: GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.