Ryan Spring

TL;DR: This work synthesizes questions and answers from a synthetic corpus generated by an 8.3 billion parameter GPT-2 model and is able to train state of the art question answering networks on entirely model-generated data that achieve higher accuracy than when using the SQuAD1.1 training set questions alone.

TL;DR: In this article, the authors combine adaptive dropouts and randomized hashing for maximum inner product search to select the nodes with the highest activation efficiently, which reduces the overall computational cost of forward and back-propagation by operating on significantly fewer (sparse) nodes.

TL;DR: This paper proposes a new sampling scheme and an unbiased estimator that estimates the partition function accurately in sub-linear time and demonstrates the effectiveness of the proposed approach against other state-of-the-art estimation techniques including IS and the efficient variant of Gumbel-Max sampling.

TL;DR: This work proves that count-sketch optimization maintains the SGD convergence rate, while gracefully reducing memory usage for large-models, and demonstrates that the technique has the same performance as the full-sized baseline, while using significantly less space for the auxiliary variables.