Katherine Lee
26 Papers
109 Citations
Katherine Lee is an academic researcher from Google. The author has contributed to research in topics: Computer science & Language model. The author has an hindex of 8, co-authored 9 publications. Previous affiliations of Katherine Lee include Princeton University.
Chat about Author
Papers
•Posted Content
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TL;DR: This systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks and achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.
Journal Article
PaLM: Scaling Language Modeling with Pathways
Aakanksha Chowdhery,Sharan Narang,Jacob Devlin,Maarten Bosma,Gaurav Mishra,Adam Roberts,Paul Barham,Hyung Won Chung,Charles Sutton,Sebastian Gehrmann,Parker Schuh,Kensen Shi,Sasha Tsvyashchenko,Joshua Maynez,Abhishek Rao,Parker Barnes,Yi Tay,Noam Shazeer,Velu Prabhakaran,Emily Reif,Nan Du,B. C. Hutchinson,Reiner Pope,James Bradbury,Jacob Austin,Michael Isard,Guy Gur-Ari,Peng Yin,Toju Duke,Anselm Levskaya,Sanjay Ghemawat,Sunipa Dev,Henryk Michalewski,Xavier Garcia,Vedant Misra,Kevin Robinson,L Fedus,Denny Zhou,Daphne Ippolito,David Luan,Hyeontaek Lim,Barret Zoph,Alexander Spiridonov,Ryan Sepassi,David Dohan,Shivani Agrawal,Mark Omernick,Andrew M. Dai,Thanumalayan Sankaranarayana Pillai,Marie Pellat,Aitor Lewkowycz,Erica Oliveira Moreira,Rewon Child,Oleksandr Polozov,Katherine Lee,Zong Tuan Zhou,Xuezhi Wang,Brennan Saeta,Mark Díaz,Orhan Firat,M. Catasta,Jason Loh Seong Wei,Kathleen S. Meier-Hellstern,Douglas Eck,Jeffrey Dean,Slav Petrov,Noah Fiedel +66 more
TL;DR: A 540-billion parameter, densely activated, Transformer language model, which is called PaLM achieves breakthrough performance, outperforming the state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark.
•Posted Content
Extracting Training Data from Large Language Models
Nicholas Carlini,Florian Tramèr,Eric Wallace,Matthew Jagielski,Ariel Herbert-Voss,Katherine Lee,Adam Roberts,Tom B. Brown,Dawn Song,Úlfar Erlingsson,Alina Oprea,Colin Raffel +11 more
TL;DR: This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model, and finds that larger models are more vulnerable than smaller models.
PaLM 2 Technical Report
Rohan Anil,Andrew M. Dai,Orhan Firat,Melvin George Johnson,Dmitry Lepikhin,Alexandre Passos,Siamak Shakeri,Emanuel Taropa,Paige Bailey,Zhi Chen,Eric Chu,Jonathan H. Clark,Laurent El Shafey,Yanping Huang,Kathleen S. Meier-Hellstern,Gaurav Mishra,Erica Oliveira Moreira,Mark Omernick,Kevin Robinson,Sebastian Ruder,Yi Tay,Kefan Xiao,Yuanzhong Xu,Yujing Zhang,Gustavo Hernandez-Abrego,Junwhan Ahn,Jacob Austin,Paul Barham,Jan A. Botha,James Bradbury,Siddhartha Brahma,Kevin Michael Brooks,M. Catasta,Yongzhou Cheng,Colin Cherry,Christopher A. Choquette-Choo,Aakanksha Chowdhery,C Crepy,Shachi Dave,Mostafa Dehghani,Sunipa Dev,Jacob Devlin,M. D'iaz,Nan Du,Ethan Dyer,Vladimir Feinberg,Fan Feng,Markus Freitag,Xavier Garcia,Sebastian Gehrmann,Guy Gur-Ari,Steven Hand,Hadi Hashemi,Le Hou,Joshua Howland,Anren Hu,Jeffrey Hui,Jeremy Scott Hurwitz,Michael Isard,Abe Ittycheriah,Matthew Jagielski,Wenhao Jia,Kathleen Kenealy,Maxim Krikun,Sneha Kudugunta,Katherine Lee,Benjamin N. Lee,Eric Li,M. Li,Wei Li,Yaguang Li,Jian Li,Hyeontaek Lim,Han Lin,Zhong-Zhong Liu,Frederick Liu,Marcello Maggioni,Aroma Mahendru,Joshua Maynez,Vedant Misra,Maysam Moussalem,Zachary Nado,John Nham,Eric Ni,Andrew Nystrom,Alicia Parrish,Marie Pellat,Martin Polacek,Alex Polozov,Reiner Pope,Siyuan Qiao,Emily Reif,Parker Riley,Alexandra Ros,Aurko Roy,Brennan Saeta,Rajkumar Samuel,Renee Shelby,Ambrose Jay Slone,Daniel Smilkov,David R. So,Daniela Sohn,Simon Tokumine,Vijay K. Vasudevan,Kiran Vodrahalli,Xuezhi Wang,Pidong Wang,Tao Wang,John Wieting,Yuhuai Wu,Ke Xu,Yu Yu Xu,Lin Wu Xue,Pengcheng Yin,Jia Yu,Biao Zhang,Steven X.F. Zheng,Ce Zheng,Wei Zhou,Denny Zhou,Slav Petrov,Yonghui Wu +121 more
TL;DR: The PaLM 2 model as mentioned in this paper is a Transformer-based model trained using a mixture of objectives, which has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM.
708
•Posted Content
WT5?! Training Text-to-Text Models to Explain their Predictions.
TL;DR: This paper uses the text-to-text framework proposed by Raffel et al. (2019) to train language models to output a natural text explanation alongside their prediction, and shows that this approach not only obtains state-of-the-art results on explainability benchmarks, but also permits learning from a limited set of labeled explanations and transferring rationalization abilities across datasets.
220