QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering.
Michihiro Yasunaga,Hongyu Ren,Antoine Bosselut,Percy Liang,Jure Leskovec +4 more
- 01 Jun 2021
- pp 535-546
601
TL;DR: This work proposes a new model, QA-GNN, which addresses the problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) through two key innovations: relevance scoring and joint reasoning.
read more
Abstract: The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG. Here we propose a new model, QA-GNN, which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to estimate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph-based message passing. We evaluate QA-GNN on the CommonsenseQA and OpenBookQA datasets, and show its improvement over existing LM and LM+KG models, as well as its capability to perform interpretable and structured reasoning, e.g., correctly handling negation in questions.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Foundation models for generalist medical artificial intelligence
Michael Moor,O. Banerjee,Zahra F.H. Abad,Harlan M. Krumholz,Jure Leskovec,Eric J. Topol,Pranav Rajpurkar +6 more
TL;DR: Generalist medical AI (GMAI) as mentioned in this paper is a new paradigm for medical AI, which is capable of carrying out a diverse set of tasks using very little or no task-specific labelled data.
Unifying Large Language Models and Knowledge Graphs: A Roadmap
TL;DR: The authors presented a forward-looking roadmap for the unification of LLMs and KGs, which consists of three general frameworks, namely, KG-enhanced LLMs, which incorporate KGs during the pre-training and inference phases of LLM, or for the purpose of enhancing understanding of the knowledge learned by LLMs; 2) LLM-augmented KGs that leverage LLMs for different KG tasks such as embedding, completion, construction, graph-to-text generation, and question answering; and 3) Synergized LLMs + KGs.
Proceedings Article
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models
Tianbao Xie,Chen Henry Wu,Peng Shi,Ruiqi Zhong,Torsten Scholak,Michihiro Yasunaga,Chien-Sheng Wu,Ming Zhong,Peng Yin,Sida Wang,Victor Zhong,Bailin Wang,Chengzu Li,Connor Boyle,Ansong Ni,Zhen Yao,Dragomir R. Radev,Caiming Xiong,Lingpeng Kong,Rui Zhang,Noah A. Smith,Luke Zettlemoyer,Tao Yu +22 more
- 16 Jan 2022
TL;DR: The U NIFIED SKG framework is proposed, which unifies 21 SKG tasks into a text-to-text format, aiming to promote systematic SKG research, instead of being exclu-sive to a single task, domain, or dataset.
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Shirui Pan,Linhao Luo,Yufei Wang,Chen Chen,Jiapu Wang,Xindong Wu +5 more
TL;DR: Unify large language models and knowledge graphs to enhance understanding and knowledge representation.
225
LinkBERT: Pretraining Language Models with Document Links
Michihiro Yasunaga,Jure Leskovec,Percy Liang +2 more
- 29 Mar 2022
TL;DR: This work proposes LinkBERT, an LM pretraining method that leverages links between documents that outperforms BERT on various downstream tasks across two domains: the general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain ( pretrained on PubMed with citation links).
References
•Posted Content
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
TL;DR: A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
81.7K
•Journal Article
Dropout: a simple way to prevent neural networks from overfitting
TL;DR: It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
•Proceedings Article
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe,Christian Szegedy +1 more
- 06 Jul 2015
TL;DR: Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
•Posted Content
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Yinhan Liu,Myle Ott,Naman Goyal,Jingfei Du,Mandar Joshi,Danqi Chen,Omer Levy,Michael Lewis,Luke Zettlemoyer,Veselin Stoyanov +9 more
TL;DR: It is found that BERT was significantly undertrained, and can match or exceed the performance of every model published after it, and the best model achieves state-of-the-art results on GLUE, RACE and SQuAD.
Graph Attention Networks
Petar Veličković,Guillem Cucurull,Arantxa Casanova,Adriana Romero,Pietro Liò,Yoshua Bengio +5 more
- 15 Feb 2018
TL;DR: Graph Attention Networks (GATs) as mentioned in this paper leverage masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations.