Knowledge Graph Embeddings: Are Relation-Learning Models Learning Relations?

TL;DR: This study provides a more robust evaluation direction for future research on relation learning models, stressing that understanding why LP models reach certain performances is a crucial step towards explaining predicted relations.

Abstract: Link Prediction (LP) is the task of inferring relations between entities in a Knowledge Graph (KG). LP is difficult, due to the sparsity and incompleteness of real-world KGs. Recent advances in Machine Learning have led to a large and rapidly growing number of relation learning models, from the seminal work of Bordes et al. [4] to the recent model in [2]. Despite the flurry of papers in this area, just a few datasets and evaluation metrics have emerged as de facto benchmarking criteria. In our work, we question the effectiveness of these benchmarks in establishing the state-of-the-art. The use of unreliable benchmarking practices can have hidden ethical implications, as it may yield distorted evaluation results and overall lead the research community into adopting ineffective design choices. To this end, we consider key desiderata of a benchmark formulated as specific questions relevant to the LP task, and provide empirical evidence to answer those questions. Our analysis shows that existing datasets and metrics fall short in capturing a model’s capability of solving LP. Specifically, we show that a model can score very high by learning to predict facts about a small fraction of the entities in the training set. Our study provides a more robust evaluation direction for future research on relation learning models, stressing that understanding why LP models reach certain performances is a crucial step towards explaining predicted relations.