Open AccessPosted Content
Using Sparse Semantic Embeddings Learned from Multimodal Text and Image Data to Model Human Conceptual Knowledge
TL;DR: This paper combines multimodal information from both text and image-based representations derived from state-of-the-art distributional models to produce sparse, interpretable vectors using Joint Non-Negative Sparse Embedding and demonstrates their ability to predict interpretable linguistic descriptions of human ground-truth semantic knowledge.
read more
Abstract: Distributional models provide a convenient way to model semantics using dense embedding spaces derived from unsupervised learning algorithms However, the dimensions of dense embedding spaces are not designed to resemble human semantic knowledge Moreover, embeddings are often built from a single source of information (typically text data), even though neurocognitive research suggests that semantics is deeply linked to both language and perception In this paper, we combine multimodal information from both text and image-based representations derived from state-of-the-art distributional models to produce sparse, interpretable vectors using Joint Non-Negative Sparse Embedding Through in-depth analyses comparing these sparse models to human-derived behavioural and neuroimaging data, we demonstrate their ability to predict interpretable linguistic descriptions of human ground-truth semantic knowledge
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Decoding semantic representations in mind and brain
TL;DR: This article reviewed cognitive theories of semantic representation and their neural instantiation, and then considered contemporary approaches to neural decoding and assessed which types of representation each can possibly detect, and identified crucial links between cognitive theory, data collection, and analysis that can help to better connect neuroimaging to mechanistic theory of semantic cognition.
33
Learning semantic sentence representations from visually grounded language without lexical knowledge.
Danny Merkx,Stefan L. Frank +1 more
TL;DR: This article used a multimodal sentence encoder trained on a corpus of images with matching text captions to produce visually grounded sentence embeddings, which achieved state-of-the-art results on two image-caption retrieval benchmark data sets.
Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations
Simone Conia,Roberto Navigli +1 more
- 01 Dec 2020
TL;DR: This paper proposes Conception, a novel technique for building language-independent vector representations of concepts which places multilinguality at its core while retaining explicit relationships between concepts, and results in high-coverage representations that outperform the state of the art in multilingual and cross-lingual Semantic Word Similarity and Word Sense Disambiguation.
Learning semantic sentence representations from visually grounded language without lexical knowledge
Danny Merkx,Stefan L. Frank +1 more
TL;DR: The authors used a multimodal sentence encoder trained on a corpus of images with matching text captions to produce visually grounded sentence embeddings, which achieved state-of-the-art results on two image-caption retrieval benchmark datasets.
VICE: Variational Interpretable Concept Embeddings
Lukas Muttenthaler,Charles Y. Zheng,Patrick McClure,Robert A. Vandermeulen,Martin N. Hebart,Francisco C. Pereira +5 more
- 02 May 2022
TL;DR: A PAC learning bound is introduced for VICE that can be used to estimate generalization performance or determine a sufficient sample size for different experimental designs and rivals or outperforms its predecessor, SPoSE, at predicting human behavior in a triplet task.
8
References
ImageNet: A large-scale hierarchical image database
Jia Deng,Wei Dong,Richard Socher,Li-Jia Li,Kai Li,Li Fei-Fei +5 more
- 20 Jun 2009
TL;DR: A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
Glove: Global Vectors for Word Representation
Jeffrey Pennington,Richard Socher,Christopher D. Manning +2 more
- 01 Oct 2014
TL;DR: A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
•Posted Content
Distributed Representations of Words and Phrases and their Compositionality
TL;DR: In this paper, the Skip-gram model is used to learn high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships and improve both the quality of the vectors and the training speed.
WordNet: a lexical database for English
TL;DR: WordNet1 provides a more effective combination of traditional lexicographic information and modern computing, and is an online lexical database designed for use under program control.
16.9K
•Posted Content
Visualizing and Understanding Convolutional Networks
Matthew D. Zeiler,Rob Fergus +1 more
TL;DR: In this article, the authors introduce a novel visualization technique that gives insight into the function of intermediate feature layers and the operation of the classifier, and perform an ablation study to discover the performance contribution from different model layers.