Open AccessPosted Content
Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
TL;DR: This article proposed an adaptive attention model with a visual sentinel to decide whether to attend to the image and where, in order to extract meaningful information for sequential word generation, which set the new state-of-the-art by a significant margin.
read more
Abstract: Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict non-visual words such as "the" and "of". Other words that may seem visual can often be predicted reliably just from the language model e.g., "sign" after "behind a red stop" or "phone" following "talking on a cell". In this paper, we propose a novel adaptive attention model with a visual sentinel. At each time step, our model decides whether to attend to the image (and if so, to which regions) or to the visual sentinel. The model decides whether to attend to the image and where, in order to extract meaningful information for sequential word generation. We test our method on the COCO image captioning 2015 challenge dataset and Flickr30K. Our approach sets the new state-of-the-art by a significant margin.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Federated Learning for Vision-and-Language Grounding Problems
Fenglin Liu,Xian Wu,Shen Ge,Wei Fan,Yuexian Zou +4 more
- 03 Apr 2020
TL;DR: This work proposes a federated learning framework to obtain various types of image representations from different tasks, which are then fused together to form fine-grained image representations that are much more powerful than the original representations alone in individual tasks.
•Posted Content
Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
TL;DR: Wang et al. as discussed by the authors proposed a hierarchical LSTM with adjusted temporal attention (hLSTMat) approach for video captioning, which utilizes the temporal attention for selecting specific frames to predict the related words.
•Posted Content
Attention on Attention for Image Captioning
TL;DR: An Attention on Attention (AoA) module is proposed, which extends the conventional attention mechanisms to determine the relevance between attention results and queries and is applied to both the encoder and the decoder of the image captioning model, which is named as AoA Network.
128
Description Generation for Remote Sensing Images Using Attribute Attention Mechanism
TL;DR: This paper presents a new model with an attribute attention mechanism for the description generation of remote sensing images, and explores the impact of the attributes extracted fromRemote Sensing images on the attention mechanism.
128
•Posted Content
Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
TL;DR: This work proposes a novel model called Explicit Sparse Transformer, able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments in the context, and achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time.
125
References
Deep Residual Learning for Image Recognition
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Microsoft COCO: Common Objects in Context
Tsung-Yi Lin,Michael Maire,Serge Belongie,James Hays,Pietro Perona,Deva Ramanan,Piotr Dollár,C. Lawrence Zitnick +7 more
- 06 Sep 2014
TL;DR: A new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding by gathering images of complex everyday scenes containing common objects in their natural context.
Bleu: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni,Salim Roukos,Todd Ward,Wei-Jing Zhu +3 more
- 06 Jul 2002
TL;DR: This paper proposed a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run.
Learning Phrase Representations using RNN Encoder--Decoder for Statistical Machine Translation
Kyunghyun Cho,Bart van Merriënboer,Caglar Gulcehre,Dzmitry Bahdanau,Fethi Bougares,Holger Schwenk,Yoshua Bengio,Yoshua Bengio,Yoshua Bengio +8 more
- 01 Jan 2014
TL;DR: In this paper, the encoder and decoder of the RNN Encoder-Decoder model are jointly trained to maximize the conditional probability of a target sequence given a source sequence.
•Proceedings Article
Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau,Kyunghyun Cho,Yoshua Bengio +2 more
- 01 Jan 2015
TL;DR: It is conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and it is proposed to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly.
25.7K
Related Papers (5)
Karen Simonyan,Andrew Zisserman +1 more
- 04 Sep 2014
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
Diederik P. Kingma,Jimmy Ba +1 more