/pdf/towards-local-visual-modeling-for-image-captioning-24c3qpjg.pdf

Towards Local Visual Modeling for Image Captioning

Natural language problems have already been investigated for around five years. Recent progress in artificial intelligence (AI) has greatly improved the performance of models. However, the results are still not sufficiently satisfying. Machines cannot imitate human brains and the way they communicate, so it remains an ongoing task. Due to the increasing amount of information on this topic, it is very difficult to keep on track with the newest researches and results achieved in the image captioning field. In this study a comprehensive Systematic Literature Review (SLR) provides a brief overview of improvements in image captioning over the last four years. The main focus of the paper is to explain the most common techniques and the biggest challenges in image captioning and to summarize the results from the newest papers. Inconsistent comparison of results achieved in image captioning was noticed during this study and hence the awareness of incomplete data collection is raised in this paper. Therefore, it is very important to compare results of a newly created model produced with the newest information and not only with the state of the art methods. This SLR is a source of such information for researchers in order for them to be precisely correct on result comparison before publishing new achievements in the image caption generation field.

A Systematic Literature Review on Image Captioning

Human eye is affected by the different eye diseases including choroidal neovascularization (CNV), diabetic macular edema (DME) and age-related macular degeneration (AMD) This work aims to design an artificial intelligence (AI) based clinical decision support system for eye disease detection and classification to assist the ophthalmologists more effectively detecting and classifying CNV, DME and drusen by using the Optical Coherence Tomography (OCT) images depicting different tissues The methodology used for designing this system involves different deep learning convolutional neural network (CNN) models and long short-term memory networks (LSTM) The best image captioning model is selected after performance analysis by comparing nine different image captioning systems for assisting ophthalmologists to detect and classify eye diseases The quantitative data analysis results obtained for the image captioning models designed using DenseNet201 with LSTM have superior performance in terms of overall accuracy of 0969, positive predictive value of 0972 and true-positive rate of 0969using OCT images enhanced by the generative adversarial network (GAN) The corresponding performance values for the Xception with LSTM image captioning models are 0969, 0969 and 0938, respectively Thus, these two models yield superior performance and have potential to assist ophthalmologists in making optimal diagnostic decision

An enhanced OCT image captioning system to assist ophthalmologists in detecting and classifying eye diseases.

Abstract In our day-to-day life, synchronizing vision and language aspects plays a crucial role in solving various real-time challenges. Image captioning is one of them, and it aims to recognise objects, activities, and their relationships in order to provide a syntactically and semantically correct visual description. There are existing works of image captioning in various directions, such as news, fashion, art, and medical domains. The core architectural idea of image captioning is based on merging CNN, RNN, and transformer models. In practice, there are many conceivable combinations, and brute forcing all of them would take a long time. As we know, there is no work on interpreting image captioning models across various usecases. In this research article, we examine and analyze different image captioning models used across various domains, and multiple insights are extracted to determine the best combinational architecture for a new application without ignoring contextual semantics. We examined numerous designs and determined that LSTM is best for image captioning across several domains.

https://www.tandfonline.com/doi/pdf/10.1080/23311916.2022.2104333?needAccess=true

Semantic interdisciplinary evaluation of image captioning models

This paper focuses on developing semantic image caption generation techniques that leverage image and scene understanding. More particularly, we are interested in addressing image captioning by developing a mixture of object detection and attention-enriched deep learning models. To extract the image features, a Convolutional Neural Network (CNN) is used, and then an extended version of Recurrent Neural Networks (LSTM) with attention-enrichment is adopted to generate the caption. We implement image captioning by considering detected objects from the image scene, and then by integrating an attention mechanism for caption generation. This can have multiple advantages from accuracy and semantics perspectives. The objective of this paper is to introduce a combined pipeline that employs several variant models for semantic caption generation. Four variant models are proposed, all of them are implemented and trained on COCO and Flickr30k datasets, and then tested on a subset of COCO dataset. Results of the different models were evaluated using a semantic similarity analysis between the generated captions and the actual ground truth captions. Our framework helps in a deeper understanding of images and decision making in diverse use-cases such as innovative and distinctive responses from multimodal data, and in analyzing and monitoring crowdsourced images from social media and other sources.

Facilitated Deep Learning Models for Image Captioning

Image captioning is a challenging task and attracting more and more attention in the field of Artificial Intelligence, and which can be applied to efficient image retrieval, intelligent blind guidance and human-computer interaction, etc. In this paper, we present a survey on advances in image captioning based on Deep Learning methods, including Encoder-Decoder structure, improved methods in Encoder, improved methods in Decoder, and other improvements. Furthermore, we discussed future research directions.

Image Captioning based on Deep Learning Methods: A Survey.

Language models based on Recurrent Neural Networks, e.g. Long Short Term Memory Network (LSTM), have shown strong ability in generating captions from image. However, in previous LSTM-based image captioning models, the image information is input to LSTM at 0th time step, and the network gradually forgets the image information, and only uses the language model to generate a simple description, leaving the potential in generating a better description. To address this challenge, in this paper, a Long-term Recurrent Merge Network (LRMN) model is proposed to merge the image feature at each step via a language model, which not only can improve the accuracy of image captioning, but also can describe the image better. Experimental results show that the proposed LRMN model has a promising improvement in image captioning.

Long-Term Recurrent Merge Network Model for Image Captioning

A visual persistence model for image captioning

Although the image captioning methods based on RNN has made great progress in recent years, these are often lacking in variability and ignore some minor information. In this paper, a novel image captioning method based on Generative Adversarial Networks is proposed, which improve the naturalness and diversity of image description. In the method, matcher is added to the generator to get the feature of the image that does not appear in the standard description, then to produce descriptions conditioned on image, and discriminator to access how well a description fits the visual content. It is noteworthy that training a sequence generator is nontrivial. Experiments on MSCOCO and Flickr30k show that it performed competitively against real people in our user study and outperformed other methods on various tasks.

A Novel Image Captioning Method Based on Generative Adversarial Networks

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Yingfei Sun

Author Tools

Chat about Author

Papers

Image Captioning based on Deep Learning Methods: A Survey.

Long-Term Recurrent Merge Network Model for Image Captioning

A visual persistence model for image captioning

A Novel Image Captioning Method Based on Generative Adversarial Networks