Using Deep Learning to Generate Semantically Correct Hindi Captions

• URL: http://arxiv.org/abs/2602.13352v1
• Date: Fri, 13 Feb 2026 05:06:30 GMT
• Title: Using Deep Learning to Generate Semantically Correct Hindi Captions
• Authors: Wasim Akram Khan, Anil Kumar Vuppala,
• Abstract summary: This research utilizes distinct models for translating the image caption into Hindi.<n>Pre-trained CNNs like VGG16, ResNet50, and Inception V3 helped in retrieving image characteristics.<n>Experiments conclude that researchs ability to produce relevant, semantically accurate image captions in Hindi.
• Score: 5.723697351415205
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Automated image captioning using the content from the image is very appealing when done by harnessing the capability of computer vision and natural language processing. Extensive research has been done in the field with a major focus on the English language which gives the scope for further developments in the same with consideration of popular foreign languages. This research utilizes distinct models for translating the image caption into Hindi, the fourth most popular language across the world. Exploring the multi-modal architectures this research comprises local visual features, global visual features, attention mechanisms, and pre-trained models. Using google cloud translator on the image dataset from Flickr8k, Hindi image descriptions have been generated. Pre-trained CNNs like VGG16, ResNet50, and Inception V3 helped in retrieving image characteristics, while the uni-directional and bi-directional techniques of text encoding are used for the text encoding process. An additional Attention layer helps to generate a weight vector and, by multiplying it, combine image characteristics from each time step into a sentence-level feature vector. Bilingual evaluation understudy scores are used to compare the research outcome. Many experiments that serve as a baseline are done for the comparative analysis of the research. An image with a score of BLEU-1 is considered sufficient, whereas one with a score of BLEU-4 is considered to have fluid image captioning. For both BLEU scores, the attention-based bidirectional LSTM with VGG16 produced the best results of 0.59 and 0.19 respectively. The experiments conclude that researchs ability to produce relevant, semantically accurate image captions in Hindi. The research accomplishes the goals and future research can be guided by this research model.

Related papers

• Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Dataset [41.02122939888977]
  This study proposes a cross-native evaluation of Transformer-based vision and language models for Brazilian Portuguese IC.<n>We use a version of Flickr30K comprised of captions manually created by native Brazilian Portuguese speakers and compare it to a version with captions automatically translated from English to Portuguese.<n>Our findings show that Swin-DistilBERTimbau consistently outperforms other models, demonstrating strong generalization across datasets.
  arXiv  Detail & Related papers  (2026-01-30T23:12:03Z)
• Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation [0.0]
  This survey reviews attention-based image captioning models, categorizing them into transformer-based, deep learning-based, and hybrid approaches.<n>It explores benchmark datasets, discusses evaluation metrics such as BLEU, METEOR, CIDEr, and ROUGE, and highlights challenges in multilingual captioning.<n>We identify future research directions, such as multimodal learning, real-time applications in AI-powered assistants, healthcare, and forensic analysis.
  arXiv  Detail & Related papers  (2025-06-03T22:18:19Z)
• Towards Retrieval-Augmented Architectures for Image Captioning [81.11529834508424]
  This work presents a novel approach towards developing image captioning models that utilize an external kNN memory to improve the generation process. Specifically, we propose two model variants that incorporate a knowledge retriever component that is based on visual similarities. We experimentally validate our approach on COCO and nocaps datasets and demonstrate that incorporating an explicit external memory can significantly enhance the quality of captions.
  arXiv  Detail & Related papers  (2024-05-21T18:02:07Z)
• Autoregressive Pre-Training on Pixels and Texts [35.82610192457444]
  We explore the dual modality of language--both visual and textual--within an autoregressive framework, pre-trained on both document images and texts. Our method employs a multimodal training strategy, utilizing visual data through next patch prediction with a regression head and/or textual data through next token prediction with a classification head. We find that a unidirectional pixel-based model trained solely on visual data can achieve comparable results to state-of-the-art bidirectional models on several language understanding tasks.
  arXiv  Detail & Related papers  (2024-04-16T16:36:50Z)
• Advancing Visual Grounding with Scene Knowledge: Benchmark and Method [74.72663425217522]
  Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Most existing VG datasets are constructed using simple description texts. We propose a novel benchmark of underlineScene underlineKnowledge-guided underlineVisual underlineGrounding.
  arXiv  Detail & Related papers  (2023-07-21T13:06:02Z)
• Improving Multimodal Datasets with Image Captioning [65.74736570293622]
  We study how generated captions can increase the utility of web-scraped datapoints with nondescript text. Our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text.
  arXiv  Detail & Related papers  (2023-07-19T17:47:12Z)
• Retrieval-Augmented Transformer for Image Captioning [51.79146669195357]
  We develop an image captioning approach with a kNN memory, with which knowledge can be retrieved from an external corpus to aid the generation process. Our architecture combines a knowledge retriever based on visual similarities, a differentiable encoder, and a kNN-augmented attention layer to predict tokens. Experimental results, conducted on the COCO dataset, demonstrate that employing an explicit external memory can aid the generation process and increase caption quality.
  arXiv  Detail & Related papers  (2022-07-26T19:35:49Z)
• On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization [89.94078728495423]
  We show that recent advances in each modality, CLIP image representations and scaling of language models, do not consistently improve multimodal self-rationalization of tasks with multimodal inputs. Our findings call for a backbone modelling approach that can be built on to advance text generation from images and text beyond image captioning.
  arXiv  Detail & Related papers  (2022-05-24T00:52:40Z)
• Bench-Marking And Improving Arabic Automatic Image Captioning Through The Use Of Multi-Task Learning Paradigm [0.0]
  This paper explores methods and techniques that could enhance the performance of Arabic image captioning. The use of multi-task learning and pre-trained word embeddings noticeably enhanced the quality of image captioning. However, the presented results shows that Arabic captioning still lags behind when compared to the English language.
  arXiv  Detail & Related papers  (2022-02-11T06:29:25Z)
• Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization [81.26077816854449]
  We first explore the use of constituency parse trees for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images.
  arXiv  Detail & Related papers  (2021-10-21T00:16:02Z)
• TextMage: The Automated Bangla Caption Generator Based On Deep Learning [1.2330326247154968]
  TextMage is a system that is capable of understanding visual scenes that belong to the Bangladeshi geographical context. This dataset contains 9,154 images along with two annotations for each image.
  arXiv  Detail & Related papers  (2020-10-15T23:24:15Z)
• UNISON: Unpaired Cross-lingual Image Captioning [17.60054750276632]
  We present a novel unpaired cross-lingual method to generate image captions without relying on any caption corpus in the source or the target language. Specifically, our method consists of two phases: (i) a cross-lingual auto-encoding process, which utilizing a sentence parallel (bitext) corpus to learn the mapping from the source to the target language in the scene graph encoding space and decode sentences in the target language, and (ii) a cross-modal unsupervised feature mapping, which seeks to map the encoded scene graph features from image modality to language modality.
  arXiv  Detail & Related papers  (2020-10-03T06:14:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.