An Integrated Vision-and-Language Pretraining (VLP) and Visual Question Answering (VQA) model to Automate Nondestructive Evaluation Image Analysis

• URL: http://arxiv.org/abs/2608.29408v2
• Date: Fri, 04 Sep 2026 15:58:20 GMT
• Title: An Integrated Vision-and-Language Pretraining (VLP) and Visual Question Answering (VQA) model to Automate Nondestructive Evaluation Image Analysis
• Abstract summary: ChatNDE Figure to Caption aims to automate the interpretation of NDE images using deep learning and natural language processing (NLP)<n>A VisionandLanguage Pretraining strategy is developed to help the model learn how to connect visual features with meaningful language.<n>The system combines a ResNet50 model to extract important features from the images and a GPT2 language model to turn those features into natural sounding text.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An AI-based approach called ChatNDE Figure to Caption is introduced, which aims to automate the interpretation of NDE images using deep learning and natural language processing (NLP). A Vision-and-Language Pretraining (VLP) strategy is developed to help the model learn how to connect visual features with meaningful language. Basically, we built a large NDE image dataset, trained the model using annotated examples, and then evaluated how well it performed using BLEU scores to compare its output to expert written descriptions. So, the system combines a ResNet50 model to extract important features from the images and a GPT2 language model to turn those features into natural sounding text. Even though the accuracy of the model has been low the generated caption results have been solid so far, the captions were shorter but mentioned some important features of images what human experts would say, which shows the model is learning to pick up on key details. Also, a Visual Question Answering (VQA) model is used as part of the system. VQA models are designed to take an image and a question about that image (like Is there a crack? or Where is the defect located?) and generate a useful answer. By adding this layer, the platform will not just describe what it sees, it can also respond to specific questions, making it even more interactive and helpful for inspectors in the field. This whole approach is a big step toward speeding up NDE workflows, reducing human error, and making the technology more accessible.

Related papers

• JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees [64.57426735981902]
  In maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions.<n>To enable fault trees stored as images to be directly processed by large language models, we propose a novel textual representation of fault trees.<n>We construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments.
  arXiv  Detail & Related papers  (2026-03-24T09:13:43Z)
• QIRL: Boosting Visual Question Answering via Optimized Question-Image Relation Learning [38.24779287568123]
  Current debiasing techniques fail to capture the superior relation between images and texts.<n>No prior work has examined the degree of input relevance in debiasing studies.<n>We propose a novel framework, which employs a generation-based self-supervised learning strategy.
  arXiv  Detail & Related papers  (2025-04-04T10:38:28Z)
• ABC: Achieving Better Control of Multimodal Embeddings using VLMs [61.396457715710774]
  Visual embedding models excel at zero-shot tasks like visual retrieval and classification.<n>These models cannot be used for tasks that contain ambiguity or require user instruction.<n>We introduce ABC, an open-source multimodal embedding model that uses a vision-language model backbone.
  arXiv  Detail & Related papers  (2025-03-01T03:29:02Z)
• Uncertainty Quantification in Retrieval Augmented Question Answering [45.573346610161195]
  We propose to quantify the uncertainty of a QA model via estimating the utility of the passages it is provided with.<n>We train a lightweight neural model to predict passage utility for a target QA model and show that while simple information theoretic metrics can predict answer correctness up to a certain extent, our approach efficiently approximates or outperforms more expensive sampling-based methods.
  arXiv  Detail & Related papers  (2025-02-25T11:24:52Z)
• Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models [36.18155629835474]
  Vision-language models (VLMs) excel at extracting and reasoning about information from images.<n>Yet, their capacity to leverage internal knowledge about specific entities remains underexplored.<n>This work investigates the disparity in model performance when answering factual questions about an entity described in text versus depicted in an image.
  arXiv  Detail & Related papers  (2024-12-18T18:22:30Z)
• Are VLMs Really Blind [3.052971829873887]
  Vision Language Models excel in handling a wide range of complex tasks. These models fail to perform well on low-level basic visual tasks. Our work presents a novel automatic pipeline designed to extract key information from images in response to specific questions.
  arXiv  Detail & Related papers  (2024-10-29T13:20:50Z)
• Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answering [2.98667511228225]
  ReRe is an encoder-decoder architecture model using a pre-trained clip vision encoder and a pre-trained GPT-2 language model as a decoder. ReRe outperforms previous methods in VQA accuracy and explanation score and shows improvement in NLE with more persuasive, reliability.
  arXiv  Detail & Related papers  (2024-08-30T04:39:43Z)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension [131.14381425260706]
  We introduce Self-Training on Image (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference for image descriptions using unlabeled images. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data.
  arXiv  Detail & Related papers  (2024-05-30T05:53:49Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions [70.70725223310401]
  This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models. The experimental results demonstrate that the proposed evaluation method effectively analyzes the robustness of VQA models.
  arXiv  Detail & Related papers  (2023-04-06T15:32:35Z)
• Knowledge-Based Counterfactual Queries for Visual Question Answering [0.0]
  We propose a systematic method for explaining the behavior and investigating the robustness of VQA models through counterfactual perturbations. For this reason, we exploit structured knowledge bases to perform deterministic, optimal and controllable word-level replacements targeting the linguistic modality. We then evaluate the model's response against such counterfactual inputs.
  arXiv  Detail & Related papers  (2023-03-05T08:00:30Z)
• Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment [81.73717488887938]
  Language-Quantized AutoEncoder (LQAE) learns to align text-image data in an unsupervised manner by leveraging pretrained language models. LQAE learns to represent similar images with similar clusters of text tokens, thereby aligning these two modalities without the use of aligned text-image pairs. This enables few-shot image classification with large language models (e.g., GPT-3) as well as linear classification of images based on BERT text features.
  arXiv  Detail & Related papers  (2023-02-02T06:38:44Z)
• Prompt-based Learning for Unpaired Image Captioning [86.44188293709307]
  Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Recent successes of Vision-Language Pre-Trained Models (VL-PTMs) have triggered the development of prompt-based learning. We present in this paper a novel scheme based on prompt to train the UIC model, making best use of the powerful generalization ability.
  arXiv  Detail & Related papers  (2022-05-26T03:13:43Z)
• On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization [89.94078728495423]
  We show that recent advances in each modality, CLIP image representations and scaling of language models, do not consistently improve multimodal self-rationalization of tasks with multimodal inputs. Our findings call for a backbone modelling approach that can be built on to advance text generation from images and text beyond image captioning.
  arXiv  Detail & Related papers  (2022-05-24T00:52:40Z)
• Visually-Augmented Language Modeling [137.36789885105642]
  We propose a novel pre-training framework, named VaLM, to Visually-augment text tokens with retrieved relevant images for Language Modeling. With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling. We evaluate the proposed model on various multimodal commonsense reasoning tasks, which require visual information to excel.
  arXiv  Detail & Related papers  (2022-05-20T13:41:12Z)
• COIN: Counterfactual Image Generation for VQA Interpretation [5.994412766684842]
  We introduce an interpretability approach for VQA models by generating counterfactual images. In addition to interpreting the result of VQA models on single images, the obtained results and the discussion provides an extensive explanation of VQA models' behaviour.
  arXiv  Detail & Related papers  (2022-01-10T13:51:35Z)
• Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering [17.51860125438028]
  We propose to use a unimodal (text-only) train and inference procedure based on automatic off-the-shelf captioning of images and pretrained language models. Our results on a visual question answering task which requires external knowledge (OK-VQA) show that our text-only model outperforms pretrained multimodal (image-text) models of comparable number of parameters.
  arXiv  Detail & Related papers  (2021-09-15T14:11:29Z)
• A Picture May Be Worth a Hundred Words for Visual Question Answering [26.83504716672634]
  In image understanding, it is essential to use concise but detailed image representations. Deep visual features extracted by vision models, such as Faster R-CNN, are prevailing used in multiple tasks. We propose to take description-question pairs as input, instead of deep visual features, and fed them into a language-only Transformer model.
  arXiv  Detail & Related papers  (2021-06-25T06:13:14Z)
• VisQA: X-raying Vision and Language Reasoning in Transformers [10.439369423744708]
  Recent research has shown that state-of-the-art models tend to produce answers exploiting biases and shortcuts in the training data. We present VisQA, a visual analytics tool that explores this question of reasoning vs. bias exploitation.
  arXiv  Detail & Related papers  (2021-04-02T08:08:25Z)
• Counterfactual Samples Synthesizing for Robust Visual Question Answering [104.72828511083519]
  We propose a model-agnostic Counterfactual Samples Synthesizing (CSS) training scheme. CSS generates numerous counterfactual training samples by masking critical objects in images or words in questions. We achieve a record-breaking performance of 58.95% on VQA-CP v2, with 6.5% gains.
  arXiv  Detail & Related papers  (2020-03-14T08:34:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.