Metamorphic Testing of Image Captioning Systems via Image-Level Reduction

• URL: http://arxiv.org/abs/2311.11791v3
• Date: Fri, 04 Oct 2024 03:50:40 GMT
• Title: Metamorphic Testing of Image Captioning Systems via Image-Level Reduction
• Authors: Xiaoyuan Xie, Xingpeng Li, Songqiang Chen,
• Abstract summary: In this paper, we propose REIC to perform metamorphic testing for IC systems with some image-level reduction transformations. With the image-level reduction transformations, REIC does not artificially manipulate any objects and hence can avoid generating unreal follow-up images.
• Score: 1.3225694028747141
• License:
• Abstract: The Image Captioning (IC) technique is widely used to describe images in natural language. Recently, some IC system testing methods have been proposed. However, these methods still rely on pre-annotated information and hence cannot really alleviate the oracle problem in testing. Besides, their method artificially manipulates objects, which may generate unreal images as test cases and thus lead to less meaningful testing results. Thirdly, existing methods have various requirements on the eligibility of source test cases, and hence cannot fully utilize the given images to perform testing. To tackle these issues, in this paper, we propose REIC to perform metamorphic testing for IC systems with some image-level reduction transformations like image cropping and stretching. Instead of relying on the pre-annotated information, REIC uses a localization method to align objects in the caption with corresponding objects in the image, and checks whether each object is correctly described or deleted in the caption after transformation. With the image-level reduction transformations, REIC does not artificially manipulate any objects and hence can avoid generating unreal follow-up images. Besides, it eliminates the requirement on the eligibility of source test cases in the metamorphic transformation process, as well as decreases the ambiguity and boosts the diversity among the follow-up test cases, which consequently enables testing to be performed on any test image and reveals more distinct valid violations. We employ REIC to test five popular IC systems. The results demonstrate that REIC can sufficiently leverage the provided test images to generate follow-up cases of good reality, and effectively detect a great number of distinct violations, without the need for any pre-annotated information.

Related papers

• SPOLRE: Semantic Preserving Object Layout Reconstruction for Image Captioning System Testing [12.895128109843071]
  SPOLRE is an automated tool for semantic-preserving object layout reconstruction in IC system testing. It eliminates the need for manual annotations and creates realistic, varied test suites. SPOLRE excels in identifying caption errors, detecting 31,544 incorrect captions across seven IC systems with an average precision of 91.62%.
  arXiv  Detail & Related papers  (2024-07-26T04:46:31Z)
• Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction [62.955327005837475]
  Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images. We present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types.
  arXiv  Detail & Related papers  (2023-10-29T13:58:57Z)
• On the Effectiveness of Image Manipulation Detection in the Age of Social Media [9.227950734832447]
  manipulation detection algorithms often rely on the manipulated regions being sufficiently'' different from the rest of the non-tampered regions in the image. We present an in-depth analysis of deep learning-based and learning-free methods, assessing their performance on benchmark datasets. We propose a novel deep learning-based pre-processing technique that accentuates the anomalies present in manipulated regions.
  arXiv  Detail & Related papers  (2023-04-19T04:05:54Z)
• GDIP: Gated Differentiable Image Processing for Object-Detection in Adverse Conditions [15.327704761260131]
  We present a Gated Differentiable Image Processing (GDIP) block, a domain-agnostic network architecture. Our proposed GDIP block learns to enhance images directly through the downstream object detection loss. We demonstrate significant improvement in detection performance over several state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-09-29T16:43:13Z)
• Exploring Resolution and Degradation Clues as Self-supervised Signal for Low Quality Object Detection [77.3530907443279]
  We propose a novel self-supervised framework to detect objects in degraded low resolution images. Our methods has achieved superior performance compared with existing methods when facing variant degradation situations.
  arXiv  Detail & Related papers  (2022-08-05T09:36:13Z)
• ObjectFormer for Image Manipulation Detection and Localization [118.89882740099137]
  We propose ObjectFormer to detect and localize image manipulations. We extract high-frequency features of the images and combine them with RGB features as multimodal patch embeddings. We conduct extensive experiments on various datasets and the results verify the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-03-28T12:27:34Z)
• RestoreDet: Degradation Equivariant Representation for Object Detection in Low Resolution Images [81.91416537019835]
  We propose a novel framework, RestoreDet, to detect objects in degraded low resolution images. Our framework based on CenterNet has achieved superior performance compared with existing methods when facing variant degradation situations.
  arXiv  Detail & Related papers  (2022-01-07T03:40:23Z)
• Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment [157.1292674649519]
  We propose a practical solution named degraded-reference IQA (DR-IQA) DR-IQA exploits the inputs of IR models, degraded images, as references. Our results can even be close to the performance of full-reference settings.
  arXiv  Detail & Related papers  (2021-08-18T02:35:08Z)
• Ensembling with Deep Generative Views [72.70801582346344]
  generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars.
  arXiv  Detail & Related papers  (2021-04-29T17:58:35Z)
• Object-based Metamorphic Testing through Image Structuring [0.6445605125467573]
  Testing software is often costly due to the need of mass-producing test cases and providing a test oracle for it. One method that has been proposed in order to alleviate the oracle problem is metamorphic testing.
  arXiv  Detail & Related papers  (2020-02-12T10:32:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.