Fine-tuning a vision-language model for fracture-surface morphology recognition

• URL: http://arxiv.org/abs/2605.07145v1
• Date: Fri, 08 May 2026 02:26:36 GMT
• Title: Fine-tuning a vision-language model for fracture-surface morphology recognition
• Abstract summary: We fine-tuned an open-source Vision-language Model (VLM) for fracture-surface image analysis using a curated dataset of 13,168 images.<n>The resulting specialist model outperforms flagship proprietary multimodal models on a benchmark of 100 manually annotated images.<n>We discuss integrated use of the fine-tuned model with proprietary models to combine fracture-specific visual accuracy with broader multimodal reasoning for autonomous fractography.
• Score: 20.872357530075153
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-language models (VLMs) have shown strong potential for scientific image understanding, but general-purpose models often lack the domain-specific visual knowledge required for reliable materials characterization. In this work, we fine-tuned an open-source VLM (Qwen3-VL-32B-Instruct) for fracture-surface image analysis using a curated dataset of 13,168 open-source, literature-mined fracture-surface images. Morphology annotations were generated by GPT-5.2-Reasoning (high) from both the images and relevant excerpts of their source papers, and the dataset was further enriched with targeted manual collection and rotation-based augmentation. The resulting specialist model outperforms flagship proprietary multimodal models on a benchmark of 100 manually annotated images. It achieves a precision of 0.92, compared to 0.35 for the base Qwen3-VL-32B-Instruct, 0.58 for GPT-5.5-Reasoning (high), and 0.78 for Gemini 3.1 Pro-Reasoning (high). Dataset ablations show that manual collection of rare-feature images and augmentation via image rotation are both beneficial to improve recognition of less common fracture morphology features. We further discuss integrated use of the fine-tuned model with proprietary models to combine fracture-specific visual accuracy with broader multimodal reasoning for autonomous fractography. Although focused on fracture-surface images, this work demonstrates how VLMs can be adapted through targeted collection and fine-tuning on novel feature images to recognize those features and support downstream decision-making in autonomous microscopy workflows.

Related papers

• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• Halfway to 3D: Ensembling 2.5D and 3D Models for Robust COVID-19 CT Diagnosis [5.645096759437525]
  We propose a deep learning framework for COVID-19 detection and disease classification from chest CT scans.<n>The framework integrates both 2.5D and 3D representations to capture complementary slice-level and volumetric information.<n>Experiments on the PHAROS-AIF-MIH benchmark demonstrate the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2026-03-16T05:24:10Z)
• Multimodal Visual Surrogate Compression for Alzheimer's Disease Classification [69.87877580725768]
  Multimodal Visual Surrogate Compression (MVSC) learns to compress and adapt large 3D sMRI volumes into compact 2D features.<n>MVSC has two key components: a Volume Context that captures global cross-slice context under textual guidance, and an Adaptive Slice Fusion module that aggregates slice-level information in a text-enhanced, patch-wise manner.
  arXiv  Detail & Related papers  (2026-01-29T13:05:46Z)
• Towards Generalizable Forgery Detection and Reasoning [23.858913560970866]
  We formulate detection and explanation as a unified Forgery Detection and Reasoning task (FDR-Task)<n>We introduce the Multi-Modal Forgery Reasoning dataset (MMFR-Dataset), a large-scale dataset containing 120K images across 10 generative models, with 378K reasoning annotations on forgery attributes.<n>Experiments across multiple generative models demonstrate that FakeReasoning achieves robust generalization and outperforms state-of-the-art methods on both detection and reasoning tasks.
  arXiv  Detail & Related papers  (2025-03-27T06:54:06Z)
• RELICT: A Replica Detection Framework for Medical Image Generation [34.82692226532414]
  memorization in generative models can lead to unintended leakage of sensitive patient information and limit model utility.<n>We propose a framework for identifying replicas, i.e. nearly identical copies of the training data, in synthetic medical image datasets.
  arXiv  Detail & Related papers  (2025-02-24T17:37:19Z)
• MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classification [36.59100450109841]
  Whole slide pathology image classification presents challenges due to gigapixel image sizes and limited annotation labels.<n>This paper introduces a prompt learning method to adapt large vision-language models for few-shot pathology classification.
  arXiv  Detail & Related papers  (2025-02-11T09:42:13Z)
• IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art Text-to-Image Models [52.73820275861131]
  Text-to-image(T2I) models have made significant progress, showcasing impressive abilities in prompt following and image generation.<n>Recent models such as FLUX.1 and Ideogram2.0 have demonstrated exceptional performance across various complex tasks.<n>This study provides valuable insights into the current state and future trajectory of T2I models as they evolve towards general-purpose usability.
  arXiv  Detail & Related papers  (2025-01-23T18:58:33Z)
• DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception [66.88792390480343]
  We propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder.<n>DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data, and a smaller base model size.
  arXiv  Detail & Related papers  (2024-05-24T05:46:04Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• Leveraging Foundation Models for Content-Based Image Retrieval in Radiology [0.14208766024828973]
  Content-based image retrieval (CBIR) has the potential to significantly improve diagnostic aid and medical research in radiology.<n>Current CBIR systems face limitations due to their specialization to certain pathologies, limiting their utility.<n>We propose using vision foundation models as powerful and versatile off-the-shelf feature extractors for content-based image retrieval.
  arXiv  Detail & Related papers  (2024-03-11T10:06:45Z)
• AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images [53.29794593104923]
  We present a novel concept of shared-context processing for whole slide histopathology images. AMIGO uses the celluar graph within the tissue to provide a single representation for a patient. We show that our model is strongly robust to missing information to an extent that it can achieve the same performance with as low as 20% of the data.
  arXiv  Detail & Related papers  (2023-03-01T23:37:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.