Multimodal Model Diffing for Feature Discovery and Control

• URL: http://arxiv.org/abs/2608.09928v1
• Date: Mon, 10 Aug 2026 17:59:30 GMT
• Title: Multimodal Model Diffing for Feature Discovery and Control
• Abstract summary: We introduce MMDiff, a framework that turns sparse autoencoders into feature-level interfaces for discovering and controlling multimodal behavior.<n>We train multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding, multimodal safety, and OCR.
• Score: 25.349396112139214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control. We introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses: (i) feature isolation, by diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training; (ii) task-specific feature detection, via per-token contrastive firing analysis that isolates causal features; and (iii) feature-level control, by causally removing or steering the discovered feature directions. We train multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding, multimodal safety, and OCR. MMDiff discovers sparse, causally specific features whose removal selectively degrades target behaviors by an average of 12% on spatial tasks and 17% on OCR, and reduces attack success rate by 24% on multimodal safety attacks, with no impact on VQA performance. Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average over a standard single-layer steering baseline. These results show that multimodal SAEs can serve not only as interpretability tools, but as mechanisms for auditing, steering, and controlling MLLMs behavior toward safer and more capable generations.

Related papers

• Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System [21.596261657771]
  UAV-MAS is a training-free multi-agent system for MLLM-based UAV aerial image understanding and reasoning.<n>It achieves 77.0% overall accuracy on UAVQA-Bench, surpassing Gemini 3 Pro by 4.0%, while the 8B variant improves 8.7% over its base model.
  arXiv  Detail & Related papers  (2026-08-12T07:25:32Z)
• Dual Sparse Aggregation Transformer for Multispectral Object Detection [34.20084627163684]
  We propose a novel Dual Sparse Aggregation Transformer (DSAFormer) for multispectral object detection.<n>Specifically, the DSAFormer is designed to exploit and boost cross-modal complementary information.<n>Our proposed DSAFormer achieves better detection performance against state-of-the-art methods on four public datasets.
  arXiv  Detail & Related papers  (2026-06-30T01:05:24Z)
• Steering Vision-Language Models with Joint Sparse Autoencoders [2.367986397868328]
  We introduce the Joint Sparse Autoencoder (JSAE) to factorize sequence-pooled vision and language activations into shared, interpretable image/caption-level features.<n>Applying JSAE to vision-language models (VLMs) yields representations that are difficult to use as controllable cross-modal steering directions.
  arXiv  Detail & Related papers  (2026-06-24T10:11:04Z)
• Sparse Visual Thought Circuits in Vision-Language Models [2.5754366051855837]
  We develop a causal pipeline to localize and test sparse visual thought circuits in Qwen3-VL-8B.<n>We train SAEs at this layer, construct task-selective sets via an explicit rule, and perform inference time scaling and ablation while quantifying accuracy and drift.
  arXiv  Detail & Related papers  (2026-03-26T06:24:36Z)
• Agentic Mixed-Source Multi-Modal Misinformation Detection with Adaptive Test-Time Scaling [41.61826091940538]
  Vision-language models (VLMs) have been proven effective for detecting multi-modal misinformation on social platforms.<n>However, a single VLM's capacity falls short in the more complex mixed-source multi-modal misinformation detection task.<n>We present AgentM3D, a multi-agent framework for zero-shot misinformation detection.
  arXiv  Detail & Related papers  (2026-03-03T02:07:52Z)
• UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perception [71.19234323863314]
  We propose a framework to simultaneously recognize driver behavior, driver emotion, vehicle behavior, and traffic context.<n>Our framework incorporates two core components: dual-branch spatial channel multimodal embedding and adaptive feature-decoupled multi-task loss.<n>We evaluate our method on the AIDE dataset, and the experimental results demonstrate that UV-M3TL achieves state-of-the-art performance across all four tasks.
  arXiv  Detail & Related papers  (2026-02-02T03:35:24Z)
• A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles [74.8162337823142]
  MM-UAV is the first large-scale benchmark for Multi-Modal UAV Tracking.<n>The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames.<n>Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework.
  arXiv  Detail & Related papers  (2025-11-23T08:42:17Z)
• Automating Steering for Safe Multimodal Large Language Models [58.36932318051907]
  We introduce a modular and adaptive inference-time intervention technology, AutoSteer, without requiring any fine-tuning of the underlying model.<n>AutoSteer incorporates three core components: (1) a novel Safety Awareness Score (SAS) that automatically identifies the most safety-relevant distinctions among the model's internal layers; (2) an adaptive safety prober trained to estimate the likelihood of toxic outputs from intermediate representations; and (3) a lightweight Refusal Head that selectively intervenes to modulate generation when safety risks are detected.
  arXiv  Detail & Related papers  (2025-07-17T16:04:55Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [79.58755811919366]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing.<n>It is designed to accurately detect horizontal or oriented objects from any sensor modality.<n>This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models [95.09157454599605]
  Large Language Models (LLMs) are becoming increasingly powerful, but they still exhibit significant but subtle weaknesses.<n>Traditional benchmarking approaches cannot thoroughly pinpoint specific model deficiencies.<n>We introduce a unified framework, AutoDetect, to automatically expose weaknesses in LLMs across various tasks.
  arXiv  Detail & Related papers  (2024-06-24T15:16:45Z)
• MKF-ADS: Multi-Knowledge Fusion Based Self-supervised Anomaly Detection System for Control Area Network [9.305680247704542]
  Control Area Network (CAN) is an essential communication protocol that interacts between Electronic Control Units (ECUs) in the vehicular network. CAN is facing stringent security challenges due to innate security risks. We propose a self-supervised multi-knowledge fused anomaly detection model, called MKF-ADS.
  arXiv  Detail & Related papers  (2024-03-07T07:40:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.