mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models

• URL: http://arxiv.org/abs/2606.07531v1
• Date: Tue, 21 Apr 2026 10:01:51 GMT
• Title: mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models
• Abstract summary: mllm-shap is a Python framework designed to extend Shapley Value explainability from text-only Large Language Models to Multimodal LLMs.<n> Modality-aware coalition masking, which manages the interleaved processing of discrete text tokens and dense audio encoder frames.<n>Phonetic alignment-based token grouping, a novel technique that reduces the coalition space by 10x to 50x.
• Score: 1.8352113484137629
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce mllm-shap, an open-source Python framework designed to extend Shapley Value (SV) explainability from text-only Large Language Models to Multimodal LLMs (MLLMs) processing joint text and audio inputs. While text-based attribution is well-studied, mllm-shap addresses three critical challenges unique to the multimodal regime: (1) Modality-aware coalition masking, which manages the interleaved processing of discrete text tokens and dense audio encoder frames. (2) Multi-turn conversation tracking, utilizing per-token metadata to maintain role and modality context. (3) Phonetic alignment-based token grouping, a novel technique that reduces the coalition space by 10x to 50x, rendering SV estimation computationally feasible for long-form audio. The platform implements five SV estimation strategies, including a Complementary Contributions (CC) estimator with Neyman-optimal allocation that demonstrates superior convergence over standard Monte Carlo baselines. mllm-shap is provided as a pip-installable package featuring an interactive web-based GUI for granular attribution visualization. To our knowledge, this is the first publicly available framework providing a complete, reproducible pipeline for SV-based explainability in text-audio MLLMs.

Related papers

• Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis [1.8352113484137629]
  Multimodal Large Language Models (MLLMs) integrate text and audio to interpret context in complex interactive dialogues.<n>We formalize a multimodal extension of the Shapley Value framework, treating discrete text tokens and aligned audio segments as cooperative features.<n>We propose Spectrogram-Guided Phonetic Alignment (SGPA), a novel preprocessing method that maps high-frequency audio streams to interpretable, word-aligned segments.
  arXiv  Detail & Related papers  (2026-04-21T11:32:24Z)
• Unlocking Financial Insights: An advanced Multimodal Summarization with Multimodal Output Framework for Financial Advisory Videos [11.550322270589952]
  FASTER (Financial Advisory Summariser with Textual Embedded Relevant images) is a framework that produces optimized, concise summaries.<n>FASTER employs BLIP for semantic visual descriptions, OCR for textual patterns, and Whisper-based transcription with Speaker diarization as BOS features.<n>A modified Direct Preference Optimization (DPO)-based loss function, equipped with BOS-specific fact-checking, ensures precision, relevance, and factual consistency.
  arXiv  Detail & Related papers  (2025-09-25T09:54:19Z)
• Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision [83.0622534215881]
  This work proposes an industry-level omni-modal large language model (LLM) pipeline that integrates auditory, visual, and linguistic modalities.<n>Our pipeline consists of three main components: First, a modular framework enabling flexible configuration of various encoder-LLM-decoder architectures.<n>Second, a lightweight training strategy that pre-trains audio-language alignment on the state-of-the-art vision-language model Qwen2.5-VL.<n>Third, an audio synthesis pipeline that generates high-quality audio-text data from diverse real-world scenarios.
  arXiv  Detail & Related papers  (2025-02-26T17:26:36Z)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation [81.78257799283777]
  We present LMFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities.<n>Compared to methods that pretrain multimodal generative models from scratch, our experiments demonstrate that, LMFusion improves image understanding by 20% and image generation by 3.6% using only 50% of the FLOPs.
  arXiv  Detail & Related papers  (2024-12-19T18:56:24Z)
• PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension [69.73137587705646]
  We introduce a multimodal textbfPunchline comprehension textbfPunchBenchmark, named textbfPunchBench, for accurate and comprehensive evaluation of punchline comprehension.<n>To enhance the evaluation accuracy, we generate synonymous and antonymous captions by modifying original captions.<n>On this basis, we conduct extensive evaluations and reveal a significant gap between state-of-the-art MLLMs and humans in punchline comprehension.
  arXiv  Detail & Related papers  (2024-12-16T15:52:59Z)
• Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation [61.91492500828508]
  Few-shot 3D point cloud segmentation (FS-PCS) aims at generalizing models to segment novel categories with minimal support samples.<n>We introduce a multimodal FS-PCS setup, utilizing textual labels and the potentially available 2D image modality.<n>We propose a simple yet effective Test-time Adaptive Cross-modal (TACC) technique to mitigate training bias.
  arXiv  Detail & Related papers  (2024-10-29T19:28:41Z)
• Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration [50.94902442781148]
  We propose a novel multi-modal large language model (LLM) that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. We construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances.
  arXiv  Detail & Related papers  (2023-06-15T12:45:25Z)
• VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs [103.99315770490163]
  We present a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio. Experiments demonstrate that our approach based on a single architecture outperforms the state-of-the-art on three video-based text-generation tasks.
  arXiv  Detail & Related papers  (2021-01-28T15:22:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.