From "What-If" to "What-Is": Counterfactual Thinking-Inspired Semantic Alignment for Visual Brain Decoding

• URL: http://arxiv.org/abs/2608.15163v1
• Date: Sat, 15 Aug 2026 10:40:53 GMT
• Title: From "What-If" to "What-Is": Counterfactual Thinking-Inspired Semantic Alignment for Visual Brain Decoding
• Abstract summary: Visual brain decoding reconstructs visual content perceived by a person from neural measurements such as fMRI.<n>We propose ConceptAlign, a counterfactual semantic alignment framework for visual brain decoding.<n>We introduce a systematic three-level semantic evaluation framework covering foundational discriminability, counterfactual semantic discrimination, and representational geometry.
• Score: 21.204310001308077
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual brain decoding reconstructs visual content perceived by a person from neural measurements such as fMRI, providing a computational approach to studying how visual information is represented in the brain. Recent multimodal representations and diffusion priors have improved reconstruction realism. However, visually plausible reconstructions may contain incorrect objects, attributes, or relations because a strong generative prior can complete content not sufficiently specified by the decoded representation. Conventional reconstruction metrics mainly assess the final image and may therefore obscure such semantic errors. We propose ConceptAlign, a counterfactual semantic alignment framework for visual brain decoding. ConceptAlign pools decoded visual tokens and projects them into a frozen text-embedding space, aligning the representation with the ground-truth caption while separating it from scene-preserving near-miss alternatives. Generated offline by an LLM, these alternatives modify one critical object, attribute, or relation while retaining the scene. A margin-based objective learns fine-grained semantic boundaries between the observed stimulus and plausible but incorrect interpretations without requiring LLM calls during inference. We introduce a systematic three-level semantic evaluation framework covering foundational discriminability, counterfactual description discrimination, and representational geometry. Experiments on the Natural Scenes Dataset show that ConceptAlign improves reconstruction measures, counterfactual semantic discrimination, and representational alignment over the MindEye2 backbone. Matched negative-source ablations, independent LLM and human-written alternatives, and human evaluation support the effectiveness and robustness of the supervision, with favorable patterns in fine-grained conflicts, limited-data decoding, and cross-subject structure.

Related papers

• MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions [0.0]
  Erasing concept-specific information from representations has been proven useful for mitigating bias or interpreting model decisions.<n>We revisit the optimal bounds of concept erasure to derive a novel class of erasure functions that naturally induce a deterministic, dual counterfactual mapping.
  arXiv  Detail & Related papers  (2026-09-10T08:47:11Z)
• Are You Thinking What I am Thinking? : Examining Conceptual Separation in Neural Architectures [0.0]
  We examine the conceptual organisation in Convolutional Neural Networks (CNNs) and Large Language Models (LLMs)<n>In CNNs, familiar ImageNet concepts form coherent and semantically ordered representations, while this weakens for unseen concepts.<n>In LLMs, clearly distinct domains remain well separated, related move closer together, and the distinction between ambiguous topics collapses.<n>These results suggest conceptual separation can reveal structure that output accuracy alone cannot, and may serve as a useful diagnostic of how robustly a model represents the concepts it is asked to identify.
  arXiv  Detail & Related papers  (2026-09-01T05:53:05Z)
• Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence [7.7737164302813895]
  MD-SigLIP aims to align brain embeddings with text embeddings in a shared semantic space, enabling retrieval-based decoding.<n>This formulation enables explicit modeling of the correspondence between neural representations and language semantics.<n> Experiments demonstrate state-of-the-art retrieval performance under both full-vocabulary and subset evaluation settings.
  arXiv  Detail & Related papers  (2026-08-17T14:47:15Z)
• CodePercept: Code-Grounded Visual STEM Perception for MLLMs [53.60065070334941]
  Our work focuses on systematically enhancing the perception capabilities of MLLMs by establishing code as a powerful perceptual medium.<n>Specifically, we construct ICC-1M, a large-scale dataset comprising 1M Image-Caption-Code triplets.<n>We further introduce STEM2Code-Eval, a novel benchmark that directly evaluates visual perception in STEM domains.
  arXiv  Detail & Related papers  (2026-03-11T13:32:58Z)
• SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstruction [52.34513874272676]
  We argue that existing methods rely too heavily on entangled visual embeddings over explicit semantic identity.<n>We parse fMRI signals into rich, sentence-level semantic descriptions that mirror the hierarchical and compositional nature of human visual understanding.<n>We propose SynMind, a framework that integrates these explicit semantic encodings with visual priors to condition a pretrained diffusion model.
  arXiv  Detail & Related papers  (2026-01-25T14:31:23Z)
• Insight: Interpretable Semantic Hierarchies in Vision-Language Encoders [52.94006363830628]
  Language-aligned vision foundation models perform strongly across diverse downstream tasks.<n>Recent works decompose these representations into human-interpretable concepts, but provide poor spatial grounding and are limited to image classification tasks.<n>We propose Insight, a language-aligned concept foundation model that provides fine-grained concepts, which are human-interpretable and spatially grounded in the input image.
  arXiv  Detail & Related papers  (2026-01-20T09:57:26Z)
• VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set [80.50996301430108]
  The alignment of vision-language representations endows current Vision-Language Models with strong multi-modal reasoning capabilities.<n>We propose VL-SAE, a sparse autoencoder that encodes vision-language representations into its hidden activations.<n>For interpretation, the alignment between vision and language representations can be understood by comparing their semantics with concepts.
  arXiv  Detail & Related papers  (2025-10-24T10:29:31Z)
• Semantic Discrepancy-aware Detector for Image Forgery Identification [21.695863229449742]
  The misalignment between the forgery and semantic concept spaces hinders the model's forgery detection performance.<n>We propose a novel Semantic Discrepancy-aware Detector (SDD) that leverages reconstruction learning to align the two spaces at a fine-grained visual level.<n>A concept-level forgery discrepancy learning module, built upon a visual reconstruction paradigm, is proposed to strengthen the interaction between visual semantic concepts and forgery traces.
  arXiv  Detail & Related papers  (2025-08-17T12:11:09Z)
• Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation [100.81837601210597]
  We propose Concept Curation (CoCu) to bridge the gap between visual and textual semantics in pre-training data. CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin.
  arXiv  Detail & Related papers  (2023-09-24T00:05:39Z)
• Text-Video Retrieval with Disentangled Conceptualization and Set-to-Set Alignment [17.423361070781876]
  We propose the Disentangled Conceptualization and Set-to-set Alignment (DiCoSA) to simulate the conceptualizing and reasoning process of human beings. For disentangled conceptualization, we divide the coarse feature into multiple latent factors related to semantic concepts. For set-to-set alignment, where a set of visual concepts correspond to a set of textual concepts, we propose an adaptive pooling method to aggregate semantic concepts.
  arXiv  Detail & Related papers  (2023-05-20T15:48:47Z)
• Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
  Concept learning constructs visual representations that are connected to linguistic semantics. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts. We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
  arXiv  Detail & Related papers  (2022-05-28T14:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.