Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

• URL: http://arxiv.org/abs/2607.13188v1
• Date: Tue, 14 Jul 2026 18:39:29 GMT
• Title: Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes
• Abstract summary: We introduce $textbfSelf-Correcting Coupled Markov Jump Processes (SC-CMJP)<n>In conjunction with SC-CMJP, we introduce $textttCO_texttt2textttJump$, a novel training-free single-pass sampler for joint multimodal geneneration.<n>For training and evaluation purposes, we have created and will release three large-scale joint multimodal generation corpora.
• Score: 70.61868608402723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human cognition does not separate understanding and generation. A teacher at a whiteboard speaks and draws $\textit{together}$, each modality reshapes the other. In this paper, we bring this coupled loop to artificial systems. Masked Diffusion Models (MDMs) are ideally suited to this task, yet existing samplers either decode text and image interleavedly or independently update them in parallel branches that share only previous-step history, but not the other modality's latest decisions $\textit{within}$ the same step; combined with MDMs' inability to remask, cross-modal contradictions are neither detected nor repaired. We introduce $\textbf{Self-Correcting Coupled Markov Jump Processes (SC-CMJP)}$, a framework in which one modality's transition rates are functionals of the other modality's confidence score, as weighted by cross-modal attention. Furthermore, a remasking jump retracts commitments the moment cross-modal evidence turns against them. In conjunction with SC-CMJP, we introduce $\texttt{CO}_\texttt{2}\texttt{Jump}$ (Self-$\underline{\text{CO}}$rrecting $\underline{\text{CO}}$upled $\underline{\text{Jump}}$), a novel training-free single-pass sampler for joint multimodal geneneration. For training and evaluation purposes, we have created and will release three large-scale joint multimodal generation corpora: $\text{JEdit-1M}$, $\text{JMaze-200K}$, $\text{JNono-200K}$, with matching in- and out-of-distribution benchmarks. $\texttt{CO}_\texttt{2}\texttt{Jump}$ achieves best joint performance for image understanding and editing as well as visual reasoning (maze and nonogram solving). The performance of the sampler scales monotonically with the number of denoising steps, evidence that the benefits of cross-modal coupling $\textit{compound}$ across the trajectory. Project page: https://coupled-jump.github.io

Related papers

• HARTS: Efficient Agentic Reinforcement Learning for Hybrid-Attention Models over Arbitrary Rollout Trees [10.936836651826722]
  We present HARTS (Hybrid-Attention RL over Tree Structures)<n>HarTS is the first system to demonstrate arbitrary-rollout-tree prefix-sharing speedups on a real hybrid-attention model.
  arXiv  Detail & Related papers  (2026-08-28T10:18:54Z)
• JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models [21.274538255180314]
  We present textbfJAGG, which reduces full transformer backward passes from $W$ to $2$ per group of $W$ consecutive steps.<n>Experiments on T2I benchmarks show JAGG delivers $sim$times$ backward speedup with negligible quality degradation.
  arXiv  Detail & Related papers  (2026-07-20T05:30:40Z)
• Temporal Memory for Resource-Constrained Agents: Continual Learning via Stochastic Compress-Add-Smooth [0.0]
  An agent that operates sequentially must incorporate new experience without forgetting old experience, under a fixed memory budget.<n>We propose a framework in which memory is not a parameter vector but a process: a Diffusion Bridge on a replay interval.<n>The framework provides a fully analytical Ising model'' of continual learning in which the mechanism, rate, and form of forgetting can be studied with mathematical precision.
  arXiv  Detail & Related papers  (2026-03-31T12:04:11Z)
• Learning Accurate Segmentation Purely from Self-Supervision [87.78965637247107]
  Selfment is a fully self-supervised framework that segments objects directly from raw images without human labels.<n>Selfment sets new state-of-the-art (SoTA) results across multiple benchmarks.
  arXiv  Detail & Related papers  (2026-02-27T07:36:32Z)
• Canonicalizing Multimodal Contrastive Representation Learning [76.15228959754727]
  We show that across model families such as CLIP, SigLIP, and FLAVA, a geometric relationship exists between their embedding spaces.<n>This finding enables backward-compatible model upgrades, avoiding costly re-embedding, and has implications for the privacy of learned representations.
  arXiv  Detail & Related papers  (2026-02-19T18:09:36Z)
• Uncovering Untapped Potential in Sample-Efficient World Model Agents [51.65485693709418]
  Simulus is a highly modular TBWM agent that integrates a multi-modality tokenization framework, intrinsic motivation, prioritized WM replay, and regression-as-classification.<n>Simulus achieves state-of-the-art sample efficiency for planning-free WMs across three diverse benchmarks.
  arXiv  Detail & Related papers  (2025-02-17T08:06:10Z)
• Reasoning to Attend: Try to Understand How <SEG> Token Works [44.33848900059659]
  We show that the $texttSEG>$ token contributes to semantic similarity within image-text pairs.<n>We present READ, which facilitates LMMs' resilient $textbfREA$soning capability of where to atten$textbfD$ under the guidance of highly activated points.
  arXiv  Detail & Related papers  (2024-12-23T17:44:05Z)
• Truncated Consistency Models [57.50243901368328]
  Training consistency models requires learning to map all intermediate points along PF ODE trajectories to their corresponding endpoints.<n>We empirically find that this training paradigm limits the one-step generation performance of consistency models.<n>We propose a new parameterization of the consistency function and a two-stage training procedure that prevents the truncated-time training from collapsing to a trivial solution.
  arXiv  Detail & Related papers  (2024-10-18T22:38:08Z)
• Federated Combinatorial Multi-Agent Multi-Armed Bandits [79.1700188160944]
  This paper introduces a federated learning framework tailored for online optimization with bandit. In this setting, agents subsets of arms, observe noisy rewards for these subsets without accessing individual arm information, and can cooperate and share information at specific intervals.
  arXiv  Detail & Related papers  (2024-05-09T17:40:09Z)
• M$^{2}$Chat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generation [24.070247575655998]
  We propose textbf$M2Chat$, a novel unified multimodal LLM framework for generating interleaved text-image conversation.<n>$M3Adapter$ integrates granular low-level visual information and high-level semantic features from multi-modality prompts.<n>$M3FT$ fine-tuning strategy optimize disjoint groups of parameters for image-text alignment and visual-instruction.
  arXiv  Detail & Related papers  (2023-11-29T11:30:33Z)
• Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models [69.31424345583537]
  Humans use cross-modal information to learn new concepts efficiently. We show that one can indeed build a better $bf visual$ dog classifier by reading about dogs and listening to them bark. We construct the first (to our knowledge) audiovisual few-shot benchmark and use cross-modal training to improve the performance of both image and audio classification.
  arXiv  Detail & Related papers  (2023-01-16T05:40:42Z)
• Simplifying and Understanding State Space Models with Diagonal Linear RNNs [56.33053691749856]
  This work disposes of the discretization step, and proposes a model based on vanilla Diagonal Linear RNNs. We empirically show that, despite being conceptually much simpler, $mathrmDLR$ is as performant as previously-proposed SSMs. We also characterize the expressivity of SSMs and attention-based models via a suite of $13$ synthetic sequence-to-sequence tasks.
  arXiv  Detail & Related papers  (2022-12-01T18:53:06Z)
• Cluster and Aggregate: Face Recognition with Large Probe Set [18.662943303044315]
  We propose a two-stage feature fusion paradigm, Cluster and Aggregate, that can both scale to large $N$ and maintain the ability to perform sequential inference with order invariance. Experiments on IJB-B and IJB-S benchmark datasets show the superiority of the proposed two-stage paradigm in unconstrained face recognition.
  arXiv  Detail & Related papers  (2022-10-19T20:01:15Z)
• Decentralized Cooperative Reinforcement Learning with Hierarchical Information Structure [14.919120396838208]
  We consider two-agent multi-armed bandits (MABs) and Markov decision processes (MDPs) with a hierarchical information structure arising in applications. In each step the leader" chooses her action first, and then the follower" decides his action after observing the leader's action. For the MDP setting, we obtain $widetildemathcalO(sqrtH7S2ABT)$ regret, where $H$ is the number of steps per episode, $S$ is the number of states
  arXiv  Detail & Related papers  (2021-11-01T09:18:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.