Stateful Visual Encoders for Vision-Language Models
Abstract Overview
This paper studies a limitation of open-weight vision-language models in multi-image and multi-turn settings: their visual encoders usually process each image independently, leaving cross-image comparison to the language model. The authors introduce a Stateful Visual Encoder (SVE) that conditions the current image representation on visual features from prior images inside the vision backbone itself. They compare several stateful designs and find that a cross-attention-plus-FFN variant is the strongest across controlled tasks for spatial aggregation, object-level visual differencing, and visual trajectory behavioral cloning. The paper also validates the approach on real-world tasks in longitudinal radiology, fine-grained image comparison, and remote sensing, while analyzing how the learned features become context-dependent and spatially selective.
Novelty
The main novelty is moving cross-image state handling into the visual encoder rather than relying on the language model to compare independently encoded images. The work also presents a practical finetuning recipe for adding this statefulness to pretrained open-weight VLMs without replacing the visual backbone or retraining the full model from scratch.
Results
Across controlled evaluations, the Cross+FFN SVE variant consistently outperforms stateless baselines and other stateful alternatives, including better spatial aggregation errors, higher CLEVR change accuracy, and lower perplexity on VisGym tasks. In real-world tests, SVE improves a Qwen3.5-4B baseline on Medical-Diff-VQA and LEVIR-CC, and on LEVIR-CC it is reported to surpass prior specialist models; the paper also reports favorable judge-based preferences on ImgEdit.
Key Points
- SVE injects causal cross-image interactions directly into the visual backbone, enabling the current image representation to use prior visual context before tokens reach the language model.
- Among the tested designs, token-level cross-attention with an added FFN performs best, and its gains depend on specific training choices such as cloned initialization, zero-initialized output branches, positional embeddings, and handling of prior-image features.
- The method appears general across resolutions, model sizes, and several VLM families, and feature analysis suggests the learned stateful updates are context-dependent, task-relevant, and spatially sparse.
References
- arXiv: https://arxiv.org/abs/2606.04433v1
- Fugu-MT: https://fugumt.com/fugumt/paper_check/2606.04433v1
- Hugging Face Papers: https://huggingface.co/papers/2606.04433
- Project: https://statefulvisualencoders.github.io/