Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality

• URL: http://arxiv.org/abs/2607.14721v1
• Date: Thu, 16 Jul 2026 08:36:54 GMT
• Title: Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
• Abstract summary: Many practical applications involve devices that are equipped with a rich set of sensors that enable multimodal sensing in their test environment.<n>This presents an opportunity to apply cross-modal learning to the multimodal data sensed by these devices to learn representations.<n>We develop Test-Space Training (TST), which performs multimodal data collection in the test environment and performs self-supervised pre-training on it.
• Score: 18.35659797985076
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-modal learning, i.e., learning to predict one modality from another, is a fundamental mechanism for self-supervision via leveraging multimodality. Many practical applications, e.g., deploying a household robot, involve devices that are equipped with a rich set of sensors that enable multimodal sensing in their test environment. This presents an opportunity to apply cross-modal learning to the multimodal data sensed by these devices to learn representations. Findings in developmental psychology also suggest that biological agents leverage it to build an effective representation of their surroundings. To study this, we propose a controlled setup, where we restrict a user device to just a given test environment. It results in a specialization setup where we attempt to develop a performant model for this specific test environment. Under this setup, we develop Test-Space Training (TST), which performs multimodal data collection in the test environment and performs self-supervised pre-training on it. We evaluate these models on various downstream tasks in the same environment. Under this setup, we find various interesting insights, such as collecting rich multimodal data only from the test environment and leveraging cross-modal learning, we can achieve competitive results with generalist models (e.g., DINOv2 and CLIP) pre-trained on large-scale internet datasets. This enables an alternative scenario where the need for external Internet-scale datasets for pre-training models is reduced. We also present a set of analyses and ablations that raise intriguing points on substituting data with (multi)modality, and how varying pre-training data enables a tradeoff between a model's abilities to specialise to a test environment, and generalize to held-out spaces.

Related papers

• More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe [52.210953881548996]
  We show that a vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks.<n>Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding.<n>Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks.
  arXiv  Detail & Related papers  (2026-07-17T13:25:44Z)
• Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models [0.0]
  We introduce LDM-v0, a multi-task, multi-modal transformer policy trained offline on trajectories collected from thousands of environments spanning multiple domains and modalities.<n>We show that a single pretrained model matches the performance of independently trained task-specific reference policies on approximately 1,000 environments including robotics, autonomous driving, inventory management, cybersecurity, trading, and video games.
  arXiv  Detail & Related papers  (2026-06-23T09:13:08Z)
• Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration [9.66105329596482]
  We propose a novel multi-modal co-learning framework capable of generalizing across various tasks without targeting a specific modality for inference.<n>Our approach combines contrastive and modality discriminative learning together to guide single-modality models to structure the internal model manifold into modality-shared and modality-specific information.
  arXiv  Detail & Related papers  (2025-10-22T13:29:32Z)
• Is Diversity All You Need for Scalable Robotic Manipulation? [50.747150672933316]
  We investigate the nuanced role of data diversity in robot learning by examining three critical dimensions-task (what to do), embodiment (which robot to use), and expert (who demonstrates)-challenging the conventional intuition of "more diverse is better"<n>We show that task diversity proves more critical than per-task demonstration quantity, benefiting transfer from diverse pre-training tasks to novel downstream scenarios.<n>We propose a distribution debiasing method to mitigate velocity ambiguity, the yielding GO-1-Pro achieves substantial performance gains of 15%, equivalent to using 2.5 times pre-training data.
  arXiv  Detail & Related papers  (2025-07-08T17:52:44Z)
• CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems [38.20651868834145]
  We propose Collaborative Auxiliary Modality Learning (CAML), a novel multi-modal multi-agent framework.<n>We show that CAML achieves up to a $bf 58.1%$ improvement in accident detection.<n>We also validate CAML on real-world aerial-ground robot data for collaborative semantic segmentation.
  arXiv  Detail & Related papers  (2025-02-25T03:59:40Z)
• Expert Routing with Synthetic Data for Continual Learning [46.01125098473039]
  Generate to Discriminate (G2D) is a continual learning method that leverages synthetic data to train a domain-discriminator.<n>We observe that G2D outperforms competitive domain-incremental learning methods on tasks in both vision and language modalities.
  arXiv  Detail & Related papers  (2024-12-22T13:16:28Z)
• Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models [11.545127156146368]
  We introduce Dual Prototype Evolving (DPE), a novel test-time adaptation approach for pre-trained vision-language models (VLMs) We create and evolve two sets of prototypes--textual and visual--to progressively capture more accurate multi-modal representations for target classes during test time. Our proposed DPE consistently outperforms previous state-of-the-art methods while also exhibiting competitive computational efficiency.
  arXiv  Detail & Related papers  (2024-10-16T17:59:49Z)
• Multi-Domain Joint Training for Person Re-Identification [51.73921349603597]
  Deep learning-based person Re-IDentification (ReID) often requires a large amount of training data to achieve good performance. It appears that collecting more training data from diverse environments tends to improve the ReID performance. We propose an approach called Domain-Camera-Sample Dynamic network (DCSD) whose parameters can be adaptive to various factors.
  arXiv  Detail & Related papers  (2022-01-06T09:20:59Z)
• Multimodal Adversarially Learned Inference with Factorized Discriminators [10.818838437018682]
  We propose a novel approach to generative modeling of multimodal data based on generative adversarial networks. To learn a coherent multimodal generative model, we show that it is necessary to align different encoder distributions with the joint decoder distribution simultaneously. By taking advantage of contrastive learning through factorizing the discriminator, we train our model on unimodal data.
  arXiv  Detail & Related papers  (2021-12-20T08:18:49Z)
• Multitask Adaptation by Retrospective Exploration with Learned World Models [77.34726150561087]
  We propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from task-agnostic storage. The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage.
  arXiv  Detail & Related papers  (2021-10-25T20:02:57Z)
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments [89.04823188871906]
  Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. Most of the existing simulators rely on randomly generating the environments. We introduce the benefits of adopting an existing formal scenario specification language, SCENIC, to assist researchers.
  arXiv  Detail & Related papers  (2021-06-18T21:49:46Z)
• Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos [69.61522804742427]
  This paper proposes a self-supervised training framework that learns a common multimodal embedding space. We extend the concept of instance-level contrastive learning with a multimodal clustering step to capture semantic similarities across modalities. The resulting embedding space enables retrieval of samples across all modalities, even from unseen datasets and different domains.
  arXiv  Detail & Related papers  (2021-04-26T15:55:01Z)
• Towards an Automatic Analysis of CHO-K1 Suspension Growth in Microfluidic Single-cell Cultivation [63.94623495501023]
  We propose a novel Machine Learning architecture, which allows us to infuse a neural deep network with human-powered abstraction on the level of data. Specifically, we train a generative model simultaneously on natural and synthetic data, so that it learns a shared representation, from which a target variable, such as the cell count, can be reliably estimated.
  arXiv  Detail & Related papers  (2020-10-20T08:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.