Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Folding

• URL: http://arxiv.org/abs/2505.07600v1
• Date: Mon, 12 May 2025 14:24:03 GMT
• Title: Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Folding
• Abstract summary: BiFold is a model that predicts language-conditioned pick-and-place actions from visual observations.<n>We present evidence that its fine-tuning and temporal context enable effective alignment between text and image regions.
• Score: 6.0914269533287175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Manipulating clothes is challenging due to their complex dynamics, high deformability, and frequent self-occlusions. Garments exhibit a nearly infinite number of configurations, making explicit state representations difficult to define. In this paper, we analyze BiFold, a model that predicts language-conditioned pick-and-place actions from visual observations, while implicitly encoding garment state through end-to-end learning. To address scenarios such as crumpled garments or recovery from failed manipulations, BiFold leverages temporal context to improve state estimation. We examine the internal representations of the model and present evidence that its fine-tuning and temporal context enable effective alignment between text and image regions, as well as temporal consistency.

Related papers

• ChronoVision: Temporal Reasoning via Latent State Reconstruction [24.305503417252485]
  ChronoVision is a framework designed to align visual logic with latent imagery.<n>Vbvr-VQA is a dataset that evaluates temporal tracking by reformulating video reasoning into a strict image-ordering task.
  arXiv  Detail & Related papers  (2026-08-06T05:58:22Z)
• The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context [76.36635039699112]
  Large language models (LLMs) are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context.<n>We find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level.<n>This aggregate stability, however, masks significant per-example instability.
  arXiv  Detail & Related papers  (2026-07-14T17:01:12Z)
• Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models [154.39583176906893]
  Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
  arXiv  Detail & Related papers  (2026-05-09T14:32:36Z)
• SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes [69.03114625064282]
  Multimodal large language models often struggle with faithful reasoning in complex visual scenes.<n>Existing preference-based approaches fail to address this challenge as they allow models to exploit language priors to bypass visual grounding.<n>We propose SceneAlign, a framework that leverages scene graphs as structured visual information to perform controllable structural interventions.
  arXiv  Detail & Related papers  (2026-01-09T07:40:39Z)
• ChronusOmni: Improving Time Awareness of Omni Large Language Models [29.685563616290352]
  Time awareness is a fundamental ability of omni large language models, especially for understanding long videos and answering complex questions.<n>We propose ChronusOmni, an omni large language model designed to enhance temporal awareness for both explicit and implicit audiovisual temporal grounding.<n>We construct ChronusAV, a temporally-accurate, modality-complete, and cross-modal-aligned dataset to support the training and evaluation on audiovisual temporal grounding task.
  arXiv  Detail & Related papers  (2025-12-10T17:22:42Z)
• Learning Time in Static Classifiers [44.358377952850994]
  We propose a simple yet effective framework that equips standard feedforward classifiers with temporal reasoning.<n>We use a novel Support-Exemplar-Query (SEQ) learning paradigm, which structures training data into temporally coherent trajectories.<n>Our approach bridges static and temporal learning in a modular and data-efficient manner, requiring only a simple on top of pre-extracted features.
  arXiv  Detail & Related papers  (2025-11-15T18:42:51Z)
• Priors in Time: Missing Inductive Biases for Language Model Interpretability [58.07412640266836]
  We show that Sparse Autoencoders impose priors that assume independence of concepts across time, implying stationarity.<n>We introduce a new interpretability objective -- Temporal Feature Analysis -- which possesses a temporal inductive bias to decompose representations at a given time into two parts.<n>Our results underscore the need for inductive biases that match the data in designing robust interpretability tools.
  arXiv  Detail & Related papers  (2025-11-03T18:43:48Z)
• Real-Time Per-Garment Virtual Try-On with Temporal Consistency for Loose-Fitting Garments [11.732243977359412]
  We propose a two-stage approach for robust semantic map estimation.<n>First, we extract a garment-invariant representation from the raw input image.<n>This representation is then passed through an auxiliary network to estimate the semantic map.
  arXiv  Detail & Related papers  (2025-06-14T04:57:21Z)
• Towards Foundation Model on Temporal Knowledge Graph Reasoning [17.165969719351125]
  Temporal Knowledge Graphs (TKGs) store temporal facts with quadruple formats (s, p, o, t)<n>New model employs sinusoidal positional encodings to capture fine-grained temporal patterns.<n>PostRA demonstrates strong zero-shot performance on unseen temporal knowledge graphs.
  arXiv  Detail & Related papers  (2025-06-04T09:19:49Z)
• Deep Temporal Reasoning in Video Language Models: A Cross-Linguistic Evaluation of Action Duration and Completion through Perfect Times [0.0]
  We introduce the textbfPerfect Times dataset, a quadrilingual (English, Italian, Russian, and Japanese) multiple-choice question-answering benchmark designed to assess video-language models (VLMs) on temporal reasoning.<n> Experimental results indicate that state-of-the-art models, despite their success on text-based tasks, struggle to mirror human-like temporal and causal reasoning grounded in video.
  arXiv  Detail & Related papers  (2025-06-01T09:45:41Z)
• SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situation [33.02002580363215]
  Vision-language temporal alignment is a crucial capability for human dynamic recognition and cognition in real-world scenarios.<n>We introduce SVLTA, the Synthetic Vision-Language Temporal Alignment derived via a well-designed and feasible control generation method within a simulation environment.<n>Our experiments reveal diagnostic insights through the evaluations in temporal question answering, distributional shift sensitiveness, and temporal alignment adaptation.
  arXiv  Detail & Related papers  (2025-04-08T11:31:37Z)
• Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models [24.586475741345616]
  We tackle the issue of temporal inconsistency in large language models (LLMs) by proposing a novel counterfactual prompting approach.<n>Our method generates counterfactual questions and enforces collective constraints, enhancing the model's consistency.<n>We evaluate our method on multiple datasets, demonstrating significant improvements in event ordering for explicit and implicit events and temporal commonsense understanding.
  arXiv  Detail & Related papers  (2025-02-17T04:37:07Z)
• Sequential Representation Learning via Static-Dynamic Conditional Disentanglement [58.19137637859017]
  This paper explores self-supervised disentangled representation learning within sequential data, focusing on separating time-independent and time-varying factors in videos. We propose a new model that breaks the usual independence assumption between those factors by explicitly accounting for the causal relationship between the static/dynamic variables. Experiments show that the proposed approach outperforms previous complex state-of-the-art techniques in scenarios where the dynamics of a scene are influenced by its content.
  arXiv  Detail & Related papers  (2024-08-10T17:04:39Z)
• On the Consistency and Robustness of Saliency Explanations for Time Series Classification [4.062872727927056]
  Saliency maps have been applied to interpret time series windows as images. This paper extensively analyzes the consistency and robustness of saliency maps for time series features and temporal attribution.
  arXiv  Detail & Related papers  (2023-09-04T09:08:22Z)
• Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation [66.86987509942607]
  We evaluate how such a paradigm should be done in imitation learning. We consider a setting where the pretraining corpus consists of multitask demonstrations. We argue that inverse dynamics modeling is well-suited to this setting.
  arXiv  Detail & Related papers  (2023-05-26T14:40:46Z)
• Generating Coherent Narratives by Learning Dynamic and Discrete Entity States with a Contrastive Framework [68.1678127433077]
  We extend the Transformer model to dynamically conduct entity state updates and sentence realization for narrative generation. Experiments on two narrative datasets show that our model can generate more coherent and diverse narratives than strong baselines.
  arXiv  Detail & Related papers  (2022-08-08T09:02:19Z)
• Video Activity Localisation with Uncertainties in Temporal Boundary [74.7263952414899]
  Methods for video activity localisation over time assume implicitly that activity temporal boundaries are determined and precise. In unscripted natural videos, different activities transit smoothly, so that it is intrinsically ambiguous to determine in labelling precisely when an activity starts and ends over time. We introduce Elastic Moment Bounding (EMB) to accommodate flexible and adaptive activity temporal boundaries.
  arXiv  Detail & Related papers  (2022-06-26T16:45:56Z)
• Interpretable Time-series Representation Learning With Multi-Level Disentanglement [56.38489708031278]
  Disentangle Time Series (DTS) is a novel disentanglement enhancement framework for sequential data. DTS generates hierarchical semantic concepts as the interpretable and disentangled representation of time-series. DTS achieves superior performance in downstream applications, with high interpretability of semantic concepts.
  arXiv  Detail & Related papers  (2021-05-17T22:02:24Z)
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference [31.97227651679233]
  Multi-object scene decomposition is a fast-emerging problem in learning. We show that our model has a high accuracy even without color information. We demonstrate the decomposition, segmentation prediction capabilities of our model and show that it outperforms the state-of-the-art on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-06-25T22:57:17Z)
• Temporal Embeddings and Transformer Models for Narrative Text Understanding [72.88083067388155]
  We present two approaches to narrative text understanding for character relationship modelling. The temporal evolution of these relations is described by dynamic word embeddings, that are designed to learn semantic changes over time. A supervised learning approach based on the state-of-the-art transformer model BERT is used instead to detect static relations between characters.
  arXiv  Detail & Related papers  (2020-03-19T14:23:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.