BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding

• URL: http://arxiv.org/abs/2605.12074v1
• Date: Tue, 12 May 2026 13:01:25 GMT
• Title: BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding
• Abstract summary: We introduce BARISTA, a densely annotated egocentric dataset and benchmark of 185 real-world coffee-preparation videos.<n>We derive zero-shot language-based tasks spanning phrase grounding, hand-object interaction recognition, referring, activity recognition, relation extraction, and temporal visual question answering.<n>Experiments reveal strong variation across task families and no consistently dominant model family, positioning BARISTA as a challenging diagnostic benchmark for procedural video understanding.
• Score: 6.330463283335849
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scene understanding is central to general physical intelligence, and video is a primary modality for capturing both state and temporal dynamics of a scene. Yet understanding physical processes remains difficult, as models must combine object localization, hand-object interactions, relational parsing, temporal reasoning, and step-level procedural inference. Existing benchmarks usually evaluate these capabilities separately, limiting diagnosis of why models fail on procedural tasks. We introduce BARISTA, a densely annotated egocentric dataset and benchmark of 185 real-world coffee-preparation videos covering fully automatic, portafilter-based, and capsule-based workflows. BARISTA provides verified per-frame scene graphs linking persistent object identities to masks, tracks, boxes, attributes, typed relations, hand-object interactions, activities, and process steps. From these graphs, we derive zero-shot language-based tasks spanning phrase grounding, hand-object interaction recognition, referring, activity recognition, relation extraction, and temporal visual question answering. Experiments reveal strong variation across task families and no consistently dominant model family, positioning BARISTA as a challenging diagnostic benchmark for procedural video understanding. Code and dataset available at https://huggingface.co/datasets/ramblr/BARISTA.

Related papers

• Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis [55.78822230833247]
  We present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis.<n>Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations.<n>We propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimize interaction reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2026-08-20T17:51:48Z)
• Unified Video Dense Prediction from Disjoint Data [71.9957801898161]
  We introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials.<n>We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors.<n>UniD achieves competitive performance against per-task specialists and multi-task baselines.
  arXiv  Detail & Related papers  (2026-07-23T17:59:50Z)
• IMBench: A Benchmark for Intuitive Robotic Manipulation [11.843368723732679]
  Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints.<n>Existing benchmarks fail to capture this integration.<n>We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution.
  arXiv  Detail & Related papers  (2026-07-17T05:34:29Z)
• TriRelVLA: Triadic Relational Structure for Generalizable Embodied Manipulation [54.81449795163812]
  Vision-language--action (VLA) models perform well on training-seen robotic tasks but struggle to generalize to unseen scenes and objects.<n>We propose TriRelVLA, a triadic relational VLA framework for generalizable embodied manipulation.<n> Experiments show strong performance on fine-tuned tasks and clear gains in cross-scene, cross-object, and cross-task generalization.
  arXiv  Detail & Related papers  (2026-05-07T05:57:49Z)
• VISTA: Video Interaction Spatio-Temporal Analysis Benchmark [47.42490151556478]
  We introduce VISTA, a benchmark for open-set, multi-action-temporal understanding in Vision-Language Models (VLMs)<n>Our benchmark integrates multiple datasets into a single interaction-aware benchmarks and comprises 12K curated video- pairs.<n>Overall, VISTA is the first, large-scale, interaction-aware diagnostic benchmark fortemporal understanding inVLMss.
  arXiv  Detail & Related papers  (2026-05-02T11:28:20Z)
• Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning [27.48318501076437]
  Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints.<n>We introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy.<n>We demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models.
  arXiv  Detail & Related papers  (2026-03-25T12:52:32Z)
• ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos [44.050522958181496]
  ToG-Bench is the first task-oriented intrinsic-temporal video grounding benchmark for ego videos.<n>Built upon videos sourced from ScanNet, ToG-Bench comprises 100 annotated clips with 2,704 task-oriented grounding instructions.<n>Extensive experiments reveal the challenges of task-oriented STVG and substantial performance gaps across explicit-implicit and multi-object grounding.
  arXiv  Detail & Related papers  (2025-12-03T10:54:44Z)
• Tracking and Segmenting Anything in Any Modality [75.32774085793498]
  We propose a universal tracking and segmentation framework named SATA, which unifies a broad spectrum of tracking and segmentation subtasks with any modality input.<n> SATA demonstrates superior performance on 18 challenging tracking and segmentation benchmarks, offering a novel perspective for more generalizable video understanding.
  arXiv  Detail & Related papers  (2025-11-22T09:09:22Z)
• SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding [48.64661382961745]
  We introduce Spatio-temporal Video Action Grounding (SVAG), a novel task that requires models to simultaneously detect, track, and temporally localize all referent objects in videos.<n>To support this task, we construct SVAG-Bench, a large-scale benchmark comprising 688 videos, 19,590 annotated records, and 903 unique verbs.<n> Empirical results show that existing models perform poorly on SVAG, particularly in dense or complex scenes.
  arXiv  Detail & Related papers  (2025-10-14T22:10:49Z)
• Contextualized Representation Learning for Effective Human-Object Interaction Detection [17.242400169885453]
  Human-Object Interaction (HOI) detection aims to simultaneously localize human-object pairs and recognize their interactions.<n>We introduce a Contextualized Representation Learning that integrates both affordance-guided reasoning and contextual prompts.<n>Our proposed method demonstrates superior performance on both the HICO-Det and V-COCO datasets in most scenarios.
  arXiv  Detail & Related papers  (2025-09-16T08:03:16Z)
• Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description [56.69740649781989]
  3D scene understanding is a long-standing challenge in computer vision and a key component in enabling mixed reality, wearable computing, and embodied AI.<n>We introduce Articulate3D, an expertly curated 3D dataset featuring high-quality manual annotations on 280 indoor scenes.<n>We also present USDNet, a novel unified framework capable of simultaneously predicting part segmentation along with a full specification of motion attributes for articulated objects.
  arXiv  Detail & Related papers  (2024-12-02T11:33:55Z)
• ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection [101.56529337489417]
  We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of human, action, object> in images. We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs. Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities.
  arXiv  Detail & Related papers  (2020-08-14T09:11:18Z)
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation [50.034189314258356]
  We propose a graph model for video captioning that exploits object interactions in space and time. Our model builds interpretable links and is able to provide explicit visual grounding. To avoid correlations caused by the variable number of objects, we propose an object-aware knowledge distillation mechanism.
  arXiv  Detail & Related papers  (2020-03-31T03:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.