Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition

• URL: http://arxiv.org/abs/2606.17627v1
• Date: Tue, 16 Jun 2026 07:31:27 GMT
• Title: Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition
• Abstract summary: Fine-grained action recognition in egocentric video is challenging for Vision-Language Models.<n>We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework.<n> Experiments show that our method positively improves zero-shot action recognition performance over the baseline.
• Score: 86.4487145812318
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-grained action recognition in egocentric video is challenging for Vision-Language Models (VLMs): actions often differ only in small visual cues, and a single model tends to be biased toward a subset of these cues. We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework in which (i) a VLM orchestrator chunks the video and proposes a top-k candidate label list per segment, (ii) an ensemble of heterogeneous VLM specialists, drawn from different open model families, engages in a structured deliberation that includes a peer-consultation round of questions, and (iii) agent rankings are aggregated with a Borda count and the orchestrator re-ranks its own prediction in light of the specialists' evidence. The entire pipeline runs locally with no fine-tuning. Experiments show that our method positively improves zero-shot action recognition performance over the baseline, highlighting the influence of a heterogeneous deliberation step, showing that the gain stems from decorrelated model priors rather than from additional compute.

Related papers

• UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning [20.624929008039402]
  UNIEGO is a unified egocentric encoder trained with nine teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four foundation models.<n>We demonstrate that structured, proxy-mediated knowledge transfer yields richer and more discriminative egocentric representations.
  arXiv  Detail & Related papers  (2026-06-18T17:59:45Z)
• DisPOSE: Projected Polystochastic Diffusion for Self-Supervised Multi-View 3D Human Pose Estimation [58.47973015036709]
  DisPOSE is a self-supervised framework that approximates the inherently discrete multi-view person-assignment problem.<n>By employing differentiable Sinkhorn projections during denoising, our model learns to guide solutions toward valid and feasible assignments.<n>The proposed approach outperforms current state-of-the-art self-supervised methods on standard datasets.
  arXiv  Detail & Related papers  (2026-06-05T16:14:39Z)
• Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks [39.645768996277184]
  Disagreement in annotation is a common phenomenon in the development of NLP datasets.<n>We propose an agreement-based clustering technique to model the disagreement between the annotators.
  arXiv  Detail & Related papers  (2026-05-11T04:04:03Z)
• Implicit Counterfactual Learning for Audio-Visual Segmentation [50.69377287012591]
  We propose the implicit counterfactual framework (ICF) to achieve unbiased cross-modal understanding.<n>Due to the lack of semantics, heterogeneous representations may lead to erroneous matches.<n>We introduce the multi-granularity implicit text (MIT) involving video-, segment- and frame-level as the bridge to establish the modality-shared space.
  arXiv  Detail & Related papers  (2025-07-28T11:46:35Z)
• Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models [30.07172193932125]
  We show that the Joint Autoencoder Modulator (JAM) induces alignment even across independently trained representations.<n>Our findings offer theoretical insight into the structure of shared semantics and practical guidance for transforming generalist unimodal foundations into specialist multimodal models.
  arXiv  Detail & Related papers  (2025-07-01T21:43:50Z)
• A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognition [71.61103962200666]
  Zero-shot named entity recognition (NER) aims to develop entity recognition systems from unannotated text corpora.<n>Recent work has adapted large language models (LLMs) for zero-shot NER by crafting specialized prompt templates.<n>We introduce the cooperative multi-agent system (CMAS), a novel framework for zero-shot NER.
  arXiv  Detail & Related papers  (2025-02-25T23:30:43Z)
• Training-Free Action Recognition and Goal Inference with Dynamic Frame Selection [51.004020874336284]
  VidTFS is a Training-free, open-vocabulary video goal and action inference framework. Our experiments demonstrate that the proposed frame selection module improves the performance of the framework significantly. We validate the performance of the proposed VidTFS on four widely used video datasets.
  arXiv  Detail & Related papers  (2024-01-23T03:45:05Z)
• Multimodal Variational Auto-encoder based Audio-Visual Segmentation [46.67599800471001]
  ECMVAE factorizes the representations of each modality with a modality-shared representation and a modality-specific representation. Our approach leads to a new state-of-the-art for audio-visual segmentation, with a 3.84 mIOU performance leap.
  arXiv  Detail & Related papers  (2023-10-12T13:09:40Z)
• Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation [63.910211095033596]
  Few-shot segmentation aims to segment unseen-class objects given only a handful of densely labeled samples. We propose a simple yet versatile framework in the spirit of divide-and-conquer. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information.
  arXiv  Detail & Related papers  (2022-04-21T06:21:14Z)
• Exploring Global Diversity and Local Context for Video Summarization [4.452227592307381]
  Video summarization aims to automatically generate a diverse and concise summary which is useful in large-scale video processing. Most of methods tend to adopt self attention mechanism across video frames, which fails to model the diversity of video frames. We propose global diverse attention by using the squared Euclidean distance instead to compute the affinities.
  arXiv  Detail & Related papers  (2022-01-27T06:56:01Z)
• MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection [76.80153360498797]
  We develop a multiple instance self-training framework (MIST) to efficiently refine task-specific discriminative representations. MIST is composed of 1) a multiple instance pseudo label generator, which adapts a sparse continuous sampling strategy to produce more reliable clip-level pseudo labels, and 2) a self-guided attention boosted feature encoder. Our method performs comparably to or even better than existing supervised and weakly supervised methods, specifically obtaining a frame-level AUC 94.83% on ShanghaiTech.
  arXiv  Detail & Related papers  (2021-04-04T15:47:14Z)
• Generalized Adversarially Learned Inference [42.40405470084505]
  We develop methods of inference of latent variables in GANs by adversarially training an image generator along with an encoder to match two joint distributions of image and latent vector pairs. We incorporate multiple layers of feedback on reconstructions, self-supervision, and other forms of supervision based on prior or learned knowledge about the desired solutions.
  arXiv  Detail & Related papers  (2020-06-15T02:18:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.