Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task

• URL: http://arxiv.org/abs/2605.00963v1
• Date: Fri, 01 May 2026 15:04:53 GMT
• Title: Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task
• Abstract summary: This manuscript extends our previous multimodal human-robot interaction system by introducing a controlled ablation study of the three modules that most strongly influence end-to-end performance.<n>The goal is not to redesign the full pipeline, but to isolate the contribution of each component under a common experimental protocol and then evaluate the best combinations end-to-end.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This manuscript extends our previous multimodal human-robot interaction system by introducing a controlled ablation study of the three modules that most strongly influence end-to-end performance: the large language model used for action extraction, the perception system used for visual grounding, and the controller used for motion execution. The goal is not to redesign the full pipeline, but to isolate the contribution of each component under a common experimental protocol and then evaluate the best combinations end-to-end. We therefore compare three language models, five perception configurations, and three controllers, followed by a second-stage factorial study over the best candidates. The resulting analysis is intended to clarify which choices primarily affect execution time, which primarily affect success rate, and where the largest engineering gains are likely to come from in future revisions of the system.

Related papers

• Robot Trajectron V3: A Probabilistic Shared Control Framework for SE(3) Manipulation [5.090943130063491]
  RT-V3 is a shared control framework designed for $SE(3)$ grasping tasks.<n>It learns a prior over user intent and combines it with real-time user commands to estimate the posterior intent distribution.<n>Experiments demonstrate that RT-V3 achieves high accuracy in trajectory prediction and competitive performance in reactive planning.
  arXiv  Detail & Related papers  (2026-07-10T11:52:18Z)
• From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping [11.17111944571978]
  This paper investigates whether a 15 percentage point gain in end-to-end task success is sufficient to produce consistent and measurable differences in user perception.<n>The baseline system combines Whisper for speech recognition, Florence-2 for open-vocabulary object detection, LLaMA 3.1 for action extraction, and an interval Type-2 fuzzy logic controller for motion execution.
  arXiv  Detail & Related papers  (2026-07-01T07:19:00Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• PCoKG: Personality-aware Commonsense Reasoning with Debate [32.49722822521962]
  Personality-aware Commonsense Knowledge Graph (PCoKG) is a structured dataset comprising 521,316 quadruples.<n>For knowledge graph construction, we leverage the role-playing capabilities of large language models.<n>We apply PCoKG to persona-based dialogue generation, where it demonstrates improved consistency between generated responses and reference outputs.
  arXiv  Detail & Related papers  (2026-01-09T15:05:01Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling [51.38330727868982]
  We show how action chunking impacts the divergence between a learner and a demonstrator.<n>We propose Bidirectional Decoding (BID), a test-time inference algorithm that bridges action chunking with closed-loop adaptation.<n>Our method boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks.
  arXiv  Detail & Related papers  (2024-08-30T15:39:34Z)
• SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation [62.58480650443393]
  Segment Anything (SAM) is a vision-foundation model for generalizable scene understanding and sequence imitation. We develop a novel multi-channel heatmap that enables the prediction of the action sequence in a single pass.
  arXiv  Detail & Related papers  (2024-05-30T00:32:51Z)
• Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection [0.0]
  Anomaly detection (AD) in images, identifying significant deviations from normality, is a critical issue in computer vision. This paper introduces a novel approach to dimensionality reduction for AD using pre-trained convolutional neural network (CNN) that incorporate EfficientNet models. Our results indicate that the proposed method surpasses both Principal Component Analysis (PCA) and Negated Principal Component Analysis (NPCA) in terms of detection accuracy.
  arXiv  Detail & Related papers  (2023-08-09T13:19:28Z)
• Body Segmentation Using Multi-task Learning [1.0832844764942349]
  We present a novel multi-task model for human segmentation/parsing that involves three tasks. The main idea behind the proposed--Pose--DensePose model (or SPD for short) is to learn a better segmentation model by sharing knowledge across different, yet related tasks. The performance of the model is analysed through rigorous experiments on the LIP and ATR datasets and in comparison to a recent (state-of-the-art) multi-task body-segmentation model.
  arXiv  Detail & Related papers  (2022-12-13T13:06:21Z)
• Multi-modal Multi-label Facial Action Unit Detection with Transformer [7.30287060715476]
  This paper describes our submission to the third Affective Behavior Analysis (ABAW) 2022 competition. We proposed a transfomer based model to detect facial action unit (FAU) in video.
  arXiv  Detail & Related papers  (2022-03-24T18:59:31Z)
• Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning [49.41766997393417]
  The system we used for Task 6 (Automated Audio Captioning)of the Detection and Classification of Acoustic Scenes and Events(DCASE) 2020 Challenge combines three elements, namely, dataaugmentation, multi-task learning, and post-processing, for audiocaptioning. The system received the highest evaluation scores, but which of the individual elements most fully contributed to its perfor-mance has not yet been clarified.
  arXiv  Detail & Related papers  (2020-09-24T01:07:33Z)
• Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation [114.48767388174218]
  This paper presents an empirical analysis on different types of dialog systems composed of different modules in different settings. Our results show that a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels.
  arXiv  Detail & Related papers  (2020-05-15T05:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.