Learning Multimodal Confidence for Intention Recognition in Human-Robot Interaction

• URL: http://arxiv.org/abs/2405.14116v1
• Date: Thu, 23 May 2024 02:43:31 GMT
• Title: Learning Multimodal Confidence for Intention Recognition in Human-Robot Interaction
• Abstract summary: We propose a novel learning-based multimodal fusion framework Batch Multimodal Confidence Learning for Opinion Pool (BMCLOP) Our approach combines Bayesian multimodal fusion method and batch confidence learning algorithm to improve accuracy, uncertainty reduction and success rate. Our desired assistive scenarios consider three modalities gestures, speech and gaze, all of which produce categorical distributions over all the finite intentions.
• Score: 6.658036685588584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid development of collaborative robotics has provided a new possibility of helping the elderly who has difficulties in daily life, allowing robots to operate according to specific intentions. However, efficient human-robot cooperation requires natural, accurate and reliable intention recognition in shared environments. The current paramount challenge for this is reducing the uncertainty of multimodal fused intention to be recognized and reasoning adaptively a more reliable result despite current interactive condition. In this work we propose a novel learning-based multimodal fusion framework Batch Multimodal Confidence Learning for Opinion Pool (BMCLOP). Our approach combines Bayesian multimodal fusion method and batch confidence learning algorithm to improve accuracy, uncertainty reduction and success rate given the interactive condition. In particular, the generic and practical multimodal intention recognition framework can be easily extended further. Our desired assistive scenarios consider three modalities gestures, speech and gaze, all of which produce categorical distributions over all the finite intentions. The proposed method is validated with a six-DoF robot through extensive experiments and exhibits high performance compared to baselines.

Related papers

• A Confidence-Aware Multimodal Fusion Framework for Industrial Human-Robot Collaboration [4.927894263207565]
  This framework fuses four heterogeneous modalities including object 6D pose, gaze, skeletal motion and IMU-based hand motion.<n>It embeds a confidence-trend-driven dynamic fusion mechanism into BiLSTM to adaptively balance bidirectional temporal features.<n>In practical assembly tasks, the framework enables proactive and stable human-robot cooperation with strong environmental adaptability.
  arXiv  Detail & Related papers  (2026-09-09T15:36:22Z)
• MARS Policy: Multimodality Only When It Matters [26.027481334743854]
  multimodality, which enables robots to capture diverse yet valid behavioral patterns, has driven the rapid emergence of generative policies as a dominant paradigm in robot learning.<n>We propose the Modality-Adaptive Robot Sampling (MARS) policy, which adaptively invokes tailored multimodality only when it is truly beneficial.<n>MARS bridges the gap between the multimodal capability of generative policies and the superior training and inference efficiency of deterministic models.
  arXiv  Detail & Related papers  (2026-05-28T11:12:30Z)
• Confidence Estimation for LLMs in Multi-turn Interactions [48.081802290688394]
  This work presents the first systematic study of confidence estimation in multi-turn interactions.<n>We establish a formal evaluation framework grounded in two key desideratas: per-turn calibration and monotonicity of confidence.<n>Our work provides a foundational methodology for developing more reliable and trustworthy conversational agents.
  arXiv  Detail & Related papers  (2026-01-05T14:58:04Z)
• Adaptive Multimodal Person Recognition: A Robust Framework for Handling Missing Modalities [2.5472580243871623]
  We propose a Trimodal person identification framework that integrates voice, face, and gesture modalities.<n>Our approach leverages multi-task learning to process each modality independently, followed by a cross-attention and gated fusion mechanism.<n>We show that our system maintains high accuracy even when one or two modalities are unavailable.
  arXiv  Detail & Related papers  (2025-12-16T22:59:24Z)
• Coupled Distributional Random Expert Distillation for World Model Online Imitation Learning [25.304836126280424]
  Imitation Learning (IL) has achieved remarkable success across various domains, including robotics, autonomous driving, and healthcare, by enabling agents to learn complex behaviors from expert demonstrations.<n>Existing IL methods often face instability challenges, particularly when relying on adversarial reward or value formulations in world model frameworks.<n>We propose a novel approach to online imitation learning that addresses these limitations through a reward model based on random network distillation (RND) for density estimation.
  arXiv  Detail & Related papers  (2025-05-04T19:32:48Z)
• Certainly Bot Or Not? Trustworthy Social Bot Detection via Robust Multi-Modal Neural Processes [28.951832771823128]
  Social bot detection is crucial for mitigating misinformation, online manipulation, and coordinated inauthentic behavior. Existing neural network-based detectors struggle with generalization due to distribution shifts across datasets. We introduce a novel Uncertainty Estimation for Social Bot Detection framework, which quantifies the predictive uncertainty of detectors beyond mere classification.
  arXiv  Detail & Related papers  (2025-03-11T01:32:52Z)
• POGEMA: A Benchmark Platform for Cooperative Multi-Agent Navigation [76.67608003501479]
  We introduce and specify an evaluation protocol defining a range of domain-related metrics computed on the basics of the primary evaluation indicators. The results of such a comparison, which involves a variety of state-of-the-art MARL, search-based, and hybrid methods, are presented.
  arXiv  Detail & Related papers  (2024-07-20T16:37:21Z)
• Confidence-aware multi-modality learning for eye disease screening [58.861421804458395]
  We propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information. Experimental results on both public and internal datasets demonstrate that our model excels in robustness.
  arXiv  Detail & Related papers  (2024-05-28T13:27:30Z)
• Neural Amortized Inference for Nested Multi-agent Reasoning [54.39127942041582]
  We propose a novel approach to bridge the gap between human-like inference capabilities and computational limitations. We evaluate our method in two challenging multi-agent interaction domains.
  arXiv  Detail & Related papers  (2023-08-21T22:40:36Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Provable Dynamic Fusion for Low-Quality Multimodal Data [94.39538027450948]
  Dynamic multimodal fusion emerges as a promising learning paradigm. Despite its widespread use, theoretical justifications in this field are still notably lacking. This paper provides theoretical understandings to answer this question under a most popular multimodal fusion framework from the generalization perspective. A novel multimodal fusion framework termed Quality-aware Multimodal Fusion (QMF) is proposed, which can improve the performance in terms of classification accuracy and model robustness.
  arXiv  Detail & Related papers  (2023-06-03T08:32:35Z)
• Achieving Fairness in Multi-Agent Markov Decision Processes Using Reinforcement Learning [30.605881670761853]
  We propose a Reinforcement Learning approach to achieve fairness in finite-horizon episodic MDPs. We show that such an approach achieves sub-linear regret in terms of the number of episodes.
  arXiv  Detail & Related papers  (2023-06-01T03:43:53Z)
• Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction [22.786774541083652]
  We propose a new Uncertainty-aware Multi-modal Learner that estimates uncertainty by measuring feature density via Cross-modal Random Network Prediction (CRNP) CRNP is designed to require little adaptation to translate between different prediction tasks, while having a stable training process.
  arXiv  Detail & Related papers  (2022-07-22T03:00:10Z)
• Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition [1.869225486385596]
  We explore the hypothesis that leveraging multiple modalities can lead to better recognition. We extend a number of recent contrastive self-supervised approaches for the task of Human Activity Recognition. We propose a flexible, general-purpose framework for performing multimodal self-supervised learning.
  arXiv  Detail & Related papers  (2022-05-20T10:39:16Z)
• Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis [18.4364234071951]
  We propose a novel framework HyCon for hybrid contrastive learning of tri-modal representation. Specifically, we simultaneously perform intra-/inter-modal contrastive learning and semi-contrastive learning. Our proposed method outperforms existing works.
  arXiv  Detail & Related papers  (2021-09-04T06:04:21Z)
• Softmax with Regularization: Better Value Estimation in Multi-Agent Reinforcement Learning [72.28520951105207]
  Overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning. We propose a novel regularization-based update scheme that penalizes large joint action-values deviating from a baseline. We show that our method provides a consistent performance improvement on a set of challenging StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2021-03-22T14:18:39Z)
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles [73.15950858151594]
  This paper presents Latent Optimistic Value Exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine latent world models with value function estimation to predict infinite-horizon returns and recover associated uncertainty via ensembling. We apply LOVE to visual robot control tasks in continuous action spaces and demonstrate on average more than 20% improved sample efficiency in comparison to state-of-the-art and other exploration objectives.
  arXiv  Detail & Related papers  (2020-10-27T22:06:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.