How Should I Pick a Foundation Model for My Robot? In Favor of a Community Evaluation Framework for Social Robots

• URL: http://arxiv.org/abs/2608.06898v1
• Date: Fri, 07 Aug 2026 07:33:25 GMT
• Title: How Should I Pick a Foundation Model for My Robot? In Favor of a Community Evaluation Framework for Social Robots
• Abstract summary: We identify five evaluation dimensions for foundation models in social robots.<n>We propose a three-tiered evaluation funnel paradigm that first filters with general metrics, then extends to simulated interactions, and terminates in more expensive, robot-specific evaluation.
• Score: 8.113321376993298
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Researchers who seek to build social robot applications on foundation models are faced with a difficult question: how should we pick a model? Public leaderboards offer little guidance: the demands of real-time, embodied social interaction lie largely outside their focus. And direct evaluation is impractical at scale: each embodied study requires scarce participant, robot, and experimenter time. In this paper, we identify five evaluation dimensions for foundation models in social robots: (i) conversational competence, (ii) user safety, (iii) embodied character, (iv) target scene effectiveness, and (v) audience appropriateness. To make model selection cheaper and better informed, we propose a three-tiered evaluation funnel paradigm that first filters with general metrics, then extends to simulated interactions, and terminates in more expensive, robot-specific evaluation. We map all five dimensions across all three tiers, chart where applicable evaluation methods exist and are missing, and close with a call to action: let's build the evaluation framework together as a community.

Related papers

• The Embodiment Gap in Robot Foundation Models [7.90590547085857]
  In robotics, a model can generalize while work still remains before it can run on a robot with a particular body.<n>We call the gap between reusable models, representations, or data and their use in execution on the target robot the embodiment gap.<n>This survey examines what can be reused across robot embodiments and what must still be implemented on a new robot.
  arXiv  Detail & Related papers  (2026-08-19T01:55:04Z)
• StARS: Socially Appropriate Robot Actions via a Recommender System-Driven Approach [9.290474217180407]
  We reformulate socially appropriate action generation as a preference modelling problem inspired by recommender systems.<n>We propose StARS, a novel model-agnostic framework that integrates collaborative filtering with learnable scene representations.<n>We evaluate StARS on two socially aware robotics datasets, MannersDB+ and SocNav1, and analyse robustness under sparse preference feedback.
  arXiv  Detail & Related papers  (2026-07-23T20:38:07Z)
• GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation [73.02743604295246]
  WMBench is a benchmark constructed from real-robot teleoperation data and matched policy rollouts.<n>We analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions.<n>Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism.
  arXiv  Detail & Related papers  (2026-07-02T17:02:43Z)
• The Robot's Inner Critic: Self-Refinement of Social Behaviors through VLM-based Replanning [2.38295275136047]
  CRISP (Critique-and-Replan for Interactive Social Presence) is an autonomous framework where a robot critiques and replans its own actions.<n>CRISP integrates (1) extraction of movable joints and constraints by analyzing the robot's description file.<n>It can generate subtly different, human-like motions on various platforms using only the robot's structure file.
  arXiv  Detail & Related papers  (2026-03-20T17:40:21Z)
• Gesture Generation (Still) Needs Improved Human Evaluation Practices: Insights from a Community-Driven State-of-the-Art Benchmark [55.41250396114216]
  We review human evaluation practices in automated, speech-driven 3D gesture generation.<n>We introduce a detailed human evaluation protocol for the widely-used BEAT2 motion-capture dataset.
  arXiv  Detail & Related papers  (2025-11-03T05:17:28Z)
• Are Large Language Models Aligned with People's Social Intuitions for Human-Robot Interactions? [7.308479353736709]
  Large language models (LLMs) are increasingly used in robotics, especially for high-level action planning. In this work, we test whether LLMs reproduce people's intuitions and communication in human-robot interaction scenarios. We show that vision models fail to capture the essence of video stimuli and that LLMs tend to rate different communicative acts and behavior higher than people.
  arXiv  Detail & Related papers  (2024-03-08T22:23:23Z)
• Real-time Addressee Estimation: Deployment of a Deep-Learning Model on the iCub Robot [52.277579221741746]
  Addressee Estimation is a skill essential for social robots to interact smoothly with humans. Inspired by human perceptual skills, a deep-learning model for Addressee Estimation is designed, trained, and deployed on an iCub robot. The study presents the procedure of such implementation and the performance of the model deployed in real-time human-robot interaction.
  arXiv  Detail & Related papers  (2023-11-09T13:01:21Z)
• SOCRATES: Text-based Human Search and Approach using a Robot Dog [6.168521568443759]
  We propose a SOCratic model for Robots Approaching humans based on TExt System (SOCRATES) We first present a Human Search Socratic Model that connects large pre-trained models in the language domain to solve the downstream task. Then, we propose a hybrid learning-based framework for generating target-cordial robotic motion to approach a person.
  arXiv  Detail & Related papers  (2023-02-10T15:35:24Z)
• Evaluating Human-Language Model Interaction [79.33022878034627]
  We develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems. We design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. We find that better non-interactive performance does not always translate to better human-LM interaction.
  arXiv  Detail & Related papers  (2022-12-19T18:59:45Z)
• Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue [92.01165203498299]
  Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. This paper argues that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dialogue research.
  arXiv  Detail & Related papers  (2022-10-10T05:51:40Z)
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach [84.02388020258141]
  We propose a new framework named ENIGMA for estimating human evaluation scores based on off-policy evaluation in reinforcement learning. ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation. Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.
  arXiv  Detail & Related papers  (2021-02-20T03:29:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.