Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environments

• URL: http://arxiv.org/abs/2509.17425v1
• Date: Mon, 22 Sep 2025 07:17:26 GMT
• Title: Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environments
• Authors: Zhenliang Zhang, Yuxi Wang, Hongzhao Xie, Shiyun Zhao, Mingyuan Liu, Yujie Lu, Xinyi He, Zhenku Cheng, Yujia Peng,
• Abstract summary: We design a set of composite tasks inspired by common daily activities observed in early childhood development.<n>Within a dynamic and simulated home environment, these tasks span three core domains: object understanding, spatial intelligence, and social activity.<n>We evaluate 17 leading proprietary and open-source MLLMs on these tasks. The results consistently showed poor performance across all three domains, indicating a substantial gap between current capabilities and general intelligence requirements.
• Score: 22.833316179836704
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A key feature differentiating artificial general intelligence (AGI) from traditional AI is that AGI can perform composite tasks that require a wide range of capabilities. Although embodied agents powered by multimodal large language models (MLLMs) offer rich perceptual and interactive capabilities, it remains largely unexplored whether they can solve composite tasks. In the current work, we designed a set of composite tasks inspired by common daily activities observed in early childhood development. Within a dynamic and simulated home environment, these tasks span three core domains: object understanding, spatial intelligence, and social activity. We evaluated 17 leading proprietary and open-source MLLMs on these tasks. The results consistently showed poor performance across all three domains, indicating a substantial gap between current capabilities and general intelligence requirements. Together, our tasks offer a preliminary framework for evaluating the general capabilities of embodied agents, marking an early but significant step toward the development of embodied MLLMs and their real-world deployment.

Related papers

• Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models [75.45940282834327]
  We introduce Viewpoint Learning, a task designed to evaluate and improve the spatial reasoning capabilities of MLLMs.<n>We present the Viewpoint-100K dataset, consisting of 100K object-centric image pairs with diverse viewpoints and corresponding question-answer pairs.<n>Our approach employs a two-stage fine-tuning strategy, resulting in significant improvements across multiple tasks.
  arXiv  Detail & Related papers  (2025-11-03T14:27:00Z)
• ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Use [64.20714385692634]
  ToolScope is an agentic framework designed to unify global planning with local multimodal perception.<n>We evaluate ToolScope on four VQA benchmarks across diverse domains, including VQA 2.0, ScienceQA, MAT-Search and MathVista.
  arXiv  Detail & Related papers  (2025-10-31T10:51:27Z)
• A Survey on Agentic Multimodal Large Language Models [84.18778056010629]
  We present a comprehensive survey on Agentic Multimodal Large Language Models (Agentic MLLMs)<n>We explore the emerging paradigm of agentic MLLMs, delineating their conceptual foundations and distinguishing characteristics from conventional MLLM-based agents.<n>To further accelerate research in this area for the community, we compile open-source training frameworks, training and evaluation datasets for developing agentic MLLMs.
  arXiv  Detail & Related papers  (2025-10-13T04:07:01Z)
• Active-O3: Empowering Multimodal Large Language Models with Active Perception via GRPO [63.140883026848286]
  Active vision refers to the process of actively selecting where and how to look in order to gather task-relevant information.<n>Recently, the use of Multimodal Large Language Models (MLLMs) as central planning and decision-making modules in robotic systems has gained extensive attention.
  arXiv  Detail & Related papers  (2025-05-27T17:29:31Z)
• Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability [12.349247962800813]
  Large language models (LLMs) have emerged as powerful tools for many AI problems. They exhibit remarkable in-context learning (ICL) capabilities. How they approach composite tasks remains an open and largely underexplored question.
  arXiv  Detail & Related papers  (2024-07-22T15:22:34Z)
• WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? [83.19032025950986]
  We study the use of large language model-based agents for interacting with software via web browsers. WorkArena is a benchmark of 33 tasks based on the widely-used ServiceNow platform. BrowserGym is an environment for the design and evaluation of such agents.
  arXiv  Detail & Related papers  (2024-03-12T14:58:45Z)
• TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage [28.554981886052953]
  Large Language Models (LLMs) have emerged as powerful tools for various real-world applications. Despite their prowess, intrinsic generative abilities of LLMs may prove insufficient for handling complex tasks. This paper proposes a structured framework tailored for LLM-based AI Agents.
  arXiv  Detail & Related papers  (2023-08-07T09:22:03Z)
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark [81.42376626294812]
  We present Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs. We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision.
  arXiv  Detail & Related papers  (2023-06-11T14:01:17Z)
• Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents [0.0]
  We present a novel framework for enhancing the capabilities of large language models (LLMs) by leveraging the power of multi-agent systems. Our framework introduces a collaborative environment where multiple intelligent agent components, each with distinctive attributes and roles, work together to handle complex tasks more efficiently and effectively.
  arXiv  Detail & Related papers  (2023-06-05T23:55:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.