A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI models

• URL: http://arxiv.org/abs/2606.06061v1
• Date: Thu, 04 Jun 2026 12:00:41 GMT
• Title: A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI models
• Abstract summary: This paper presents a distributed conversational framework for human-robot collaborative manipulation.<n>It integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack. Language understanding, visual grounding, orchestration, and motion execution run as separate ROS 2 nodes, enabling flexible deployment across distributed hardware while maintaining a responsive control loop. From free-form user commands, the system generates structured action requests for pick, place, and handover. It uses a VLM to return image-space targets, which are converted into metric robot-frame goals using depth and calibration. A web dashboard exposes intermediate intent and grounding overlays (pixel, depth, and robot-frame) and requires explicit operator confirmation before any motion is executed. Experiments on a Franka FR3 platform evaluate end-to-end task reliability and latency under increasing working table scene ambiguity and compare alternative LLM/VLM configurations in the same pipeline. Code and full documentation are available at [github.com/cogrob-tuni/franka-llm](https://github.com/cogrob-tuni/franka-llm).

Related papers

• DSWAM: A Dual-System World Action Foundation Model for Fine-Grained Robot Manipulation [14.193285862278913]
  World Action Models (WAMs) provide a promising alternative to Vision-Language-Action (VLA) policies.<n>Existing WAMs excel at physically grounded execution, but typically lack the explicit language-level planning interface.<n>We introduce DSWAM, a Dual-System World Action Foundation Model for fine-grained robot manipulation.
  arXiv  Detail & Related papers  (2026-07-06T10:57:01Z)
• Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation [0.6927055673104934]
  The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution.<n>The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning.
  arXiv  Detail & Related papers  (2026-06-22T11:01:56Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• CLAW: Composable Language-Annotated Whole-body Motion Generation [55.99805728566105]
  We present CLAW, a pipeline for scalable generation of language-annotated whole-body motion data.<n>CLAW composes motion primitives from a kinematic planner, parameterized by movement, heading, speed, pelvis height, and duration.<n>A low-level controller tracks these references in MuJoCo simulation, yielding physically grounded trajectories.
  arXiv  Detail & Related papers  (2026-04-13T10:02:04Z)
• A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systems [0.0]
  This article presents a ROS 2 wrapper for Florence-2 that exposes the model through three complementary interaction modes.<n>The wrapper is designed for local execution and supports both native installation and Docker container deployment.
  arXiv  Detail & Related papers  (2026-04-01T17:29:59Z)
• Agentic AI for Robot Control: Flexible but still Fragile [32.43712661386947]
  We present an agentic control system in which a reasoning-capable language model plans and executes tasks.<n>We deploy the system on two physical robot platforms in two settings: (i) tabletop grasping, placement, and box insertion in indoor mobile manipulation (Mobipick) and (ii) autonomous agricultural navigation and sensing (Valdemar)
  arXiv  Detail & Related papers  (2026-02-13T16:43:34Z)
• VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement [66.13644883379087]
  We tackle three key challenges in 3D object arrangement task using MLLMs.<n>First, to address the weak visual grounding of MLLMs, we introduce an MCP-based API.<n>Second, we augment the MLLM's 3D scene understanding with a suite of specialized visual tools.<n>Third, to manage the iterative, error-prone updates, we propose a collaborative multi-agent framework.
  arXiv  Detail & Related papers  (2025-12-26T19:22:39Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning [74.58666091522198]
  We present a framework for intuitive robot programming by non-experts. We leverage natural language prompts and contextual information from the Robot Operating System (ROS) Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface.
  arXiv  Detail & Related papers  (2024-06-28T08:28:38Z)
• MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting [97.52388851329667]
  We introduce Marking Open-world Keypoint Affordances (MOKA) to solve robotic manipulation tasks specified by free-form language instructions. Central to our approach is a compact point-based representation of affordance, which bridges the VLM's predictions on observed images and the robot's actions in the physical world. We evaluate and analyze MOKA's performance on various table-top manipulation tasks including tool use, deformable body manipulation, and object rearrangement.
  arXiv  Detail & Related papers  (2024-03-05T18:08:45Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.