When Robots Exchange Meaning: A Demo of Goal-Oriented Semantic Communications for Collaborative Robotics

• URL: http://arxiv.org/abs/2607.28256v1
• Date: Thu, 30 Jul 2026 14:16:23 GMT
• Title: When Robots Exchange Meaning: A Demo of Goal-Oriented Semantic Communications for Collaborative Robotics
• Abstract summary: This demo paper presents a robot-edge semantic communication (SemCom) testbed integrating robot-side visual compression, edge-side semantic mapping, and dashboard-based mission interaction.<n>A mobile robot equipped with RGB-D sensing and LiDAR runs ROS 2, while a Jetson Orin edge node performs reconstruction, semantic object handling, and visualization.<n>The reconstructed visual stream is further associated with depth, pose, and 3D mapping information to generate a semantic map for downstream robotic applications.
• Score: 2.7158085195874713
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Collaborative robotics is a representative task-oriented 6G use-case, where communication quality should be reflected in mission execution, environment understanding, and closed-loop operation rather than packet delivery alone. This demo paper presents a robot-edge semantic communication (SemCom) testbed integrating robot-side visual compression, edge-side semantic mapping, and dashboard-based mission interaction. A mobile robot equipped with RGB-D sensing and LiDAR runs ROS 2, while a Jetson Orin edge node performs reconstruction, RTAB-Map mapping, semantic object handling, and browserbased visualization. As an initial proof of concept, RGB frames are encoded on the robot into VQ-VAE tokens using an ONNX Runtime encoder and reconstructed on the edge using a PyTorch decoder. A 320 X 240 image is represented by an 80 X 60 token grid with a packed payload of 5400 bytes, corresponding to a 42.67X reduction relative to model-input RGB bytes. The reconstructed visual stream is further associated with depth, pose, and 3D mapping information to generate a semantic map for downstream robotic applications. The demo exposes the full path from semantic visual transport to object-level map interaction, and provides a practical platform for future task-aware 6G networking studies at the intersection of SemCom, embodied AI, and physical AI-enabled robotics. A video of the demo is available at https://tinyurl.com/Tos09

Related papers

• Sparse2Act: Learning Action-Aligned Sparse 3D Representations for Cross-Domain Robot Manipulation [52.00134393320209]
  We introduce Sparse2Act, an observation-action alignment framework for pretraining sparse point-cloud encoders.<n>Masked sparse 3D tokens are trained to organize scene features around the workspace motion paired with the observation.<n>On the LIBERO-10 benchmark, our method achieves 86.9% average success after 500 fine-tuning steps.
  arXiv  Detail & Related papers  (2026-06-10T23:56:01Z)
• Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation [60.07205156194741]
  Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen.<n>We present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families and to four heterogeneous real robots.
  arXiv  Detail & Related papers  (2026-05-26T18:52:04Z)
• Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics [0.0]
  We present an offline hand-shadowing and mapping pipeline from a single egocentric RGB-D camera mounted on 3D-printed glasses.<n>The pipeline detects 21 hand landmarks per hand using MediaPipe Hands.<n>A gripper controller maps thumb-index finger geometry to grasp aperture with a four-level fallback hierarchy.
  arXiv  Detail & Related papers  (2026-03-11T23:53:28Z)
• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• Robotic Visual Instruction [10.110333805611974]
  We introduce the Robotic Visual Instruction (RoVI) to guide robotic tasks.<n>RoVI encodes spatial-temporal information into human-interpretable visual instructions.<n>We present Visual Instruction Embodied (VIEW) to enable robots to understand RoVI better.
  arXiv  Detail & Related papers  (2025-05-01T17:55:05Z)
• EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation [55.26713167507132]
  We introduce a generative robotics foundation model that constructs and interprets embodied spaces.<n>EnerVerse employs an autoregressive video diffusion framework to predict future embodied spaces from instructions, enhanced by a sparse context memory for long-term reasoning.<n>We present EnerVerse-D, a data engine pipeline combining the generative model with 4D Gaussian Splatting, forming a self-reinforcing data loop to reduce the sim-to-real gap.
  arXiv  Detail & Related papers  (2025-01-03T17:00:33Z)
• Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models [53.22792173053473]
  We introduce an interactive robotic manipulation framework called Polaris. Polaris integrates perception and interaction by utilizing GPT-4 alongside grounded vision models. We propose a novel Synthetic-to-Real (Syn2Real) pose estimation pipeline.
  arXiv  Detail & Related papers  (2024-08-15T06:40:38Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• Object Goal Navigation Based on Semantics and RGB Ego View [9.702784248870522]
  This paper presents an architecture and methodology to empower a service robot to navigate an indoor environment with semantic decision making, given RGB ego view. The robot navigates based on GeoSem map - a relational combination of geometric and semantic map. The presented approach was found to outperform human users in gamified evaluations with respect to average completion time.
  arXiv  Detail & Related papers  (2022-10-20T19:23:08Z)
• Visual Navigation in Real-World Indoor Environments Using End-to-End Deep Reinforcement Learning [2.7071541526963805]
  We propose a novel approach that enables a direct deployment of the trained policy on real robots. The policy is fine-tuned on images collected from real-world environments. In 30 navigation experiments, the robot reached a 0.3-meter neighborhood of the goal in more than 86.7% of cases.
  arXiv  Detail & Related papers  (2020-10-21T11:22:30Z)
• Deep Reinforcement learning for real autonomous mobile robot navigation in indoor environments [0.0]
  We present our proof of concept for autonomous self-learning robot navigation in an unknown environment for a real robot without a map or planner. The input for the robot is only the fused data from a 2D laser scanner and a RGB-D camera as well as the orientation to the goal. The output actions of an Asynchronous Advantage Actor-Critic network (GA3C) are the linear and angular velocities for the robot.
  arXiv  Detail & Related papers  (2020-05-28T09:15:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.