FG-CLTP: Fine-Grained Contrastive Language Tactile Pretraining for Robotic Manipulation

• URL: http://arxiv.org/abs/2603.10871v1
• Date: Wed, 11 Mar 2026 15:21:54 GMT
• Title: FG-CLTP: Fine-Grained Contrastive Language Tactile Pretraining for Robotic Manipulation
• Abstract summary: We develop a 3D tactile-language-action (3D-TLA) architecture driven by a flow matching policy to enable reasoning and control.<n>The proposed FG-C model yields a 95.9% classification accuracy and reduces the regression error (MAE) by 52.6%.<n>Building upon this fine-grained representation, we develop a 3D tactile-language-action (3D-TLA) architecture driven by a flow matching policy to enable reasoning and control.
• Score: 10.959531491338552
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in integrating tactile sensing into vision-language-action (VLA) models have demonstrated transformative potential for robotic perception. However, existing tactile representations predominantly rely on qualitative descriptors (e.g., texture), neglecting quantitative contact states such as force magnitude, contact geometry, and principal axis orientation, which are indispensable for fine-grained manipulation. To bridge this gap, we propose FG-CLTP, a fine-grained contrastive language tactile pretraining framework. We first introduce a novel dataset comprising over 100k tactile 3D point cloud-language pairs that explicitly capture multidimensional contact states from the sensor's perspective. We then implement a discretized numerical tokenization mechanism to achieve quantitative-semantic alignment, effectively injecting explicit physical metrics into the multimodal feature space. The proposed FG-CLTP model yields a 95.9% classification accuracy and reduces the regression error (MAE) by 52.6% compared to state-of-the-art methods. Furthermore, the integration of 3D point cloud representations establishes a sensor-agnostic foundation with a minimal sim-to-real gap of 3.5%. Building upon this fine-grained representation, we develop a 3D tactile-language-action (3D-TLA) architecture driven by a flow matching policy to enable multimodal reasoning and control. Extensive experiments demonstrate that our framework significantly outperforms strong baselines in contact-rich manipulation tasks, providing a robust and generalizable foundation for tactile-language-action models.

Related papers

• DCGrasp: Distance-aware Controllable Grasp Generation [64.30284326160826]
  We present DCGrasp, a controllable grasp generation system built on a novel grasp energy term.<n>This term computes Distance Profile, a signed distance from each hand to the nearest object point.<n>Our experiments show that DCGrasp produces high-quality, physically plausible grasps with flexible user control.
  arXiv  Detail & Related papers  (2026-06-29T08:02:02Z)
• Decoupling Semantics and Geometric Grounding: Spatial Visual Prompts for Language-Conditioned Imitation Learning [21.67233368353003]
  We propose SVP-IL, a decoupled architecture that explicitly extracts spatial visual grounding from the action generation loop.<n>In experiments, SVP-IL significantly outperforms state-of-the-art VLAs and pure visuomotor baselines.
  arXiv  Detail & Related papers  (2026-06-24T03:45:06Z)
• T-Rex: Tactile-Reactive Dexterous Manipulation [139.71263755530654]
  We introduce a variable-rate Mixture-of-Transformers (MoT) architecture equipped with a novel temporal tactile VQ-VAE encoder.<n>We demonstrate the effectiveness of tactile-reactive policies on 12 manipulation tasks requiring delicate force control and deformable object manipulation.
  arXiv  Detail & Related papers  (2026-06-15T17:59:55Z)
• PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought [17.13654442098613]
  Chain-of-Thought (CoT) reasoning has shown strong effectiveness in LLMs and image-based MLLMs.<n>We propose a data-centric framework for constructing large-scale CoT supervision tailored to 3D point cloud understanding.<n>Fine-tuning PointLLM on PoCoTI yields PointLLM-R, a reasoning-capable 3D multimodal language model.
  arXiv  Detail & Related papers  (2026-05-21T05:19:51Z)
• PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction [57.63073414949329]
  We propose PointACT, a dual-system 3D-aware VLA policy that integrates hierarchical 3D point cloud representations directly into the action decoding process.<n>PointACT employs a multi-scale point-action interaction mechanism with efficient bottleneck window self-attention, enabling evolving action tokens to densely attend to both local geometric detail and global scene structure.
  arXiv  Detail & Related papers  (2026-05-20T17:10:31Z)
• A Model-based Visual Contact Localization and Force Sensing System for Compliant Robotic Grippers [11.232660351404684]
  Integrating force sensing into deformable grippers negotiates trade-offs in cost, complexity, mechanical robustness, and performance.<n>Current approaches mostly utilize end-to-end deep learning, which can be brittle when generalizing to new scenarios.<n>We developed a model-based visual force sensing approach integrating an iterative contact localization with generalization to unseen objects.
  arXiv  Detail & Related papers  (2026-05-01T00:34:19Z)
• Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds [53.82500407523346]
  PointINS is an instance-oriented self-supervised framework that enriches point cloud representations through geometry-aware learning.<n>PointINS achieves on average +3.5% mAP improvement for indoor instance segmentation and +4.1% PQ gain for outdoor panoptic segmentation.
  arXiv  Detail & Related papers  (2026-03-26T08:31:06Z)
• SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning [54.232148007248874]
  Current Sign Language Production (SLP) frameworks face a stark trade-off: direct text-to-pose models suffer from regression-to-the-mean effects.<n>We propose a novel training paradigm that leverages sparses to capture the true underlying distribution of human signing.<n>By predicting dense motion from these discrete anchors, our approach mitigates regression-to-the-mean while ensuring fluid articulation.
  arXiv  Detail & Related papers  (2026-03-11T06:02:36Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• Smooth Operator: Smooth Verifiable Reward Activates Spatial Reasoning Ability of Vision-Language Model [18.526821056010384]
  Vision-Language Models (VLMs) face a critical bottleneck in achieving precise numerical prediction for 3D scene understanding.<n>Traditional reinforcement learning approaches, primarily based on relative ranking, often suffer from severe reward sparsity and gradient instability.<n>We introduce the Smooth Numerical Reward Activation (SNRA) operator and the Absolute-Preserving GRPO framework.
  arXiv  Detail & Related papers  (2026-01-12T16:26:42Z)
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model [76.08429266631823]
  We propose an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM)<n>URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction.<n> Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-11-02T13:45:51Z)
• Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysis [51.37795317716487]
  We propose the Adaptive Point-Prompt Tuning (APPT) method, which fine-tunes pre-trained models with a modest number of parameters.<n>We convert raw point clouds into point embeddings by aggregating local geometry to capture spatial features followed by linear layers.<n>To calibrate self-attention across source domains of any modality to 3D, we introduce a prompt generator that shares weights with the point embedding module.
  arXiv  Detail & Related papers  (2025-08-30T06:02:21Z)
• NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embedding [8.131547418489534]
  We propose NeuroVoxel-LM, a novel framework that integrates Neural Radiance Fields (NeRF) with dynamic resolution voxelization and lightweight meta-embedding.<n>Specifically, we introduce a Dynamic Resolution Multiscale Voxelization (DR-MSV) technique that adaptively adjusts voxel based on geometric and structural complexity.<n>We also propose the Token-level Adaptive Pooling for Lightweight Meta-Embedding (TAP-LME) mechanism, which enhances semantic representation through attention-based weighting and residual fusion.
  arXiv  Detail & Related papers  (2025-07-27T03:11:08Z)
• MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation [28.75149480374178]
  MEgoHand is a framework that synthesizes physically plausible hand-object interactions from egocentric RGB, text, and initial hand pose.<n>It achieves substantial reductions in wrist translation error and joint rotation error, highlighting its capacity to accurately model fine-grained hand joint structures.
  arXiv  Detail & Related papers  (2025-05-22T12:37:47Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Semantic Consistent Language Gaussian Splatting for Point-Level Open-vocabulary Querying [25.32838673665989]
  Open-vocabulary 3D scene understanding is crucial for robotics applications, such as natural language-driven manipulation.<n>Existing methods for querying 3D Gaussian Splatting often struggle with inconsistent 2D mask supervision.<n>We present a novel point-level querying framework that performs tracking on segmentation masks to establish a semantically consistent ground-truth.
  arXiv  Detail & Related papers  (2025-03-27T17:59:05Z)
• Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding [58.38294408121273]
  We propose Cross-modal and Uncertainty-aware Agglomeration for Open-vocabulary 3D Scene Understanding dubbed CUA-O3D.<n>Our method addresses two key challenges: (1) incorporating semantic priors from VLMs alongside the geometric knowledge of spatially-aware vision foundation models, and (2) using a novel deterministic uncertainty estimation to capture model-specific uncertainties.
  arXiv  Detail & Related papers  (2025-03-20T20:58:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.