NIFT: Neural Interaction Field and Template for Object Manipulation

• URL: http://arxiv.org/abs/2210.10992v2
• Date: Fri, 21 Oct 2022 01:56:47 GMT
• Title: NIFT: Neural Interaction Field and Template for Object Manipulation
• Abstract summary: NIFT is a descriptive and robust interaction representation to facilitate imitation learning. NIF is a neural field which encodes the relationship between each spatial point and a given object. NIT effectively guides the feature matching in the NIFs of the new object instances to optimize the object poses.
• Score: 24.42098331719611
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce NIFT, Neural Interaction Field and Template, a descriptive and robust interaction representation of object manipulations to facilitate imitation learning. Given a few object manipulation demos, NIFT guides the generation of the interaction imitation for a new object instance by matching the Neural Interaction Template (NIT) extracted from the demos to the Neural Interaction Field (NIF) defined for the new object. Specifically, the NIF is a neural field which encodes the relationship between each spatial point and a given object, where the relative position is defined by a spherical distance function rather than occupancies or signed distances, which are commonly adopted by conventional neural fields but less informative. For a given demo interaction, the corresponding NIT is defined by a set of spatial points sampled in the NIF of the demo object with associated neural features. To better capture the interaction, the points are sampled on the interaction bisector surface, which consists of points that are equidistant to two interacting objects and has been used extensively for interaction representation. With both point selection and pointwise features defined for better interaction encoding, NIT effectively guides the feature matching in the NIFs of the new object instances to optimize the object poses to realize the manipulation while imitating the demo interactions. Experiments show that our NIFT solution outperforms state-of-the-art imitation learning methods for object manipulation and generalizes better to objects from new categories.

Related papers

• FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection [26.006274699996027]
  FineHOI is a zero-shot HOI framework that explicitly models interactions from dense patch-level features.<n>Our approach is motivated by the observation that human-object interactions are defined by localized spatial relationships.<n>Experiments demonstrate that FineHOI consistently outperforms existing zero-shot HOI methods.
  arXiv  Detail & Related papers  (2026-09-05T07:53:11Z)
• Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation [45.67585781847749]
  We propose surfacepoint trajectories as an object motion representation for each rigid component.<n>This representation handles multi-object coordination and diverse articulation mechanisms directly from point dynamics.<n>We factorize HOI generation into three stages: generating object motions from text or waypoints, predicting the contact distance field, and synthesizing the contact distance field.
  arXiv  Detail & Related papers  (2026-08-04T05:47:07Z)
• Learning Human-Object Interaction as Groups [52.28258599873394]
  GroupHOI is a framework that propagates contextual information in terms of geometric proximity and semantic similarity.<n>It exhibits leading performance on the more challenging Nonverbal Interaction Detection task.
  arXiv  Detail & Related papers  (2025-10-21T07:25:10Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction. We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images. We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Visual-Geometric Collaborative Guidance for Affordance Learning [63.038406948791454]
  We propose a visual-geometric collaborative guided affordance learning network that incorporates visual and geometric cues. Our method outperforms the representative models regarding objective metrics and visual quality.
  arXiv  Detail & Related papers  (2024-10-15T07:35:51Z)
• Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detection [57.883265488038134]
  We propose a hierarchical graph interaction network termed HGINet for camouflaged object detection. The network is capable of discovering imperceptible objects via effective graph interaction among the hierarchical tokenized features. Our experiments demonstrate the superior performance of HGINet compared to existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-27T12:53:25Z)
• ORMNet: Object-centric Relationship Modeling for Egocentric Hand-object Segmentation [14.765419467710812]
  Egocentric hand-object segmentation (EgoHOS) is a promising new task aiming at segmenting hands and interacting objects in egocentric images. This paper proposes a novel Object-centric Relationship Modeling Network (ORMNet) to fulfill end-to-end and effective EgoHOS.
  arXiv  Detail & Related papers  (2024-07-08T03:17:10Z)
• Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange [50.45953583802282]
  We introduce a novel self-supervised learning (SSL) strategy for point cloud scene understanding. Our approach leverages both object patterns and contextual cues to produce robust features. Our experiments demonstrate the superiority of our method over existing SSL techniques.
  arXiv  Detail & Related papers  (2024-04-11T06:39:53Z)
• ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors [73.26004792375556]
  This paper shows that robustness and generalisation to novel scene objects in 3D object-aware character synthesis can be achieved by training a motion model with as few as one reference object. We leverage an implicit feature representation trained on object-only datasets, which encodes an SE(3)-equivariant descriptor field around the object. We demonstrate substantial improvements in 3D virtual character motion and interaction quality and robustness to scenarios with unseen objects.
  arXiv  Detail & Related papers  (2023-08-24T17:59:51Z)
• InterTracker: Discovering and Tracking General Objects Interacting with Hands in the Wild [40.489171608114574]
  Existing methods rely on frame-based detectors to locate interacting objects. We propose to leverage hand-object interaction to track interactive objects. Our proposed method outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-06T09:09:17Z)
• Learning-based Relational Object Matching Across Views [63.63338392484501]
  We propose a learning-based approach which combines local keypoints with novel object-level features for matching object detections between RGB images. We train our object-level matching features based on appearance and inter-frame and cross-frame spatial relations between objects in an associative graph neural network.
  arXiv  Detail & Related papers  (2023-05-03T19:36:51Z)
• Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation [10.684104348212742]
  We present a method to generalize object manipulation skills acquired from a limited number of demonstrations. Our approach, Local Neural Descriptor Fields (L-NDF), utilizes neural descriptors defined on the local geometry of the object. We illustrate the efficacy of our approach in manipulating novel objects in novel poses -- both in simulation and in the real world.
  arXiv  Detail & Related papers  (2023-02-07T16:37:19Z)
• Neural Message Passing for Visual Relationship Detection [45.595505481091074]
  We propose a neural message passing (NMP) algorithm for visual relationship detection. We explicitly model objects and interactions by an interaction graph and then propose a message-passing-style algorithm to propagate the contextual information. Experimental results on two benchmark datasets demonstrate the superiority of our proposed method.
  arXiv  Detail & Related papers  (2022-08-08T14:06:23Z)
• A Graph-based Interactive Reasoning for Human-Object Interaction Detection [71.50535113279551]
  We present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs. We construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Our framework is end-to-end trainable and free from costly annotations like human pose.
  arXiv  Detail & Related papers  (2020-07-14T09:29:03Z)
• A Deep Learning Approach to Object Affordance Segmentation [31.221897360610114]
  We design an autoencoder that infers pixel-wise affordance labels in both videos and static images. Our model surpasses the need for object labels and bounding boxes by using a soft-attention mechanism. We show that our model achieves competitive results compared to strongly supervised methods on SOR3D-AFF.
  arXiv  Detail & Related papers  (2020-04-18T15:34:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.