ShapeLib: designing a library of procedural 3D shape abstractions with Large Language Models

• URL: http://arxiv.org/abs/2502.08884v1
• Date: Thu, 13 Feb 2025 01:52:02 GMT
• Title: ShapeLib: designing a library of procedural 3D shape abstractions with Large Language Models
• Authors: R. Kenny Jones, Paul Guerrero, Niloy J. Mitra, Daniel Ritchie,
• Abstract summary: We present ShapeLib, a library of 3D shape abstraction functions.<n>We train a recognition network that learns to infer shape programs based on our library from different visual modalities.<n>Our shape functions have parameters that are semantically interpretable and can be modified to produce plausible shape variations.
• Score: 46.372361125013974
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Procedural representations are desirable, versatile, and popular shape encodings. Authoring them, either manually or using data-driven procedures, remains challenging, as a well-designed procedural representation should be compact, intuitive, and easy to manipulate. A long-standing problem in shape analysis studies how to discover a reusable library of procedural functions, with semantically aligned exposed parameters, that can explain an entire shape family. We present ShapeLib as the first method that leverages the priors of frontier LLMs to design a library of 3D shape abstraction functions. Our system accepts two forms of design intent: text descriptions of functions to include in the library and a seed set of exemplar shapes. We discover procedural abstractions that match this design intent by proposing, and then validating, function applications and implementations. The discovered shape functions in the library are not only expressive but also generalize beyond the seed set to a full family of shapes. We train a recognition network that learns to infer shape programs based on our library from different visual modalities (primitives, voxels, point clouds). Our shape functions have parameters that are semantically interpretable and can be modified to produce plausible shape variations. We show that this allows inferred programs to be successfully manipulated by an LLM given a text prompt. We evaluate ShapeLib on different datasets and show clear advantages over existing methods and alternative formulations.

Related papers

• Learning Fine-to-Coarse Cuboid Shape Abstraction [7.152103069753289]
  abstraction of 3D objects with simple geometric primitives like cuboids allows to infer structural information from complex geometry.<n>We introduce a novel fine-to-coarse unsupervised learning approach to abstract collections of 3D shapes.<n>Our results confirm an improvement over previous cuboid-based shape abstraction techniques.
  arXiv  Detail & Related papers  (2025-02-03T22:06:26Z)
• Sketch: A Toolkit for Streamlining LLM Operations [51.33202045501429]
  Large language models (LLMs) have achieved remarkable success. The flexibility of their output format poses challenges in controlling and harnessing the model's outputs. We present Sketch, an innovative toolkit designed to streamline LLM operations across diverse fields.
  arXiv  Detail & Related papers  (2024-09-05T08:45:44Z)
• AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph [62.685920585838616]
  abstraction ability is essential in human intelligence, which remains under-explored in language models. We present AbsPyramid, a unified entailment graph of 221K textual descriptions of abstraction knowledge.
  arXiv  Detail & Related papers  (2023-11-15T18:11:23Z)
• LILO: Learning Interpretable Libraries by Compressing and Documenting Code [71.55208585024198]
  We introduce LILO, a neurosymbolic framework that iteratively synthesizes, compresses, and documents code. LILO combines LLM-guided program synthesis with recent algorithmic advances in automated from Stitch. We find that AutoDoc boosts performance by helping LILO's synthesizer to interpret and deploy learned abstractions.
  arXiv  Detail & Related papers  (2023-10-30T17:55:02Z)
• ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives [44.01940125080666]
  We present ShapeCoder, the first system capable of taking a dataset of shapes, represented with unstructured primitives. We show how ShapeCoder discovers a library of abstractions that capture high-level relationships, remove extraneous degrees of freedom, and achieve better dataset compression.
  arXiv  Detail & Related papers  (2023-05-09T17:55:48Z)
• PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation [59.70430570779819]
  We introduce a data-driven shape completion approach that focuses on completing geometric details of missing regions of 3D shapes. Our key insight is to copy and deform patches from the partial input to complete missing regions. We leverage repeating patterns by retrieving patches from the partial input, and learn global structural priors by using a neural network to guide the retrieval and deformation steps.
  arXiv  Detail & Related papers  (2022-07-24T18:59:09Z)
• Towards Implicit Text-Guided 3D Shape Generation [81.22491096132507]
  This work explores the challenging task of generating 3D shapes from text. We propose a new approach for text-guided 3D shape generation, capable of producing high-fidelity shapes with colors that match the given text description.
  arXiv  Detail & Related papers  (2022-03-28T10:20:03Z)
• Leveraging Language to Learn Program Abstractions and Search Heuristics [66.28391181268645]
  We introduce LAPS (Language for Abstraction and Program Search), a technique for using natural language annotations to guide joint learning of libraries and neurally-guided search models for synthesis. When integrated into a state-of-the-art library learning system (DreamCoder), LAPS produces higher-quality libraries and improves search efficiency and generalization.
  arXiv  Detail & Related papers  (2021-06-18T15:08:47Z)
• Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds [8.156355030558172]
  Representing complex 3D objects as simple geometric primitives, known as shape abstraction, is important for geometric modeling, structural analysis, and shape synthesis. We propose an unsupervised shape abstraction method to map a point cloud into a compact cuboid representation.
  arXiv  Detail & Related papers  (2021-06-07T09:15:16Z)
• ShapeMOD: Macro Operation Discovery for 3D Shape Programs [41.973945628565104]
  We present ShapeMOD, an algorithm for automatically discovering macros that are useful across large datasets of 3D shape programs. ShapeMOD operates on shape programs expressed in an imperative, statement-based language. We show that it automatically discovers a concise set of macros that abstract out common structural and parametric patterns.
  arXiv  Detail & Related papers  (2021-04-13T17:54:03Z)
• ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis [38.27280837835169]
  We propose ShapeAssembly, a domain-specific "assembly-language" for 3D shape structures. We show how to extract ShapeAssembly programs from existing shape structures in the PartNet dataset. We evaluate our approach by comparing shapes output by our generated programs to those from other recent shape structure models.
  arXiv  Detail & Related papers  (2020-09-17T02:26:45Z)
• PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations [75.42959184226702]
  We present a new mid-level patch-based surface representation for object-agnostic training. We show several applications of our new representation, including shape and partial point cloud completion.
  arXiv  Detail & Related papers  (2020-08-04T15:34:46Z)
• Learning Generative Models of Shape Handles [43.41382075567803]
  We present a generative model to synthesize 3D shapes as sets of handles. Our model can generate handle sets with varying cardinality and different types of handles. We show that the resulting shape representations are intuitive and achieve superior quality than previous state-of-the-art.
  arXiv  Detail & Related papers  (2020-04-06T22:35:55Z)
• DualSDF: Semantic Shape Manipulation using a Two-Level Representation [54.62411904952258]
  We propose DualSDF, a representation expressing shapes at two levels of granularity, one capturing fine details and the other representing an abstracted proxy shape. Our two-level model gives rise to a new shape manipulation technique in which a user can interactively manipulate the coarse proxy shape and see the changes instantly mirrored in the high-resolution shape.
  arXiv  Detail & Related papers  (2020-04-06T17:59:15Z)
• Deep Geometric Functional Maps: Robust Feature Learning for Shape Correspondence [31.840880075039944]
  We present a novel learning-based approach for computing correspondences between non-rigid 3D shapes. Key to our method is a feature-extraction network that learns directly from raw shape geometry.
  arXiv  Detail & Related papers  (2020-03-31T15:20:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.