VLMaterial: Procedural Material Generation with Large Vision-Language Models

• URL: http://arxiv.org/abs/2501.18623v2
• Date: Tue, 18 Feb 2025 16:53:58 GMT
• Title: VLMaterial: Procedural Material Generation with Large Vision-Language Models
• Abstract summary: We leverage the ability to convert procedural materials into standard Python programs.<n>We fine-tune a large pre-trained vision-language model to generate such programs from input images.<n>Our method outperforms previous methods on both synthetic and real-world examples.
• Score: 43.60515109432699
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Procedural materials, represented as functional node graphs, are ubiquitous in computer graphics for photorealistic material appearance design. They allow users to perform intuitive and precise editing to achieve desired visual appearances. However, creating a procedural material given an input image requires professional knowledge and significant effort. In this work, we leverage the ability to convert procedural materials into standard Python programs and fine-tune a large pre-trained vision-language model (VLM) to generate such programs from input images. To enable effective fine-tuning, we also contribute an open-source procedural material dataset and propose to perform program-level augmentation by prompting another pre-trained large language model (LLM). Through extensive evaluation, we show that our method outperforms previous methods on both synthetic and real-world examples.

Related papers

• Reflecting Process Expertise in Procedural Material Generation [47.15641864882028]
  Procedural material creation underpins applications in digital content creation, visual effects, and 3D asset design.<n>We represent expert synthesis as process traces: textual records of construction steps, parameters, and design intent.<n>We leverage tutorial videos as a source of process knowledge and extract textual traces using automated video analysis tools.
  arXiv  Detail & Related papers  (2026-07-14T22:59:32Z)
• Automated Visualization Makeovers with LLMs [0.716879432974126]
  Visualisation makeovers are exercises where the community exchange feedback to improve charts and data visualizations.<n>Can multi-modal large language models (LLMs) emulate this task?<n>Our system is centred around prompt engineering of a pre-trained model, relying on a combination of user guidelines and any latent knowledge of data visualization practices.
  arXiv  Detail & Related papers  (2025-07-21T11:51:20Z)
• ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models [103.25208095165486]
  Existing practices rely on powerful but costly large language models (LLMs) or multimodal language models (MLMs) to produce instruction data. We present a programmatic approach that employs scene graphs as symbolic representations of images and human-written programs to systematically synthesize vision-centric instruction data. Our approach ensures the interpretability and controllability of the data generation process and scales efficiently while maintaining factual accuracy.
  arXiv  Detail & Related papers  (2024-12-09T21:44:02Z)
• PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions [65.73013453373349]
  PixWizard is an image-to-image visual assistant designed for image generation, manipulation, and translation based on free-from language instructions. We tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning dataset. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions.
  arXiv  Detail & Related papers  (2024-09-23T17:59:46Z)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension [131.14381425260706]
  We introduce Self-Training on Image (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference for image descriptions using unlabeled images. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data.
  arXiv  Detail & Related papers  (2024-05-30T05:53:49Z)
• Synth$^2$: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings [16.28853186016663]
  We create synthetic image-text pairs for efficient and effective Visual-Language Models (VLMs) training. Our method employs a pretrained text-to-image model to synthesize image embeddings from captions generated by an LLM. Our VLM, finetuned on synthetic data achieves comparable performance to models trained solely on human-annotated data.
  arXiv  Detail & Related papers  (2024-03-12T15:36:42Z)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists [66.85125112199898]
  We develop a unified language interface for computer vision tasks that abstracts away task-specific design choices. Our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models.
  arXiv  Detail & Related papers  (2023-09-30T14:26:43Z)
• Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models [50.07056960586183]
  We propose Position-enhanced Visual Instruction Tuning (PVIT) to extend the functionality of Multimodal Large Language Models (MLLMs) This integration promotes a more detailed comprehension of images for the MLLM. We present both quantitative experiments and qualitative analysis that demonstrate the superiority of the proposed model.
  arXiv  Detail & Related papers  (2023-08-25T15:33:47Z)
• GPT4Image: Can Large Pre-trained Models Help Vision Models on Perception Tasks? [51.22096780511165]
  We present a new learning paradigm in which the knowledge extracted from large pre-trained models are utilized to help models like CNN and ViT learn enhanced representations. We feed detailed descriptions into a pre-trained encoder to extract text embeddings with rich semantic information that encodes the content of images.
  arXiv  Detail & Related papers  (2023-06-01T14:02:45Z)
• Neural Photometry-guided Visual Attribute Transfer [4.630419389180576]
  We present a deep learning-based method for propagating visual material attributes to larger samples of the same or similar materials. For training, we leverage images of the material taken under multiple illuminations and a dedicated data augmentation policy. Our model relies on a supervised image-to-image translation framework and is agnostic to the transferred domain.
  arXiv  Detail & Related papers  (2021-12-05T09:22:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.