SmartAvatar: Text- and Image-Guided Human Avatar Generation with VLM AI Agents

• URL: http://arxiv.org/abs/2506.04606v1
• Date: Thu, 05 Jun 2025 03:49:01 GMT
• Title: SmartAvatar: Text- and Image-Guided Human Avatar Generation with VLM AI Agents
• Authors: Alexander Huang-Menders, Xinhang Liu, Andy Xu, Yuyao Zhang, Chi-Keung Tang, Yu-Wing Tai,
• Abstract summary: SmartAvatar is a vision-language-agent-driven framework for generating fully rigged, animation-ready 3D human avatars.<n>A key innovation is an autonomous verification loop, where the agent renders draft avatars.<n>The generated avatars are fully rigged and support pose manipulation with consistent identity and appearance.
• Score: 91.26239311240873
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: SmartAvatar is a vision-language-agent-driven framework for generating fully rigged, animation-ready 3D human avatars from a single photo or textual prompt. While diffusion-based methods have made progress in general 3D object generation, they continue to struggle with precise control over human identity, body shape, and animation readiness. In contrast, SmartAvatar leverages the commonsense reasoning capabilities of large vision-language models (VLMs) in combination with off-the-shelf parametric human generators to deliver high-quality, customizable avatars. A key innovation is an autonomous verification loop, where the agent renders draft avatars, evaluates facial similarity, anatomical plausibility, and prompt alignment, and iteratively adjusts generation parameters for convergence. This interactive, AI-guided refinement process promotes fine-grained control over both facial and body features, enabling users to iteratively refine their avatars via natural-language conversations. Unlike diffusion models that rely on static pre-trained datasets and offer limited flexibility, SmartAvatar brings users into the modeling loop and ensures continuous improvement through an LLM-driven procedural generation and verification system. The generated avatars are fully rigged and support pose manipulation with consistent identity and appearance, making them suitable for downstream animation and interactive applications. Quantitative benchmarks and user studies demonstrate that SmartAvatar outperforms recent text- and image-driven avatar generation systems in terms of reconstructed mesh quality, identity fidelity, attribute accuracy, and animation readiness, making it a versatile tool for realistic, customizable avatar creation on consumer-grade hardware.

Related papers

• VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis [70.76837748695841]
  We propose VisualSpeaker, a novel method that bridges the gap using photorealistic differentiable rendering, supervised by visual speech recognition, for improved 3D facial animation.<n>Our contribution is a perceptual lip-reading loss, derived by passing 3D Gaussian Splatting avatar renders through a pre-trained Visual Automatic Speech Recognition model during training.<n> Evaluation on the MEAD dataset demonstrates that VisualSpeaker improves both the standard Lip Vertex Error metric by 56.1% and the perceptual quality of the generated animations, while retaining the controllability of mesh-driven animation.
  arXiv  Detail & Related papers  (2025-07-08T15:04:17Z)
• AdaHuman: Animatable Detailed 3D Human Generation with Compositional Multiview Diffusion [56.12859795754579]
  AdaHuman is a novel framework that generates high-fidelity animatable 3D avatars from a single in-the-wild image.<n>AdaHuman incorporates two key innovations: a pose-conditioned 3D joint diffusion model and a compositional 3DGS refinement module.
  arXiv  Detail & Related papers  (2025-05-30T17:59:54Z)
• EVA: Expressive Virtual Avatars from Multi-view Videos [51.33851869426057]
  We introduce Expressive Virtual Avatars (EVA), an actor-specific, fully controllable, and expressive human avatar framework.<n>EVA achieves high-fidelity, lifelike renderings in real time while enabling independent control of facial expressions, body movements, and hand gestures.<n>This work represents a significant advancement towards fully drivable digital human models.
  arXiv  Detail & Related papers  (2025-05-21T11:22:52Z)
• Multimodal Generation of Animatable 3D Human Models with AvatarForge [67.31920821192323]
  AvatarForge is a framework for generating animatable 3D human avatars from text or image inputs using AI-driven procedural generation.<n>Our evaluations show that AvatarForge outperforms state-of-the-art methods in both text- and image-to-avatar generation.
  arXiv  Detail & Related papers  (2025-03-11T08:29:18Z)
• GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars [44.8290935585746]
  Photo-realistic and controllable 3D avatars are crucial for various applications such as virtual and mixed reality (VR/MR), telepresence, gaming, and film production. Traditional methods for avatar creation often involve time-consuming scanning and reconstruction processes for each avatar. We propose a text-conditioned generative model that can generate photo-realistic facial avatars of diverse identities.
  arXiv  Detail & Related papers  (2024-08-24T21:25:22Z)
• DivAvatar: Diverse 3D Avatar Generation with a Single Prompt [95.9978722953278]
  DivAvatar is a framework that generates diverse avatars from a single text prompt. It has two key designs that help achieve generation diversity and visual quality. Extensive experiments show that DivAvatar is highly versatile in generating avatars of diverse appearances.
  arXiv  Detail & Related papers  (2024-02-27T08:10:31Z)
• AgentAvatar: Disentangling Planning, Driving and Rendering for Photorealistic Avatar Agents [16.544688997764293]
  Our framework harnesses LLMs to produce a series of detailed text descriptions of the avatar agents' facial motions. These descriptions are processed by our task-agnostic driving engine into continuous motion embeddings. Our framework adapts to a variety of non-verbal avatar interactions, both monadic and dyadic.
  arXiv  Detail & Related papers  (2023-11-29T09:13:00Z)
• Text-Conditional Contextualized Avatars For Zero-Shot Personalization [47.85747039373798]
  We propose a pipeline that enables personalization of image generation with avatars capturing a user's identity in a delightful way. Our pipeline is zero-shot, avatar texture and style agnostic, and does not require training on the avatar at all. We show, for the first time, how to leverage large-scale image datasets to learn human 3D pose parameters.
  arXiv  Detail & Related papers  (2023-04-14T22:00:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.