Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution

• URL: http://arxiv.org/abs/2305.03216v2
• Date: Thu, 10 Aug 2023 01:59:55 GMT
• Title: Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution
• Authors: Hyojoon Park, Sangeetha Grama Srinivasan, Matthew Cong, Doyub Kim, Byungsoo Kim, Jonathan Swartz, Ken Museth, Eftychios Sifakis
• Abstract summary: We present a neural network-based simulation framework that can efficiently and realistically enhance a facial performance produced by a low-cost, realtime physics-based simulation. We use face animation as an exemplar of such a simulation domain, where creating this semantic congruence is achieved by simply dialing in the same muscle actuation controls and skeletal pose in the two simulators. Our proposed neural network super-resolution framework generalizes from this training set to unseen expressions, compensates for modeling discrepancies between the two simulations due to limited resolution or cost-cutting approximations in the real-time variant, and does not require any semantic descriptors or parameters to
• Score: 7.14576106770047
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a neural network-based simulation super-resolution framework that can efficiently and realistically enhance a facial performance produced by a low-cost, realtime physics-based simulation to a level of detail that closely approximates that of a reference-quality off-line simulator with much higher resolution (26x element count in our examples) and accurate physical modeling. Our approach is rooted in our ability to construct - via simulation - a training set of paired frames, from the low- and high-resolution simulators respectively, that are in semantic correspondence with each other. We use face animation as an exemplar of such a simulation domain, where creating this semantic congruence is achieved by simply dialing in the same muscle actuation controls and skeletal pose in the two simulators. Our proposed neural network super-resolution framework generalizes from this training set to unseen expressions, compensates for modeling discrepancies between the two simulations due to limited resolution or cost-cutting approximations in the real-time variant, and does not require any semantic descriptors or parameters to be provided as input, other than the result of the real-time simulation. We evaluate the efficacy of our pipeline on a variety of expressive performances and provide comparisons and ablation experiments for plausible variations and alternatives to our proposed scheme.

Related papers

• BeSimulator: A Large Language Model Powered Text-based Behavior Simulator [28.112491177744783]
  We introduce BeSimulator as an attempt towards behavior simulation in the context of text-based environments. BeSimulator can generalize across scenarios and achieve long-horizon complex simulation.
  arXiv  Detail & Related papers  (2024-09-24T08:37:04Z)
• A Realistic Simulation Framework for Analog/Digital Neuromorphic Architectures [73.65190161312555]
  ARCANA is a spiking neural network simulator designed to account for the properties of mixed-signal neuromorphic circuits. We show how the results obtained provide a reliable estimate of the behavior of the spiking neural network trained in software.
  arXiv  Detail & Related papers  (2024-09-23T11:16:46Z)
• Accelerate Neural Subspace-Based Reduced-Order Solver of Deformable Simulation by Lipschitz Optimization [9.364019847856714]
  Reduced-order simulation is an emerging method for accelerating physical simulations with high DOFs. We propose a method for finding optimized subspace mappings, enabling further acceleration of neural reduced-order simulations. We demonstrate the effectiveness of our approach through general cases in both quasi-static and dynamics simulations.
  arXiv  Detail & Related papers  (2024-09-05T12:56:03Z)
• Informal Safety Guarantees for Simulated Optimizers Through Extrapolation from Partial Simulations [0.0]
  Self-supervised learning is the backbone of state of the art language modeling. It has been argued that training with predictive loss on a self-supervised dataset causes simulators.
  arXiv  Detail & Related papers  (2023-11-29T09:32:56Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation [110.4255414234771]
  Existing solutions require massive training data or lack generalizability to unknown rendering configurations. We propose a novel approach that marries domain randomization and differentiable rendering gradients to address this problem. Our approach achieves significantly lower reconstruction errors and has better generalizability among unknown rendering configurations.
  arXiv  Detail & Related papers  (2022-05-11T17:59:51Z)
• Predicting Loose-Fitting Garment Deformations Using Bone-Driven Motion Networks [63.596602299263935]
  We present a learning algorithm that uses bone-driven motion networks to predict the deformation of loose-fitting garment meshes at interactive rates. We show that our method outperforms state-of-the-art methods in terms of prediction accuracy of mesh deformations by about 20% in RMSE and 10% in Hausdorff distance and STED.
  arXiv  Detail & Related papers  (2022-05-03T07:54:39Z)
• DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects [18.266002992029716]
  Research in manipulation of deformable objects is typically conducted on a limited range of scenarios. Realistic simulators with support for various types of deformations and interactions have the potential to speed up experimentation. For highly deformable objects it is challenging to align the output of a simulator with the behavior of real objects.
  arXiv  Detail & Related papers  (2022-04-07T00:45:26Z)
• Generic Lithography Modeling with Dual-band Optics-Inspired Neural Networks [52.200624127512874]
  We introduce a dual-band optics-inspired neural network design that considers the optical physics underlying lithography. Our approach yields the first published via/metal layer contour simulation at 1nm2/pixel resolution with any tile size. We also achieve 85X simulation speedup over traditional lithography simulator with 1% accuracy loss.
  arXiv  Detail & Related papers  (2022-03-12T08:08:50Z)
• Augmenting Differentiable Simulators with Neural Networks to Close the Sim2Real Gap [15.1962264049463]
  We present a differentiable simulation architecture for articulated rigid-body dynamics that enables the augmentation of analytical models with neural networks at any point of the computation.
  arXiv  Detail & Related papers  (2020-07-12T17:27:11Z)
• Learning to Simulate Complex Physics with Graph Networks [68.43901833812448]
  We present a machine learning framework and model implementation that can learn to simulate a wide variety of challenging physical domains. Our framework---which we term "Graph Network-based Simulators" (GNS)--represents the state of a physical system with particles, expressed as nodes in a graph, and computes dynamics via learned message-passing. Our results show that our model can generalize from single-timestep predictions with thousands of particles during training, to different initial conditions, thousands of timesteps, and at least an order of magnitude more particles at test time.
  arXiv  Detail & Related papers  (2020-02-21T16:44:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.