Neural Fields in Visual Computing and Beyond
- URL: http://arxiv.org/abs/2111.11426v2
- Date: Tue, 23 Nov 2021 20:44:43 GMT
- Title: Neural Fields in Visual Computing and Beyond
- Authors: Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan,
Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, Srinath
Sridhar
- Abstract summary: Recent advances in machine learning have created increasing interest in solving visual computing problems using coordinate-based neural networks.
neural fields have seen successful application in the synthesis of 3D shapes and image, animation of human bodies, 3D reconstruction, and pose estimation.
This report provides context, mathematical grounding, and an extensive review of literature on neural fields.
- Score: 54.950885364735804
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent advances in machine learning have created increasing interest in
solving visual computing problems using a class of coordinate-based neural
networks that parametrize physical properties of scenes or objects across space
and time. These methods, which we call neural fields, have seen successful
application in the synthesis of 3D shapes and image, animation of human bodies,
3D reconstruction, and pose estimation. However, due to rapid progress in a
short time, many papers exist but a comprehensive review and formulation of the
problem has not yet emerged. In this report, we address this limitation by
providing context, mathematical grounding, and an extensive review of
literature on neural fields. This report covers research along two dimensions.
In Part I, we focus on techniques in neural fields by identifying common
components of neural field methods, including different representations,
architectures, forward mapping, and generalization methods. In Part II, we
focus on applications of neural fields to different problems in visual
computing, and beyond (e.g., robotics, audio). Our review shows the breadth of
topics already covered in visual computing, both historically and in current
incarnations, demonstrating the improved quality, flexibility, and capability
brought by neural fields methods. Finally, we present a companion website that
contributes a living version of this review that can be continually updated by
the community.
Related papers
- Graph Neural Networks for Brain Graph Learning: A Survey [53.74244221027981]
Graph neural networks (GNNs) have demonstrated a significant advantage in mining graph-structured data.
GNNs to learn brain graph representations for brain disorder analysis has recently gained increasing attention.
In this paper, we aim to bridge this gap by reviewing brain graph learning works that utilize GNNs.
arXiv Detail & Related papers (2024-06-01T02:47:39Z) - Hebbian Learning based Orthogonal Projection for Continual Learning of
Spiking Neural Networks [74.3099028063756]
We develop a new method with neuronal operations based on lateral connections and Hebbian learning.
We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities.
Our method consistently solves for spiking neural networks with nearly zero forgetting.
arXiv Detail & Related papers (2024-02-19T09:29:37Z) - Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization
Method [4.981452040789784]
We present a novel auto-encoder-based non-linear landscape visualization method called Neuro-Visualizer.
Our findings show that Neuro-Visualizer outperforms other linear and non-linear baselines.
arXiv Detail & Related papers (2023-09-26T01:10:16Z) - Intrinsic Neural Fields: Learning Functions on Manifolds [45.269698580847916]
Intrinsic neural fields combine the advantages of neural fields with the spectral properties of the Laplace-Beltrami operator.
We show that intrinsic neural fields can reconstruct high-quality textures from images with state-of-the-art quality.
We demonstrate the versatility of intrinsic neural fields by tackling various applications.
arXiv Detail & Related papers (2022-03-15T14:52:52Z) - Overcoming the Domain Gap in Neural Action Representations [60.47807856873544]
3D pose data can now be reliably extracted from multi-view video sequences without manual intervention.
We propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations.
To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions.
arXiv Detail & Related papers (2021-12-02T12:45:46Z) - Neural Actor: Neural Free-view Synthesis of Human Actors with Pose
Control [80.79820002330457]
We propose a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses.
Our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses.
arXiv Detail & Related papers (2021-06-03T17:40:48Z) - Action in Mind: A Neural Network Approach to Action Recognition and
Segmentation [0.0]
This thesis presents a novel computational approach for human action recognition through different implementations of multi-layer architectures based on artificial neural networks.
The proposed action recognition architecture is composed of several processing layers including a preprocessing layer, an ordered vector representation layer and three layers of neural networks.
For each level of development the system is trained with the input data consisting of consecutive 3D body postures and tested with generalized input data that the system has never met before.
arXiv Detail & Related papers (2021-04-30T09:53:28Z) - Neural population geometry: An approach for understanding biological and
artificial neural networks [3.4809730725241605]
We review examples of geometrical approaches providing insight into the function of biological and artificial neural networks.
Neural population geometry has the potential to unify our understanding of structure and function in biological and artificial neural networks.
arXiv Detail & Related papers (2021-04-14T18:10:34Z) - Understanding Information Processing in Human Brain by Interpreting
Machine Learning Models [1.14219428942199]
The thesis explores the role machine learning methods play in creating intuitive computational models of neural processing.
This perspective makes the case in favor of the larger role that exploratory and data-driven approach to computational neuroscience could play.
arXiv Detail & Related papers (2020-10-17T04:37:26Z) - State of the Art on Neural Rendering [141.22760314536438]
We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photo-realistic outputs.
This report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free-viewpoint video, and the creation of photo-realistic avatars for virtual and augmented reality telepresence.
arXiv Detail & Related papers (2020-04-08T04:36:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.