Neural Fields in Visual Computing and Beyond
- URL: http://arxiv.org/abs/2111.11426v2
- Date: Tue, 23 Nov 2021 20:44:43 GMT
- Title: Neural Fields in Visual Computing and Beyond
- Authors: Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan,
Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, Srinath
Sridhar
- Abstract summary: Recent advances in machine learning have created increasing interest in solving visual computing problems using coordinate-based neural networks.
neural fields have seen successful application in the synthesis of 3D shapes and image, animation of human bodies, 3D reconstruction, and pose estimation.
This report provides context, mathematical grounding, and an extensive review of literature on neural fields.
- Score: 54.950885364735804
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent advances in machine learning have created increasing interest in
solving visual computing problems using a class of coordinate-based neural
networks that parametrize physical properties of scenes or objects across space
and time. These methods, which we call neural fields, have seen successful
application in the synthesis of 3D shapes and image, animation of human bodies,
3D reconstruction, and pose estimation. However, due to rapid progress in a
short time, many papers exist but a comprehensive review and formulation of the
problem has not yet emerged. In this report, we address this limitation by
providing context, mathematical grounding, and an extensive review of
literature on neural fields. This report covers research along two dimensions.
In Part I, we focus on techniques in neural fields by identifying common
components of neural field methods, including different representations,
architectures, forward mapping, and generalization methods. In Part II, we
focus on applications of neural fields to different problems in visual
computing, and beyond (e.g., robotics, audio). Our review shows the breadth of
topics already covered in visual computing, both historically and in current
incarnations, demonstrating the improved quality, flexibility, and capability
brought by neural fields methods. Finally, we present a companion website that
contributes a living version of this review that can be continually updated by
the community.
Related papers
- NeuroFly: A framework for whole-brain single neuron reconstruction [17.93211301158225]
We introduce NeuroFly, a validated framework for large-scale automatic single neuron reconstruction.
NeuroFly breaks down the process into three distinct stages: segmentation, connection, and proofreading.
Our goal is to foster collaboration among researchers to address the neuron reconstruction challenge.
arXiv Detail & Related papers (2024-11-07T13:56:13Z) - Neural Fields in Robotics: A Survey [39.93473561102639]
Neural Fields have emerged as a transformative approach for 3D scene representation in computer vision and robotics.
This survey explores their applications in robotics, emphasizing their potential to enhance perception, planning, and control.
Their compactness, memory efficiency, and differentiability, along with seamless integration with foundation and generative models, make them ideal for real-time applications.
arXiv Detail & Related papers (2024-10-26T16:26:41Z) - Graph Neural Networks for Brain Graph Learning: A Survey [53.74244221027981]
Graph neural networks (GNNs) have demonstrated a significant advantage in mining graph-structured data.
GNNs to learn brain graph representations for brain disorder analysis has recently gained increasing attention.
In this paper, we aim to bridge this gap by reviewing brain graph learning works that utilize GNNs.
arXiv Detail & Related papers (2024-06-01T02:47:39Z) - Hebbian Learning based Orthogonal Projection for Continual Learning of
Spiking Neural Networks [74.3099028063756]
We develop a new method with neuronal operations based on lateral connections and Hebbian learning.
We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities.
Our method consistently solves for spiking neural networks with nearly zero forgetting.
arXiv Detail & Related papers (2024-02-19T09:29:37Z) - Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization
Method [4.981452040789784]
We present a novel auto-encoder-based non-linear landscape visualization method called Neuro-Visualizer.
Our findings show that Neuro-Visualizer outperforms other linear and non-linear baselines.
arXiv Detail & Related papers (2023-09-26T01:10:16Z) - Overcoming the Domain Gap in Neural Action Representations [60.47807856873544]
3D pose data can now be reliably extracted from multi-view video sequences without manual intervention.
We propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations.
To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions.
arXiv Detail & Related papers (2021-12-02T12:45:46Z) - Neural Actor: Neural Free-view Synthesis of Human Actors with Pose
Control [80.79820002330457]
We propose a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses.
Our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses.
arXiv Detail & Related papers (2021-06-03T17:40:48Z) - Neural population geometry: An approach for understanding biological and
artificial neural networks [3.4809730725241605]
We review examples of geometrical approaches providing insight into the function of biological and artificial neural networks.
Neural population geometry has the potential to unify our understanding of structure and function in biological and artificial neural networks.
arXiv Detail & Related papers (2021-04-14T18:10:34Z) - Understanding Information Processing in Human Brain by Interpreting
Machine Learning Models [1.14219428942199]
The thesis explores the role machine learning methods play in creating intuitive computational models of neural processing.
This perspective makes the case in favor of the larger role that exploratory and data-driven approach to computational neuroscience could play.
arXiv Detail & Related papers (2020-10-17T04:37:26Z) - State of the Art on Neural Rendering [141.22760314536438]
We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photo-realistic outputs.
This report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free-viewpoint video, and the creation of photo-realistic avatars for virtual and augmented reality telepresence.
arXiv Detail & Related papers (2020-04-08T04:36:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.