What makes an image realistic?

• URL: http://arxiv.org/abs/2403.04493v4
• Date: Tue, 21 May 2024 14:44:46 GMT
• Title: What makes an image realistic?
• Authors: Lucas Theis,
• Abstract summary: We discuss the problem of quantifying realism, that is, designing functions that can reliably tell realistic data from unrealistic data. We discuss why this problem is challenging, why a good generative model alone is insufficient to solve it, and what a good solution would look like. We introduce the notion of a universal critic, which unlike adversarial critics does not require adversarial training.
• Score: 5.656581242851759
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The last decade has seen tremendous progress in our ability to generate realistic-looking data, be it images, text, audio, or video. Here, we discuss the closely related problem of quantifying realism, that is, designing functions that can reliably tell realistic data from unrealistic data. This problem turns out to be significantly harder to solve and remains poorly understood, despite its prevalence in machine learning and recent breakthroughs in generative AI. Drawing on insights from algorithmic information theory, we discuss why this problem is challenging, why a good generative model alone is insufficient to solve it, and what a good solution would look like. In particular, we introduce the notion of a universal critic, which unlike adversarial critics does not require adversarial training. While universal critics are not immediately practical, they can serve both as a North Star for guiding practical implementations and as a tool for analyzing existing attempts to capture realism.

Related papers

• Enter: Graduated Realism: A Pedagogical Framework for AI-Powered Avatars in Virtual Reality Teacher Training [0.0]
  We argue that hyper-realism is not always optimal, as high-fidelity avatars can impose excessive extraneous cognitive load on novices.<n>A significant gap exists between the technological drive for photorealism and the pedagogical need for scaffolded learning.<n>We propose Graduated Realism, a framework advocating for starting trainees with lower-fidelity avatars.
  arXiv  Detail & Related papers  (2025-06-13T15:37:36Z)
• Information-Theoretic Foundations for Machine Learning [20.617552198581024]
  We propose a theoretical framework which attempts to answer what exists outside of the cave. We provide a framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning. We conclude with a section dedicated to characterizing the performance of misspecified algorithms.
  arXiv  Detail & Related papers  (2024-07-17T03:18:40Z)
• Efficient Imitation Learning with Conservative World Models [54.52140201148341]
  We tackle the problem of policy learning from expert demonstrations without a reward function. We re-frame imitation learning as a fine-tuning problem, rather than a pure reinforcement learning one.
  arXiv  Detail & Related papers  (2024-05-21T20:53:18Z)
• Sim2real Transfer Learning for Point Cloud Segmentation: An Industrial Application Case on Autonomous Disassembly [55.41644538483948]
  We present an industrial application case that uses sim2real transfer learning for point cloud data. We provide insights on how to generate and process synthetic point cloud data. A novel patch-based attention network is proposed additionally to tackle this problem.
  arXiv  Detail & Related papers  (2023-01-12T14:00:37Z)
• Understanding Self-Predictive Learning for Reinforcement Learning [61.62067048348786]
  We study the learning dynamics of self-predictive learning for reinforcement learning. We propose a novel self-predictive algorithm that learns two representations simultaneously.
  arXiv  Detail & Related papers  (2022-12-06T20:43:37Z)
• Resolving Copycat Problems in Visual Imitation Learning via Residual Action Prediction [10.275717930942989]
  We show that imitation from observation histories performs worse than imitation from the most recent observation. We propose a novel imitation learning neural network architecture that does not suffer from this issue by design.
  arXiv  Detail & Related papers  (2022-07-20T07:15:32Z)
• Causal Reasoning Meets Visual Representation Learning: A Prospective Study [117.08431221482638]
  Lack of interpretability, robustness, and out-of-distribution generalization are becoming the challenges of the existing visual models. Inspired by the strong inference ability of human-level agents, recent years have witnessed great effort in developing causal reasoning paradigms. This paper aims to provide a comprehensive overview of this emerging field, attract attention, encourage discussions, bring to the forefront the urgency of developing novel causal reasoning methods.
  arXiv  Detail & Related papers  (2022-04-26T02:22:28Z)
• Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation [88.8963330073454]
  We propose a novel monocular 6D pose estimation approach by means of self-supervised learning. We leverage current trends in noisy student training and differentiable rendering to further self-supervise the model. Our proposed self-supervision outperforms all other methods relying on synthetic data.
  arXiv  Detail & Related papers  (2022-03-19T15:12:06Z)
• On The Empirical Effectiveness of Unrealistic Adversarial Hardening Against Realistic Adversarial Attacks [9.247680268877795]
  We study whether unrealistic adversarial examples can be used to protect models against realistic examples. Our results reveal discrepancies across the use cases, where unrealistic examples can either be as effective as the realistic ones or may offer only limited improvement. We shed light on the patterns that discriminate which unrealistic examples can be used for effective hardening.
  arXiv  Detail & Related papers  (2022-02-07T15:08:10Z)
• Practical Imitation Learning in the Real World via Task Consistency Loss [18.827979446629296]
  This paper introduces a self-supervised loss that encourages sim and real alignment both at the feature and action-prediction levels. We achieve 80% success across ten seen and unseen scenes using only 16.2 hours of teleoperated demonstrations in sim and real.
  arXiv  Detail & Related papers  (2022-02-03T21:43:06Z)
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense [142.53911271465344]
  We argue that the next generation of AI must embrace "dark" humanlike common sense for solving novel tasks. We identify functionality, physics, intent, causality, and utility (FPICU) as the five core domains of cognitive AI with humanlike common sense.
  arXiv  Detail & Related papers  (2020-04-20T04:07:28Z)
• Adversarial Texture Optimization from RGB-D Scans [37.78810126921875]
  We present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views. The key idea of our approach is to learn a patch-based conditional discriminator which guides the texture optimization to be tolerant to misalignments.
  arXiv  Detail & Related papers  (2020-03-18T18:00:05Z)
• CRAVES: Controlling Robotic Arm with a Vision-based Economic System [96.56564257199474]
  Training a robotic arm to accomplish real-world tasks has been attracting increasing attention in both academia and industry.<n>This work discusses the role of computer vision algorithms in this field.<n>We present an alternative solution, which uses a 3D model to create a large number of synthetic data.
  arXiv  Detail & Related papers  (2018-12-03T13:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.