A New Type of Foundation Model Based on Recordings of People's Emotions and Physiology

• URL: http://arxiv.org/abs/2408.00030v1
• Date: Wed, 31 Jul 2024 11:14:45 GMT
• Title: A New Type of Foundation Model Based on Recordings of People's Emotions and Physiology
• Authors: David Gamez, Dionis Barcari, Aliya Grig,
• Abstract summary: A first-person foundation model would map environmental stimuli to a person's emotional and physiological states. We have developed a recording rig that captures what the wearer is seeing and hearing as well as their emotional and physiological states. This novel source of data could help to address the shortage of new data for building the next generation of foundation models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Foundation models have had a big impact in recent years and billions of dollars are being invested in them in the current AI boom. The more popular ones, such as Chat-GPT, are trained on large amounts of data from the Internet, and then reinforcement learning, RAG, prompt engineering and cognitive modelling are used to fine-tune and augment their behavior. This technology has been used to create models of individual people, such as Caryn Marjorie. However, these chatbots are not based on people's actual emotional and physiological responses to their environment, so they are, at best, surface-level approximations to the characters they are imitating. This paper describes how a new type of foundation model - a first-person foundation model - could be created from recordings of what a person sees and hears as well as their emotional and physiological reactions to these stimuli. A first-person foundation model would map environmental stimuli to a person's emotional and physiological states, and map a person's emotional and physiological states to their behavior. First-person foundation models have many exciting applications, including a new type of recommendation engine, personal assistants, generative adversarial networks, dating and recruitment. To obtain training data for a first-person foundation model, we have developed a recording rig that captures what the wearer is seeing and hearing as well as their emotional and physiological states. This novel source of data could help to address the shortage of new data for building the next generation of foundation models.

Related papers

• Human-Centric Foundation Models: Perception, Generation and Agentic Modeling [79.97999901785772]
  Human-centric Foundation Models unify diverse human-centric tasks into a single framework. We present a comprehensive overview of HcFMs by proposing a taxonomy that categorizes current approaches into four groups. This survey aims to serve as a roadmap for researchers and practitioners working towards more robust, versatile, and intelligent digital human and embodiments modeling.
  arXiv  Detail & Related papers  (2025-02-12T16:38:40Z)
• Cosmos World Foundation Model Platform for Physical AI [136.1002343616157]
  Physical AI needs a digital twin of itself, the policy model, and a digital twin of the world, the world model. We present the Cosmos World Foundation Model Platform to help developers build customized world models for their Physical AI setups.
  arXiv  Detail & Related papers  (2025-01-07T06:55:50Z)
• A Review of Human Emotion Synthesis Based on Generative Technology [14.92674135999986]
  Human emotion synthesis is a crucial aspect of affective computing. It involves using computational methods to mimic and convey human emotions through various modalities. Recent advancements in generative models have significantly contributed to the development of this field.
  arXiv  Detail & Related papers  (2024-12-10T02:06:10Z)
• Human-like Affective Cognition in Foundation Models [28.631313772625578]
  We introduce an evaluation framework for testing affective cognition in foundation models. We generate 1,280 diverse scenarios exploring relationships between appraisals, emotions, expressions, and outcomes. Our results show foundation models tend to agree with human intuitions, matching or exceeding interparticipant agreement.
  arXiv  Detail & Related papers  (2024-09-18T06:42:13Z)
• Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations [52.11801730860999]
  In recent years, the robot learning community has shown increasing interest in using deep generative models to capture the complexity of large datasets. We present the different types of models that the community has explored, such as energy-based models, diffusion models, action value maps, or generative adversarial networks. We also present the different types of applications in which deep generative models have been used, from grasp generation to trajectory generation or cost learning.
  arXiv  Detail & Related papers  (2024-08-08T11:34:31Z)
• Recording First-person Experiences to Build a New Type of Foundation Model [0.0]
  We have developed a recording rig that captures what the wearer is seeing and hearing as well as their skin conductance. AI algorithms are used to process this data into a rich picture of the environment and internal states of the subject. This type of model has many potential applications, including recommendation, personal assistance, GAN systems, dating and recruitment.
  arXiv  Detail & Related papers  (2024-07-31T11:51:26Z)
• On the Stability of Iterative Retraining of Generative Models on their own Data [56.153542044045224]
  We study the impact of training generative models on mixed datasets. We first prove the stability of iterative training under the condition that the initial generative models approximate the data distribution well enough. We empirically validate our theory on both synthetic and natural images by iteratively training normalizing flows and state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-09-30T16:41:04Z)
• Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation [54.68893964373141]
  Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos. Recent developments in diffusion-based generative models allow for more realistic and stable data synthesis. We present an autoregressive diffusion model that requires only one identity image and audio sequence to generate a video of a realistic talking human head.
  arXiv  Detail & Related papers  (2023-01-06T14:16:54Z)
• PAL: Persona-Augmented Emotional Support Conversation Generation [54.069321178816686]
  Due to the lack of human resources for mental health support, there is an increasing demand for employing conversational agents for support. Recent work has demonstrated the effectiveness of dialogue models in providing emotional support. We propose a framework for dynamically inferring and modeling seekers' persona.
  arXiv  Detail & Related papers  (2022-12-19T04:12:54Z)
• Emotion Recognition of the Singing Voice: Toward a Real-Time Analysis Tool for Singers [0.0]
  Current computational-emotion research has focused on applying acoustic properties to analyze how emotions are perceived mathematically. This paper seeks to reflect and expand upon the findings of related research and present a stepping-stone toward this end goal.
  arXiv  Detail & Related papers  (2021-05-01T05:47:15Z)
• The BIRAFFE2 Experiment. Study in Bio-Reactions and Faces for Emotion-based Personalization for AI Systems [0.0]
  We present an unified paradigm allowing to capture emotional responses of different persons. We provide a framework that can be easily used and developed for the purpose of the machine learning methods.
  arXiv  Detail & Related papers  (2020-07-29T18:35:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.