VoCopilot: Voice-Activated Tracking of Everyday Interactions

• URL: http://arxiv.org/abs/2312.10265v1
• Date: Fri, 15 Dec 2023 23:46:52 GMT
• Title: VoCopilot: Voice-Activated Tracking of Everyday Interactions
• Authors: Sheen An Goh, Manoj Gulati, Ambuj Varshney
• Abstract summary: This paper presents our efforts to design a new vocal tracking system we call VoCopilot. VoCopilot is an end-to-end system centered around an energy-efficient acoustic hardware and firmware combined with advanced machine learning models.
• Score: 1.0435741631709405
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Voice plays an important role in our lives by facilitating communication, conveying emotions, and indicating health. Therefore, tracking vocal interactions can provide valuable insight into many aspects of our lives. This paper presents our ongoing efforts to design a new vocal tracking system we call VoCopilot. VoCopilot is an end-to-end system centered around an energy-efficient acoustic hardware and firmware combined with advanced machine learning models. As a result, VoCopilot is able to continuously track conversations, record them, transcribe them, and then extract useful insights from them. By utilizing large language models, VoCopilot ensures the user can extract useful insights from recorded interactions without having to learn complex machine learning techniques. In order to protect the privacy of end users, VoCopilot uses a novel wake-up mechanism that only records conversations of end users. Additionally, all the rest of pipeline can be run on a commodity computer (Mac Mini M2). In this work, we show the effectiveness of VoCopilot in real-world environment for two use cases.

Related papers

• Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software [9.881955481813465]
  Large Language Model (LLM)-based in-application assistants, or copilots, can automate software tasks. We investigated two automation paradigms by designing and implementing a fully automated copilot and a semi-automated copilot. GuidedCopilot automates trivial steps while offering step-by-step visual guidance.
  arXiv  Detail & Related papers  (2025-04-22T03:11:10Z)
• HACTS: a Human-As-Copilot Teleoperation System for Robot Learning [47.9126187195398]
  We introduce HACTS (Human-As-Copilot Teleoperation System), a novel system that establishes bilateral, real-time joint synchronization between a robot arm and teleoperation hardware. This simple yet effective feedback mechanism, akin to a steering wheel in autonomous vehicles, enables the human copilot to intervene seamlessly while collecting action-correction data for future learning.
  arXiv  Detail & Related papers  (2025-03-31T13:28:13Z)
• Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultation [96.22329536480976]
  We introduce the construction of a Healthcare Copilot designed for medical consultation. The proposed Healthcare Copilot comprises three main components: 1) the Dialogue component, responsible for effective and safe patient interactions; 2) the Memory component, storing both current conversation data and historical patient information; and 3) the Processing component, summarizing the entire dialogue and generating reports. To evaluate the proposed Healthcare Copilot, we implement an auto-evaluation scheme using ChatGPT for two roles: as a virtual patient engaging in dialogue with the copilot, and as an evaluator to assess the quality of the dialogue.
  arXiv  Detail & Related papers  (2024-02-20T22:26:35Z)
• Demystifying Practices, Challenges and Expected Features of Using GitHub Copilot [3.655281304961642]
  We conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub Discussions. We identified the programming languages, technologies used with Copilot, functions implemented, benefits, limitations, and challenges when using Copilot. Our results suggest that using Copilot is like a double-edged sword, which requires developers to carefully consider various aspects when deciding whether or not to use it.
  arXiv  Detail & Related papers  (2023-09-11T16:39:37Z)
• Bootstrapping Adaptive Human-Machine Interfaces with Offline Reinforcement Learning [82.91837418721182]
  Adaptive interfaces can help users perform sequential decision-making tasks. Recent advances in human-in-the-loop machine learning enable such systems to improve by interacting with users. We propose a reinforcement learning algorithm to train an interface to map raw command signals to actions.
  arXiv  Detail & Related papers  (2023-09-07T16:52:27Z)
• Multi-model fusion for Aerial Vision and Dialog Navigation based on human attention aids [69.98258892165767]
  We present an aerial navigation task for the 2023 ICCV Conversation History. We propose an effective method of fusion training of Human Attention Aided Transformer model (HAA-Transformer) and Human Attention Aided LSTM (HAA-LSTM) models.
  arXiv  Detail & Related papers  (2023-08-27T10:32:52Z)
• A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers [0.797970449705065]
  We propose a novel virtual simulation-pilot engine for speeding up air traffic controller (ATCo) training. The engine receives spoken communications from ATCo trainees, and it performs automatic speech recognition and understanding. To the best of our knowledge, this is the first work fully based on open-source ATC resources and AI tools.
  arXiv  Detail & Related papers  (2023-04-16T17:45:21Z)
• Towards Cooperative Flight Control Using Visual-Attention [61.99121057062421]
  We propose a vision-based air-guardian system to enable parallel autonomy between a pilot and a control system. Our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention.
  arXiv  Detail & Related papers  (2022-12-21T15:31:47Z)
• Speech and Natural Language Processing Technologies for Pseudo-Pilot Simulator [0.5480546613836199]
  This paper describes a simple yet efficient repetition-based modular system for speeding up air-traffic controllers (ATCos) training. E.g., a human pilot is still required in EURO's ESCAPE lite simulator (see https://www.eurocontrol.int/simulator/escape) during ATCo training. This need can be substituted by an automatic system that could act as a pilot.
  arXiv  Detail & Related papers  (2022-12-14T11:34:59Z)
• Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming [28.254978977288868]
  We studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.
  arXiv  Detail & Related papers  (2022-10-25T20:01:15Z)
• Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning [62.83590925557013]
  We learn a set of challenging partially-observed manipulation tasks from visual and audio inputs. Our proposed system learns these tasks by combining offline imitation learning from tele-operated demonstrations and online finetuning. In a set of simulated tasks, we find that our system benefits from using audio, and that by using online interventions we are able to improve the success rate of offline imitation learning by 20%.
  arXiv  Detail & Related papers  (2022-05-30T04:52:58Z)
• Stop Bugging Me! Evading Modern-Day Wiretapping Using Adversarial Perturbations [47.32228513808444]
  Mass surveillance systems for voice over IP (VoIP) conversations pose a great risk to privacy. We present an adversarial-learning-based framework for privacy protection for VoIP conversations.
  arXiv  Detail & Related papers  (2020-10-24T06:56:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.