Exploring Mobile Touch Interaction with Large Language Models

• URL: http://arxiv.org/abs/2502.07629v1
• Date: Tue, 11 Feb 2025 15:17:00 GMT
• Title: Exploring Mobile Touch Interaction with Large Language Models
• Authors: Tim Zindulka, Jannek Sekowski, Florian Lehmann, Daniel Buschek,
• Abstract summary: We propose to control Large Language Models via touch gestures performed directly on the text. Results demonstrate that touch-based control of LLMs is both feasible and user-friendly. This work lays the foundation for further research into gesture-based interaction with LLMs on touch devices.
• Score: 26.599610206222142
• License:
• Abstract: Interacting with Large Language Models (LLMs) for text editing on mobile devices currently requires users to break out of their writing environment and switch to a conversational AI interface. In this paper, we propose to control the LLM via touch gestures performed directly on the text. We first chart a design space that covers fundamental touch input and text transformations. In this space, we then concretely explore two control mappings: spread-to-generate and pinch-to-shorten, with visual feedback loops. We evaluate this concept in a user study (N=14) that compares three feedback designs: no visualisation, text length indicator, and length + word indicator. The results demonstrate that touch-based control of LLMs is both feasible and user-friendly, with the length + word indicator proving most effective for managing text generation. This work lays the foundation for further research into gesture-based interaction with LLMs on touch devices.

Related papers

• Mimir: Improving Video Diffusion Models for Precise Text Understanding [53.72393225042688]
  Text serves as the key control signal in video generation due to its narrative nature. The recent success of large language models (LLMs) showcases the power of decoder-only transformers. This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser.
  arXiv  Detail & Related papers  (2024-12-04T07:26:44Z)
• Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLM [37.640412098917636]
  We introduce a novel interactive furniture layout generation system that extends the functionality of multimodal language models (MLLMs) Within this framework, we present a novel training-free visual mechanism that assists MLLMs in reasoning about plausible layout plans. Experimental results demonstrate that our approach facilitates language-interactive generation and arrangement for diverse and complex 3D furniture.
  arXiv  Detail & Related papers  (2024-07-31T04:49:46Z)
• Training a Vision Language Model as Smartphone Assistant [1.3654846342364308]
  We present a visual language model (VLM) that can fulfill diverse tasks on mobile devices. Our model functions by interacting solely with the user interface (UI) Unlike previous methods, our model operates not only on a single screen image but on vision-language sentences created from sequences of past screenshots.
  arXiv  Detail & Related papers  (2024-04-12T18:28:44Z)
• OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition [79.852642726105]
  We propose a unified paradigm for parsing visually-situated text across diverse scenarios. Specifically, we devise a universal model, called Omni, which can simultaneously handle three typical visually-situated text parsing tasks. In Omni, all tasks share the unified encoder-decoder architecture, the unified objective point-conditioned text generation, and the unified input representation.
  arXiv  Detail & Related papers  (2024-03-28T03:51:14Z)
• Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Dataset [50.09271028495819]
  multimodal research related to touch focuses on visual and tactile modalities. We construct a touch-language-vision dataset named TLV (Touch-Language-Vision) by human-machine cascade collaboration.
  arXiv  Detail & Related papers  (2024-03-14T19:01:54Z)
• Using Large Language Models to Accelerate Communication for Users with Severe Motor Impairments [17.715162857028595]
  We present SpeakFaster, consisting of large language models (LLMs) and a co-designed user interface for text entry in a highly-abbreviated form. Pilot study with 19 non-AAC participants typing on a mobile device by hand demonstrated gains in motor savings in line with the offline simulation. Lab and field testing on two eye-gaze typing users with amyotrophic lateral sclerosis (ALS) demonstrated text-entry rates 29-60% faster than traditional baselines.
  arXiv  Detail & Related papers  (2023-12-03T23:12:49Z)
• BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs [101.50522135049198]
  BuboGPT is a multi-modal LLM with visual grounding that can perform cross-modal interaction between vision, audio and language. Our contributions are two-fold: 1) An off-the-shelf visual grounding module based on SAM that extracts entities in a sentence and find corresponding masks in the image. Our experiments show that BuboGPT achieves impressive multi-modality understanding and visual grounding abilities during the interaction with human.
  arXiv  Detail & Related papers  (2023-07-17T15:51:47Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Enabling Conversational Interaction with Mobile UI using Large Language Models [15.907868408556885]
  To perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task. This paper investigates the feasibility of enabling versatile conversational interactions with mobile UIs using a single language model.
  arXiv  Detail & Related papers  (2022-09-18T20:58:39Z)
• Vision-Language Pre-Training for Boosting Scene Text Detectors [57.08046351495244]
  We specifically adapt vision-language joint learning for scene text detection. We propose to learn contextualized, joint representations through vision-language pre-training. The pre-trained model is able to produce more informative representations with richer semantics.
  arXiv  Detail & Related papers  (2022-04-29T03:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.