From Words to Widgets for Controllable LLM Generation

• URL: http://arxiv.org/abs/2604.10925v2
• Date: Fri, 10 Jul 2026 21:13:12 GMT
• Title: From Words to Widgets for Controllable LLM Generation
• Abstract summary: We propose Malleable Prompting, a new interactive prompting technique for controllable large language models (LLMs) generation.<n>It reifies preference expressions in natural language prompts into GUI widgets that users can configure to steer generation.<n>We show that Malleable Prompting helps participants achieve target preferences more precisely and is perceived as more controllable and transparent than natural language prompting alone.
• Score: 23.17092768490849
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Natural language remains the predominant way people interact with large language models (LLMs). However, users often struggle to precisely express and control subjective preferences (e.g., tone, style, and emphasis) through prompting. We propose Malleable Prompting, a new interactive prompting technique for controllable LLM generation. It reifies preference expressions in natural language prompts into GUI widgets (e.g., sliders, dropdowns, and toggles) that users can directly configure to steer generation, while visualizing each control's influence on the output to support attribution and comparison across iterations. To enable this interaction, we introduce an LLM decoding algorithm that modulates the token probability distribution during generation based on preference expressions and their widget values. Through a user study, we show that Malleable Prompting helps participants achieve target preferences more precisely and is perceived as more controllable and transparent than natural language prompting alone.

Related papers

• Position: Vector Prompt Interfaces Should Be Exposed to Enable Customization of Large Language Models [79.63399445098759]
  We argue that text-only prompting does not constitute a suitable control interface for scalable, stable, and inference-only customization.<n>We discuss why exposing vector prompts need not fundamentally increase model leakage risk under a standard black-box threat model.
  arXiv  Detail & Related papers  (2026-03-04T17:08:47Z)
• Test-Time Computing for Referring Multimodal Large Language Models [143.49848714354698]
  We propose ControlMLLM++, a novel test-time adaptation framework.<n>It injects learnable visual prompts into frozen multimodal large language models.
  arXiv  Detail & Related papers  (2026-02-23T04:42:10Z)
• CIE: Controlling Language Model Text Generations Using Continuous Signals [19.55268309121786]
  We show how an LM can be finetuned to expect a control vector that is interpolated between a "low" and a "high" token embedding.<n>Our method more reliably exerts response-length control than in-context learning methods or fine-tuning methods that represent the control signal as a discrete signal.
  arXiv  Detail & Related papers  (2025-05-19T17:59:58Z)
• Steerable Chatbots: Personalizing LLMs with Preference-Based Activation Steering [4.3537491807568465]
  We leverage activation steering to guide large language models to align with user preferences during inference.<n>In contrast to memory-based personalization methods that require longer user history, steering is extremely lightweight and can be easily controlled by the user.<n>Results demonstrate the effectiveness of preference-based steering for aligning real-world conversations with hidden user preferences.
  arXiv  Detail & Related papers  (2025-05-07T09:10:51Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)
• Let Models Speak Ciphers: Multiagent Debate through Embeddings [84.20336971784495]
  We introduce CIPHER (Communicative Inter-Model Protocol Through Embedding Representation) to address this issue. By deviating from natural language, CIPHER offers an advantage of encoding a broader spectrum of information without any modification to the model weights. This showcases the superiority and robustness of embeddings as an alternative "language" for communication among LLMs.
  arXiv  Detail & Related papers  (2023-10-10T03:06:38Z)
• Language Models as Black-Box Optimizers for Vision-Language Models [62.80817942316398]
  Vision-language models (VLMs) pre-trained on web-scale datasets have demonstrated remarkable capabilities on downstream tasks when fine-tuned with minimal data. We aim to develop a black-box approach to optimize VLMs through natural language prompts.
  arXiv  Detail & Related papers  (2023-09-12T04:03:41Z)
• Guiding Large Language Models via Directional Stimulus Prompting [114.84930073977672]
  We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model to generate an auxiliary directional stimulus prompt for each input instance.
  arXiv  Detail & Related papers  (2023-02-22T17:44:15Z)
• PADL: Language-Directed Physics-Based Character Control [66.517142635815]
  We present PADL, which allows users to issue natural language commands for specifying high-level tasks and low-level skills that a character should perform. We show that our framework can be applied to effectively direct a simulated humanoid character to perform a diverse array of complex motor skills.
  arXiv  Detail & Related papers  (2023-01-31T18:59:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.