Position: Vector Prompt Interfaces Should Be Exposed to Enable Customization of Large Language Models

• URL: http://arxiv.org/abs/2603.04292v1
• Date: Wed, 04 Mar 2026 17:08:47 GMT
• Title: Position: Vector Prompt Interfaces Should Be Exposed to Enable Customization of Large Language Models
• Authors: Liangwei Yang, Shiyu Wang, Haolin Chen, Rithesh Murthy, Ming Zhu, Jielin Qiu, Zixiang Chen, Juntao Tan, Jianguo Zhang, Zhiwei Liu, Wenting Zhao, Silvio Savarese, Caiming Xiong, Huan Wang, Shelby Heinecke,
• Abstract summary: We argue that text-only prompting does not constitute a suitable control interface for scalable, stable, and inference-only customization.<n>We discuss why exposing vector prompts need not fundamentally increase model leakage risk under a standard black-box threat model.
• Score: 79.63399445098759
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models (LLMs) transition from research prototypes to real-world systems, customization has emerged as a central bottleneck. While text prompts can already customize LLM behavior, we argue that text-only prompting does not constitute a suitable control interface for scalable, stable, and inference-only customization. This position paper argues that model providers should expose \emph{vector prompt inputs} as part of the public interface for customizing LLMs. We support this position with diagnostic evidence showing that vector prompt tuning continues to improve with increasing supervision whereas text-based prompt optimization saturates early, and that vector prompts exhibit dense, global attention patterns indicative of a distinct control mechanism. We further discuss why inference-only customization is increasingly important under realistic deployment constraints, and why exposing vector prompts need not fundamentally increase model leakage risk under a standard black-box threat model. We conclude with a call to action for the community to rethink prompt interfaces as a core component of LLM customization.

Related papers

• Test-Time Computing for Referring Multimodal Large Language Models [143.49848714354698]
  We propose ControlMLLM++, a novel test-time adaptation framework.<n>It injects learnable visual prompts into frozen multimodal large language models.
  arXiv  Detail & Related papers  (2026-02-23T04:42:10Z)
• Beyond Single Prompts: Synergistic Fusion and Arrangement for VICL [4.215181054941225]
  Vision In-Context Learning (VICL) enables inpainting models to quickly adapt to new visual tasks from only a few prompts.<n>VICL methods suffer from two key issues: (1) selecting only the most similar prompt discards complementary cues from other high-quality prompts; and (2) failing to exploit the structured information implied by different prompt arrangements.<n>We propose an end-to-end VICL framework to overcome these limitations. Firstly, an adaptive Fusion Module aggregates critical patterns and annotations from multiple prompts to form more precise contextual prompts.
  arXiv  Detail & Related papers  (2026-01-15T06:53:59Z)
• Modular Prompt Optimization: Optimizing Structured Prompts with Section-Local Textual Gradients [0.8604557306886812]
  We introduce a schema-based prompt optimization framework that treats prompts as structured objects composed of fixed semantic sections.<n>We evaluate MPO on two reasoning benchmarks, ARC-Challenge and MMLU, using LLaMA-3 8B-Instruct and Mistral-7B-Instruct as solver models.
  arXiv  Detail & Related papers  (2026-01-07T16:20:08Z)
• Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization [75.88719716002014]
  Domain Generalization (DG) seeks to develop a versatile model capable of performing effectively on unseen target domains.<n>Recent advances in pre-trained Visual Foundation Models (VFMs) have demonstrated considerable potential in enhancing the generalization capabilities of deep learning models.<n>We propose addressing this challenge by leveraging the controllable and flexible language prompt of the VFM.
  arXiv  Detail & Related papers  (2025-07-03T03:52:37Z)
• Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models [54.85405423240165]
  We introduce Interactive Reasoning, an interaction design that visualizes chain-of-thought outputs as a hierarchy of topics.<n>We implement interactive reasoning in Hippo, a prototype for AI-assisted decision making in the face of uncertain trade-offs.
  arXiv  Detail & Related papers  (2025-06-30T10:00:43Z)
• Revisiting Prompt Pretraining of Vision-Language Models [13.888505919946578]
  We propose a general framework termed Revisiting Prompt Pretraining (RPP) RPP targets at improving the fitting and generalization ability from two aspects: prompt structure and prompt supervision. We additionally utilize soft labels derived from zero-shot probability predictions provided by a pretrained Contrastive Language Image Pretraining (CLIP) teacher model.
  arXiv  Detail & Related papers  (2024-09-10T02:36:13Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)
• Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs [80.48606583629123]
  PASTA is a method that allows large language models to read text with user-specified emphasis marks. It can substantially enhance an LLM's ability to follow user instructions or integrate new knowledge from user inputs.
  arXiv  Detail & Related papers  (2023-11-03T22:56:43Z)
• Learning to Rewrite Prompts for Personalized Text Generation [27.50476377270294]
  We propose a novel method to automatically revise prompts for personalized text generation. The proposed method takes the initial prompts generated by a state-of-the-art, multistage framework for personalized generation and rewrites a few critical components. In-depth analysis of the rewritten prompts shows that they are not only human readable, but also able to guide manual revision of prompts.
  arXiv  Detail & Related papers  (2023-09-29T21:15:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.