Navigating User Behavior toward Personalized Multimodal Generation

• URL: http://arxiv.org/abs/2606.24196v2
• Date: Wed, 24 Jun 2026 04:54:52 GMT
• Title: Navigating User Behavior toward Personalized Multimodal Generation
• Abstract summary: We study personalized content generation, which turns a user's interaction history into an executable instruction for downstream synthesis.<n>We propose NaviGen, which represents each item with a dual legible coupling a collaborative code and a textual code.<n>Experiments across product, game, and short-video domains show that NaviGen improves personalized image and video generation, strengthens next-item prediction, and yields more specific, relevant, and visually generatable instructions.
• Score: 65.58245647055583
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern AIGC pipelines deliver high-fidelity images and videos but presuppose a well-formed creation instruction, while end users rarely articulate visual details, leaving generators misaligned with user demand. We study personalized content generation, which turns a user's interaction history into an executable instruction for downstream synthesis, and identify two obstacles: behavior must be encoded in a form legible to language reasoning, and the model must acquire instruction-writing skill absent from both pretraining and behavior data. We propose NaviGen, which represents each item with a dual identifier coupling a collaborative code and a textual code as a behavioral substrate and a semantic bridge in one token stream. On this representation, a two-stage SFT+RL pipeline first distills preference reasoning and instruction writing from evolutionarily searched supervision, then aligns generation with user intent through hierarchical and self-consistent rewards. Experiments across product, game, and short-video domains show that NaviGen improves personalized image and video generation, strengthens next-item prediction, and yields more specific, relevant, and visually generatable instructions. Our code is released at: https://github.com/iLearn-Lab/NaviGen.

Related papers

• A Reason-then-Describe Instruction Interpreter for Controllable Video Generation [88.95178842901095]
  We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators.<n>We show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs.
  arXiv  Detail & Related papers  (2025-11-25T17:59:07Z)
• In-Video Instructions: Visual Signals as Generative Control [79.44662698914401]
  We investigate whether capabilities can be harnessed for controllable image-to-video generation by interpreting visual signals embedded within the frames as instructions.<n>In-Video Instruction encodes user guidance directly into the visual domain through elements such as overlaid text, arrows, or trajectories.<n>Experiments on three state-of-the-art generators, including Veo 3.1, Kling 2.5, and Wan 2.2, show that video models can reliably interpret and execute such visually embedded instructions.
  arXiv  Detail & Related papers  (2025-11-24T18:38:45Z)
• Learning Decomposed Contextual Token Representations from Pretrained and Collaborative Signals for Generative Recommendation [17.061613097917217]
  We propose a unified framework that preserves pretrained semantics while enhancing the adaptability of token embeddings.<n>Experiments on three real-world datasets demonstrate that DECOR consistently outperforms state-of-the-art baselines in recommendation performance.
  arXiv  Detail & Related papers  (2025-08-22T18:50:38Z)
• Lana: A Language-Capable Navigator for Instruction Following and Generation [70.76686546473994]
  LANA is a language-capable navigation agent which is able to execute human-written navigation commands and provide route descriptions to humans. We empirically verify that, compared with recent advanced task-specific solutions, LANA attains better performances on both instruction following and route description. In addition, endowed with language generation capability, LANA can explain to humans its behaviors and assist human's wayfinding.
  arXiv  Detail & Related papers  (2023-03-15T07:21:28Z)
• Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation [26.933683814025475]
  We introduce two novel multimodal datasets: the synthetic CLEVR-ATVC dataset (620K) and the manually pictured Fruit-ATVC dataset (50K). These datasets incorporate both visual and text-based inputs and outputs. To facilitate the accountability of multimodal systems in rejecting human requests, similar to language-based ChatGPT conversations, we introduce specific rules as supervisory signals within the datasets.
  arXiv  Detail & Related papers  (2023-03-10T15:35:11Z)
• Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation [172.15808300686584]
  We describe an approach that learns the two tasks simultaneously and exploits their intrinsic correlations to boost the training of each. Our approach improves the performance of various follower models and produces accurate navigation instructions.
  arXiv  Detail & Related papers  (2022-03-30T18:15:26Z)
• GenNI: Human-AI Collaboration for Data-Backed Text Generation [102.08127062293111]
  Table2Text systems generate textual output based on structured data utilizing machine learning. GenNI (Generation Negotiation Interface) is an interactive visual system for high-level human-AI collaboration in producing descriptive text.
  arXiv  Detail & Related papers  (2021-10-19T18:07:07Z)
• Improving Tree-Structured Decoder Training for Code Generation via Mutual Learning [27.080718377956693]
  Code generation aims to automatically generate a piece of code given an input natural language utterance. We first throughly analyze the context modeling difference between neural code generation models with different decodings. We propose to introduce a mutual learning framework to jointly train these models.
  arXiv  Detail & Related papers  (2021-05-31T08:44:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.