Multi-Objective Instruction-Aware Representation Learning in Procedural Content Generation RL

• URL: http://arxiv.org/abs/2508.09193v1
• Date: Fri, 08 Aug 2025 09:41:42 GMT
• Title: Multi-Objective Instruction-Aware Representation Learning in Procedural Content Generation RL
• Abstract summary: We propose MIPCGRL, a multi-objective representation learning method for instructed content generators.<n>The proposed method achieves up to a 13.8% improvement in controllability with multi-objective instructions.
• Score: 4.949352356120632
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in generative modeling emphasize the importance of natural language as a highly expressive and accessible modality for controlling content generation. However, existing instructed reinforcement learning for procedural content generation (IPCGRL) method often struggle to leverage the expressive richness of textual input, especially under complex, multi-objective instructions, leading to limited controllability. To address this problem, we propose \textit{MIPCGRL}, a multi-objective representation learning method for instructed content generators, which incorporates sentence embeddings as conditions. MIPCGRL effectively trains a multi-objective embedding space by incorporating multi-label classification and multi-head regression networks. Experimental results show that the proposed method achieves up to a 13.8\% improvement in controllability with multi-objective instructions. The ability to process complex instructions enables more expressive and flexible content generation.

Related papers

• AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce [16.733243065736023]
  We propose Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning (AFMRL)<n>It defines product fine-grained understanding as an attribute generation task.<n>It leverages the generative power of Multimodal Large Language Models (MLLMs) to extract key attributes from product images and text.
  arXiv  Detail & Related papers  (2026-04-22T03:02:21Z)
• PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation [58.1914505657064]
  This paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the challenge of knowledge interference between class-level semantics and spatial context.<n>Experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
  arXiv  Detail & Related papers  (2026-03-18T09:26:43Z)
• Reconstructing Content via Collaborative Attention to Improve Multimodal Embedding Quality [59.651410243721045]
  CoCoA is a Content reconstruction pre-training paradigm based on Collaborative Attention for multimodal embedding optimization.<n>We introduce an EOS-based reconstruction task, encouraging the model to reconstruct input from the corresponding EOS> embeddings.<n>Experiments on MMEB-V1 demonstrate that CoCoA built upon Qwen2-VL and Qwen2.5-VL significantly improves embedding quality.
  arXiv  Detail & Related papers  (2026-03-02T05:34:45Z)
• Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation [53.18286807225952]
  Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework.<n>We propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method.<n>Our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.
  arXiv  Detail & Related papers  (2026-01-29T08:42:25Z)
• Multi-Aspect Cross-modal Quantization for Generative Recommendation [27.92632297542123]
  We propose Multi-Aspect Cross-modal quantization for generative Recommendation (MACRec)<n>We first introduce cross-modal quantization during the ID learning process, which effectively reduces conflict rates.<n>We also incorporate multi-aspect cross-modal alignments, including the implicit and explicit alignments.
  arXiv  Detail & Related papers  (2025-11-19T04:55:14Z)
• IPCGRL: Language-Instructed Reinforcement Learning for Procedural Level Generation [11.71881275085903]
  IPCGRL is an instruction-based procedural content generation method via reinforcement learning.<n>IPCGRL fine-tunes task-specific embedding representations to effectively compress game-level conditions.
  arXiv  Detail & Related papers  (2025-03-16T04:53:38Z)
• Unified Generative and Discriminative Training for Multi-modal Large Language Models [88.84491005030316]
  Generative training has enabled Vision-Language Models (VLMs) to tackle various complex tasks. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval. This paper proposes a unified approach that integrates the strengths of both paradigms.
  arXiv  Detail & Related papers  (2024-11-01T01:51:31Z)
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training [55.54020926284334]
  Multimodal Large Language Models (MLLMs) have recently received substantial interest, which shows their emerging potential as general-purpose models for various vision-language tasks. Retrieval augmentation techniques have proven to be effective plugins for both LLMs and MLLMs. In this study, we propose multimodal adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training (RA-BLIP), a novel retrieval-augmented framework for various MLLMs.
  arXiv  Detail & Related papers  (2024-10-18T03:45:19Z)
• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception [63.03288425612792]
  We propose bfAnyRef, a general MLLM model that can generate pixel-wise object perceptions and natural language descriptions from multi-modality references. Our model achieves state-of-the-art results across multiple benchmarks, including diverse modality referring segmentation and region-level referring expression generation.
  arXiv  Detail & Related papers  (2024-03-05T13:45:46Z)
• Towards More Unified In-context Visual Understanding [74.55332581979292]
  We present a new ICL framework for visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them.
  arXiv  Detail & Related papers  (2023-12-05T06:02:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.