Universal Post-Processing Networks for Joint Optimization of Modules in Task-Oriented Dialogue Systems

• URL: http://arxiv.org/abs/2502.00747v1
• Date: Sun, 02 Feb 2025 10:46:37 GMT
• Title: Universal Post-Processing Networks for Joint Optimization of Modules in Task-Oriented Dialogue Systems
• Abstract summary: We propose a joint optimization method for post-processing the outputs of all modules using universal post-processing networks (UniPPNs)<n>Our RL algorithm, which employs a module-level Markov decision process, enables fine-grained value and advantage estimation for each module, thereby stabilizing joint learning for post-processing the outputs of all modules.
• Score: 3.8443430569753034
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post-processing networks (PPNs) are components that modify the outputs of arbitrary modules in task-oriented dialogue systems and are optimized using reinforcement learning (RL) to improve the overall task completion capability of the system. However, previous PPN-based approaches have been limited to handling only a subset of modules within a system, which poses a significant limitation in improving the system performance. In this study, we propose a joint optimization method for post-processing the outputs of all modules using universal post-processing networks (UniPPNs), which are language-model-based networks that can modify the outputs of arbitrary modules in a system as a sequence-transformation task. Moreover, our RL algorithm, which employs a module-level Markov decision process, enables fine-grained value and advantage estimation for each module, thereby stabilizing joint learning for post-processing the outputs of all modules. Through both simulation-based and human evaluation experiments using the MultiWOZ dataset, we demonstrated that UniPPN outperforms conventional PPNs in the task completion capability of task-oriented dialogue systems.

Related papers

• Interaction-Grounded Learning for Contextual Markov Decision Processes with Personalized Feedback [59.287761696290865]
  We propose a computationally efficient algorithm that achieves a sublinear regret guarantee for contextual episodic Markov Decision Processes (MDPs) with personalized feedback.<n>We demonstrate the effectiveness of our method in learning personalized objectives from multi-turn interactions through experiments on both a synthetic episodic MDP and a real-world user booking dataset.
  arXiv  Detail & Related papers  (2026-02-09T06:29:54Z)
• Multi-module GRPO: Composing Policy Gradients and Prompt Optimization for Language Model Programs [77.22973302887435]
  Group Relative Policy Optimization (GRPO) has proven to be an effective tool for post-training language models (LMs)<n>We present mmGRPO, a simple multi-module of GRPO that groups LM calls by module across rollouts and handles variable-length and interrupted trajectories.<n>We find that mmGRPO, composed with automatic prompt optimization, improves accuracy by 11% on average across classification, many-hop search, and privacy-preserving delegation tasks.
  arXiv  Detail & Related papers  (2025-08-06T17:28:31Z)
• Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models [50.19188692497892]
  Traditional alignment methods often require retraining large pretrained models.<n>We propose a novel textitResidual Alignment Model (textitRAM) that formalizes the alignment process as a type of importance sampling.<n>We develop a resampling algorithm with iterative token-level decoding to address the common first-token latency issue in comparable methods.
  arXiv  Detail & Related papers  (2025-05-26T08:53:02Z)
• Optimization of Module Transferability in Single Image Super-Resolution: Universality Assessment and Cycle Residual Blocks [4.937699452538975]
  We introduce the concept of "Universality" and its associated definitions, which extend the traditional notion of "Generalization"<n>We then propose the Universality Assessment Equation (UAE), a metric that quantifies how readily a given module can be transplanted across models.<n>We demonstrate that networks embedded with the proposed plug-and-play modules outperform several state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-06T13:35:59Z)
• Optimizing Model Selection for Compound AI Systems [76.69936664916061]
  We propose an efficient framework for model selection in compound systems. It iteratively selects one module and allocates to it the model with the highest module-wise performance. It confers 5%-70% accuracy gains compared to using the same LLM for all modules.
  arXiv  Detail & Related papers  (2025-02-20T18:36:25Z)
• Who's the MVP? A Game-Theoretic Evaluation Benchmark for Modular Attribution in LLM Agents [37.69474034158126]
  CapaBench is an evaluation framework grounded in cooperative game theory's Shapley Value.<n>It measures the marginal impact of individual modules and their interactions within an agent's architecture.<n>CapabilityBench bridges the gap between component-level evaluation and holistic system assessment.
  arXiv  Detail & Related papers  (2025-02-01T18:07:34Z)
• Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models [56.93608812478369]
  We present L2R, a method that isolates the training of new PEFT modules to ensure their task specialization. L2R then learns to compose the learned modules by training a network of routers that leverages a small memory containing examples of previously seen tasks. Our results demonstrate that L2R provides an effective composition of PEFT modules, leading to improved generalization and performance compared to other methods.
  arXiv  Detail & Related papers  (2024-08-16T23:57:29Z)
• Composing Parameter-Efficient Modules with Arithmetic Operations [20.119291936493788]
  We propose to compose parameter-efficient modules through linear arithmetic operations in the weight space. Our approach requires emphno additional training and enables highly flexible module composition. We extend our approach to detoxify Alpaca-LoRA, the latest instruction-tuned large language model based on LLaMA.
  arXiv  Detail & Related papers  (2023-06-26T17:33:21Z)
• Post-processing Networks: Method for Optimizing Pipeline Task-oriented Dialogue Systems using Reinforcement Learning [2.421451893574618]
  We propose a method for optimizing a pipeline system composed of modules implemented with arbitrary methods for dialogue performance. With our method, neural-based components called post-processing networks (PPNs) are installed inside such a system to post-process the output of each module. All PPNs are updated to improve the overall dialogue performance of the system by using reinforcement learning, not necessitating each module to be differentiable.
  arXiv  Detail & Related papers  (2022-07-25T13:22:40Z)
• Proximal Policy Optimization-based Transmit Beamforming and Phase-shift Design in an IRS-aided ISAC System for the THz Band [90.45915557253385]
  IRS-aided integrated sensing and communications (ISAC) system operating in the terahertz (THz) band is proposed to maximize the system capacity. Transmit beamforming and phase-shift design are transformed into a universal optimization problem with ergodic constraints.
  arXiv  Detail & Related papers  (2022-03-21T09:15:18Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers [84.57980167400513]
  Neural Function Modules (NFM) aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm.
  arXiv  Detail & Related papers  (2020-10-15T20:43:17Z)
• Determinantal Point Process as an alternative to NMS [24.802857965503982]
  A subsetal point process (DPP) inspired alternative to non-maximum suppression (NMS) has become an integral step in all state-of-the-art object detection frameworks. We pose NMS as a subset selection problem and posit that directly incorporating DPP like framework can improve the overall performance of the object detection system.
  arXiv  Detail & Related papers  (2020-08-26T09:06:11Z)
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs [55.31018404591743]
  We leverage ideas of common structure from the HiP-MDP setting to enable robust state abstractions inspired by Block MDPs. We derive instantiations of this new framework for both multi-task reinforcement learning (MTRL) and meta-reinforcement learning (Meta-RL) settings.
  arXiv  Detail & Related papers  (2020-07-14T17:25:27Z)
• Iterative Algorithm Induced Deep-Unfolding Neural Networks: Precoding Design for Multiuser MIMO Systems [59.804810122136345]
  We propose a framework for deep-unfolding, where a general form of iterative algorithm induced deep-unfolding neural network (IAIDNN) is developed. An efficient IAIDNN based on the structure of the classic weighted minimum mean-square error (WMMSE) iterative algorithm is developed. We show that the proposed IAIDNN efficiently achieves the performance of the iterative WMMSE algorithm with reduced computational complexity.
  arXiv  Detail & Related papers  (2020-06-15T02:57:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.