Efficient Utility Function Learning for Multi-Objective Parameter Optimization with Prior Knowledge

• URL: http://arxiv.org/abs/2208.10300v2
• Date: Tue, 25 Apr 2023 11:34:37 GMT
• Title: Efficient Utility Function Learning for Multi-Objective Parameter Optimization with Prior Knowledge
• Authors: Farha A. Khan, J\"org P. Dietrich, Christian Wirth
• Abstract summary: We learn a utility function offline, using expert knowledge by means of preference learning. In contrast to other works, we do not only use (pairwise) result preferences, but also coarse information about the utility function space. We show the sample efficiency and quality gains of the proposed method in 4 domains.
• Score: 0.225596179391365
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The current state-of-the-art in multi-objective optimization assumes either a given utility function, learns a utility function interactively or tries to determine the complete Pareto front, requiring a post elicitation of the preferred result. However, result elicitation in real world problems is often based on implicit and explicit expert knowledge, making it difficult to define a utility function, whereas interactive learning or post elicitation requires repeated and expensive expert involvement. To mitigate this, we learn a utility function offline, using expert knowledge by means of preference learning. In contrast to other works, we do not only use (pairwise) result preferences, but also coarse information about the utility function space. This enables us to improve the utility function estimate, especially when using very few results. Additionally, we model the occurring uncertainties in the utility function learning task and propagate them through the whole optimization chain. Our method to learn a utility function eliminates the need of repeated expert involvement while still leading to high-quality results. We show the sample efficiency and quality gains of the proposed method in 4 domains, especially in cases where the surrogate utility function is not able to exactly capture the true expert utility function. We also show that to obtain good results, it is important to consider the induced uncertainties and analyze the effect of biased samples, which is a common problem in real world domains.

Related papers

• Learning Linear Utility Functions From Pairwise Comparison Queries [35.01228510505625]
  We study learnability of linear utility functions from pairwise comparison queries. We show that in the passive learning setting, linear utilities are efficiently learnable with respect to the first objective. In this case, we show that even the second objective is efficiently learnable, and present algorithms for both the noise-free and noisy query response settings.
  arXiv  Detail & Related papers  (2024-05-04T08:43:45Z)
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning [57.38874587065694]
  We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks.
  arXiv  Detail & Related papers  (2024-01-17T19:31:21Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Learning to Rank for Active Learning via Multi-Task Bilevel Optimization [29.207101107965563]
  We propose a novel approach for active learning, which aims to select batches of unlabeled instances through a learned surrogate model for data acquisition. A key challenge in this approach is developing an acquisition function that generalizes well, as the history of data, which forms part of the utility function's input, grows over time.
  arXiv  Detail & Related papers  (2023-10-25T22:50:09Z)
• Leveraging Prior Knowledge in Reinforcement Learning via Double-Sided Bounds on the Value Function [4.48890356952206]
  We show how an arbitrary approximation for the value function can be used to derive double-sided bounds on the optimal value function of interest. We extend the framework with error analysis for continuous state and action spaces.
  arXiv  Detail & Related papers  (2023-02-19T21:47:24Z)
• Network Utility Maximization with Unknown Utility Functions: A Distributed, Data-Driven Bilevel Optimization Approach [25.47492126908931]
  Existing solutions almost exclusively assume each user utility function is known and concave. This paper seeks to answer the question: how to allocate resources when utility functions are unknown, even to the users? We provide a new solution using a distributed and data-driven bilevel optimization approach.
  arXiv  Detail & Related papers  (2023-01-04T19:50:39Z)
• Towards Sequence Utility Maximization under Utility Occupancy Measure [53.234101208024335]
  In the database, although utility is a flexible criterion for each pattern, it is a more absolute criterion due to neglect of utility sharing. We first define utility occupancy on sequence data and raise the problem of High Utility-Occupancy Sequential Pattern Mining. An algorithm called Sequence Utility Maximization with Utility occupancy measure (SUMU) is proposed.
  arXiv  Detail & Related papers  (2022-12-20T17:28:53Z)
• Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficient [65.08966446962845]
  offline reinforcement learning, which aims at optimizing decision-making strategies with historical data, has been extensively applied in real-life applications. We take a step by considering offline reinforcement learning with differentiable function class approximation (DFA) Most importantly, we show offline differentiable function approximation is provably efficient by analyzing the pessimistic fitted Q-learning algorithm.
  arXiv  Detail & Related papers  (2022-10-03T07:59:42Z)
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data [100.83688818427915]
  We present training objectives that use offline datasets to learn a factored transition model. Our theoretical analysis shows that the learned latent action space can boost the sample-efficiency of downstream imitation learning. To learn the latent action space in practice, we propose TRAIL (Transition-Reparametrized Actions for Imitation Learning), an algorithm that learns an energy-based transition model.
  arXiv  Detail & Related papers  (2021-10-27T21:05:00Z)
• Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration [143.43658264904863]
  We show how iteration under a more standard notion of low inherent Bellman error, typically employed in least-square value-style algorithms, can provide strong PAC guarantees on learning a near optimal value function. We present a computationally tractable algorithm for the reward-free setting and show how it can be used to learn a near optimal policy for any (linear) reward function.
  arXiv  Detail & Related papers  (2020-08-18T04:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.