Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge

• URL: http://arxiv.org/abs/2407.20506v1
• Date: Tue, 30 Jul 2024 02:51:21 GMT
• Title: Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge
• Authors: Yupei Yang, Biwei Huang, Shikui Tu, Lei Xu,
• Abstract summary: causal exploration is a strategy that leverages the underlying causal knowledge for both data collection and model training. We focus on enhancing the sample efficiency and reliability of the world model learning within the domain of task-agnostic reinforcement learning. We demonstrate that causal exploration aids in learning accurate world models using fewer data and provide theoretical guarantees for its convergence.
• Score: 15.588014017373048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The effectiveness of model training heavily relies on the quality of available training resources. However, budget constraints often impose limitations on data collection efforts. To tackle this challenge, we introduce causal exploration in this paper, a strategy that leverages the underlying causal knowledge for both data collection and model training. We, in particular, focus on enhancing the sample efficiency and reliability of the world model learning within the domain of task-agnostic reinforcement learning. During the exploration phase, the agent actively selects actions expected to yield causal insights most beneficial for world model training. Concurrently, the causal knowledge is acquired and incrementally refined with the ongoing collection of data. We demonstrate that causal exploration aids in learning accurate world models using fewer data and provide theoretical guarantees for its convergence. Empirical experiments, on both synthetic data and real-world applications, further validate the benefits of causal exploration.

Related papers

• Causal Information Prioritization for Efficient Reinforcement Learning [21.74375718642216]
  Current Reinforcement Learning (RL) methods often suffer from sample-inefficiency. Recent causal approaches aim to address this problem, but they lack grounded modeling of reward-guided causal understanding of states and actions. We propose a novel method named Causal Information Prioritization (CIP) that improves sample efficiency by leveraging factored MDPs.
  arXiv  Detail & Related papers  (2025-02-14T11:44:17Z)
• B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners [18.960920426485163]
  Self-improvement has emerged as a primary method for enhancing performance. We identify and propose methods to monitor two pivotal factors in this iterative process. We introduce B-STaR, a Self-Taught Reasoning framework that adjusts configurations across iterations to balance exploration and exploitation.
  arXiv  Detail & Related papers  (2024-12-23T03:58:34Z)
• KBAlign: Efficient Self Adaptation on Specific Knowledge Bases [73.34893326181046]
  Large language models (LLMs) usually rely on retrieval-augmented generation to exploit knowledge materials in an instant manner. We propose KBAlign, an approach designed for efficient adaptation to downstream tasks involving knowledge bases. Our method utilizes iterative training with self-annotated data such as Q&A pairs and revision suggestions, enabling the model to grasp the knowledge content efficiently.
  arXiv  Detail & Related papers  (2024-11-22T08:21:03Z)
• Mamba4KT:An Efficient and Effective Mamba-based Knowledge Tracing Model [8.432717706752937]
  Knowledge tracing enhances student learning by leveraging past performance to predict future performance. Due to the growing amount of data in smart education scenarios, this poses a challenge in terms of time and space consumption for knowledge tracing models. Mamba4KT is the first to explore enhanced efficiency and resource utilization in knowledge tracing.
  arXiv  Detail & Related papers  (2024-05-26T12:26:03Z)
• Collaborative Knowledge Infusion for Low-resource Stance Detection [83.88515573352795]
  Target-related knowledge is often needed to assist stance detection models. We propose a collaborative knowledge infusion approach for low-resource stance detection tasks.
  arXiv  Detail & Related papers  (2024-03-28T08:32:14Z)
• Learning Objective-Specific Active Learning Strategies with Attentive Neural Processes [72.75421975804132]
  Learning Active Learning (LAL) suggests to learn the active learning strategy itself, allowing it to adapt to the given setting. We propose a novel LAL method for classification that exploits symmetry and independence properties of the active learning problem. Our approach is based on learning from a myopic oracle, which gives our model the ability to adapt to non-standard objectives.
  arXiv  Detail & Related papers  (2023-09-11T14:16:37Z)
• Thrust: Adaptively Propels Large Language Models with External Knowledge [58.72867916604562]
  Large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters. The inherent knowledge in PTLMs can be opaque or static, making external knowledge necessary. We propose the instance-level adaptive propulsion of external knowledge (IAPEK), where we only conduct the retrieval when necessary.
  arXiv  Detail & Related papers  (2023-07-19T20:16:46Z)
• GLUECons: A Generic Benchmark for Learning Under Constraints [102.78051169725455]
  In this work, we create a benchmark that is a collection of nine tasks in the domains of natural language processing and computer vision. We model external knowledge as constraints, specify the sources of the constraints for each task, and implement various models that use these constraints.
  arXiv  Detail & Related papers  (2023-02-16T16:45:36Z)
• Evaluation of Induced Expert Knowledge in Causal Structure Learning by NOTEARS [1.5469452301122175]
  We study the impact of expert knowledge on causal relations in the form of additional constraints used in the formulation of the nonparametric NOTEARS model. We found that (i) knowledge that corrects the mistakes of the NOTEARS model can lead to statistically significant improvements, (ii) constraints on active edges have a larger positive impact on causal discovery than inactive edges, and surprisingly, (iii) the induced knowledge does not correct on average more incorrect active and/or inactive edges than expected.
  arXiv  Detail & Related papers  (2023-01-04T20:39:39Z)
• Causal Reinforcement Learning using Observational and Interventional Data [14.856472820492364]
  Learning efficiently a causal model of the environment is a key challenge of model RL agents operating in POMDPs. We consider a scenario where the learning agent has the ability to collect online experiences through direct interactions with the environment. We then ask the following questions: can the online and offline experiences be safely combined for learning a causal model.
  arXiv  Detail & Related papers  (2021-06-28T06:58:20Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.