Q-Cogni: An Integrated Causal Reinforcement Learning Framework
- URL: http://arxiv.org/abs/2302.13240v1
- Date: Sun, 26 Feb 2023 05:50:26 GMT
- Title: Q-Cogni: An Integrated Causal Reinforcement Learning Framework
- Authors: Cris Cunha, Wei Liu, Tim French, Ajmal Mian
- Abstract summary: We present Q-Cogni, an algorithmically integrated causal reinforcement learning framework.
Q-Cogni achieves optimal learning with a pre-learned structural causal model of the environment.
We report results that demonstrate better policies, improved learning efficiency and superior interpretability of the agent's decision making.
- Score: 29.196739858730567
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: We present Q-Cogni, an algorithmically integrated causal reinforcement
learning framework that redesigns Q-Learning with an autonomous causal
structure discovery method to improve the learning process with causal
inference. Q-Cogni achieves optimal learning with a pre-learned structural
causal model of the environment that can be queried during the learning process
to infer cause-and-effect relationships embedded in a state-action space. We
leverage on the sample efficient techniques of reinforcement learning, enable
reasoning about a broader set of policies and bring higher degrees of
interpretability to decisions made by the reinforcement learning agent. We
apply Q-Cogni on the Vehicle Routing Problem (VRP) and compare against
state-of-the-art reinforcement learning algorithms. We report results that
demonstrate better policies, improved learning efficiency and superior
interpretability of the agent's decision making. We also compare this approach
with traditional shortest-path search algorithms and demonstrate the benefits
of our causal reinforcement learning framework to high dimensional problems.
Finally, we apply Q-Cogni to derive optimal routing decisions for taxis in New
York City using the Taxi & Limousine Commission trip record data and compare
with shortest-path search, reporting results that show 85% of the cases with an
equal or better policy derived from Q-Cogni in a real-world domain.
Related papers
- Mimicking Human Intuition: Cognitive Belief-Driven Q-Learning [5.960184723807347]
We propose Cognitive Belief-Driven Q-Learning (CBDQ), which integrates subjective belief modeling into the Q-learning framework.
CBDQ enhances decision-making accuracy by endowing agents with human-like learning and reasoning capabilities.
We evaluate the proposed method on discrete control benchmark tasks in various complicate environments.
arXiv Detail & Related papers (2024-10-02T16:50:29Z) - Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making [51.737762570776006]
LLM-ACTR is a novel neuro-symbolic architecture that provides human-aligned and versatile decision-making.
Our framework extracts and embeds knowledge of ACT-R's internal decision-making process as latent neural representations.
Our experiments on novel Design for Manufacturing tasks show both improved task performance as well as improved grounded decision-making capability.
arXiv Detail & Related papers (2024-08-17T11:49:53Z) - A Study of Situational Reasoning for Traffic Understanding [63.45021731775964]
We devise three novel text-based tasks for situational reasoning in the traffic domain.
We adopt four knowledge-enhanced methods that have shown generalization capability across language reasoning tasks in prior work.
We provide in-depth analyses of model performance on data partitions and examine model predictions categorically.
arXiv Detail & Related papers (2023-06-05T01:01:12Z) - MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion
Control in Real Networks [63.24965775030673]
We propose a novel Reinforcement Learning (RL) approach to design generic Congestion Control (CC) algorithms.
Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return.
We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch.
arXiv Detail & Related papers (2023-02-02T18:27:20Z) - Utilizing Background Knowledge for Robust Reasoning over Traffic
Situations [63.45021731775964]
We focus on a complementary research aspect of Intelligent Transportation: traffic understanding.
We scope our study to text-based methods and datasets given the abundant commonsense knowledge.
We adopt three knowledge-driven approaches for zero-shot QA over traffic situations.
arXiv Detail & Related papers (2022-12-04T09:17:24Z) - Hierarchically Structured Task-Agnostic Continual Learning [0.0]
We take a task-agnostic view of continual learning and develop a hierarchical information-theoretic optimality principle.
We propose a neural network layer, called the Mixture-of-Variational-Experts layer, that alleviates forgetting by creating a set of information processing paths.
Our approach can operate in a task-agnostic way, i.e., it does not require task-specific knowledge, as is the case with many existing continual learning algorithms.
arXiv Detail & Related papers (2022-11-14T19:53:15Z) - Rethinking Learning Dynamics in RL using Adversarial Networks [79.56118674435844]
We present a learning mechanism for reinforcement learning of closely related skills parameterized via a skill embedding space.
The main contribution of our work is to formulate an adversarial training regime for reinforcement learning with the help of entropy-regularized policy gradient formulation.
arXiv Detail & Related papers (2022-01-27T19:51:09Z) - The Statistical Complexity of Interactive Decision Making [126.04974881555094]
We provide a complexity measure, the Decision-Estimation Coefficient, that is proven to be both necessary and sufficient for sample-efficient interactive learning.
A unified algorithm design principle, Estimation-to-Decisions (E2D), transforms any algorithm for supervised estimation into an online algorithm for decision making.
arXiv Detail & Related papers (2021-12-27T02:53:44Z) - Reinforcement Learning for Autonomous Driving with Latent State
Inference and Spatial-Temporal Relationships [46.965260791099986]
We show that explicitly inferring the latent state and encoding spatial-temporal relationships in a reinforcement learning framework can help address this difficulty.
We encode prior knowledge on the latent states of other drivers through a framework that combines the reinforcement learner with a supervised learner.
The proposed framework significantly improves performance in the context of navigating T-intersections compared with state-of-the-art baseline approaches.
arXiv Detail & Related papers (2020-11-09T08:55:12Z) - Model-based Multi-Agent Reinforcement Learning with Cooperative
Prioritized Sweeping [4.5497948012757865]
We present a new model-based reinforcement learning algorithm, Cooperative Prioritized Sweeping.
The algorithm allows for sample-efficient learning on large problems by exploiting a factorization to approximate the value function.
Our method outperforms the state-of-the-art algorithm sparse cooperative Q-learning algorithm, both on the well-known SysAdmin benchmark and randomized environments.
arXiv Detail & Related papers (2020-01-15T19:13:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.