Reinforcement Learning with Graph Attention for Routing and Wavelength Assignment with Lightpath Reuse

• URL: http://arxiv.org/abs/2502.14741v2
• Date: Fri, 18 Apr 2025 16:51:53 GMT
• Title: Reinforcement Learning with Graph Attention for Routing and Wavelength Assignment with Lightpath Reuse
• Abstract summary: We investigate reinforcement learning for routing and spectrum assignment on fixed-grid networks with flex-rate transponders.<n>We show that RWA-LR has 6% increased throughput when candidate paths are ordered by number of hops, rather than total length.<n>We train an RL agent for RWA-LR with graph attention networks for the policy and value functions to exploit the graph-structured data.
• Score: 49.1574468325115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many works have investigated reinforcement learning (RL) for routing and spectrum assignment on flex-grid networks but only one work to date has examined RL for fixed-grid with flex-rate transponders, despite production systems using this paradigm. Flex-rate transponders allow existing lightpaths to accommodate new services, a task we term routing and wavelength assignment with lightpath reuse (RWA-LR). We re-examine this problem and present a thorough benchmarking of heuristic algorithms for RWA-LR, which are shown to have 6% increased throughput when candidate paths are ordered by number of hops, rather than total length. We train an RL agent for RWA-LR with graph attention networks for the policy and value functions to exploit the graph-structured data. We provide details of our methodology and open source all of our code for reproduction. We outperform the previous state-of-the-art RL approach by 2.5% (17.4 Tbps mean additional throughput) and the best heuristic by 1.2% (8.5 Tbps mean additional throughput). This marginal gain highlights the difficulty in learning effective RL policies on long horizon resource allocation tasks.

Related papers

• Reversal Q-Learning [57.43956630070019]
  Reversal Q-learning is an off-policy RL algorithm that trains a flow policy based on prior data.<n>We show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
  arXiv  Detail & Related papers  (2026-06-16T05:56:10Z)
• SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling [54.276306194000405]
  We propose SortedRL, an online length-aware scheduling strategy for scaling reinforcement learning.<n>SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates.<n>Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, show that SortedRL reduces RL training bubble ratios by over 50%.
  arXiv  Detail & Related papers  (2026-03-24T16:48:31Z)
• Off-Policy Value-Based Reinforcement Learning for Large Language Models [25.962820072445222]
  ReVal is a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification.<n>On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO.
  arXiv  Detail & Related papers  (2026-03-24T15:55:02Z)
• PromptRL: Prompt Matters in RL for Flow-Based Image Generation [44.81148439118129]
  We present PromptRL, a framework that incorporates language models (LMs) as trainable prompt refinement agents directly within the flow-based reinforcement learning loop.<n>PromptRL achieves state-of-the-art performance across multiple benchmarks, obtaining scores of 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore.
  arXiv  Detail & Related papers  (2026-02-01T18:31:06Z)
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle [53.239242017802056]
  Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM)<n>However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing and Rollout Silencing.<n>We propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition.
  arXiv  Detail & Related papers  (2025-08-07T17:53:47Z)
• Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay [69.67914133280296]
  Reinforcement learning (RL) has become an effective approach for fine-tuning large language models (LLMs)<n>We propose two techniques to improve data efficiency in LLM RL fine-tuning: difficulty-targeted online data selection and rollout replay.<n>Our method reduces RL fine-tuning time by 23% to 62% while reaching the same level of performance as the original GRPO algorithm.
  arXiv  Detail & Related papers  (2025-06-05T17:55:43Z)
• Enhancing Spectrum Efficiency in 6G Satellite Networks: A GAIL-Powered Policy Learning via Asynchronous Federated Inverse Reinforcement Learning [67.95280175998792]
  A novel adversarial imitation learning (GAIL)-powered policy learning approach is proposed for optimizing beamforming, spectrum allocation, and remote user equipment (RUE) association ins. We employ inverse RL (IRL) to automatically learn reward functions without manual tuning. We show that the proposed MA-AL method outperforms traditional RL approaches, achieving a $14.6%$ improvement in convergence and reward value.
  arXiv  Detail & Related papers  (2024-09-27T13:05:02Z)
• Knowledge Graph Reasoning with Self-supervised Reinforcement Learning [30.359557545737747]
  We propose a self-supervised pre-training method to warm up the policy network before the RL training stage. In our supervised learning stage, the agent selects actions based on the policy network and learns from generated labels. We show that our SSRL model meets or exceeds current state-of-the-art results on all Hits@k and mean reciprocal rank (MRR) metrics.
  arXiv  Detail & Related papers  (2024-05-22T13:39:33Z)
• RLHF Workflow: From Reward Modeling to Online RLHF [79.83927049253924]
  We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report. RLHF is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature. We show that supervised fine-tuning (SFT) and iterative RLHF can obtain state-of-the-art performance with fully open-source datasets.
  arXiv  Detail & Related papers  (2024-05-13T15:50:39Z)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL [80.10358123795946]
  We develop a framework for building multi-turn RL algorithms for fine-tuning large language models. Our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks.
  arXiv  Detail & Related papers  (2024-02-29T18:45:56Z)
• RL$^3$: Boosting Meta Reinforcement Learning via RL inside RL$^2$ [12.111848705677142]
  We propose RL$3$, a hybrid approach that incorporates action-values, learned per task via traditional RL, in the inputs to meta-RL.<n>We show that RL$3$ earns greater cumulative reward in the long term compared to RL$2$ while drastically reducing meta-training time and generalizes better to out-of-distribution tasks.
  arXiv  Detail & Related papers  (2023-06-28T04:16:16Z)
• Train Hard, Fight Easy: Robust Meta Reinforcement Learning [78.16589993684698]
  A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. In this work, we define a robust MRL objective with a controlled level. The data inefficiency is addressed via the novel Robust Meta RL algorithm (RoML)
  arXiv  Detail & Related papers  (2023-01-26T14:54:39Z)
• Boosting Offline Reinforcement Learning via Data Rebalancing [104.3767045977716]
  offline reinforcement learning (RL) is challenged by the distributional shift between learning policies and datasets. We propose a simple yet effective method to boost offline RL algorithms based on the observation that resampling a dataset keeps the distribution support unchanged. We dub our method ReD (Return-based Data Rebalance), which can be implemented with less than 10 lines of code change and adds negligible running time.
  arXiv  Detail & Related papers  (2022-10-17T16:34:01Z)
• Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories [37.14064734165109]
  Natural agents can learn from multiple data sources that differ in size, quality, and types of measurements. We study this in the context of offline reinforcement learning (RL) by introducing a new, practically motivated semi-supervised setting.
  arXiv  Detail & Related papers  (2022-10-12T18:22:23Z)
• RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads [4.575381867242508]
  We propose RL-Scope, a cross-stack profiler that scopes low-level CPU/GPU resource usage to high-level algorithmic operations. We demonstrate RL-Scope's utility through in-depth case studies.
  arXiv  Detail & Related papers  (2021-02-08T15:42:48Z)
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem [37.38316954355031]
  We re-examine the challenges posed by distributed reinforcement learning. We show that viewing RL as a dataflow problem leads to highly composable and performant implementations. We propose RLlib Flow, a hybrid actor-dataflow programming model for distributed RL.
  arXiv  Detail & Related papers  (2020-11-25T13:28:16Z)
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement [137.29281352505245]
  We show that hindsight relabeling is inverse RL, an observation that suggests that we can use inverse RL in tandem for RL algorithms to efficiently solve many tasks. Our experiments confirm that relabeling data using inverse RL accelerates learning in general multi-task settings.
  arXiv  Detail & Related papers  (2020-02-25T18:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.