Asynchronous Multi-Agent Reinforcement Learning for Scheduling in Subnetworks

Ashvin Srinivasan; Junshan Zhang; Olav Tirkkonen

doi:10.1109/VTC2025-Spring65109.2025.11174624

Asynchronous Multi-Agent Reinforcement Learning for Scheduling in Subnetworks

Arizona State University
University of California, Davis

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

2 Citations (Scopus)

119 Downloads (Pure)

Abstract

We address radio resource scheduling in a network of
multiple in-X subnetworks providing wireless Ultra-Reliable Low-
Latency Communication (URLLC) service. Each subnetwork is
controlled by an agent responsible for scheduling resources to its
devices. Agents rely solely on interference measurements for information
about other agents, with no explicit coordination. Subnetwork
mobility and fast-fading effects create a non-stationary
environment, adding to the complexity of the scheduling problem.
This scenario is modeled as a multi-agent Markov Decision
Process (MDP). To address the problem, we propose a Multi-
Agent Deep Reinforcement Learning (MADRL) approach under
URLLC constraints, which integrates Long Short-Term Memory
(LSTM) with the Deep Deterministic Policy Gradient (DDPG)
algorithm to manage non-stationarity and high-dimensional action
spaces. We apply an asynchronous update strategy, where
one agent is updating at a time. This reduces learning variability,
resolves policy conflicts, and improves the interpretability of the
MADRL approach. Simulation results demonstrate that the asynchronous
update mechanism outperforms synchronous updates
and baseline methods, achieving superior reliability, resource
utilization, and explainability.

Original language	English
Title of host publication	Proceedings of the IEEE 101st Vehicular Technology Conference
Publisher	IEEE
Number of pages	6
ISBN (Electronic)	979-8-3315-3147-8
DOIs	https://doi.org/10.1109/VTC2025-Spring65109.2025.11174624
Publication status	Published - 2025
MoE publication type	A4 Conference publication
Event	IEEE Vehicular Technology Conference - Oslo, Norway Duration: 17 Jun 2025 → 20 Jun 2025 Conference number: 101

Publication series

Name	IEEE Vehicular Technology Conference
ISSN (Electronic)	2577-2465

Conference

Conference	IEEE Vehicular Technology Conference
Abbreviated title	VTC
Country/Territory	Norway
City	Oslo
Period	17/06/2025 → 20/06/2025

Funding

This work was funded in part by Business Finland under the project "Extreme Machine Type Communications for 6G" and by the Research Council of Finland (grant 345109). We also acknowledge the Aalto Scientific Computing for the resources.

Keywords

6G
Dynamic Spectrum Allocation
Explainable AI
In-X subnetworks
Multi-agent Reinforcement Learning
Non-Stationary MDP
URLLC

Access to Document

10.1109/VTC2025-Spring65109.2025.11174624

Asynchronous_Multi-Agent_Reinforcement_Learning_for_Scheduling_in_SubnetworksAccepted author manuscript, 565 KB

Cite this

@inproceedings{8a786562eb364675b5189a7b76ea5f1c,

title = "Asynchronous Multi-Agent Reinforcement Learning for Scheduling in Subnetworks",

abstract = "We address radio resource scheduling in a network of multiple in-X subnetworks providing wireless Ultra-Reliable Low- Latency Communication (URLLC) service. Each subnetwork is controlled by an agent responsible for scheduling resources to its devices. Agents rely solely on interference measurements for information about other agents, with no explicit coordination. Subnetwork mobility and fast-fading effects create a non-stationary environment, adding to the complexity of the scheduling problem. This scenario is modeled as a multi-agent Markov Decision Process (MDP). To address the problem, we propose a Multi- Agent Deep Reinforcement Learning (MADRL) approach under URLLC constraints, which integrates Long Short-Term Memory (LSTM) with the Deep Deterministic Policy Gradient (DDPG) algorithm to manage non-stationarity and high-dimensional action spaces. We apply an asynchronous update strategy, where one agent is updating at a time. This reduces learning variability, resolves policy conflicts, and improves the interpretability of the MADRL approach. Simulation results demonstrate that the asynchronous update mechanism outperforms synchronous updates and baseline methods, achieving superior reliability, resource utilization, and explainability.",

keywords = "6G, Dynamic Spectrum Allocation, Explainable AI, In-X subnetworks, Multi-agent Reinforcement Learning, Non-Stationary MDP, URLLC",

author = "Ashvin Srinivasan and Junshan Zhang and Olav Tirkkonen",

year = "2025",

doi = "10.1109/VTC2025-Spring65109.2025.11174624",

language = "English",

series = " IEEE Vehicular Technology Conference",

publisher = "IEEE",

booktitle = "Proceedings of the IEEE 101st Vehicular Technology Conference",

address = "United States",

note = "IEEE Vehicular Technology Conference, VTC ; Conference date: 17-06-2025 Through 20-06-2025",

}

Asynchronous Multi-Agent Reinforcement Learning for Scheduling in Subnetworks. / Srinivasan, Ashvin; Zhang, Junshan; Tirkkonen, Olav.
Proceedings of the IEEE 101st Vehicular Technology Conference . IEEE, 2025. ( IEEE Vehicular Technology Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - Asynchronous Multi-Agent Reinforcement Learning for Scheduling in Subnetworks

AU - Srinivasan, Ashvin

AU - Zhang, Junshan

AU - Tirkkonen, Olav

N1 - Conference code: 101

PY - 2025

Y1 - 2025

N2 - We address radio resource scheduling in a network of multiple in-X subnetworks providing wireless Ultra-Reliable Low- Latency Communication (URLLC) service. Each subnetwork is controlled by an agent responsible for scheduling resources to its devices. Agents rely solely on interference measurements for information about other agents, with no explicit coordination. Subnetwork mobility and fast-fading effects create a non-stationary environment, adding to the complexity of the scheduling problem. This scenario is modeled as a multi-agent Markov Decision Process (MDP). To address the problem, we propose a Multi- Agent Deep Reinforcement Learning (MADRL) approach under URLLC constraints, which integrates Long Short-Term Memory (LSTM) with the Deep Deterministic Policy Gradient (DDPG) algorithm to manage non-stationarity and high-dimensional action spaces. We apply an asynchronous update strategy, where one agent is updating at a time. This reduces learning variability, resolves policy conflicts, and improves the interpretability of the MADRL approach. Simulation results demonstrate that the asynchronous update mechanism outperforms synchronous updates and baseline methods, achieving superior reliability, resource utilization, and explainability.

AB - We address radio resource scheduling in a network of multiple in-X subnetworks providing wireless Ultra-Reliable Low- Latency Communication (URLLC) service. Each subnetwork is controlled by an agent responsible for scheduling resources to its devices. Agents rely solely on interference measurements for information about other agents, with no explicit coordination. Subnetwork mobility and fast-fading effects create a non-stationary environment, adding to the complexity of the scheduling problem. This scenario is modeled as a multi-agent Markov Decision Process (MDP). To address the problem, we propose a Multi- Agent Deep Reinforcement Learning (MADRL) approach under URLLC constraints, which integrates Long Short-Term Memory (LSTM) with the Deep Deterministic Policy Gradient (DDPG) algorithm to manage non-stationarity and high-dimensional action spaces. We apply an asynchronous update strategy, where one agent is updating at a time. This reduces learning variability, resolves policy conflicts, and improves the interpretability of the MADRL approach. Simulation results demonstrate that the asynchronous update mechanism outperforms synchronous updates and baseline methods, achieving superior reliability, resource utilization, and explainability.

KW - 6G

KW - Dynamic Spectrum Allocation

KW - Explainable AI

KW - In-X subnetworks

KW - Multi-agent Reinforcement Learning

KW - Non-Stationary MDP

KW - URLLC

UR - https://www.scopus.com/pages/publications/105019057890

U2 - 10.1109/VTC2025-Spring65109.2025.11174624

DO - 10.1109/VTC2025-Spring65109.2025.11174624

M3 - Conference article in proceedings

T3 - IEEE Vehicular Technology Conference

BT - Proceedings of the IEEE 101st Vehicular Technology Conference

PB - IEEE

T2 - IEEE Vehicular Technology Conference

Y2 - 17 June 2025 through 20 June 2025

ER -

Asynchronous Multi-Agent Reinforcement Learning for Scheduling in Subnetworks

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

6G-eMTC: Extreme Machine Type Communications for 6G

RILREW: Reinforcement Learning for Real-time Wireless Scheduling and Edge Caching: Theory and Algorithm Design

Cite this

Asynchronous Multi-Agent Reinforcement Learning for Scheduling in Subnetworks

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Projects

6G-eMTC: Extreme Machine Type Communications for 6G

RILREW: Reinforcement Learning for Real-time Wireless Scheduling and Edge Caching: Theory and Algorithm Design

Cite this