Constrained Deep Reinforcement Learning for Cost-Efficient and Timely Packet Delivery

Overview

Real-time interactive services such as extended reality need packets delivered before strict per-packet deadlines, and network operators want to do so using as few resources as possible. Stochastic network optimization methods, such as backpressure, give efficient routing and scheduling under average delay constraints. They fall short when each packet has its own deadline.

This project formulates minimum-cost, deadline-constrained network control as a constrained Markov decision process. Each packet is tracked by its remaining lifetime, and packets that expire are dropped. The resulting method, CDRL-NC, uses constrained deep reinforcement learning with a dual subgradient algorithm. It has a multi-agent design: a centralized routing agent and distributed scheduling agents at each node. In network simulations, CDRL-NC keeps timely throughput above the target reliability where baselines fall short, at lower resource cost than throughput-maximizing methods.

The work is a collaboration with Antonia Tulino and Jaime Llorca (NYU), Vincenzo Norman Vitale (University of Naples Federico II) and Hao Feng (Intel Labs). It was supported in part by NYU WIRELESS and by the NSF Resilient & Intelligent NextG Systems (RINGS) program.