Global Convergence for Multi-agent Reinforcement Learning in Unreliable Communication Networks

Abstract

Multi-agent reinforcement learning (MARL) has been widely adopted in embodied artificial intelligence (embodied AI) systems, such as cooperative robotics, autonomous swarms, and wireless-enabled intelligent agents, where decision-making relies on local perception, physical interaction, and limited communication. However, most existing MARL algorithms require centralized training with global state-action information, which is often infeasible in real-world embodied settings due to communication unreliability and execution constraints. In this paper, we propose a distributed and communication-efficient MARL algorithm for embodied multi-agent systems. Each agent employs an approximated policy gradient using only its own action and locally available state and reward information. To further reduce communication and computation costs, we adopt a linear function approximation based on κ-hop neighbor states, naturally matching the local perception and interaction range of embodied agents. We prove global convergence with bounded errors induced by unreliable communication. Simulation results show that our method closely approaches centralized performance while achieving up to 99.3% runtime reduction, and outperforms Soft Actor-Critic (SAC) with a 76.0% reduction in computation time. The source code of the proposed method is available at: https://github.com/Pengcheng-Dai/DACA.