Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning

Abstract

The study of behavioral diversity in Multi-Agent Reinforcement Learning (MARL) is a nascent yet promising field. In this context, the present work deals with the question of how to control the diversity of a multi-agent system. With no existing approaches to control diversity to a set value, current solutions focus on blindly promoting it via intrinsic rewards or additional loss functions, effectively changing the learning objective and lacking a principled measure for it. To address this, we introduce Diversity Control (DiCo), a method able to control diversity to an exact value of a given metric by representing policies as the sum of a parameter-shared component and dynamically scaled per-agent components. By applying constraints directly to the policy architecture, DiCo leaves the learning objective unchanged, enabling its applicability to any actor-critic MARL algorithm. We theoretically prove that DiCo achieves the desired diversity, and we provide several experiments, both in cooperative and competitive tasks, that show how DiCo can be employed as a novel paradigm to increase performance and sample efficiency in MARL. Multimedia results are available on the paper’s website.

Matteo Bettini

PhD Candidate

Matteo’s research is focused on studying heterogeneity and resilience in multi-agent and multi-robot systems.

Ryan Kortvelesy

PhD Candidate

Ryan’s work focuses on multi-agent reinforcement learning. He is interested in the credit assignment problem, new graph neural network architectures and explainability (applying symbolic regression to multi-agent systems).

Amanda Prorok

Professor

Amanda’s research focuses on multi-agent and multi-robot systems. Our mission is to find new ways of coordinating artificially intelligent agents (e.g., robots, vehicles, machines) to achieve common goals in shared physical and virtual spaces.