Component-action deep Q-learning for real-time strategy game AI

Kelly, Richard (2021) Component-action deep Q-learning for real-time strategy game AI. Masters thesis, Memorial University of Newfoundland.

[English] PDF - Accepted Version Available under License - The author retains copyright ownership and moral rights in this thesis. Neither the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author's permission. Download (2MB)

Abstract

Real-time Strategy (RTS) games provide a challenging environment for AI research, due to their large state and action spaces, hidden information, and real-time gameplay. The RTS game StarCraft II has become a new test-bed for deep reinforcement learning (RL) systems using the StarCraft II Learning Environment (SC2LE). Recently the full game of StarCraft II has been approached with a complex multi-agent RL system only possible with extremely large financial investments. In this thesis we will describe existing work in RTS AI and motivate our work adapting the deep Q-learning (DQN) RL algorithm to accommodate the multi-dimensional action-space of the SC2LE. We then present the results of our experiments using custom combat scenarios. First, we compare methods for calculating DQN training loss with action components. Second, we show that policies trained with component-action DQN for five hours perform comparably to scripted policies in smaller scenarios and outperform them in larger scenarios. Third, we explore several ways to transfer policies between scenarios, and show that it is a viable method to reduce training time. We show that policies trained on scenarios with fewer units can be applied to larger scenarios and to scenarios with different unit types with only a small loss in performance.

Actions (login required)

View Item