Reinforcement learning (RL) has been an active research area in AI for many years. Recently there has been growing interest in extending RL to the multiagent domain. From the technical point of view,this has taken the community from the realm of Markov Decision Problems (MDPs) to the realm of game theory,and in particular stochastic (or Markov) games (SGs). The body of work in AI on multi-agent RL is still small,with only a couple of dozen papers on the topic as of the time of writing. This contrasts with the literature on single-agent learning in AI,as well as the literature on learning in game
theory – in both cases one finds hundreds if not thousands of articles,and several books. Despite the small number we still cannot discuss each of these papers. Instead will trace a representative historical path through this literature. We will
concentrate on what might be called the “Bellman heritage” in multi-agent RL
– work that is based on Q-learning [