Evolution of Cooperation in Multiagent Systems

Transcription

1 Evolution of Cooperation in Multiagent Systems Brian Mayoh Department of Computer Science, University of Aarhus, Ny Munkegade, bldg. 540, 8000 Aarhus C, Denmark Abstract. Many multiagent systems are simulations of aspects of real world societies. In any such simulation of a real world society one must either evolve or design an appropriate balance between cooperation and competition among the individual agents. The question of when a laborious design approach can be replaced by a selforganising evolutionary approach is illuminated by the investigation of a simple game theoretical model in this paper. Many multiagent systems are simulations of aspects of real world societies. Some are simulations of natural or ecological societies, while others are simulations of economic or communal activity in human societies. One can find examples of both kinds of multiagent systems in our earlier paper ( Mayoh & Junping 1999 ) on the requirements for an agent behaviour language. In any such simulation of a real world society one must either evolve or design an appropriate balance between cooperation and competition among the individual agents. Cooperation is widespread in nature; there are many theories as to why it is beneficial and some theories why it has evolved. For an example of evolution of cooperating robots see (Watson et al.2002). In a prizewinning paper (Jennings 2000 ) N. Jennings advocates designing an organisational level in multiagent systems. The question of when such a laborious design approach can be replaced by a selforganising evolutionary approach is illuminated by the investigation of a simple game theoretical model in this paper. In any multiagent system an agent A may benefit from actions done by the other agents in the system and agent A may do actions that benefit the other agents. Often agent A s willingness to do actions for another agent depends only its previous interactions with that agent, so one can separate the multiagent cooperation policy problem to the two agent cooperation policy problem. Thus our model is the game of repeated interactions between two individuals, known as the reciprocal altruism game. Usually the payoff matrix in the repeated games does not change, but some have followed May s suggestion (May, 1987) and allowed the game players to vary their investment in the repeated games. This much widens the players choice of strategies and in the tournaments described in (Roberts & Sherratt, 1999) the RTS, Raise The Stakes, strategy does very well. However Killingback and Doebeli (Killingback & M.H. Shafazand and A M. Tjoa (Eds.): EurAsia-ICT 2002, LNCS 2510, pp , Springer-Verlag Berlin Heidelberg 2002

2 702 B. Mayoh Doebeli, 1999) claim that RTS inevitably evolves into defection, so strategies like RTS are not the reason why reciprocal altruism has evolved. The tournaments described in this paper show that there are strategies that lead to reciprocal altruism even faster than RTS. This complements the results in (Killingback et al., 1999) that altruism arises in spatially distributed ecosystems even when individuals cannot recognise each other so repeated games are inappropriate. For an example of evolution of cooperating robots using RTS and other strategies see (Birk & Wiernik.2002). The main innovation in this paper is that we allow the two game players to have very different cost-benefit ratios. All previous studies assume identical cost-benefit ratios, but this assumption is not realistic for multiagent systems. 1 Reciprocal Altruism Game In the reciprocal altruism game there are R rounds between Leader and Follower. In round i each player benefits from an altruistic payment of the other player and the payoffs are P L (i) = k L x pay F (i) - pay L (i) P F (i) = k F x pay L (i) - pay F (i) where k F and k L are the players altruistic cost-benefit constants. The strategies of the players determine the amounts pay L and pay F ; these amounts cannot be negative but they can be zero- corresponding to deception in the much studied Prisoners Dilemma game. Our game reduces to that in (Roberts & Sherratt, 1999) when the altruistic benefit is real and the same for all players: 1 < k F =k L.The total payoffs for the players are: Reward L = k L x Cost F - Cost L Reward F = k F x Cost L Cost F where Cost L = S pay L (i) and Cost F = S pay F (i). Altruism is rewarding for both players if k F > Cost F /Cost L > 1/k L, so it will be interesting to see if altruistic strategies evolve when k F x k L > 1 but k F or k L is under 1. Example: Imagine two agents alternatively delegating tasks to one another. Cost L measures the resources used by the Leader agent in doing tasks for the Follower agent. The reciprocal altruism game reduces to the much studied Iterative Prisoners Dilemma when k F = k L = k > 1 and all pay L (i) and pay F (i) are either 0 or 1. The payoff to two cooperators is R = k-1, the payoff to two deceivers is P = 0, the traitor payoff is T = k and the sucker payoff is S = -1. Clearly the crucial inequalities T > R > P > S and

3 Evolution of Cooperation in Multiagent Systems 703 R >(S + T)/2 are satisfied. Our distinction between Leaders and Followers only matters if a follower strategy is allowed to use pay L (i) in determining pay F (i). Estimating partner quality seems to be biologically realistic and it is incorporated in our new strategies for the reciprocal altruism game in the form: l = kown / kopponent. 2 Strategies in the Reciprocal Altruism Game The strategy of a game playing agent can be considered as its personality (Gmytrasiewicz& Lisetti.2002), so the strategy names indicate the personality and the labels indicate the emotions of the agent.the strategies we consider are NA) Non-altruism: pay = 0 always GGG) Give-as-good-as-you-get: (1) then a else opponent s last payment SC) Short-changer: (1) then 1 else opponent s last payment - 1 RTS) Raise-the stakes: (1) then a else if opponent undercut then sad: opponent s last payment else if opponent matched then own last payment + b else happy; own last payment + 2 b OSC) Occasional-short-changer: chance determines either RTS or RTS -1 OC) Occasional-cheat: chance determines either RTS or 0 AWD) Anything-will-do: pay = a always AON) All-or-nothing if opponent undercut then 0 else a RTG) Generous raise-the stakes: (1) then a else if opponent undercut then sad: opponent s last payment else if opponent matched then dubious: own last payment + b else happy; opponent last payment + 2 b OSG) Generous occasional-short-changer: chance determines either RTG or RTG -1 OCG) Generous occasional-cheat: chance determines either RTG or 0 RTL) Flexible Mean raise-the stakes: l = kown / kopponent (1) then a else if own last payment > l opponent last payment then sad: opponent s last payment else if own last payment = l opponent last payment

4 704 B. Mayoh OSL) OCL) then dubious: own last payment + b else happy; own last payment + 2 b Flexible raise-the stakes: l = kown / kopponent (1) then a else if own last payment > l opponent last payment then sad: l opponent s last payment else if own last payment = l opponent last payment then dubious: own last payment + b else happy; own last payment + 2 b Flexible Generous raise-the stakes: l = kown / kopponent (1) then a else if own last payment > l opponent last payment then sad: own last payment else if own last payment = l opponent last payment then dubious: own last payment + b else happy; own last payment + 2 b The first eight of these strategies are taken from (Roberts & Sherratt, 1999). Some of them are familiar from Iterative Prisoners Dilemma Game; NA is always deceive, AWD is always cooperate and the others are like Tit For Tat. In the robot experiment (Birk & Wiernik, 2002) we find some of our strategies but also Justified snobbism and several others. In our coevolutionary implementation there was a population of 50 followers and a population of 50 leaders. In each generation every follower (leader) plays at least 5 randomly chosen leaders (followers); the average number of opponents is 10. We separated followers and leaders to see if this distinction has any influence on which strategy that evolves. The parameters of the leaders were: strategy type, a,b; the parameters of the followers were:strategy type,b and the number of iteration rounds. We fixed follower s a to 1 as we wanted to see how the number of iterations evolved. 3 Results To run an experiment one must fix the cost-benefit ratios, k F and k L, choose a variety of strategies and run the simulation for a number of generations. During the simulation not only can the distribution of follower and leader strategies be tracked, but one can also plot the fitness and other parameters of individuals in various ways. The results of our first experiments are given in table 1 and illustrated in fig.1. The last row is not a separate run; it gives the averages over the last 10 generations so it checks that the stability of the final strategy distributions and fitnesses in rowl. The table gives no information on the evolution of strategy parameters because the only patterns observed were initial investment a increases but large variation between 2 and 8, increment b keeps wide variation between 0 and 4.

5 Evolution of Cooperation in Multiagent Systems 705 Fig. 1. Experimental Results-error bars shortened and K-fitness made positive The last column of the table shows that the number of iterations does evolve, increasing appreciably from the initial 25. This is surprising as the number of iterations is determined by the follower, who is unlikely to benefit from more iterations when k F is low. It is satisfying that altruism usually evolves when it should, but it is surprising that the specially designed adjustable payment strategies RTL,OSL and OCL were never the most popular follower or leader strategy when more altruistic strategies were available. We shall see why in the next section. In (Bazzan et al., 1999) experiments with the Iterated Prisoner s Dilemma game were interpreted as showing that moral sentiments, being benevolent to build up trust, play a role in altruism. Some of our experiments can also be interpreted like this, the player who benefits most from mutual altruism tolerates shortchanging strategy SC and cheating by the other player. Mathematical investigations of a simple altruism game (Eshel et al. 1999) show that sometimes altruism can be an unbeatable strategy, just as if one was interacting with close kin. 4 Analysis of Altruism Game Experiments The ideal result of a game model is that players strategies evolve to evolutionary stable strategies, but this rarely happens with realistic models. Even with such a

6 706 B. Mayoh Table 1. Experimental Results for each experiment upper row gives follower results, lower row gives leader results classical simple game as Hawk-Dove simulations show that players strategies do not evolve to evolutionary stable strategies (Fogel et al, 1997). However one often has self-organising criticality (Bak, 1996) in that the player s strategies evolve to a critical level where each player has a reasonable fitness. The pattern of strategies played

7 Evolution of Cooperation in Multiagent Systems 707 varies while at this critical level and more or less serious avalanches happen from time to time; during an avalanche one or more players get low fitness and change their strategies appropriately. Sometimes evolution leads to mediocre stable states which are adequate but far from optimal for all players. Rarely does evolution lead to periodic oscillation or random wandering of strategies. There is some biological evidence for oscillation in lizard mating strategies (Sinervo & Lively, 1997), and some altruism studies show cycling from mild altruism to too-generous altruism to cheating to mild altruism again. All of these patterns occur in our experiments but space limitations prevent us presenting the detailed Theory of moves analysis of our experiments here (see publication The evolution of altruism at The evolutionary behaviour of games is complex and depends strongly on the possible strategies available to each player, but analysis and understanding is not impossible. However one should distrust experimental results when they are not repeatable. To check for this the experimental results for 12 runs in the case k F = 0.8, k L = 2 and all 14 strategies are shown in table 2 and figure 2.The case k F = 0.8, k L = 2 was chosen because it gave the most different results in earlier runs when we had varying numbers of strategies ( compare rows D,K,L in table 1or fig.1). Fig. 2. Experimental Results-error bars quartered (except for K and K? ) Figure 2 suggests that our experimental fitnesses are mutually consistent and this is confirmed by the fact that each follower fitness has [-19,65] and each leader fitness has [174,328] within their fitness error bounds. Table 2 shows that leader strategy distributions are mutually consistent, follower strategy distributions are mutually consistent, but follower strategy distributions are very different from leader strategy

8 708 B. Mayoh distributions. Figure 3 shows the crosscorrelation of the followers strategies against the leaders strategies. In figure 3 there are two regions of high positive crosscorrelation: (Mediocre) top corner peak -leader strategies 1-3 (RTG, OSG, OCG) follower strategies (GGG,AWD, AON, NA, SC) (Fair) middle peak follower & leader strategies 7 9 (RTL, OSL, OCL) Both followers and leaders get higher payoffs in the Fair region, but then leaders become greedy, the followers retaliate by not cooperating and both end in the Mediocre region. Evolution selforganises the delicate balance between the Mediocre and the Fair. Fig. 3. Crosscorrelation (z-axis) of follower strategies (y-axis) against leader strategies (x-axis) 5 Conclusions Spatial distribution of interacting individuals encourages the development of altruistic strategies, but it is not a necessary precondition. Our experiments show that cooperation can evolve, even when partners have very different cost-benefit ratios and must make very different investments. Our experiments give surprising insights into the advantages and disadvantages of the strategy families we tried:

9 Evolution of Cooperation in Multiagent Systems 709 Table 2. Experimental Results K10 is the same run as K9 but the results are after 500 generations NA never evolved even when it was best SC often evolved ( benefits more than NA in low kf) Leaders often prefer the greedy strategies to the fair RTL, OSL and OCL so the followers choose shortchanging strategies OC and OCG. They also suggest an explanation of why followers should be more cooperative than leaders, even when followers are not coerced by leaders.

10 710 B. Mayoh Our game results indicate that designers of multiagent systems should not expect to find an optimal autonomous agent strategy for responding to task requests and they should consider allowing individual agents to choose their strategy on the basis of their experience with each of the other agents in the system. References BAK, P. (1996) How Nature works: the science of self-organised criticality. Springer BAZZAN, A.L.C. et al. (1999) Moral sentiments in multi-agent systems, ATAL98 Springer LNAI 1555, pp BIRK, A. & WIERNIK, J. (2002) An N-player prisoner s dilemma in a robotic ecosystem, Robotics and Autonomous Systems 39, pp ESHEL, I. et al. (1999) The emergence of kinship behaviour in structured populations of unrelated individuals. Int.J.Game Theory 28, pp FOGEL, D.B. et al. (1997) On the instability of evolutionary stable strategies. Biosystems 44, pp GMYTRASIEWICZ, P.J. & LISETTI, C.L. (2002) Emotions and personality in agent design and modelling, Intelligent Agents VIII, Springer LNAI 2333 pp JENNINGS, N.R.. (2000)On agent-based software engineering. Artificial Intelligence.117 pp KILLINGBACK, T. & DOEBELI, M. (1998) Raise the stakes evolves into a defector. Nature 400 p. 518 KILLINGBACK, T. et al. (1999) Variable investment, the continuous prisoner s dilemma, and the origin of cooperation. Proc. R. Soc. Lond. B 266 pp MAY, R.M. (1987) More evolution of cooperation. Nature 327 pp MAYOH, B.H. & JUNPING, DU. (1999) How flexible should agent behaviour languages be?. Proc. IAT99,World Scientific Pub. ROBERTS, G. & SHERRATT, T.N. (1998) Development of cooperative relationships through increasing investment. Nature 394 pp ROBERTS, G. & SHERRATT, T.N. (1999) The emergence of quantitatively responsive cooperative trade. J. Theor. Bio. 200 pp SHERRATT, T.N. & ROBERTS, G. (1998) The evolution of generosity and choosiness in cooperative exchanges. J. Theor. Bio. 193 pp SINERVO, B & LIVELY, CM (1997) The rock-paper-scissors game and the evolution of alternative male strategies. Nature 380, pp WATSON, R.A, FILICI, S.G. & POLLACK, J.B. (2002) Embodied Evolution: distributing an evolutionary algorithm in a population of robots. Robotics and Autonomous Systems 39 pp. 1 18