Developing a Steem Monsters Bot

Developing a bot is always a trade-off between time invested and time saved in the end. In this post, I'll discuss my experience developing a bot for Steem Monsters.

The whole Steem ecosystem never piqued my interest until this spring. That's when I've gotten into Steemit and the lovely ssg-community. After a month or so I developed my first bot, a curation bot for Steemit. While watching this bot do its thing I noticed some of the Steem dapps. Since I've played a couple of fantasy card games in my youth Steem Monsters piqued my interest pretty quickly. Even though at first I thought it was just another Cryptokitties clone.
First impressions can deceive. The game was surprisingly fun.

Making of a Bot

I've had a couple of lectures on reinforcement learning and general artificial intelligence at University, so I had a rough idea of how to make a gameplay bot. But still, applying all this theory to an actual game is a different story. I finally settled for a Monte Carlo Tree Search (MCTS) as the base of the bot.

The main issue with the MCTS is that you need a simulation of the game which you want to play. The faster the simulation the better. The algorithm selects one team member at a time, after each selection it fills the remaining slots with random team members and simulates a match. By doing this the algorithm scores each partial team and should find the optimal team against an enemy team selection.

Simulation

The simulation was the most coding and trial and error intensive part. Evaluating matches step by step to check whether or not they were both doing the same thing. There are many nuances in the game in turn order,´e.g. edge cases where some monsters have the same speed. In this case, magic goes first, followed by ranged and then melee. Otherwise, the game decides randomly who goes first.
Another nuance which you usually don't encounter in normal gameplay is the fatigue mechanic. After turn 20 one more damage is applied to each monster after each round.

After finishing the first version, the simulation computed ~80% of the matches correctly. I don't think it will get much better than that, because there are many chance mechanics in place. For example, a series of unlucky dodges can turn the tide of a game. Even just one missed poison or a dodge can decide a game. One can work with expectancy values in this case, but then you just get the average outcome of the match. And an unlucky dodge is not the average outcome but an outlier.

For those of you who are interested, I pushed the code onto GitHub where you can check it out.

https://github.com/Bauloewe/SM_Simulation

The Bot

After finishing the first decent Simulation I moved onto the bot. The actual mechanism is pretty straight forward, but the devil's in the detail. It all starts with the team selection which the bot should be evaluated against. Do you just look at the last 5 teams of the enemy? The last 5 winning teams? Do you use the bot's team history as well? Maybe just evaluate against the most successful teams of the season? It's not an easy question to answer. I settled for the last 5 winning teams with the same ruleset of both my bot and the opponent.

But once you go down that route you come across another issue: rewarding the bot. After each step, the Monte Carlo Tree Search evaluates the team through the simulation against the "opponent" team selection. The tricky part is how to reward each win. You could reward each team equally with 1 point for a win, 0 points for a draw and -1 point for a loss. But this leaves out a lot of information about how good the team performed. E.g. is a loss where the team almost wiped out the whole enemy team better than a loss, where the hit points of the enemy are untouched? Should a win which took 25 turns give equal reward than one which occurred only after 5 turns? Many of those things depend on each other and should be used in a good reward function.

After some playing around I came up with this function. It takes most of those things into account.

First Result

The bot has been running with many different settings over the last couple days and I noticed the fewer splinters it can choose from, the better the result usually gets. The dragon splinter adds a lot of confusion, that's why I omitted them for now.

With every splinter available (except dragon) the bot reaches a win rate of 65% until its deck limits it's progress. Then the win rate falls to 50% and it stagnates +-100 points. With my last deck that was around Silver II, with an upgraded deck I'm currently at Gold II. Not too bad for the amount of work.

If the opponent always uses the same team the bot is very good at finding optimal counters. For example this match :

Example match

https://steemmonsters.com?p=battle&id=708e8c4e9cdccf5246a68c049b65c97b7fc254fa&ref=aicu. The enemy uses a pretty strong team with double sneak and a high level legendary. Cerberus with retaliate devastates the double sneak team and tanks the back. The second tank mitigated the damage with dodge and shield from the lord of darkness after the first was defeated.

Discussion

I'm surprised by the performance of such a simple approach. I've talked with other people who actively develop bots for Steem Monsters and I've seen similar numbers from various approaches. There might just be a good portion of luck involved in some matches.

After working on this bot I can definitely say two things about this all:

1. That my bot is still far from perfect.
   There is still room for improvement. I'll work on those approaches in the upcoming weeks. One field which looks promising would be the use of machine learning as a heuristic to guide the team selection towards promising nodes more quickly.
2. To play Steem Monsters in the higher leagues is an expensive hobby!

Some Beta Pack Porn

Afterwards, I was curious if it's worth your money to open 100 Beta packs with two brilliant potions (gold foil and legendary) and I got lucky! One Gold Chromatic Dragon and one Gold Hydra! Gotta keep those two around. Two gold foil legendaries. Although having opened a couple beta packs before, I don't think that's the norm. Even with those two potions.

Closing Word

You can see for yourself how my bot is doing under the account "aicu" on Steem Monsters.

And one more thing: I thought long and hard whether I should make the code for the bot public on GitHub as well. For now, I leave it private, because it would be too easy for someone to build a bot army with that and ruin the game. Even with a win ratio of just 0.5 to 0.75. I'd be interested on your thoughts down below on that topic. Should I make the bots code public as well?