I transformed Pokelike.xyz into a LLM and RL benchmark!

Hey everyone! I'm a data scientist and I've been pretty fascinated by reinforcement learning for a while. A few days ago, my friends showed me Pokelike . Obviously, my first thought was: "wait, I could build an harness around this and make bots play it." So that's what I did. My original idea was to build an environment for training reinforcement learning agents. That's already working, and I've built a few bots to test it out. They are still pretty terrible though lol. If you're interested in trying it yourself, everything is documented in the repo and it should be fairly easy to build your own agent and run it against the game. Then, while talking about the project with a friend, we realized that this could actually be pretty interesting as a benchmark for LLMs too. So I built the harness for that as well. The idea is that an LLM-based bot gets: a system prompt a set of tools some previous turns the current game state a few other configurable inputs Pretty much all of this can be changed. You can write your own strategy in the system prompt, decide how the game state is represented, add or remove tools, and use whatever LLM you want. There are already a few bot implementations in the repo that you can use as a starting point. And honestly, it's much harder than I expected . I've tried a few models already. With my current setup, GLM 5.2 gets to the second map at best and then dies. Opus does roughly the same. So now I'm wondering what's actually holding them back. Is it the strategy? Is the model missing some important information? Is the state representation bad? Are the tools not good enough? That's the part I'd really like to explore. I'd love to see if someone can get significantly better results with smaller models by improving the prompt, the tools, or the way the game state is presented to the model. I'm also running more traditional RL agents alongside the LLMs, so right now both approaches are being tracked on the leaderboard. For me the interesting part is not necessarily building the strongest bot, but seeing what kind of agent actually works well in this environment and how much you can get out of a relatively small model with the right setup. For now the project is being developed by me, a friend, and Claude Code 😅 If anyone wants to try it, build a bot, contribute, report a bug, or has an idea for something that could be interesting to test, I'd love to hear about it. Repo: github.com/pierpierpy/pokelike.xyz.bot One last thing: the bot doesn't use the internet and doesn't interact with the actual online game. Everything runs completely offline. During setup it downloads the game and a good chunk of its assets, and then the simulations run locally.