Claude isn’t a great Pokémon player, and that’s okay

If Claude Plays Pokémon is supposed to offer a glimpse of AI's future, it's not a very convincing showcase. For the past month and counting, Twitch has watched Anthropic's chatbot struggle to play Pokémon Red. Across multiple runs, Claude has failed to beat the nearly 30 year old game. And yet for David Hershey, the project's lead developer, the showcase has been a success. "I wanted some place where I could understand how Claude handles situations where it needs to work over a very long period of time," Hershey explains to me over a video call. As part of his day job at Anthropic, Hershey works on the go-to-market team where he helps the company's clients create their own agents (more on those in a moment). He first began working on Claude Plays Pokémon as a side project around the time Anthropic released 3.5 Sonnet last June. As you can probably guess from the name, the project was partly inspired by Twitch Plays Pokémon, which debuted in 2014 and saw 1.16 million participate in a crowdsourced attempt to beat Pokémon Red using only the inputs viewers typed into the stream's chatbox. Hershey wasn't the first Anthropic employee to try to mold Claude into a Pokémon League Champion, but the project took on a life of its own right around the time he got involved. In the early days of the project, it was a big deal when Claude managed to leave Red's home and find Professor Oak. "I spent some ungodly number of hours tinkering to get it to make that kind of progress," Hershey tells me. He would update his co-workers on Claude's progress in an internal Slack channel. At that point, most of the company wasn't paying attention, and it wasn't something Anthropic planned to share with the world. However, Hershey has made it a habit to revisit the project with each new major model release from Anthropic, starting with the upgraded version of Claude 3.5 Sonnet last fall and again more recently with 3.7 Sonnet. "It's the way I go to see 'What is this new model?' 'How does it work?' 'What can I learn about it?'" Hershey explains. And with Claude 3.7 Sonnet, the version of Claude playing the game right now, it was the first time "you could squint and see signs of life." Inside Anthropic the hope was that Claude would become better at trying different strategies and adjusting its approach when things didn't go according to plan. With Pokémon Red, the company saw Claude do those things in real-time. "[Claude 3.7 Sonnet] spends less time stuck on assumptions," says Hershey. "You'll still see it make a guess and then spend some number of hours believing that's true and making dumb decisions in the meanwhile, but previous models would kind of go on doing that forever." Antrhopic And you can, quite literally, see Claude develop and run with those assumptions. Each ploddingly slow move in the game is preceded by a paragraph of text output from the AI — "I've encountered a wild ZUBAT while trying to navigate to (24,24). As per my strategy, I should flee from this battle to conserve resources" — followed by one single button press. Then it reassess the game state and does that all over again. If you've been watching Claude fumble through Pokémon Red as a fan of the game, a model that spends "less time stuck on assumptions" appears minor, especially when the chatbot will frequently get stuck in areas like Viridian Forest, sometimes for days, due to the maze-like level design. Nonetheless, it is a milestone for the type of AI system that Claude 3.7 represents. Like a lot of recent frontier AI systems, Claude 3.7 Sonnet is a reasoning model, meaning it's designed to tackle problems by breaking them down into smaller pieces. "A lot of our customers care about how effective Claude is an agent," explains Hershey. For the uninitiated, agents or agentic AIs are systems that are designed to plan and carry out complicated tasks without human supervision. Right now, most people think of AI as a blank chat box waiting to answer a question, but chatbots are only the consumer face of the industry; agentic systems represent an incremental but important step towards the promise of artificial general intelligence. From that perspective, there are a couple of things that make Claude Plays Pokémon interesting. First, there's the surprising fact Hershey delegated a lot of the programming that made the project possible to Anthropic's coding agent including an overlay that allows Claude to make sense of Pokémon Red's game world. Second, and more importantly, Claude was not pretrained to play Pokémon Red. The chatbot knows some fundamentals about the game, such as the name of each gym leader and the order the player must beat them in, but it doesn't have hundreds of years worth of game knowledge like some specialized AI systems. "You can throw a model at a game with no preparation, no guidance and it can learn everything itself," he says. "I aim to be as close to that side as possible." Hershey had to give Claude some help. I already mentioned the over

Apr 9, 2025 - 16:49

Claude isn’t a great Pokémon player, and that’s okay

If Claude Plays Pokémon is supposed to offer a glimpse of AI's future, it's not a very convincing showcase. For the past month and counting, Twitch has watched Anthropic's chatbot struggle to play Pokémon Red. Across multiple runs, Claude has failed to beat the nearly 30 year old game. And yet for David Hershey, the project's lead developer, the showcase has been a success.

"I wanted some place where I could understand how Claude handles situations where it needs to work over a very long period of time," Hershey explains to me over a video call. As part of his day job at Anthropic, Hershey works on the go-to-market team where he helps the company's clients create their own agents (more on those in a moment). He first began working on Claude Plays Pokémon as a side project around the time Anthropic released 3.5 Sonnet last June.

As you can probably guess from the name, the project was partly inspired by Twitch Plays Pokémon, which debuted in 2014 and saw 1.16 million participate in a crowdsourced attempt to beat Pokémon Red using only the inputs viewers typed into the stream's chatbox. Hershey wasn't the first Anthropic employee to try to mold Claude into a Pokémon League Champion, but the project took on a life of its own right around the time he got involved.

In the early days of the project, it was a big deal when Claude managed to leave Red's home and find Professor Oak. "I spent some ungodly number of hours tinkering to get it to make that kind of progress," Hershey tells me. He would update his co-workers on Claude's progress in an internal Slack channel. At that point, most of the company wasn't paying attention, and it wasn't something Anthropic planned to share with the world.

However, Hershey has made it a habit to revisit the project with each new major model release from Anthropic, starting with the upgraded version of Claude 3.5 Sonnet last fall and again more recently with 3.7 Sonnet. "It's the way I go to see 'What is this new model?' 'How does it work?' 'What can I learn about it?'" Hershey explains. And with Claude 3.7 Sonnet, the version of Claude playing the game right now, it was the first time "you could squint and see signs of life."

Inside Anthropic the hope was that Claude would become better at trying different strategies and adjusting its approach when things didn't go according to plan. With Pokémon Red, the company saw Claude do those things in real-time. "[Claude 3.7 Sonnet] spends less time stuck on assumptions," says Hershey. "You'll still see it make a guess and then spend some number of hours believing that's true and making dumb decisions in the meanwhile, but previous models would kind of go on doing that forever."

A chart showing the progress in playing Pokemon Red.

And you can, quite literally, see Claude develop and run with those assumptions. Each ploddingly slow move in the game is preceded by a paragraph of text output from the AI — "I've encountered a wild ZUBAT while trying to navigate to (24,24). As per my strategy, I should flee from this battle to conserve resources" — followed by one single button press. Then it reassess the game state and does that all over again.

If you've been watching Claude fumble through Pokémon Red as a fan of the game, a model that spends "less time stuck on assumptions" appears minor, especially when the chatbot will frequently get stuck in areas like Viridian Forest, sometimes for days, due to the maze-like level design. Nonetheless, it is a milestone for the type of AI system that Claude 3.7 represents.

Like a lot of recent frontier AI systems, Claude 3.7 Sonnet is a reasoning model, meaning it's designed to tackle problems by breaking them down into smaller pieces. "A lot of our customers care about how effective Claude is an agent," explains Hershey. For the uninitiated, agents or agentic AIs are systems that are designed to plan and carry out complicated tasks without human supervision. Right now, most people think of AI as a blank chat box waiting to answer a question, but chatbots are only the consumer face of the industry; agentic systems represent an incremental but important step towards the promise of artificial general intelligence.

From that perspective, there are a couple of things that make Claude Plays Pokémon interesting. First, there's the surprising fact Hershey delegated a lot of the programming that made the project possible to Anthropic's coding agent including an overlay that allows Claude to make sense of Pokémon Red's game world.

Second, and more importantly, Claude was not pretrained to play Pokémon Red. The chatbot knows some fundamentals about the game, such as the name of each gym leader and the order the player must beat them in, but it doesn't have hundreds of years worth of game knowledge like some specialized AI systems. "You can throw a model at a game with no preparation, no guidance and it can learn everything itself," he says. "I aim to be as close to that side as possible."

Hershey had to give Claude some help. I already mentioned the overlay that allows it to interpret Pokémon Red's interface. Pixel art is something all AI systems struggle with, and 3.7 Sonnet is no expectation. As humans, our imagination does a great job of filling in the details suggested by just a few pixels. What’s more, Claude doesn't "see" the way we do.

If you watch it closely, you'll notice each time it moves the player character, it will make a few inputs before reevaluating its position. Between those frames, Claude doesn’t have any sensory input. It can't see Red walking, nor does it "hear" when its inputs cause him to crash into a tree or some other obstacle. Claude's "poor vision" is one of the primary reasons it struggles with the game; in fact, Hershey had to give the chatbot a way to read the game's memory so it was less likely to get confused if it misinterpreted the screen.

If the goal of the project was for Claude to beat Pokémon Red, that would have been easy. Hershey could have programmed a route through the game for the chatbot to follow, but at that point all he would have been testing is how well Claude follows a rigid set of instructions. "Claude is pretty good at that," Hershey says. "I knew that. We all knew that."

Instead, in leaving Claude to its own devices, the new model has shown it's better at planning, coming up with new strategies and ultimately trying something different when its assumptions prove to be wrong. One of the more novel solutions Claude developed during its third run through the game was to intentionally cause all of its Pokémon to faint so that it could escape from Mt. Moon.

Still, Claude could be a lot better at both short- and long-term planning. In the same example I just mentioned, Claude deleted all of its notes on Mt. Moon after respawning at a nearby Pokémon Center, incorrectly believing it had successfully navigated the cave. One of its more promising runs ended after Claude failed to recognize it needed to talk to Bill to progress the game. It got stuck in an endless loop of bad decision making.

"Moving forward, I don't know how useful this will be internally as a benchmark. It's possible that with a small, tiny set of skills, Claude gets a little bit better and beats the game, and then the benchmark is not that interesting," Hershey admits. "It could also be the case that there are things I don't quite understand yet about what's going to make our next model good, and then we'll still be learning a lot more incremental things along the way."

As for what happens next, Hershey says he doesn't have a long-term strategy for Claude Plays Pokémon. "I've just spent so much time — my wife would say too much time — staring at this thing," he says, laughing. I also get the sense Hershey's not quite ready to close the book on the project. "I would imagine whenever a new model comes out, I'll be playing Pokémon with it, and I will probably show the world that too."

Until then, Anthropic, following a recent reset, continues to stream Claude Plays Pokémon on Twitch. The project has been successful enough to inspire an independent developer to program a Gemini Plays Pokémon stream, and if I had to guess, we'll see more imitators before long.This article originally appeared on Engadget at https://www.engadget.com/ai/claude-isnt-a-great-pokemon-player-and-thats-okay-151522448.html?src=rss