AI benchmarking just got weirder—and more controversial—as it found an unexpected battleground: the original Pokémon game trilogy.
On April 10, a Twitch livestream featuring Google’s Gemini AI playing Pokémon went viral on X (formerly Twitter). In the stream, Gemini had reached the eerie Lavender Town, while Anthropic’s Claude AI remained stuck in Mount Moon as of February.
“Gemini is literally ahead of Claude atm in Pokémon after reaching Lavender Town,” wrote X user Jush (@Jush21e8), highlighting the real-time gameplay progress. The post racked up thousands of views and sparked debate.
Unfair Advantage? Gemini Used a Custom Minimap
However, users on Reddit quickly pointed out a critical flaw in the comparison: Gemini had help. The developer running the Gemini stream had implemented a custom minimap that allowed the AI to identify in-game elements such as cuttable trees and key tiles—without needing to analyze screenshots frame by frame. This gave Gemini a significant edge in decision-making speed and accuracy.
In contrast, Claude’s Pokémon gameplay appears to rely on raw visual processing, resulting in slower progress through the game world.
AI Benchmarks Are Getting Murky
Although Pokémon isn’t a formal benchmark in AI research, this quirky competition underscores a growing concern in the industry: benchmarks can be easily skewed by custom implementations.
For instance, Anthropic recently shared two different performance scores for its Claude 3.7 Sonnet model on the SWE-bench Verified benchmark, which evaluates code-generation skills:
- 62.3% accuracy without custom tooling
- 70.3% accuracy with a “custom scaffold” designed to support the model
Similarly, Meta fine-tuned its Llama 4 Maverick model to perform exceptionally well on LM Arena, a popular language model evaluation. Yet the standard version of Llama 4 performed far worse—demonstrating how tailoring models for specific benchmarks can inflate results.
The Bigger Picture: Why Benchmark Integrity Matters
AI benchmarks are meant to offer consistent, comparable metrics for model performance. But when developers add non-standard tools or custom enhancements—like Gemini’s minimap or Claude’s scaffolding—it becomes difficult to trust headline results. This could mislead enterprises, developers, and researchers who rely on these scores to choose the best model for their needs.
With each new AI release, the arms race to “win” on benchmarks like MMLU, HellaSwag, and now Pokémon (!) continues. But as models get smarter, so too must our methods of evaluating them.
