Date:
April 07, 2025
Meta's Llama 4 benchmark claims face scrutiny as researchers spot differences between public models and those tested on leaderboards.
Meta made waves with its new Llama 4 models, hyping them up as top-tier AI contenders. But there’s just one catch: some of the benchmarks Meta is flaunting might not be telling the full story.
According to a recent TechCrunch report, the version of Llama 4 Maverick that’s been topping charts, specifically on LM Arena, isn’t the same model developers actually get access to. That version? It’s a custom-tuned, “experimental chat version”, designed to shine in conversational tasks, several AI researchers reported on X.
Benchmark Games?
LM Arena, a crowdsourced platform where human reviewers rate AI model responses, ranked Maverick second overall. But the version Meta submitted was tweaked specifically for that format. It tends to give longer responses, use emojis more liberally, and focus on being more personable—traits that score well with human judges but aren’t necessarily reflective of what devs will experience under the hood.
That’s a big deal because benchmarks like LM Arena are used to compare how models stack up across the board. If one version is specially tuned to do well on a test, but another is what people actually use, it muddies the waters.
An Ongoing Problem
This isn’t just about Meta. It reflects a broader issue in the AI industry: benchmark inflation. As competition heats up, companies are more motivated to squeeze every drop of performance out of their models for headline results—even if it means gaming the test.
"What you see on benchmark leaderboards isn’t always what you get in real-world performance," one AI researcher told TechCrunch. "We need more transparency."
Meta hasn’t responded directly to the discrepancy yet, but the takeaway here is clear: if you’re building on these models, be aware of what version you’re actually using—and take the leaderboard hype with a grain of salt.
Because in the race to dominate AI, even the benchmarks aren’t immune to a little marketing spin.
