One Researcher Just Beat GPT-3.5 on a Hard AI Benchmark for 67 Cents

A small transformer trained in 1.5 hours on a single consumer GPU scored 44% on ARC-AGI, raising uncomfortable questions about why well-funded labs haven't cracked this six-year-old, million-dollar challenge.

A single researcher just accomplished what million-dollar AI labs have failed to do in six years: meaningfully crack ARC-AGI, an intentionally brutal benchmark designed to be nearly impossible for AI. The result? 44% accuracy on ARC-AGI-1, beating GPT-3.5, trained in 1.5 hours on a consumer GPU for $0.67. Watch the full breakdown or listen to the podcast episode.

The setup is almost comically modest. One RTX 5090 GPU. No massive data center. No nine-figure training budget. Just a small transformer, trained from scratch, open source, and ready for anyone to inspect. This is the third in a series of posts on ARC-AGI from the same researcher—faster, cheaper, and better than the previous version, which itself sparked public debates between heavyweight researchers like Lucas Beyer, Jeremy Howard, and Rohan Anil over whether such results were even legitimate.

Why This Matters Beyond Bragging Rights

The uncomfortable question hanging over this result is simple: Why didn't anyone else figure this out first?

The researcher's own diagnosis is blunt. Modern AI progress is brute force—bigger models, more data, more compute, more money. But ARC-AGI flips that entirely. It's a metalearning benchmark with only about a thousand puzzles total, each using a different rule. It requires almost no prior knowledge. It's still far from saturated, especially with real data efficiency (setting aside LLMs that just throw synthetic data at the problem). And it doesn't need cutting-edge architecture—just careful representation design and optimization choices.

The Breakthrough: Output-Only Training

The jump from 40% to 44% came from several stacked improvements: SwiGLU instead of GELU, RMSNorm instead of LayerNorm, scaling to 8 layers, and better data diversity. But the single change that surprised even the researcher was stopping training on input tokens and training only on output tokens. That alone pushed the score up four percentage points.

The catch? The researcher doesn't fully know why. His best guess is finite model capacity. And here's the weirder part: after the change, test loss actually got worse, yet the model became more stable and scored higher. This is a warning flag for anyone optimizing sample efficiency: chasing lower validation loss doesn't guarantee a better model. Sometimes it actively misleads you.

What the Ablations Reveal

The ablation studies paint a clear picture of what actually drives performance. Stripping 3D RoPE positional embeddings down to 1D drops the score to 24%. Remove per-task embeddings entirely—same collapse to 24%. Train the way competitors like CompressARC do it (from scratch on each task, unsupervised) and it falls to 18%. Make that supervised and it actually gets worse, bottoming out at 15%.

The researcher also discovered that swapping from vanilla Muon to NorMuon eliminated a learning-rate stalling problem that AdamW couldn't solve without manual intervention—a small technical win that matters more than you'd think when you're optimizing for sample efficiency.

The Real Kicker

The researcher admits he didn't expect a plain transformer to reach 45% at all. He thought it would require entirely new ideas. He definitely didn't expect to get there at this cost. And now he's confident that 65% is reachable using the standard transformer framework, no exotic new architecture required.

Critics have claimed the work amounts to "training on test," but the researcher directly refutes this: the actual test labels were never touched. Transductive reasoning on evaluation puzzles is both the intended design of ARC as a metalearning benchmark and a well-established technique studied since Vapnik.

The takeaway: the next frontier in AI research may not be scale. It may be paying attention.