Every time a new AI model tops a leaderboard, we celebrate. Then, quietly, someone leaked that it saw the test answers during training. Or it crumbles the moment the data shifts by six months. We’ve been here before: chess, Go, ImageNet, translation. Each time, we declare “intelligence achieved,” only to watch the goalposts sprint away.
John Hawkins, a data science veteran from the Pingla Institute, isn’t impressed. In his new paper, Improving AGI Evaluation: A Data Science Perspective , he argues that AGI isn’t a high score, it's surviving deployment in the wild. And the tools to prove it already exist in every production ML pipeline.
Forget IQ tests for robots. Hawkins wants to treat AGI like a bank fraud detector: if it can’t generalize beyond its training window, flag uncertainty, and adapt to new patterns without human babysitting, it’s not intelligent, it's a lookup table with delusions.
The Data Science Stress Test: Three Battle-Hardened Protocols
Hawkins borrows from the unglamorous but unbreakable world of model validation in finance, insurance, and recommendation systems. These aren’t theoretical puzzles, they're the checks that decide whether a system gets deployed or deleted.
1. Out-of-Time Testing: Can It Invent Bitcoin in 2007?
Task: Recreate Satoshi Nakamoto’s Bitcoin whitepaper using only research published before 2008.
This isn’t trivia. It’s a temporal firewall. Every paper, code snippet, or forum post after October 31, 2008, is radioactive. If your AGI can synthesize digital cash from hash chains, Merkle trees, and proof-of-work papers without ever seeing the answer, you’re looking at causal reasoning, not copy-paste.
This is standard in the time-series forecasting train on 2005–2007, predict 2009. But applied to research, it exposes memorization vs. understanding. No more “it read the arXiv during pretraining” excuses.
2. Cohort Testing: Can It Teach Quantum Physics in Swahili?
Task: Write a college-level quantum mechanics textbook in Swahili, trained only on English, French, and German physics corpora.
Now flip it: High-agency mode the system scans global literature, detects the Swahili gap autonomously, and fills it.
This is leave-one-group-out cross-validation, a staple in fairness and domain adaptation. It tests knowledge transfer across language, culture, and modality boundaries. If your AGI can’t generalize concepts without native data, it’s not general, it's linguistically privileged.
3. Uncertainty Quantification: When to Say “I Need a Human”
Task: Process 10,000 mock insurance claims using a 200-page rulebook. 5% contain unresolvable rule conflicts.
A real employee doesn’t guess they escalate. An AGI must do the same.
Hawkins proposes an administrative task simulator with:
- Confidence-calibrated outputs (via conformal prediction or ensembles)
- Explicit escalation triggers
- Audit logs of reasoning chains
This isn’t just safety, it's cognitive maturity. Overconfident AIs are dangerous. Calibrated ones are promotable.
Agency: The Missing Slider in AGI Evaluation
Hawkins introduces a deceptively simple scale:
| Level | Human Analogy | AGI Task Example |
| High | Lab director | “Improve renewable energy storage” → finds gaps, designs experiments, writes grants |
| Medium | Postdoc | “Design a battery with >500 Wh/kg” → proposes chemistry, runs simulations |
| Low | Intern | “Run this simulation with given parameters” → executes, reports |
Most benchmarks test Level 0: “Here’s the prompt, here’s the rubric, go.”
True AGI must climb the agency ladder and we need metrics (planning depth, intervention rate, goal decomposition entropy) to track it.
What’s Still Missing (Even in This Framework)
Hawkins’ playbook is brilliant but incomplete:
- No public test suite: Where’s the GitHub repo with timestamped corpora, conflict-rule engines, and scoring scripts?
- No robot support: Digital tasks only. What about AGI that cooks in unfamiliar kitchens (Wozniak’s Coffee Test)?
- No cost model: Curating out-of-time datasets at scale is expensive. Who pays?
- No gaming resistance proof: Can agents exploit the evaluation (e.g., inferring the cutoff date)?
The Bottom Line
AGI isn’t a leaderboard. It’s a deployment log: It’s intelligence.
Hawkins doesn’t want philosophical debates about consciousness. He wants proof of work in the data science sense. And until we run AGI through out-of-time, cross-cohort, uncertainty-calibrated stress tests at scale, we’re not evaluating intelligence.
We’re grading homework.
Conclusion
John Hawkins forces the AGI field to confront an uncomfortable truth: while we crown models for acing curated benchmarks, production data science has long demanded the only metric that matters surviving real-world deployment without crumbling. His out-of-time validation, cohort generalization, uncertainty calibration, and agency ladder aren’t revolutionary; they’re the unglamorous rituals that already separate deployable systems from fragile toys. Until these stress tests replace leaderboards as the default bar for “intelligence,” we’re not evaluating AGI. We’re just running the world’s most expensive memorization contest. Time to stop grading homework and start logging real deployments.
