In April 1965, Soviet news agencies announced the discovery of a supercivilization. The panic all came from the work of one man: Nikolai Kardashev, a soviet astronomer who had been obsessed with the following question: if you did detect a civilization far beyond your own, would you be able to say anything rigorous about it?
He became famous for his solution. Instead of focusing on intelligence, intentionality or behaviour, all very complex and difficult to assess, he created a theory on how civilizations use energy. Thermodynamics is indifferent to intelligence - he argued -.
It all turned out to be a quasar. The measurement was fine; there was just not an alien behind that measurement.
BENCHMARKING IN THE AGE OF SUPERINTELLIGENCE
61 years later and now we have an oddly similar panic: we seem to be on the cusp of creating an intelligence which is about to surpass the smartest human to have ever lived. And yet it is of vital importance that we be able to say something meaningful about them, i.e. we need to benchmark them.
The reason is simple: resource allocation. Regardless of their far superior intelligence, we will still want the best return for our time and money. Nobody uses Stockfish 13 in chess, for deep analysis, when the 2026 model is just better.
So why can't we just use our current benchmarks?
The immediate problem is saturation. A fixed test stops being useful the moment every model scores 100% on it, and that is already happening: benchmarks that were considered hard three years ago are now cleared by cheap models. You can keep writing harder tests, but only up to the hardest test a human can write. Which brings us to the real problem.
Up to now we have mostly measured intelligence using closed tests. When I say "closed" i refer to tests which we know the score of before the test has began (IQ tests, mathematical tests, programming...). We use these type of tests for obvious reasons 1. it is objective. 2. It is relatively quick to assess. 3. Inertia: it's how we usually test humans.
However as soon as you start thinking about applying these tests to smarter AIs the issue seems obvious - these exams are designed for a superior intelligence to rank an inferior one: a teacher and a pupil. This is exactly why we need to start thinking about and designing a new type of assessment. We, the inferior intelligence - the pupil - , will have to evaluate a SI - the teacher - while still guaranteeing objectivity.
This problem, how an inferior intelligence can rigorously assess a superior one, I name the Kardashev problem, and is what I am personally most passionate about (and the reason I have decided to create Kardashevian.com).
FUTURE SUPER-INTELLIGENCE BENCHMARKS
We are helped by one saving principle: checking a candidate answer is radically easier than finding one. A Sudoku takes effort to solve; verifying a filled-in grid takes thirty seconds. And in most cases, an external verifier will help us be as objective as possible as to the quality and intelligence of these highly advanced systems.
4 different external verification systems I see for the future of benchmarking:
- The World Is the Verifier. Open Ended Quests: objectives; not questions and answers. The world grades the objective. The higher or the more optimized, the better.
**-Example: paper trading value investing in the stock market.
**
In this scenario the complexity of the real world is so vast and dynamic (constantly changing and reacting) that it makes the saturation of this challenge practically impossible. I am planning on writing another essay further along on why I believe value investing to be a great test for intelligence; the main gist is in the assumption that a system capable of reliably predicting how the world, including humans and markets will behave, is one of the clearest signs of intelligence.
2. The rules and the game are the verifier. Competition. Chess, again, shows that despite having superior players than humans, we are still able to know that Model X is better than Model Z. The ELO system, which we use in Kardashevian.com also allows us to not just see the winner but be able to keep a statistically significant scoreboard.
-Example: Chess, Go, or any infinitely more complex game that can be created just for this purpose.
**
- Physics and Chemistry are the verifier. Complex Engineering. We limit the test to physics and chemistry and let nature do the grading. **
-Example: design a material with a given property, then synthesise it in an automated lab and test it.
4. Logic is the verifier. Formal Proof. Mathematics written in a language a computer can check.
-Example: Improving a known bound or solving an open conjecture in Lean.
Risks are: the limit of known bounds are human, as well as posed conjectures. Still very hard to saturate.
KARDASHEVIAN.COM
**What we are doing:
**Kardashevian is a first attempt at all four of those ideas, built in my spare time and on a small budget :)
Every arena works the same way: inputs are frozen, the models commit, and then something they cannot see or influence does the grading. None of the tests have a specific "answer".
As expected, markets value investing is the open-ended quest. It's a smaller version of the one mentioned in the essay: they receive trading data from a packet of 300 US stocks, ETFs and crypto pairs. Their objective is to create 15 long-only portfolios: five horizons from 1 month to 2 years, at three risk levels, each with a return forecast. We then track every portfolio daily against the S&P 500. Again, the market does the grading. So far no model has lost any money (on average) and nearly 1 month in 3/5 are beating the S&P500 - albeit the statistical significance of this is still limited.
Sandbox is the competition we've created. Each model writes the controller for a rover in a simulated world with hidden physics (motor imbalance, command lag, sensor noise, wind, ice), watches its run, and tries again, eight times. Its best controller is then scored on ten courses it has never seen. Nobody knows the perfect controller for that world, but we can tell exactly which modelâs is better, and how fast each one learned.
In both arenas the models are rated against each other with the same statistics that chess uses: every pair plays a match in every cell, and a BradleyâTerry fit on the Elo scale turns the results into a rating with a 95% interval. The S&P 500 is a player fixed at 1,500. A model that cannot beat an index fund has a rating to show for it.
Two very clear limitations we currently face:
- First, we are self-funded, so we run the low-cost models from each frontier lab (GPT-6 Luna, Claude Haiku 5.5, DeepSeek Flash, MiMo), not their flagships. The method is the point; the flagships come when the budget does.
- We need time. So 358 portfolios and 6 worlds, with new books every Monday and the first portfolios maturing on 25 October. The intervals are still about ±100 points wide.
Everything is public. Each number comes from a numbered release you can download, and the method page describes exactly how it was computed.
We are working hard on increasing the number and difficulty of these tasks.
If anyone is interested in working together on this project feel free to send me a DM or an email at [email protected]
