On July 6, 2026, an article appeared in ITmedia.
"The bottleneck for AI utilization is shifting from model performance to 'evaluation,' 'governance,' and 'cost efficiency,'" says Jonathan Frankle, Chief AI Scientist at Databricks and co-founder of MosaicML.
His argument is simple:
- AI is already smart enough. Just thinking about how to use existing models leaves "decades of work."
- What's needed for better AI implementation is not developing smarter models, but investing in evaluation and governance.
- AI is "not enough at 70% or 90% accuracy"; it needs orders of magnitude more rigorous evaluation.
- And the biggest challenge is translating the human standard of "what is good work" into a checklist. This "is much harder than developing the next giant model and could take more than 10 years."
Reading this, I felt a sharp pang of realization.
Recently, I've been experimenting with measuring AI "usage volume." I have a dashboard that pulls "Databricks" x "Claude Code/Codex/Cowork" usage telemetry via OpenTelemetry into Databricks, updating every morning with "who used which tool, how much, and at what cost."
But the article's point wasn't that. "Was that work good?" — Evaluation of Agent quality was a gaping hole.
I gave the article to Fable and asked, "Think about what we should evaluate"
So, I gave the URL of the article to Claude (Fable 5), which I use regularly, and asked:
Read this article and think about what we should be evaluating.
After examining our internal log infrastructure and dashboard configuration, Fable came up with a design consisting of "three layers of things to evaluate."
Fable's Answer: Three Layers of "What to Evaluate"
Advice below """""""""""""""""""""""""""""""""
Layer 1: Friction Signals (Daily, Everyone, Mechanical Judgment)
There are "signs of poor quality" that can be picked up mechanically from logs without even asking an LLM.
- Suspected Loops: Hitting the same tool more than 5 times in a row.
- Cost Outliers: Melting more than $5 in a single session.
- Block Frequency: Stopping many times waiting for human confirmation.
- Cache Efficiency: Poor reuse of context (i.e., re-reading every time).
- Tool Connection Failures: Continued failure to connect to MCP servers.
Layer 2: Work Quality (Weekly, LLM Judge)
Create a system that passes AI agent work logs (traces) to an LLM Judge to score on 8 axes.
- Goal Achievement / Accuracy / Tool Selection / Fidelity to Evidence / Validity of Steps / Recovery from Failure / Safety / Efficiency
- Pass if the average is 3.5 or higher. However, if "Goal Achievement" or "Safety" is 2 or lower, it's an immediate failure (hard fail) even if everything else is perfect.
Automatically score the most recent sessions every Sunday night so that the "pass rate for the week" appears in the Monday regular review. Only safety hard fails trigger an email alert the next morning.
Layer 3: Making Evaluation Itself a Product
The work the article says "takes 10 years" — "translating the criteria for good work into a checklist" — is exactly what we should be providing to customers as AI consultants and corporate trainers. The dashboard running internally becomes a sales demo as is.
"""""""""""""""""""""""""""""""""
And it was implemented that same day
This is the amazing part of the agent era: all three layers started running that very day.
- A SQL view for Layer 1 was added as a "Quality" tab to the organizational dashboard and published.
- Weekly execution for Layer 2 was scheduled, and safety alerts were integrated into the monitoring system.
- An offering design document for Layer 3 was documented.
All I did was choose the policy, click a few approval buttons, and run the scheduler registration command once.
"Found" on the very first day
And on the first day of operation, there were two immediate discoveries.
First: We were losing badly when measured.
When we scored 8 recent internal agent sessions using the 8-axis Judge, the result was — 1 pass, 7 failures. There was a distance between "making AI do a lot of work" and "AI doing good work" that only became visible once measured.
Second: "Friction that can be fixed if taught" was found.
In one member's Cowork (Claude's agent workspace), the dashboard detected that MCP server connection failures had worsened from 4 the previous day to 12 that day. The plugin authentication had been broken for two days, and they kept using it.
The person probably just thought, "It's acting a bit weird," and kept working. It can be fixed in 5 minutes by speaking up and fixing the authentication. "Friction that can be fixed if taught" accumulates without being reported to anyone — this was exactly what was never visible on the usage dashboard.
Three things I learned by trying it
1. Evaluation is an operation, not a tool
If you just build the scoring mechanism and stop, it's the same as not having it. It's only when you connect it to weekly regular execution and alerts, and the numbers appear in the Monday meeting, that you can say you are "evaluating." The "investment in evaluation" Frankle mentions is probably about operations, not tools.
2. Place a mechanical judgment layer before the LLM Judge
The 8-axis Judge evaluation is powerful, but it costs time and money because it runs an LLM. It was realistic to run deterministic signals like loops, cost outliers, and connection failures daily for the full volume, and use the Judge for weekly sampling.
3. Implementing "90% is not enough" means hard fails
When looking at average scores, safety issues get buried in other points. Only by designing it so that "if safety is 2 or lower, it's a failure even if others are perfect" do we take a step closer to the "orders of magnitude more rigorous evaluation" mentioned in the article. The 1/8 pass rate is painful, but this pain is the starting point for improvement.
Conclusion
"AI is already smart enough."
— That's why the next battle is who scores the work entrusted to smart AI and how.
Frankle said this is a 10-year job.
Things that take 10 years become a differentiator the earlier you start. And starting itself was possible in one day with an AI agent.
At my company (AI Brain Partners), we help build this "AI usage measurement and evaluation" system through Claude Code-specialized corporate training and AI consulting. If you're wondering, "What's actually happening with our AI utilization?" please check the links below.
- AI Agent Training for Corporations: https://ai-agent.camp/ja/lab/corporate-training
- Free Report "How to Introduce AI to Enterprises": https://ai-agent.camp/ja/lab/whitepaper-ai-enterprise
(Original article: ITmedia AI+ "The bottleneck for AI utilization is shifting to evaluation and governance" July 6, 2026)
