Methodology for Evaluation, Design, and Safety to "Not Let AI End as Just a Convenient Tool"
If you view Yusuke Narita only as a "scholar who talks a lot about AI," you are missing the essence. His way of facing AI does not fit within the general range of productivity improvement, such as creating sentences with chatbots, summarizing meetings, or writing prompts well. Rather, his characteristic lies in viewing AI as a "device that carries out decision-making" and thinking collectively about how to design it, how to evaluate it, and how to safely implement it in society.
On his official website, he explains his specialty as "designing business and public policy using data, algorithms, and thought," and "developing methods to design social decision-making algorithms in a data-driven way." In his official Yale University profile, the center of his research is the design of decision-making algorithms in policy and business, using a combination of causal inference, machine learning, and structural estimation. In other words, for him, AI is not a standalone app, but the foundation of "intelligence that drives real-world judgments" such as recommendations, advertising, search, and policy allocation.
And finally, just one thing.
The usage introduced in this article—"not letting AI write the answers, but letting it arrange the materials for you to judge"—will return to the original usage tomorrow if you just read and agree. It only becomes meaningful once you actually run it in your own work.
Therefore, I have prepared a free practical kit so that you can drop this way of thinking directly into your own operations.
You can receive the following:
You don't need to participate in free consultations or seminars. You can receive it directly after adding the LINE.
Get it here:
Now, let's go.
1. The Core of Narita-style AI Utilization is "Judging the AI's Judgment" Rather Than "Asking the AI"
Many people use AI as a superior version of a search engine or as an outsourcing destination for writing. Of course, that in itself is effective, but the Narita-style way of facing AI lies beyond that. In his thinking, AI does not just answer questions; it carries out "decision-making" itself, such as which products to recommend, which advertisements to show, and which coupons to distribute. And the important thing is not to leave that judgment unchecked, but to design it so that it can always be scored later.
Fact shown by Narita: AI is not an "answering box" but a "subject that makes judgments"
In a co-authored paper by Narita, it is stated that "algorithms are coming to carry out many of the decision-makings in policy and business." The first utilization technique seen from here is to treat AI not as a "box that returns answers" but as a "subject that makes judgments," and to first create a system to measure the quality of that judgment. If you only use AI to shorten emails, the competitive advantage is small. However, if you create a structure where you entrust decision-making to AI, verify the quality of that judgment with data, and improve it while preventing deterioration, AI becomes not just an efficiency tool but a judgment engine for the business.
Application to individuals and companies
If you replace this way of thinking with individuals or companies, it becomes like this. Before letting AI do something, decide "what judgment is this to improve?" and "how will we measure later if that judgment was good?" For sales, don't just create proposals; set the judgment of which proposal to give to which customer and the verification of that closing rate as a set. For EC, don't just describe products; score the judgment of which product to show to whom, including not only the purchase rate but also inventory efficiency. Narita-style is the idea of designing the "judgment system" rather than the "answer" of the AI.
2. Deciding AI Usage with "Evaluation First"
The most consistent part of Narita's way of thinking is the evaluation-first idea. In his co-authored papers, he points out that while A/B tests are reliable, they take time and money and involve the risk of failure. Therefore, he repeatedly argues that instead of trying it out in production suddenly, you should first estimate "what would have happened if you had done it differently" from past log data.
What is evaluation-first?
Evaluation-first in AI utilization is not "introducing AI because it's popular." First, you decide "how to measure whether this AI's judgment was good" beforehand.
For example, consider the job of AI-fying customer support. Superficially, it's a "job to automatically generate answers," but when broken down with evaluation-first, the design comes first: what is considered a good answer (resolution rate, satisfaction, or response time?), what data will be used to measure it, and how will you notice when it deteriorates?
Application to individuals and companies
Narita-style AI utilization is sharp in this separation. Instead of throwing everything at the AI, you first define "what is a good judgment," prepare a yardstick to measure it, and then move the AI for the first time. Before thinking about what to let the AI do, you question what you want to improve in the first place. This is evaluation-first AI introduction.
3. "Scoring with Past Data Before Going to Production" = The Idea of Counterfactual Evaluation
At the core of Narita's research is a technology called Off-Policy Evaluation (OPE). It's a difficult word, but the content is simple: "scoring measures that haven't been done yet from past log data beforehand."
Why is "suddenly going to production" dangerous?
This way of thinking can be used directly for AI utilization. Many organizations fail because they suddenly run new methods they thought of in production. It's fine if it works, but if it misses, it worsens customer response and loses time and cost.
Application to individuals and companies
If you think in the Narita-style, the order of AI introduction is this. First, if you think of a new prompt or policy, don't put it all into production suddenly. Next, use logs of similar past cases to estimate "what would have happened if it were that new method." Then, only those that have been confirmed not to clearly deteriorate are put into production little by little.
AI is powerful, but if you put it into production without verification, the failure reaches the user in its entirety. Conversely, if you output after scoring with past data first, you can greatly reduce the accident rate. In other words, what is important as a pre-stage of AI utilization is not to try vigorously, but to safely read ahead with history data.
4. "Questioning the Yardstick Itself" = Not Believing in a Single Metric
Indispensable in the Narita-style way of facing AI is skepticism toward the evaluation method itself. In his co-authored papers, there is one to the effect that "which evaluation method is best changes depending on the task, and there is no single winner." Therefore, you should prepare multiple yardsticks and choose the one that fits best for each situation.
Why is a single metric dangerous?
This shows an important pillar of Narita-style AI utilization. That is, not judging the results of AI by "a single number" alone. In business, it's easy to jump at a single metric like the click rate went up or the reaction was good. But is that number really measuring what you want to improve?
Application to individuals and companies
If an individual mimics this, when evaluating the results of AI, always look at it from multiple angles. For example, if you measure a chatbot only by the "resolution rate," even if the resolution rate goes up, if the user feels it's "cold" and leaves, it's actually a failure. So, look at the resolution rate, satisfaction, churn rate, and response time separately.
Narita-style AI utilization questions whether the yardstick is correct before comparing models. AI will optimize toward the yardstick you set. Therefore, if the yardstick is off, the smarter it gets, the more it will run in the wrong direction. Solidifying this first is the Narita-style.
5. Not Postponing "Annoying Real-World Constraints"
Common to Narita's recent research is incorporating real-world troubles into the evaluation system from the beginning, not as an afterthought. The problem of new products and articles continuing to increase, the problem of upper limits on inventory and coupon budgets, and the problem of behavior being different for each user. He takes these constraints into account from the start.
Why AI made with idealism breaks in production
What's important here is that AI utilization does not end with "ideal conditions." Real-world operations always have budgets, man-hours, NG conditions, and user diversity. Even if you create a prompt that works well in an ideal state, it breaks when it hits real-world constraints.
For example, the upper limit of inventory. If you only command the AI to "keep putting out products that seem to have a good reaction," popular products will dry up in an instant, and you won't be able to put out anything for customers who come later. Narita's research takes such supply constraints into account from the start and evaluates by looking at "allocation to future users" rather than just "the reaction at this moment."
Application to individuals and companies
This idea can also be applied to general work. When thinking about AI measures, instead of first building with idealism and adjusting later, incorporate "usable budget," "man-hours that can be spent," "things that must never be done," and "the range of target users" as premises from the start. Narita-style AI utilization does not postpone real-world constraints. The more beautifully an AI is made, the more it will fail in production if constraints are ignored.
6. Treating AI Not as an "Answering Machine" but as a "Deterioration-Avoiding Machine"
What well represents Narita's research is the idea of treating "not getting worse" itself as a result. In his research, there are things that impose a constraint that it does not fall below the currently moving policy with a high probability, and things that loosen the safety brake little by little within a small number of introductions.
The idea of counting "avoiding deterioration" as a result
Here is a big leap in Narita-style AI utilization. Many AI utilizations only look at "how much better it got." But Narita treats "how much deterioration could be avoided" as an equally important result.
Application to individuals and companies
This is also suggestive in the AI utilization of companies. When trying to improve answer quality, instead of suddenly switching everything to a new method, first guarantee that it does not fall below the current method, and then try little by little. Specifically, try the new policy with only 1-5% of the total, and decide a stop line like "stop if it clearly deteriorates" beforehand.
What makes a difference in AI utilization is not just the flashiness of the offense. It's how much you can expand the range of exploration while suppressing the probability of deterioration. In Narita's words, only by designing AI quality improvement and the suppression of runaway, incorrect answers, and bias at the same table at the same time can AI grow safely.
7. Incorporating "Ethics" into Calculations, Not as a Note
A characteristic of Narita's view of AI is the way ethics are handled. While ethics tend to be a "note added at the end" in many fields, in Narita's research, ethics are incorporated into the optimization problem.
Incorporating ethics into optimization
For example, in research on medical experiment design, he points out that conventional methods have ethical problems such as assigning treatments known to be less effective or treatments that participants dislike. Therefore, he incorporates participant preferences and predicted effects into the allocation calculation from the start to try to increase participant satisfaction.
Application to individuals and companies
If you pull this back to AI utilization, it becomes an essential story. It's not "OK if the accuracy is high," but "putting the feelings of the people who use it and the cost of damage into the evaluation from the start."
For example, when putting out new recommendations with AI. If you don't put out new candidates, the system will stagnate, but if you put out too many, you'll miss and have an accident. Narita's research tries to satisfy both this "newness (fair exposure)" and "safety" at the same time. If you only chase efficiency, the burden will go to someone somewhere. Narita-style incorporates the cost of that burden into the calculation from the start. It treats ethics not as a brake but as part of the design.
8. Evaluating the Evaluator Itself = One-Step Meta-Optimization
What's interesting about Narita's research is that before comparing AI models, he places a stage to verify "whether the comparison method (evaluator) itself is correct." In his research, there is a method to automatically choose which evaluation method is good according to the task.
Verifying the "yardstick" before the model
What can be seen from this is that Narita's view of performance optimization is one step above model tuning. Many people compete for "which model is superior," but Narita confirms "whether the yardstick itself for scoring the model is properly correct" before that.
Application to individuals and companies
If a general company mimics this, before comparing AI tools, they question the criteria for comparison. For example, when trying to compare two AIs by "speed of response," they first ask whether speed is the most important thing in this business in the first place. If you compare with the criteria shifted, even if you adopt the one that won, it will actually deteriorate.
AI utilization seems like a battle of "which model to choose," but it's actually a battle of "what criteria to choose by." If you learn from the Narita-style, you should measure the validity of the yardstick before measuring the performance of the model.
9. Knowing That Conventional Methods Break in "Large Choices"
In Narita's research, there is a recognition that conventional evaluation methods break in situations where there are very many choices (actions). He argues that in situations dealing with large choices like recommendations and searches with too many candidates, and language models, evaluation using features and embeddings is necessary rather than simple win-rate comparison.
Why evaluation breaks with a large number of choices
This hits today's LLM utilization directly. Generative AI has a huge number of candidates for output and choices for tools that can be used. In such a situation, if you simply compare "which was better, A or B," the comparison itself becomes unstable.
Application to individuals and companies
If an individual applies this, the more work there is with many AI choices, the more they avoid rough two-choice comparisons. For example, don't immediately decide "which of the 10 prompt candidates is best" with a small sample. When there are many candidates, don't rush the judgment and look carefully under multiple conditions.
Narita-style AI utilization assumes that evaluation becomes more difficult as the number of choices increases. Therefore, in situations with many candidates, they stick to designed evaluation rather than simple comparison.
10. Understanding the Importance of "Reducing Input Friction"
At the root of Narita's research is the idea of continuing to leave data correctly in order to continuously run the judgment system. In the data infrastructure he is involved in, information such as which choice was put out with what probability is recorded so that "why that result occurred" can be fairly scored later.
Continuing to leave a "verifiable state"
AI utilization here is not just efficiency. It is "continuing to create a state that can be verified later without finding it annoying." If verification is annoying, people will stop doing it. That's why a design that lowers the hurdle for recording and evaluation is needed.
Application to individuals and companies
If you drop this idea into daily AI utilization, what's important is to "reduce the friction of verification." Manually scoring the results of AI every time won't last. Therefore, template the prompts you use often, decide the metrics to measure the results in advance, and make it so the results are automatically left. The more you shorten the distance to verification, the more the AI improvement cycle will continue to turn.
Narita-style way of facing AI is ultimately directed toward "humans defining good judgment, AI executing it, the results always being recorded, and being improved again."
11. Using with a Sense of Crisis
In Narita's view of AI, there is a sense of tension about entrusting judgment to algorithms at the same time as paying attention to the possibilities. In a dialogue, he says to the effect that "money" is just a rough one-dimensional representation of what people have done in the past, and if there is more detailed data, it can be replaced by that. He views AI and data as a foundation that can replace the judgment criteria of society itself.
Responsibility becomes ambiguous the more you entrust judgment
This point is also important as an AI utilization technique. It's dangerous to just use AI because it's convenient. The more you entrust judgment to AI, the more ambiguous it becomes whose responsibility that judgment is and by what criteria it was made. Information leakage, misinformation, bias, location of responsibility, and excessive dependence. If you entrust decision-making to AI while ignoring these, you lose long-term trust in exchange for short-term efficiency.
Application to individuals and companies
If you learn from the Narita-style, don't stop out of fear of AI, but design with risks as a premise. Create rules not to put in confidential information. Leave human confirmation for important judgments. Leave AI judgment logs. Decide the scope of responsibility when an incorrect judgment occurs. AI utilization is designing not only the accelerator but also the brake.
12. Repeating "Trying with Limited Introduction"
What is consistent in Narita's research is the idea of starting from a small number of limited introductions rather than a full-scale switch. In his research, the flow of not applying a new policy to the whole suddenly, but first trying it in a part, looking at the results, re-learning, and expanding little by little appears repeatedly.
"Trying small" works better in an era where prototyping is fast
In the AI era, this "trying small" attitude becomes even more important. This is because the cost of prototyping drops dramatically due to AI, but the temptation to run without verification also becomes stronger. Planning documents, code, advertising copy, analysis reports. Things that used to take several days now become a first draft in a few minutes. That's why it's effective to not be grateful for the first draft and to try small and verify.
Application to individuals and companies
In Narita-style AI utilization techniques, AI is not a "magic that puts out a finished product in one shot." Rather, it is a device to increase the number of times you try small and verify. Try with 1-5%. Look at the results. Confirm deterioration. Return. Improve. Try again. People who can safely speed up this cycle will receive the benefits of AI. People who use AI but don't get results are betting too much on a single full-scale introduction.
13. Practical Method for Individuals to Mimic Narita-style AI Utilization
You don't need to have a university research foundation or large-scale data like Narita. If it's just the idea, even an individual can mimic it from today.
5 steps you can do from today
First, before letting AI do something, write in one sentence "what do I want to improve?" Everyone gets lost because they skip this. Next, decide in advance "by what number will I measure that quality?" Third, even if you think of a new prompt, don't use it for everything suddenly; first try it in similar past cases or a part. Fourth, don't judge the results by a single number alone; look at it from multiple angles. Fifth, prepare a system beforehand where you can notice and stop when it deteriorates.
If you continue this flow, AI will change from a mere convenience tool to your own judgment foundation that continues to improve without breaking.
14. If a Company Mimics, Create a "Judgment System" Rather Than "Answer Accuracy"
The biggest point companies should learn from the Narita-style is not to focus only on increasing the accuracy of AI answers. What Narita's research has been refining all along was the system for "making a judgment, scoring it, and safely improving it" rather than the correctness of individual answers.
"Judgment system" rather than model smartness
In many companies, AI introduction stops at a comparison of "which model is smart." But if you think in the Narita-style, what's important is not the smartness of the model, but whether there is a design to evaluate that judgment, prevent deterioration, follow real-world constraints, and incorporate ethics. If you introduce AI while the purpose and KPI are ambiguous, it will end as an internal event using the latest technology.
Culture that companies should have
If a company is serious about using AI, it first needs to define "what is considered a good judgment" as management, organize data to measure it, create a system to detect deterioration, and have a culture of expanding from limited introduction. AI is not a theme only for the information systems department. The "judgment system" is questioned in sales, development, manufacturing, legal, HR, finance, and customer response. In other words, AI utilization is the design of decision-making itself.
15. Pitfalls of Narita-style AI Utilization, and What Should Be Said Honestly
Of course, there is no need to praise the Narita-style as it is. While the attitude of thorough evaluation, verification, and safety reduces accidents, there are also situations where it slows down speed. If you are cautiously scoring everything, there are times when you won't be able to move in situations where you should try quickly. What's important in AI utilization is not to superficially copy the Narita-style, but to incorporate the principles according to your own environment.
To be honest: this part is "unconfirmed"
And there is one more thing I want to write honestly. The "Narita-style method" introduced in this article is not something Narita himself said, "This is how I use AI." It is a "manner" reconstructed with high probability by carefully reading his public materials (site, CV, papers). Personal routines, such as which LLM Narita uses in his daily life and what prompts he types, cannot be confirmed in public information. So, I won't fill that with imagination and will honestly leave it as "unconfirmed."
Principles that can still be incorporated
Beyond that, the principles to incorporate are: thinking with evaluation-first, scoring with past data before production, questioning the yardstick itself, incorporating real-world constraints from the start, treating avoiding deterioration itself as a result, and incorporating ethics into calculations. And not pretending not to see the risks.
Conclusion: Yusuke Narita's AI Utilization Technique is "Growing a Good Judgment System Without Breaking It"
If you express Yusuke Narita's way of facing AI in one word, it is to treat AI not as an "answering machine" but as a "judgment device," and to create a system to score that judgment and improve it without breaking it. Instead of looking for the strongest prompts or god tools, design the foundation of what the AI decides, how to measure if it was good, and how to prevent deterioration.
Summary of Narita-style principles
The principles for that are clear. With evaluation-first, define what you want to improve first. Score with past data before going to production. Don't believe in a single number and question the yardstick itself. Don't postpone real-world constraints and incorporate them from the start. Regard avoiding deterioration itself as a result. Incorporate ethics into calculations rather than as a note. And try small from limited introduction and expand safely.
What really makes a difference in the AI era is not just "which AI you are using." It's how much you can evaluate and continue to improve the AI's judgment without breaking it. Narita's strength lies in the fact that he did not chase the answers of AI trends. Instead, he has consistently faced growing the AI's judgment system without letting it crash.
Therefore, what we should learn is not to "do the same research as Narita." It is to choose one judgment to entrust to AI in your own work, decide the criteria for wanting to improve it, and improve it little by little while preventing deterioration. Instead of being satisfied with just letting AI write sentences, measure the quality of the AI's judgment and grow it without breaking it. Prompt trends change in half a year, but this evaluation-first way of thinking will continue to work no matter how much AI evolves.
That is the most practical thing that can be learned from Yusuke Narita's AI utilization techniques.
