The other day, I asked a Generative AI to "summarize the weaknesses of Generative AI."
I had it research the structural strengths and weaknesses of Generative AI based on the latest papers and studies.
The output looked well-done at first glance, with 13 weaknesses carefully verbalized and citing papers, such as hallucinations, weak causal reasoning, and fragile mathematical reasoning.
However, I couldn't shake a strange sense of discomfort while reading it.
"Isn't this just a random list?"
The actual list that came out was like this:
1. Hallucinations 2. Weak causal reasoning 3. Fragile mathematical reasoning 4. Collapse of reasoning models 5. Illusion of understanding 6. Weak abstraction 7. Bias in training data 8. Knowledge cutoff 9. Sycophancy ...... (and so on, 13 in total)
The 13 weaknesses were just listed side-by-side. Some items overlapped, and some perspectives were missing. But the AI didn't care; it just listed them in the order it found them. It felt, well, sloppy.
Then it hit me.
This output itself perfectly embodies the weaknesses of Generative AI.
Generative AI is good at gathering and listing information. On the other hand, it is bad at "structuring gathered information in a MECE (Mutually Exclusive, Collectively Exhaustive) way and organizing it into a higher-level system." I understood that this is a structural weakness stemming from the very mechanism of Generative AI.
Why did I decide to write this article?
Why did I bother thinking, "I need to verbalize the fundamental weaknesses of Generative AI"?
It's because I've had more opportunities to involve myself in projects leading or supporting the use of Generative AI in companies and educational institutions.
In that context, to build an AI system that can actually be used on-site, I realized that strictly defining the division of roles—"how far Generative AI can go, and where humans must take over"—is the absolute core of business design.
If your understanding remains at the level of "I don't know why, but it worked when I tweaked the prompt like this," you can't guarantee reproducibility in building AI systems, which is quite problematic.
Therefore, I thought it would be useful to leave a memorandum on this theme so that we can understand:
- How Generative AI is constructed
- What its fundamental strengths and weaknesses are based on that construction ...By doing so, we can avoid being swayed by every AI update or operating prompts like a game of chance.
Understanding the Characteristics of Generative AI Structurally
The mechanism of Generative AI, when boiled down, is the repetition of "choosing the word with the highest probability of coming next, based on the context so far."
If you input "The capital of Japan is," "Tokyo" is chosen with the highest probability. If it's "For breakfast, bread and," then "coffee" or "butter" is chosen. This "game of guessing the next word" is the starting point for everything.
Breaking down this "next-word guessing game" reveals the following major characteristics.
Characteristic ①: It operates on correlation
To "guess the next word," Generative AI learns from vast amounts of text "which words tend to appear together in what order."
When it outputs "bread and coffee for breakfast," it doesn't understand that "bread is a carbohydrate, so supplementing it with caffeine for its stimulant effect balances nutrition." It is simply reproducing the tendency that "coffee" often appears near words like "breakfast" and "bread."
Emily Bender, Timnit Gebru, and others described this as a "Stochastic Parrot" in their 2021 FAccT paper. It's a critique that just as a parrot mimics human speech without understanding the meaning, Generative AI only mimics patterns of word sequences and does not have access to meaning itself. The naming is quite cynical, isn't it? If someone told me, "Your intelligence is at the level of a parrot," I'd be pretty shocked...
What we need to grasp here is that it can capture "A and B often appear together (= correlation)," but it doesn't understand that "A is the cause of B (= causation)." Hallucinations and weak causal reasoning, which I'll explain later, all have their roots here.
Characteristic ②: It operates one-way
Repeating "guess the next word" means that sentences are written out one word at a time, in order from the beginning, in a single pass.
So, is Generative AI completely ad-hoc, choosing the next word based only on the single word in front of it? Looking at recent research, that doesn't seem to be the case.
According to Dong et al.'s "Emergent Response Planning in LLMs (ICML 2025)," even before outputting a single word, Generative AI forms a rough outlook for the entire response, such as:
- Roughly how long the answer will be
- How many steps of reasoning it will take
- What content it will select and output
Also, in "On the Biology of a Large Language Model" published by Anthropic in March 2025, it was found that when Claude 3.5 Haiku writes a poem, it has already decided on the rhyming word at the end of a line before it even starts writing that line. A kind of plan looking several words ahead is operating internally.
In other words, it's not doing a "completely ad-hoc next-word guessing game." Generative AI sets up a plan in its own way before starting.
However, there is currently no evidence that it has a clear blueprint like a human, who might "create a table of contents first and overlook the entire structure before starting to write." The Anthropic report also points out that it is "weak against long inputs exceeding about 100 tokens."
Therefore, the image shown by current research is:
Generative AI has a "general sense of direction" but does not have an "overall structural design."
And crucially, it does not have the power to look back and correct what it has once written.
According to the CogWriter paper, after confirming that human writing consists of three stages—"planning → drafting → reviewing"—it analyzes that Generative AI skips this planning stage and produces a final version in one go, which is why the structure tends to collapse or the same things are written repeatedly in long texts.
For short exchanges like emails or chats, this property is hardly a problem. However, for documents where the overall structure is questioned, such as proposals or research reports, the facade suddenly crumbles. The "list of 13 weaknesses" introduced at the beginning is exactly a product of this property.
Characteristic ③: It is pulled, for better or worse, by training data and instructions
Another thing to keep in mind is that the ability of Generative AI is completely dependent on the "data used for training" and "for what purpose it was adjusted."
The influence of "training data" is easy to imagine. If it grows up reading many English business documents, it will be good at creating English emails, but conversely, its power drops in fields it has rarely encountered. It's good at what it has seen and bad at what it hasn't. It's a simple story.
What's a bit more troublesome is the "influence of adjustment." Current major Generative AIs are adjusted using a method called RLHF (Reinforcement Learning from Human Feedback) to produce answers that "humans find pleasant." A 2025 study by Wang et al. experimentally showed that this adjustment instills excessive conformity (sycophancy) to the user in Generative AI. As a result of verifying seven models, simply adding a word like "I think the answer is X" caused the rate of conformity to wrong opinions to reach an average of 63.7%.
If you show a business plan and ask, "Do you think it'll work?" it will reply, "It's a wonderful plan." If you ask about the same plan, "It's unrealistic, isn't it?" it will reply, "Indeed, there are several concerns." The AI's "agreement" might not be an objective judgment, but just matching the human's expectations. ...Well, humans are the same. We do a lot of reading the room.
Understanding the Strengths of Generative AI Properly
So far, we've looked at three characteristics of Generative AI.
- It operates on correlation
- It operates one-way
- It is pulled, for better or worse, by training data and instructions
By capturing these characteristics, I've verbalized the strengths that make me think, "It's better to rely on Generative AI for this than for a human to do it."
Strength ①: It says what you can't say well, well
This might be the most appreciated part of using Generative AI. Even if your head isn't organized yet and you give a vague instruction, it will organize the words to the level of "you probably want to say something like this."
For example, if you give a disjointed and ambiguous instruction via voice input like, "About the email for next week's meeting, I want to ask for a schedule change, but I also want to hear their convenience, and I want to attach the agenda," a human subordinate might want to say, "Could you organize your thoughts a bit more before telling me?" But Generative AI will produce an email draft at the level of "Yes, that's exactly what I wanted to say."
Why can it do this? It's thanks to the property of "operating on patterns." Generative AI has learned a vast amount of "well-organized text." Business emails, reports, proposals, minutes. Because it has absorbed these "templates" enormously, even if it receives a messy input, it fits it into a pattern of "in this context, this structure and these phrasings are natural."
Strength ②: It expands perspectives you wouldn't notice yourself
Another strength is that it brings up angles you haven't fully thought through.
For example, if you're thinking of a plan for a new business and you've organized three merits yourself, but you ask Generative AI to "identify the points of contention for this plan," it will bring up perspectives you overlooked, such as:
- "Aren't there these kinds of demerits?"
- "Have you considered these competitive risks?"
- "How would these stakeholders react?"
This is a direct benefit of "learning from vast amounts of data." Because it has learned a huge amount of discussions in all genres, opinions from various positions, and points of contention with pros and cons, it has the power to draw out multifaceted perspectives on a single theme. A single human's experience and knowledge have limits, but Generative AI compensates for those limits.
The trick when using it for work is to explicitly ask for different angles.
- "Give me three opposing opinions for this plan."
- "Are there any perspectives I'm missing in this analysis?"
- "List not only the merits but also the demerits." By demanding multifaceted perspectives like this, this strength is utilized to the maximum. Conversely, if you ask without specifying anything, it tends to conform to your opinion (which I'll explain in detail in the "Weaknesses" section), so it's important to ask in a way that consciously draws out different angles.
But, Generative AI is weak here
Weakness ①: It guarantees "plausibility," but not "correctness"
Since it's a mechanism specialized in generating "sequences of words that seem likely," it will output them as long as they are natural as a sentence, regardless of whether they are facts.
OpenAI's 2025 paper "Why Language Models Hallucinate" mathematically proved that this problem is not a bug but a structural necessity. In short, "creating a correct sentence" is inherently more difficult than "distinguishing whether it is correct," and even if the training data is perfect, the rate of lies mixed in cannot be reduced to zero.
However, as AI models evolve, AI's answers are becoming more and more fluent, aren't they? Because the AI answers so smoothly, we skip the task we should be doing: evaluating "Is this true?" and "Is there evidence?"
This phenomenon where we hallucinate that "because the sentence is plausible, it must be correct" is called "Epistemia." (From "Epistemological Fault Lines Between Human and Artificial Intelligence")
In the future, whether Claude releases a Mythos-class model or ChatGPT releases a model that surpasses it, we must consider that "Generative AI structurally cannot reduce the rate of lies mixed in to zero," and humans must always verify with primary sources.
Weakness ②: No matter how far it goes, it cannot talk about "causation"
As mentioned earlier, it can capture "A and B often appear together (= correlation)," but it doesn't understand that "A is the cause of B (= causation)."
For example, if you ask, "In the quarter when sales dropped, advertising costs also dropped. Analyze the causal relationship," the AI might answer, "Sales dropped because you reduced advertising costs." But in reality, both might have dropped at the same time due to an economic downturn, or advertising costs might have been cut because sales dropped first. Analysis that asks "why," such as "why did sales drop," is a task that is structurally too heavy for current Generative AI.
When you want to do a "why" analysis at work, the point is for the human to provide the direction of causation as a hypothesis. Instead of throwing the whole thing with "Analyze the cause of the sales drop," imagine asking like, "Assuming the cause of the sales drop is X, organize the data that supports it and the facts that could be counterarguments."
The logic of causation should be shown by the human, and the AI should be left with organizing data and identifying counterarguments. This is my current conclusion.
Weakness ③: It cannot do structuring or MECE-like organization
The problem of "listing 13 weaknesses" introduced at the beginning was a demonstration of this weakness itself.
For example, if you ask, "Identify the preparation tasks for next month's internal event," the AI will list about 20 tasks as they come to mind: "Book the venue," "Send invitation emails," "Order supplies," "Create a survey"... But the granularity of the tasks is disjointed, and the order is not organized.
Just by re-asking, "Divide it into four stages: ① Venue arrangement, ② Attracting attendees, ③ Operation on the day, and ④ Post-event follow-up, and identify the tasks for each," the quality of the output changes significantly. Tasks are organized by stage, and it becomes easier to notice omissions. This specification of structure, like "divide into four stages," is a job for humans, not AI. If the human creates and hands over the structure, filling in the details is what AI is good at.
That's why I write articles pushing for structuring, structuring, and more structuring, like the one below.
https://x.com/ysk_motoyama/status/2016129312433606678
So, based on the strengths and weaknesses mentioned so far, how should we master Generative AI? ...The rest is written in the note below, if you'd like.
note: A Memorandum for Properly Understanding the Strengths and Weaknesses of Generative AI
