Hot takes on AI memory

One of the beautiful things about building in a new space is that there are no right answers yet. This also means that to build anything inherently involves making bets on where the ecosystem will evolve. We've compiled a (non-exhaustive) list below of questions that we discuss often with those in this space along our predictions on what the answer is. We would love to hear your thoughts, predictions, and disagreements!

Is there room for memory and knowledge base companies beyond the labs?

• Prediction: Companies doing vertical memory scaling (i.e. helping agents run for longer) will have a hard time competing and will be squeezed by the labs and other agentic harnesses. Companies doing horizontal scaling (i.e. across teams or entire organizations) will find a better landscape. This is because enterprise deal cycles take longer and the problems (data isolation, security, company ontology) cannot be solved by the newest model update or research idea.

Should memory layers operate in weight vs token space?

• Token space has a lot of advantages. It's interpretable. It's model-agnostic. It's cheap. We have decades of infrastructure built to handle storage, data isolation, modularity, etc.
• Weight however seems to be more expressive and there may be a class of problems that we cannot solve purely in token space. In particular, procedural memory involving fuzzy lines and complex branching paths do not seem well suited for token space (e.g. think of trying to read the rules to a board game vs being shown how to play it)
• Prediction: Most memory will operate in token space (e.g. agent traces, semantic information, etc.) but there will be certain problems (e.g. writing style, taste, procedural skills, etc.) that will have adapters which can be fit into models. Mech interp techniques will enable us to interpret them.

Is memory simply a search and retrieval problem?

• Most memory systems today are focused on retrieval. They are focused on finding the right information at the right time for agents to do work (e.g. LoCoMo benchmark focuses on needle in the haystack retrieval).
• The question is if this is sufficient for to solve the memory problem. Put it another way, if you hook on SOTA search (e.g. Google or Exa or Perplexity) to a private data store, is that enough to call memory solved?
• Prediction: There is a growing consensus of researchers and builders working at the cutting edge that memory is not merely storage of information and retrieval over that information. We call this problem "blast radius" internally. Information's usefulness is bounded by scope (time or context). Humans have no problem reading tons of irrelevant text and only applying the proper weight to the most useful information. A pure retrieval system (even with smart reranking) falls short of that.

Should we inject information into context automatically?

• The argument against is context rot or pollution. Injecting information into an agent, especially if it is not the right information could cause degraded performance. It also causes the agent to over index on connections between your sessions which may not actually be real. This is why many people turn off memory features for chatGPT or claude code.
• Prediction: Injecting information into context is critical because it enables the agent to deal with "unknown unknowns". You can have a perfect memory tool but if the agent doesn't know to use it, you haven't solved the problem. For humans, this type of "injection" happens all the time. Past memories appear in your consciousness without your active choice. The problems with this today are likely downstream from the blast radius problem outlined above.

What are the right benchmarks for memory?

• There is a general sense that existing benchmarks like LoCoMo and LongMemEval are not sufficient. We've hit on ~85% performance on them and memory still feels as unsolved today as it did a year ago. Moreover, better performance on the benchmarks doesn't seem to correlate with "better feeling" memory from a user perspective.
• Moreover, benchmarks in this space are difficult build since the inherently long time horizons memory operates over create data availability and cost/scaling problems.
• Prediction: The company or lab that solves this problem will likely not do so by hill climbing on a benchmark but by betting on some customer/user insight that current benchmarks are not measuring. This is similar to Wisprflow where they threw out the word-error-rate metric that other transcription tools anchored on.

Will longer context windows will solve everything?

• We made a prediction in Janthat context windows won't actually solve the problem and it has turned out to be mostly right so far.

Strong models combined with data integrations makes memory systems- useless

• The argument for is that you can retrieve any information you want if you have a frontier model + agent harness + MCP data connectors. And it turns out that the retrieval quality doesn't change much compared to other systems (e.g. LLM wiki, hybrid retrieval, etc.)
• Prediction: In the short term, memory systems are still useful because they reduce latency and cost compared to having frontier models search over everything all the time. In the medium to long term, memory systems enable consistency over retrievals which enable compounding. Put it another way, we still have agents write code which they improve over time rather than have them manifest say an app directly.

Agentic search over file systems are all you need

• Letta predicted this last year and it has turned out to be quite prophetic. In the short-medium term, agents are extremely good at operating over file systems due to the post-training aimed at coding performance. Leveraging that post-training yields rewards today.
• Prediction: In the long term, it's hard not to imagine a type hybrid index in addition to a file system. The main intuition behind why this is necessary is that file systems perform worse in when there are higher volumes of data or in federated use-cases. Agent "monologues" over raw data will also become increasingly important and we will need principled and structured ways to support that.