Holy cow, I would call this project the infrastructure of the Metaverse.
Remember back in 2021 when Facebook changed its name to Meta and announced it would become a Metaverse company in the future?
A few years have passed, and the Metaverse seems to have gradually vanished. To be honest, I once thought this concept might have died out.
Until recently, when I saw AlayaWorld.
1. The Missing Infrastructure of the Metaverse
Many people understand the Metaverse as entering a 3D world with VR glasses, or just a simple 3D game shell.
A building needs modeling and textures, an NPC needs appearance design, animations, and behavior trees, an explosion needs special effects and scripts, and even what a player sees after pushing open a door must be pre-defined by developers.
This industrial process can produce a beautiful game, but it is difficult to support a true Metaverse.
A true Metaverse should not just be a larger fixed map, but a parallel universe with near-infinite scale, constantly changing scenes, and co-created by a large number of users and Agents.
2. AlayaWorld: Letting the World Generate in Real-Time Based on User Actions
AlayaWorld aims to solve four core problems of the Metaverse simultaneously:
Real-time interaction, spatial memory, long-term stability, and open-ended content generation.
- First is real-time interaction.
AlayaWorld supports 6DoF camera control, allowing users to move forward, backward, and turn to explore the generated world from different angles.
It also supports dynamically switching Prompts during the generation process.
This means users are no longer restricted to following a script written by developers; they can directly change the world using natural language.
- Second is spatial memory.
The biggest problem with ordinary video models is that they don't "remember the way."
If a user turns around and comes back, the original building might disappear, or the road might change completely.
AlayaWorld uses 3D Cache to record generated spatial information, combined with compressed historical frames, allowing the model to remember: what this place looked like and what just happened.
This gives AI-generated content the basic attributes of a "world" rather than just a video playing forward.
- Third is long-term stability.
A generative world needs to constantly use its just-generated frames as input for the next round.
Any tiny error will accumulate, eventually leading to character distortion, scene drift, or even the collapse of the entire world.
AlayaWorld uses drift history training and an Error Bank to let the model learn how to identify and fix these cumulative errors in advance, improving the consistency of long-term generation.
- Finally, real-time operation.
If a user takes a step forward and the AI has to wait dozens of seconds to generate a frame, it's not a Metaverse; it's just a video generation tool.
AlayaWorld uses model distillation and streaming video block generation to boost world generation speed to 720p at 24 FPS.
In other words, it is approaching a state where "the world generates as the user operates."
This is why I believe AlayaWorld is suitable as the underlying infrastructure for the Metaverse.
It isn't solving "how to get people into the Metaverse," but rather a more fundamental question: how should the world within the Metaverse be created, remembered, and sustained.
3. The Metaverse Fairy Tale: When Humans and Agents Co-Create a Digital Civilization
The truly exciting future of AlayaWorld isn't just helping users generate a city, but letting users and AI Agents live together in a city that continuously evolves.
Imagine one day entering a world without a fixed script.
The tavern on the street is run by an Agent who remembers you owed money last time and adjusts business based on wars, prices, and foot traffic; your personal Agent continues to manage assets, attend meetings, and handle collaborations after you go offline.
Meanwhile, there are other humans here.
You walk out of the tavern and find the whole city is holding a vote.
The voters include both human users and Agents.
One group of humans and Agents advocates for expanding city boundaries to let the world model continue generating new islands; another group wants to limit expansion and use computing power to repair the old district.
They will give speeches, campaign for votes, and even organize other humans and Agents to lobby.
And the final result will truly change the future shape of the city.
If the expansionists win, when you wake up the next day, a new continent might have already been generated outside the city.
Human players and construction Agents start building ports there, exploration Agents enter unknown areas to map them, trading Agents quickly establish markets, and security Agents constantly analyze potential risks in the new area.
A few months later, this continent that didn't exist might have developed new countries, professions, trade routes, and political systems because of a single Prompt.
Agents here are no longer just tools serving humans.
They could be merchants, journalists, architects, mercenaries, artists, researchers, or even city managers.
And world models like AlayaWorld are responsible for turning user imagination, Agent actions, and collective community decisions into new spaces, events, and history in real-time.
This is the truly fascinating part of the Metaverse:
It is not a pre-made map, but a digital civilization where humans and Agents live together, govern together, and grow together.
We used to think the Metaverse was dead.
Now it seems it was just waiting for two key technologies to mature:
One is AI Agents capable of autonomous action and long-term memory;
The other is a world model capable of hosting these Agents and humans living together.
When these two truly combine, the Metaverse can transform from a marketing concept into a continuously running second world.
And the technology represented by AlayaWorld may be one of the infrastructures leading to that world.
4. Official Demos
- Summoning dark magic to destroy buildings with one Prompt
- Casting fireball magic
- Roaming and exploring in the Metaverse world
If you want to embrace the future of the Metaverse, follow this project!
- Official X Account: https://x.com/alayastd
- GitHub Repo: https://github.com/AlayaLab/AlayaWorld
- Project Homepage: https://alaya-lab.github.io/AlayaWorld/
- Alaya Lab Official Website: https://alaya-lab.github.io/
