Your Inference Server is Secretly a Learner: Open-Sourcing Reef for Continual Self-Improving Agents

Reef is fully open source: https://github.com/Human-Agent-Society/reef

1. Before all the “RSI” hype becomes real

Before today's excitement around RSI is fully realized, we want to open-source Reef, an infrastructure we've been building for the same broader problem: enabling agents (harness + model) to continuously evolve from their experience.

The goal is to make it easier for the open-source community to experiment with continual self-improvement and readily access production-grade infrastructure for it. We use continual self-improvement as the broader and more practical framing here, with RSI representing a more fully recursive form of the same idea.¹

2. Why does continual self-improvement need new infrastructure?

GIF

Most LLM infrastructure assumes a relatively simple lifecycle. We train a model, evaluate it, deploy it, and then use it for inference. For continually self-improving agents, we think two assumptions behind this setup start to break.

First, inference is no longer the end of the pipeline. Agents generate useful experience while they work, including trajectories, execution results, user feedback, and other signals that can drive future improvement. What happens at inference time is no longer something we simply serve and discard. It becomes part of the learning process itself.

Second, the model is no longer the only thing that evolves. An agent is more than a model, and continual self-improvement should not be restricted to model weights. Prompts, memory, skills, tools, and orchestration logic can all potentially improve from experience. A stronger model expands the agent's capabilities, while a better harness helps elicit those capabilities more effectively and exercise them more reliably in complex tasks.

Together, these changes turn what was once a mostly sequential pipeline into a continuous evolution loop. Agents interact, generate experience, improve different parts of themselves, evaluate whether those changes are worth keeping, and return to serving as new versions of the system.

That is also why we do not think of Reef as merely training infrastructure. Training is only one part of the loop. We think infrastructure for continual self-improvement has to start from live inference and support the evolution of the whole agent.

GIF

3. What does such infrastructure need, and how does Reef implement it?

Reef architecture

Infrastructure for continual self-improvement needs to own three things end to end: the experience, the agent, and the updates. That means (1) learning from live traffic, (2) updating both the model and the harness, and (3) properly evaluating, versioning, and controlling how updates are released.

Own the experience — learning has to be built on top of live serving

Inference and training have historically been decoupled. Some RL infrastructure (e.g., Slime, veRL) incorporates an inference engine for generating data, but these systems are designed for model training rather than model serving. We posit that a continual self-improving infra should first be an inference infra and build its training capacity around live inference: the system serves real applications, collects test-time experience, and lets learning recipes continuously consume it. This belief leads to a rethinking of many design choices, including training signal generation and training sample selection.

Inference is native to Reef. Reef exposes standard inference endpoints, making it easy to integrate into existing applications and turn them into self-evolving systems.

1# client = xxx  # initialize httpx client to Reef's serving endpoint
2
3# Standard inference call through Reef, Open-AI format
4response = client.post(
5        "/v1/chat/completions",
6    json={"model": xxx, "messages": xxx},
7)
8
9# Reference to the inference record stored by Reef
10receipt = response.headers["x-reef-agent-record-id"]

Applications can also report rewards, evaluator feedback, or other signals associated with specific inference calls through:

1# Attach feedback to the corresponding inference record
2client.post(
3        "/reef/report",
4    json={"feedback": "wrong answer", "references": [receipt]},
5)

Different from existing inference engines that stay static throughout their lifecycles, Reef offers stateful inference: Reef stores inference traces and feedback as a structured experience stream, handling issues such as off-policy staleness, session merging, and deduplication. Learning recipes define how this stream is processed, which learning algorithm is used, and when updates are evaluated and deployed. This lets different applications evolve with their own learning strategies on top of the same infrastructure.

Own the whole agent - both the model and the harness are evolved via stateful inference

An AI agent capable of delivering end-to-end results consists not only of a model, but also of a harness. The model provides the underlying capabilities needed to solve tasks, while the harness enables reliable execution across complex, long-horizon trajectories by managing tools, context, memory, feedback, and orchestration. The two are tightly coupled: the harness determines how the model’s capabilities are elicited, grounded, and exercised, while the model determines what forms of execution the harness can reliably support. Advances in either can therefore change the optimal design of the other. A continual self-improving infrastructure should support the joint evolution of the entire agent stack, including not only model weights, but also prompts, memory, skills, tools, and orchestration logic.

Reef supports updates to both the model and the harness.

On the harness side, Reef uses Cordis as a "training backend." A harness evolution recipe typically analyzes agent trajectories and feedback, then proposes edits to the harness. This process happens entirely within Reef, and each evolved harness version is released to the user as an installable update (assuming Reef is served on localhost:8900):

1curl -fsS -H "Authorization: Bearer $REEF_TOKEN" \
2  'http://localhost:8900/reef/harness/install?adapter=pi' | bash

The command above installs a Reef-wrapped Pi harness in much the same way as a typical coding agent. The harness is configured to use Reef's stateful inference endpoint, so its inference traffic flows through Reef. As the user works, Cordis evolves the harness following the configured harness evolution recipe, and new versions are made available through Reef. The next time the user opens the harness, they may see:

On the model side, learning recipes consume Reef records to update model weights. Training runs asynchronously with live serving using a distributed training backend, currently adapted from Slime, and produces candidate weight updates such as checkpoints or LoRA adapters.

Once a candidate passes evaluation and is approved for deployment, Reef publishes it as a new version of the scenario's model artifact and hot-updates the serving engine using NCCL-based weight synchronization, without restarting the service.

Own the updates - evolved releases are evaluated and versioned

A continuously evolving agent is subject to service quality degradation especially as the evolution is not guaranteed to bring performance improvement. As such, each evolved candidate should be evaluated before release. Reef controls whether an evolved candidate is allowed to replace the artifact currently serving a scenario. If the candidate is rejected, serving remains unchanged. Otherwise Reef publishes it as a new, auditable release.

Anything Reef can evolve—such as a model checkpoint, LoRA adapter, harness tree, or routing policy—is represented as an artifact managed by a version controller. Reef adopts Git LFS for managing the artifacts, especially ones taking up large disk space such as model weights. The release path is:

Each scenario has an append-only release chain. Reef advances the scenario’s release head using compare-and-swap, so a stale publisher cannot overwrite a newer release. The release pipeline allows Reef to efficiently track continuous version changes and release processes.

4. Continual self-improving methods in Reef

The infrastructure above provides the common abstractions for continual self-improvement. The actual logic of how an agent improves is implemented through modular learning recipes.

We have seen a growing zoo of methods for turning signals generated at test time into better agents: online reinforcement learning, test-time training, skill evolution, harness evolution, self-play, and more. Despite their different names and mechanisms, they share the same basic pattern: signals generated at test time are turned into updates to the system that generates the next interaction.

These recipes differ mainly along three dimensions:

Learning signal: What form of signal drives improvement, and where does it come from?

Experience acquisition: How is learning experience generated? Is it proactively sought by the agent, or reactively generated from an external task or interaction?

Evolving target: What actually changes: the model, the harness, or both?

Recipes already supported or coming soon to Reef

Reef is designed so that these methods can largely be expressed through different learning recipes on top of the same infrastructure. Using a recipe is simple: choose one and configure it when starting the Reef service.

1# serve.yaml
2reef:
3  recipe: recipes.sao.recipe:SAORecipe   # Plug in an evolution recipe
4  batch_size: 1                          # Recipe-specific configuration
5  max_staleness: 18

Then start Reef with the configuration:

1reef serve -c recipes/sao/examples/sao/serve.yaml

Below, we show two examples, OpenClaw-RL and TTT-Discover, which follow very different evolution strategies but can both be implemented in Reef.

GIF

The visual demonstrates an integration of OpenClaw-RL in Reef. The user interacts with an agent whose model is continuously evolved by Reef asynchronously without interrupting the user. As more and more rounds accumulate, the agent gradually learns to correctly interpret user’s preference and gives a satisfactory answer.

GIF

The visual demonstrates how TTT iteratively improves a Packing 32 solution. As optimization proceeds, increasingly effective solutions are discovered and retained, leading to progressively higher packing scores.

5. Conclusion

Reef is our attempt to turn the broad idea of continual self-improvement into a concrete systems problem. By open-sourcing Reef, we hope to make this problem easier to study and give the community a practical foundation for building agents that do not merely serve, but continually learn and evolve from experience.

We invite you to try Reef, build your own learning recipes, and integrate it with your agents. Explore the code, documentation, and example recipes at https://github.com/Human-Agent-Society/reef. If you find Reef useful, we'd appreciate a star on the repository. If you want to build with us or discuss continual self-improvement more broadly, you're more than welcome to join our Discord: https://discord.gg/5y8e5f937k.

1. We use continual self-improvement as a broader and more practical framing than RSI. It covers systems that repeatedly improve from experience without requiring the fully closed loop often associated with RSI, where AI itself participates in building and improving the system that produces its next version. Reef is designed for this broader problem, while leaving room for more recursive forms of self-improvement to emerge on top of it.

Growing List of Contributors (alphabetical order): Chai Wenhao (@wenhaocha1), Ding Shuangrui (@ShuangruiDing), He Hao, He Haoze, Jiang Chonhe (@JiangChonghe), Jiang Nan (@nanjiangwill), Jiang Xuan, Li Xiaochen (@jacobli99), Liang Paul (@pliang279), Liu Bo (@Benjamin_eecs), Long Boyuan, Mang Qiuyang (@MangQiuyang), Qi Zhenting (@ZhentingQi), Qu Ao (@ao_qu18465), Qu Mingruo, Wang Zhaokai, Yan Xuezhi, Yu Hanfei (@yhfchitanda), Yu Haofei (@haofeiyu44), Yu Simon (@simon_ycl), Zheng Han (@hanzheng_7), Zhou Kaichen (@alex_kai2020), Zhou Zijian (@BobbyZhouZijian), Zhu Jiacheng (@JiachengZhu_ML), Zhuang Dingyi