TL;DR
In an AI-native engineering organization, generating code is no longer the slowest part of shipping software. Review, verification, repair, and human decision-making become the bottlenecks, and ultimately define how much faster the organization can move. Standalone AI coding tools may deliver a 30–40% speedup, but achieving a step-change in throughput requires optimizing everything that happens after the code is written.
In our previous post about solving code review with Cosmos, we described a review system that helped our engineering organization increase code output by 3× while reducing median merge time and maintaining quality. We have since expanded that system beyond review into the full PR-to-merge loop: specialized agents handle mechanical work, verify correctness, and address feedback, while humans provide judgment and knowledge transfer, and always own the final merge decision.
This is loop engineering: improving the complete system that turns generated code into a verified, understood, merge-ready change rather than optimizing any single tool in isolation.
Why an “AI code review tool” is not enough
A conventional AI code review tool like CodeRabbit or Greptile analyzes a diff and posts comments. Useful, but review is only one step. The real drag is the chain of human handoffs:
- Reconstructing intent and architecture.
- Triaging low-risk changes.
- Applying review feedback.
- Fixing CI failures and merge conflicts.
- Verifying feature behavior end to end.
- Gathering enough evidence to ship safely.
- Re-reviewing after each push.
A PR-to-merge loop keeps specialized Experts working through review, repair, verification, and decision support until a human can merge with confidence. Each Expert in the fleet owns one or more of these handoffs, turning a fragmented sequence of manual tasks into a coordinated system. The goal is to optimize the entire loop: human time, cost, quality, and merge latency, not time to first comment.
1. Overview: engineering the complete loop
read image description
ALT
An AI-native code review loop: agents find and fix bugs, verify changes, and evaluate policy; humans resolve judgment calls and make the final merge decision.
Our original system separated risk analysis (Risk Analyzer), line-by-line correctness review (Deep Reviewer), and human-guided design review (Pair Review). The expanded system adds repair, runtime verification, and expanded auto-approval, reducing the human bottleneck further. For an overview of Cosmos and its configurable Experts, see our previous post about solving code review with Cosmos.
Experts and capabilities, not one general-purpose reviewer
Expert or capability
Responsibility
Risk Analyzer
Classifies risk and applies auto-approval policy
Deep Reviewer
Performs exhaustive, line-by-line analysis for objective correctness defects
Pair Reviewer
Reconstructs intent, architecture, product context, and tradeoffs
Memory Manager
Remember feedback from the PR and Pair Review sessions to improve future runs
Verifier
<sup>
NEW
</sup>
Exercises affected behavior end-to-end in a test environment (
)
PR Fixer
<sup>
NEW
</sup>
Repairs review findings, CI failures, and merge conflicts
Review Dashboard
<sup>
NEW
</sup>
Observes and summarizes expert state
cosmos approve
<sup>
NEW
</sup>
Evaluates a configurable approval policy on author request
The distinction between Deep Reviewer and Pair Reviewer is particularly important:
- Deep Reviewer asks: “Is there an objective bug in this implementation?” It runs autonomously and checks the PR against AGENTS.md or CLAUDE.md guidelines.
- Pair Reviewer asks: “Does this change make sense in the broader system, and which decisions require human judgment?” It runs interactively with the human. The human can also instruct it to monitor the PR after posting review comments and approve on their behalf once those comments are addressed.
The PR Author Expert previously did double duty: writing a PR and fixing review comments, CI failures, and other follow-up work. In the decoupled design, PR Author stops at draft PR creation and PR Fixer takes over. This gives users more control over how fixes run and supports PRs created without PR Author.
The Review Dashboard brings Expert status, reviewed commits, evidence, and available actions into one view.
- Designing the human-in-the-loop
Human review and verification are becoming the scarce resources in AI-native engineering organizations. The goal is not to remove humans indiscriminately. It is to spend human attention only where it has the highest leverage.
Why humans remain in the loop
Agents can perform a large share of mechanical analysis and execution, but they do not have complete business and organizational context. Humans remain essential for:
- Judgment calls: Should this logic live in the frontend or backend? Is this tradeoff appropriate for the product? Is this risk acceptable now?
- Knowledge transfer: Review is one of the ways engineers build a shared understanding of architecture and product behavior.
- Ownership and accountability: Agents do not own the software after it ships; human developers and engineering organizations do. A human therefore always makes the final decision and clicks Merge. None of these experts merges a PR.
The design target is therefore:
Agents do the mechanical work. Humans make the consequential decisions.
The author and reviewer workflow: before and after
Traditional workflow
Human-in-the-loop workflow
Manually triage every PR and identify low-risk changes
Let
Risk Analyzer
classify risk and apply the organization’s auto-approval policy
Read the PR line by line
Trust
Deep Reviewer
to perform exhaustive line-by-line analysis
Reconstruct context, intent, and architecture from the diff
Use
Pair Reviewer
’s briefing to understand the change and identify judgment calls
Manually deploy and exercise the feature
Inspect
Verifier
evidence: screenshots, logs, traces, and captured outputs
Re-review every fix from scratch
Let
Pair Reviewer
monitor whether authorized comments were addressed
Triage feedback, implement fixes, repair CI, resolve conflicts, and explain each change
Let
PR Fixer
—or
PR Author
when it owns the full lifecycle—handle the mechanical follow-up and report what changed
Search across comments and checks for current state
Use the
Review Dashboard
as the entry point
Manually gather review, ownership, and verification evidence before requesting approval
Invoke
cosmos approve
to evaluate the configured approval policy against current evidence
Decide whether to merge
Still decide whether to merge
The economic intuition of using multiple experts
Tokens that measurably reduce human time in the review bottleneck are worth the investment. When Experts turn hours of review and PR shepherding into minutes, they free scarce engineering judgment for the consequential decisions only humans can make, while also helping features reach customers faster.
One of Augment’s core commitments is helping organizations optimize costs. That means optimizing total cost per task: human effort plus token cost. It does not mean minimizing token usage at the expense of successful outcomes. A single Expert burdened with six responsibilities will do a substandard job of each and require more human intervention. Six dedicated Experts can each focus on a different part of PR review, produce higher-quality review artifacts, and drive more of the process autonomously.
3. Cost and quality: optimize cost per successful outcome
The cheapest model by token price is often a false economy. A missed defect, bad repair, or retry can cost more than getting the task right once. We benchmark representative work and choose the lowest-cost model that clears each Expert’s quality bar, the same cost-per-success principle we use across Cosmos.
Today, we use GPT-5.6 Sol for judgment-heavy work such as Risk analysis, Deep and Pair review, and code repair. Bounded, mechanically verifiable tasks such as dashboard aggregation and merge-conflict monitoring run on GPT-5.6 Luna. Models with longer cache TTLs also favor long-running agents because cached input is typically discounted at 90%.
4. A configurable approval path with cosmos approve
The Risk Analyzer has always been able to approve intrinsically low-risk changes under a conservative policy. We now support a second, disabled-by-default approval mode for other changes.
The PR author can comment cosmos approve to request an evaluation against an approval policy. A human still owns and performs the final merge.
Organizations define their own approval policy. Our internal policy checks:
- Ownership: The requester is the PR author, and the author is an effective CODEOWNER for every modified file.
- Current-head review: No unresolved Deep Reviewer findings, Pair Reviewer blockers, or unaddressed human comments.
- No contradictory runtime evidence: The Verifier has not reported an unaddressed defect against the current commit.
5. Customizability is part of the architecture
We designed the fleet so teams can customize the loop with Cosmos Advisor.
- Fleet: Add, remove, or replace experts; choose the model and prompt for each responsibility.
- Operation: Configure triggers, tools, integrations, and verification environments.
- Control: Restrict who can invoke actions and which credentials, repositories, and systems each expert can access.
- Policy: Define approval rules, required checks, CODEOWNER requirements, and acceptable evidence.
The operating model for AI-native PR Review
The lesson from our first version was that code review could not scale by asking humans to read AI-generated code faster. The lesson from this version is broader: no individual review agent can optimize the full path to merge.
A high-functioning PR-to-merge loop needs:
- Agent Specialization: Separate risk, correctness, design judgment, runtime verification, and repair.
- Minimize human checkpoints: Pull humans in for consequential judgment and knowledge transfer only.
- Evidence: Give reviewers inspectable proof rather than unsupported verdicts.
- Repair loops: Let findings flow back into implementation without waiting for manual intervention.
- Observability: Make the state of the entire fleet legible in one place.
- Cost discipline: Use the cheapest model that clears the quality bar for each responsibility.
- Configurability: Customize the loop to each organization’s unique requirements.
- Human ownership: Keep the final merge decision with the people accountable for the software.
This is loop engineering applied to the PR lifecycle: optimize the system that produces a verified, understood, merge-ready change, not the amount of output produced by any single agent.
Build your own PR-to-merge loop
Cosmos gives engineering teams the shared context, runtime controls, integrations, and human checkpoints to run agents across review, verification, repair, and the rest of the software lifecycle.
