LLM Inference — Past, Now, and Future: Where Does the Value Move?
For the past two years, LLM inference has been one of the most contested layers in AI infrastructure. Dozens of inference providers, GPU clouds, open-source projects, and chip vendors have all chased the same goal: serve a trained model faster and more cheaply than the next company.
The attraction rested on a plausible idea. Inference looked like a software problem with a software moat. Better kernels, a cleverer scheduler, or a stronger speculative decoding could support premium pricing or better margins at the same price. For a time, an in-house engine was a real competitive weapon, and benchmarks won deals.
That period is ending. Inference matters more than ever, and the market continues to grow, but defensible value has moved away from the engine. The engine layer is commoditizing quickly. Value is shifting first into serving operations and platforms, then toward capital, GPU capacity, and eventually the data centers themselves.
Software performance still counts. A slow or unreliable engine can disqualify a provider. Good performance, however, has become widely available, making it harder for any one company to charge for it. The question is no longer whether the engine creates value, but who captures that value once the engine becomes common infrastructure.
Three hardware generations make the shift visible. The same pattern explains the competition among inference providers today and where it is likely to go next.
Part I: The Past — When the Engine Was the Moat
Three generations, three checklists
Each NVIDIA hardware generation came with a checklist for an in-house engine. What changed was how quickly the checklist became public knowledge, and how little advantage remained once everyone had completed it.
The Ampere era (A100). The early bar was concrete. An engine that supported CUDA Graphs to remove launch overhead, implemented speculative decoding such as EAGLE-1 or Medusa, and shipped solid W8A8 INT8 quantization stood ahead of most of the field. The engineering was difficult but bounded, and completing that short list put a provider in the top tier. Those features won deals.
The Hopper era (H100/H200). The list grew and split in two. For a single deployment, whether one replica, one node, or a few, the differentiators were FlashAttention-3, FP8 attention, EAGLE-3 speculative decoding, and W8A8 FP8 quantization. Strong implementations produced standout single-node results.
Hopper also opened a second front in disaggregated deployment. Support for prefill–decode (PD) disaggregation, expert parallelism (EP) for the increasingly dominant MoE architectures, and KV cache offload across the memory hierarchy mattered at cluster scale, where the largest contracts lived. This tier demanded systems engineering as well as kernel work. For a while, it separated the strongest providers from everyone else.
The two tiers rewarded different capabilities. Kernel teams could still win a benchmark on a contained deployment, while the largest production workloads demanded coordination across machines, memory pools, and failure domains. That second tier took longer to copy and gave providers a larger window in which to turn engineering work into revenue.
The Blackwell era (B200/B300/GB200/GB300). Here the checklist has narrowed to one main item: NVFP4 optimization. The leading route is no longer an independent implementation. Teams integrate the trtllm-gen cubins shipped by NVIDIA or build on TensorRT-LLM directly. Speculative decoding, FP8 and FP4 paths, disaggregation support, and MoE parallelism are already in the reference stack.
Blackwell changes the make-or-buy decision. Building the stack once demonstrated technical depth; now it can amount to recreating vendor work while competitors spend the same engineers elsewhere. A proprietary implementation may still suit an unusual model or deployment, but it is no longer the default path to leading performance.
Inference has no secrets. Every important technique has a paper, an open-source implementation, or a vendor binary. Knowledge that once circulated through a handful of performance teams is packaged into code that any capable group can inspect or integrate. The checklist still qualifies an engine, but no longer sets it apart.
Why TRT-LLM became the Blackwell baseline
TensorRT-LLM's position in the Blackwell era follows from the incentives on both sides of the market.
NVIDIA needs TRT-LLM to perform well. The software underpins its new-hardware benchmarks, including InferenceX submissions, launch-day claims, and keynote results. Optimizations for a new chip therefore arrive in TRT-LLM on day one, backed by a large kernel-engineering organization and a capable runtime.
Independent engines cannot reproduce that advantage through effort alone. NVIDIA sees the hardware roadmap, controls the lowest software layers, and has a direct commercial reason to make each generation look strong at launch. TRT-LLM is where those incentives meet.
At the same time, a handful of frontier open-weight model families now account for the overwhelming majority of serious production traffic. Providers have less need to support hundreds of architectures. Given that narrower model set and Blackwell hardware, TRT-LLM offers the highest available performance ceiling. Breadth of model support, the traditional case for a general-purpose engine, matters less when demand itself has narrowed.
Production now rewards specialization. An engine that handles a long tail gracefully is useful, but a provider earns most of its revenue from the models customers actually request. On a small matrix of popular models and current NVIDIA hardware, peak performance outweighs architectural breadth.
Since mid-2025, more inference teams have stopped maintaining fully independent engines and moved to secondary development on TRT-LLM. Its persistent weakness is usability. The developer experience is rough, but a team paid to extract the last 20 percent from a GPU fleet will tolerate a difficult toolchain. Usability breaks ties; TRT-LLM on Blackwell has no equals to break ties with.
Those teams have not stopped engineering. They still tune models, patch runtime behavior, and build the production systems around the engine. What changed is the layer at which they begin. Starting from NVIDIA's baseline directs more effort toward their workload and less toward reproducing general machinery.
The accelerant: coding agents
Open-source convergence and NVIDIA's vertical push were already shortening the life of proprietary advantages. Over the past six months, coding agents have shortened it further by reducing the cost of inference engineering.
Kernel work, runtime changes, and serving infrastructure can all be completed faster with AI assistance. Intent Lab spent about a week, working agent-assisted on TRT-LLM, to ship optimizations that produced very large end-to-end gains. Previously, work of that scope might have occupied a dedicated engineer for a quarter.
The economics of proprietary engine work change with that speed. A technique that required six engineer-months and bought nine months of exclusivity could justify the investment. If it takes two weeks to build and competitors reproduce it in three, the result is not a moat; it is a treadmill. The work remains technically hard, but the useful life of the advantage is approaching zero.
What matters economically is how long the lead lasts. A difficult optimization can still be commercially weak if it diffuses before the company recovers the cost of building it. Coding agents do not make the engineering trivial; they make exclusivity expire faster.
What's left for the open engines: community, usability, and very little loyalty
vLLM, SGLang, and other open-source engines serve the market left open by TRT-LLM's rough developer experience. Many users outside providers with dedicated inference teams are researchers or run offline generation: throughput-oriented batch workloads rather than latency-sensitive online serving. In those settings, the performance gap with TRT-LLM is modest, often negligible.
Those users optimize for a different workflow. They need to bring up a model quickly, change architectures without rewriting the stack, and find answers when something breaks. A few percentage points of throughput rarely justify days spent fighting a runtime, especially when the workload has no interactive latency target.
Adoption turns instead on ease of use, documentation, and community. Install the package, point it at a Hugging Face repository, and expose an OpenAI-compatible endpoint. For a researcher or a batch-generation pipeline, that is the whole purchase decision.
Adoption is not loyalty. Standardization on the OpenAI-compatible API reduces an engine switch, in the simplest case, to a change in base_url. A team can run vLLM today, try SGLang tomorrow, and benchmark both later in the week. Open engines must keep competing for workloads they have already won.
Users benefit from that portability; projects seeking durable control do not. Community size can attract a workload, and documentation can keep it for a while, but neither prevents a team from rerunning the comparison when a competitor releases a faster version.
The real switching costs appear in online production. They come from monitoring and alerting, deployment pipelines, disaster recovery, automated failure recovery, accumulated bug fixes, and the long tail of production edge cases. Tool-call parsing alone supplies plenty of them. This operational layer creates lock-in, but it is migration friction rather than a capability moat. A competent team can reproduce it around another engine in weeks. More important, that knowledge belongs to the user's SRE organization, so the engine project captures none of it.
Commercially, operational knowledge makes a deployment sticky without giving the engine vendor pricing power. The user bears the migration cost and owns most of the surrounding systems. Even where replacement is annoying, the engine itself has not secured the account.
Open source as a public good
The major open-source engines have largely accepted this role. vLLM and SGLang return nearly all of their work to the community. Their strategic aim is adoption; the result is a steadily better free baseline in features, performance, and stability. In practice, the ecosystem is subsidizing state-of-the-art inference for everyone.
There is a cost to the projects themselves. Each improvement to the baseline shrinks the room for differentiation, including for in-house engines and the open engines that made the improvement. By raising the floor, these projects also compress the value of their own layer.
The tell: engine companies are moving up the stack
The behavior of engine authors offers the clearest evidence that an engine alone cannot capture much value.
The companies behind the two leading open-source engines have begun taking production serving work, reportedly signing inference contracts with prominent model labs and consumer platforms. Their stated reason makes sense: running production is the fastest way to expose the failures an engine must handle. Dogfooding at scale finds the corner cases.
It also puts those companies in competition with providers that were once their most important users and advocates. The project drives adoption, while serving contracts produce revenue. Those roles are difficult to reconcile when the users of the project sell the same service.
Engine companies are understandably careful about the provider label. Their ecosystem depends on companies that want a neutral upstream project, not a subsidized competitor. Yet once the engine company operates production for customers, the overlap is real regardless of how the work is described.
Moving into serving says more than the accompanying explanation. Engine authors do not expect the engine layer to support a business by itself. The question is whether serving is any more defensible.
Part II: The Present — What Actually Distinguishes Providers
The incumbents' real advantages have nothing to do with kernels
The durable advantages of leading inference providers such as Together AI, Fireworks, and Baseten do not appear on a engine benchmark.
First-mover advantage and brand. When a model lab needs a launch partner or an AI-native startup needs production inference, these companies make the first shortlist. Mindshare sounds soft until it decides contract after contract. Default consideration in a fast market is worth more than a narrow benchmark lead.
The platform. Years of work have accumulated in deployment tooling, observability, enterprise controls, and compliance. A new entrant must rebuild that surface one piece at a time while incumbents continue to extend it.
None of these features wins a public speed chart, but together they decide whether a customer can move a workload into production. They also compound. Each deployment exposes another missing control or failure mode, and the fix becomes part of the platform offered to the next customer.
GPU capacity and its supply chain. This is the hardest advantage and the least discussed. Providers must secure allocations across hardware generations, negotiate with clouds and neoclouds, manage heterogeneous fleets, and plan capacity against uneven demand. Incumbents have learned to do this under load. When a large contract arrives, the deciding question is often not whose engine is faster, but who can bring thousands of GPUs online next month.
Late entrants are responding to those economics. Modal has launched an inference service. Nebius acquired Eigen AI to add serving to its cloud. Inference carries better margins than raw GPU-hours, so GPU clouds are moving up while engine companies move into serving from below. Providers, clouds, and engine companies are converging on the same layer because that is where value currently collects.
Each group begins with a different advantage. Engine companies bring software expertise, GPU clouds bring capacity, and established providers bring customers and operating experience. Their movement toward the same product makes the remaining differences easier to see: distribution, capital, and the ability to run a reliable service at scale.
Why small-batch TPS rankings will fade
The small-concurrency output-speed ranking on Artificial Analysis still attracts attention, and for years it was a fair proxy for engineering quality. It says less with each hardware cycle. The standard API tests use either one request or ten parallel requests. At that load, a provider can combine newer hardware with load-aware speculative decoding such as DSpark, spend more batch capacity on each request while the machine is otherwise idle, and post extraordinary per-user TPS. The number is real but narrow: it shows how fast a lightly loaded endpoint can emit one user's tokens, not how efficiently a fleet serves a business.
Production has a different objective. First, keep per-user TPS above the level the application requires. Then maximize total tokens per minute per GPU (TPM/GPU) without dropping below that floor. Once the user experience is fast enough, another increment of lightly loaded TPS may be worth far less than serving more simultaneous users on the same GPU. Cost per delivered token matters more than the headline rank.
Speculative decoding sharpens the distinction. Verification work that is cheap at low batch can consume valuable batch capacity under concurrency, so a configuration tuned for the highest lightly loaded TPS need not sit on the best production cost curve. The useful result is a Pareto frontier: per-user TPS on one axis and TPM/GPU on the other, with the application's speed floor selecting the operating point and cost per token following from it.
Small-batch TPS is not useless. It establishes an interactivity floor and exposes endpoints that are plainly too slow. Artificial Analysis's own ramped-concurrency testing points in the more useful direction, measuring what the system as a whole can sustain. What will fade is the winner-takes-all reading of the small-batch ranking. Production buyers will care less about who posts the highest TPS and more about how much paid traffic each GPU carries while per-user TPS remains above the required floor. It is the essay's pattern in miniature: the most visible number stops predicting where the money goes.
Day-0 support: the relationship moat tightens
A change in 2026 has strengthened the incumbents further: model developers increasingly partner with inference providers directly.
Day-0 support once ran through open-source engines. Before a launch, a lab would coordinate with vLLM or SGLang; the engine would merge support; providers downstream would pick it up. This gave open engines a central place in the distribution chain.
The old dependence is fading. Major labs now grant inference providers early access before release, sometimes ahead of open-source engine teams. A lab needs more than a merged pull request on launch day. It needs tuned and load-tested production capacity with an SLA. A provider can deliver the package; an engine supplies only its first component.
Early access is a relationship rather than a published technique, so competitors cannot reproduce it from a paper or a kernel. Labs grant it to partners they already trust, reinforcing providers that already have brand and capacity. Open-source engines, meanwhile, are losing their place at the front of the release pipeline. Providers increasingly lead Day-0 work and engines follow.
A successful launch makes a provider more likely to receive early access for the next one, and early access improves its odds again. Unlike an optimization, the relationship does not become available to the rest of the market after publication.
The endgame at the serving layer: a capital game
Once competitors reach the serving layer, they face a common bar. A credible provider must hold its SLA under pressure, deliver top-tier latency, and preserve accuracy. These are difficult operational requirements, and they separate providers from GPU resellers. They are also table stakes: meeting them grants entry, while missing them eliminates a provider.
Shared engines and public techniques pull serious providers toward comparable performance, reliability, and pricing. Above that line, GPU capacity becomes the deciding variable. In other words, the contest turns into a contest for capital.
The flywheel is direct. Capital secures GPU capacity. Capacity enables larger model-lab and enterprise contracts that constrained competitors must decline. Those contracts produce inference volume and ARR. Higher ARR supports a larger financing round and valuation, which pays for the next block of capacity.
A ready fleet also changes the sales conversation. A customer choosing a provider for a major launch cannot rely on capacity that may appear later. It needs machines reserved, networked, and ready. A provider that can commit those resources has an advantage before latency and token price enter the negotiation.
For engineers, this is an uncomfortable outcome. The serving endgame looks less like a software competition and more like capital-intensive infrastructure, closer to early cloud computing, airlines, or telecoms. Operational excellence determines survival; access to capital determines scale.
Operations still separate viable companies from failed ones. Poor utilization, weak reliability, or bad capacity planning can destroy a well-funded provider. Once every serious competitor meets the technical bar, though, another kernel improvement cannot substitute for the balance sheet required to accept the next large contract.
The market has already moved this way. Leading providers have raised large rounds and seen their valuations multiply over the past year. Investors are paying for position in a capacity land grab, not for kernel engineering.
Part III: The Future — Value Sinks Into Concrete
Prediction: they will all buy data centers
The capital logic points toward ownership. At sufficient scale, a company whose cost of goods is dominated by GPUs and electricity loses margin whenever it rents compute or data-center space. Each rented layer transfers part of the gross margin to another balance sheet.
Leading inference providers will therefore buy or acquire data centers, including the facilities and the power contracts beneath them, rather than stopping at GPU allocations or long-term cloud commitments. The war chests they have raised make sense under no other endgame.
The inference provider began as a software company whose engine was the product. It became a managed-service company selling a platform and an SLA. It is now becoming an infrastructure company selling capacity. Across that progression, value has moved from kernels to serving operations and then to steel, land, and power-purchase agreements.
Each step changes the skills and economics of the company. Software iteration gives way to fleet management, procurement, financing, and energy strategy. The interface may remain an API, but the business underneath it becomes heavier and more exposed to utilization.
The story began with the loss of a software moat. It closes with the same companies competing for the oldest moats in the world: land, energy, and capital. The moat was never really in the software.
Still "inference providers," never "neoclouds"
Even after buying data centers, these companies will continue to describe themselves as inference providers rather than neoclouds. Capital markets award a higher multiple to an AI infrastructure company that sells tokens than to a neocloud that rents GPU-hours.
Under the label, they will resemble a new kind of neocloud: the same asset-heavy foundation with a different interface, selling tokens, SLAs, and APIs instead of bare GPUs. The CoreWeaves of the world built the asset base first and are moving toward serving. The Fireworks of the world built the interface first and are moving toward the asset base. Both paths lead to the same corporate shape; the starting point determines the label and the multiple attached to it.
Selling through an API changes the packaging. Tokens bundle software, operations, and capacity into a product a customer can consume directly. GPU-hours expose more of the underlying commodity. Two companies may own similar assets yet receive very different valuations because one has packaged those assets higher in the stack.
They will own the data centers and still call themselves inference providers because the label is worth more than the building.
The real competitor is coming from below
If the destination is a capital-intensive, data-center-owning token business, ranking the current players by their distance from it changes the competitive picture.
Together AI, Fireworks, and Baseten have brands, platforms, and contracts. They still need to build or buy an asset base of facilities, power, and a neocloud-scale supply chain, all constrained by multiyear construction and power lead times.
Nebius already owns data centers and operates its own GPU supply chain. As a public company, it also has a financing channel that private providers can approximate only through repeated large rounds. Its missing piece was the serving layer. The acquisition of Eigen AI brought in software and a team, and a focused year closes most of the software gap because inference has no secrets.
By contrast, a software-first provider cannot compress the physical timetable in the same way. Power interconnects, construction, and equipment delivery follow multiyear schedules. Capital can secure a place in those queues, but it cannot turn them into a software release cycle.
Software can be added to an asset base much faster than an asset base can be added to a software company. Engine commoditization helped providers grow, but it also equips their most dangerous challenger.
The biggest rival facing Together AI, Fireworks, and Baseten is not another provider. It is Nebius, followed by any neocloud willing to climb the stack.
Where the engineering hours go instead
Inference engineering will still matter inside providers, but its allocation will change. Open-source engines make that reallocation possible.
As the free engine baseline improves, providers can move expensive engineering time out of kernels and runtimes. The hours go into infrastructure reliability, which gives an SLA substance; platform experience, which supports renewals; and increasingly RL. Reinforcement-learning rollouts are inference-intensive, so serving expertise transfers directly into RL infrastructure for the growing post-training market.
Providers are not retreating from technical work. Reliability across a heterogeneous fleet, fast recovery under load, and efficient RL rollouts are difficult systems problems. Their solutions remain closer to the provider's customers and operations, making them more useful as sources of differentiation.
The value in the engine layer has not vanished. Open source has folded it into a free foundation, allowing engineering value to migrate into the systems built above it. The strategic question for most companies is no longer whether to build an engine. It is where to spend the time that a free engine saves. The beneficiaries of open-source public-goods work show up in other companies' roadmaps.
The story is not over
The endgame remains unsettled. Vera Rubin, MI455, LPU, and other inference-first hardware will reset the checklist, as every previous hardware transition has done. Each reset briefly reopens room for engine differentiation through new numeric formats, memory hierarchies, and parallelism trade-offs. The techniques then spread and the window closes.
New model architectures and workloads will create similar openings. Agents with large reusable contexts, and RL at production scale all impose new constraints. Engine commoditization is a recurring cycle. The interval between invention and diffusion is shrinking, but it has not disappeared.
During each opening, a fast team can still win meaningful business. Temporary advantages matter when they arrive alongside a major hardware or model transition, because the revenue and customer relationships can persist after competitors catch up. What has changed is the time available to recognize and exploit the opening.
Engines and providers still face two tests. The first is reaction speed: can they catch a transition while the advantage remains available? Every generation has left behind well-run companies that arrived one hardware cycle late. Teams that moved early on disaggregation or Blackwell's NVFP4 economics earned temporary but compounding gains.
The second test is financial: can they preserve margin while ARR grows? A provider can book more revenue and lose more money by buying growth with underpriced capacity. As competition tightens and gross margins approach infrastructure economics, operational discipline becomes a condition of survival. Fleet utilization, power costs, demand forecasting, capacity planning, and the balance between committed and spot revenue will decide the next group of winners. Raising billions is only the beginning; spending those billions well is the craft.
Value in this market keeps moving: from models to engines, from engines to serving, and from serving toward capacity and power. It leaves layers where no durable moat remains. The next winners will recognize the shift before the checklist becomes common knowledge.
