Chasing Microseconds: Lighter's Latency Engineering

Introduction

Over months of iteration, Lighter's end-to-end p99 dropped from 280 ms to a flat 55 ms. Transaction processing p99 went from 20–30 ms spikes to under 1 ms. Hot path apply time sits at 100–250 µs. All on mainnet, at scale.

How much scale? On June 5, 2026, Lighter hit an all-time high of 811 million transactions in a single day, averaging 9,388 TPS with peaks of 20,740 TPS.

Lighter's sequencer is written in Go. Transactions are executed on Lighter L2 with full cryptographic proofs settled to L1. The key architectural insight is that proving is decoupled from execution: proof generation runs asynchronously and never blocks trading. That means the latency traders experience is determined entirely by Lighter's execution and API layers.

Lighter's API layer is the service traders interact with directly. It maintains a complete, up-to-date view of the exchange state in memory and serves REST and WebSocket connections from that local state. A critical goroutine receives state updates from the execution engine and applies them to in-memory caches in real-time. Everything downstream (WebSocket pushes, API reads, transaction simulations) serves from these caches.

Key Concepts Covered

The following concepts provide a high-level overview of the key topics covered in the rest of this technical write-up:

Find it before mainnet: An identical loadtest environment with synthetic accounts and realistic load, instrumented far more aggressively than production, catches bottlenecks before traders notice them.

Dual-environment observability: Loadtest carries metrics too expensive for production; mainnet is monitored in real-time for freshness, simulation latency, and end-to-end transaction lifecycle. Together, they give complete coverage.

Death by a thousand allocations: Systematic replacement of heap-heavy types and deep copies with stack-allocated alternatives and immutable snapshots. In a GC'd language, fewer allocations = fewer pauses = lower tail latency.

OS-level control: Pinning the vital threads to dedicated CPU cores with near real-time priority.

Hardware-aware deployment: NUMA-aware placement ensures the hot path has fast, local memory access. The final layer in the latency stack.

Binary over convenience: Hand-rolled binary serialization on the hot path eliminates reflection and allocation overhead. Every microsecond saved compounds across thousands of updates per second.

Finding Bottlenecks Before They Hit Mainnet

You can't fix what you can't see. Before optimizing anything, we invested heavily in tooling to identify exactly where time was being spent.

The Loadtest Environment

When we want to test something, we spin up a dedicated loadtest environment: an identical copy of the mainnet infrastructure running the services with identical configs under the same deployment topology. It's not always running; we bring it up on demand for a specific test and tear it down afterward.

Once this environment is up, we create synthetic accounts and generate realistic trading load to simulate real market conditions under stress. This allows:

• Extra-granular timing: Per-step timing inside the hot path, per-operation cache build durations, and transaction lifecycle timestamps tracking every stage from submission to confirmation.
• On-demand profiling & flight recorder: We capture CPU, memory, and execution trace profiles on request during load. Go's flight recorder gives us always-on trace collection. When a slow event is detected, the last few seconds of execution are captured automatically, letting us diagnose transient latency spikes after the fact.
• Distributed tracing: We instrumented every major function in the hot path with trace spans, giving us fine-grained visibility into exactly where time is spent within a single update cycle, across service boundaries.

Dual-Environment Monitoring

We monitor both environments closely (mainnet and loadtest) but at different levels of granularity.

Mainnet is tightly monitored in real-time. We track everything that matters to the trading experience:

• Freshness tracking: We measure latency through the \order_book\ WebSocket channel on the most active markets. Essentially, the drift between when an order book update is produced and when a client receives it. This is the metric that most directly reflects the experience traders feel: how stale is the order book you're looking at? We track this at two points. The most important is execution engine → client: the full pipeline from when the execution engine processes a state change to the moment the resulting \order_book\ update lands on the client. We also measure API layer → client: the API layer timestamps each outgoing update and the client compares it against its own clock, giving us the last-mile latency in isolation.
• Dry-run latency: The API layer dry-runs every transaction (validating signatures, nonces, and balances) before passing it to the execution engine. Every dry-run is timed and reported.
• End-to-end transaction lifecycle: Full-cycle latency histograms from submission to confirmation.
• Cache effectiveness: How often we serve from memory vs. falling back to slower storage.

Loadtest is monitored even more granularly. Because it's not serving real traders, we can crank up the instrumentation without worrying about overhead: per-step timing inside the hot path, allocation profiling, extra histogram buckets on every cache mutation. This finer granularity catches micro-level regressions that mainnet's production-safe metrics wouldn't reveal.

Mainnet monitoring tells us how the system is performing for real traders right now. Loadtest monitoring catches regressions before they reach production.

Deep Copy Elimination & Heap Allocation War

Go's garbage collector is a latency tax. Every heap allocation eventually becomes a GC pause, and GC pauses on a trading engine are latency spikes for traders. So we went through the hot path and cut allocations wherever we could. The result was fewer latency spikes and more predictable performance.

Deep Copy Elimination

Deep copies are allocation factories. We went after them in a few ways:

• Immutable snapshots: Made order book caches immutable. Reads return a pointer to the current snapshot, no copy needed. Updates create a new version and swap it in atomically via \atomic.Pointer\.
• Removed unnecessary copies: Found code paths that deep-copied data that was never subsequently mutated. Removed them entirely.

Allocation-Conscious Design

• Stack-allocated numerics: Replaced heap-heavy \big. Int\ and \big.Rat\ with stack-allocated alternatives (\int128\, \float64\, \int64\ division) across the hot path: price conversions, order book depth maps, size calculations. Up to 8.3× faster on key formatting functions, zero heap allocations per operation.
• Conditional updates: Skip allocation when nothing changed.
• Pre-sized collections: Eliminate grow-and-copy cycles by sizing data structures up front.
• Buffer reuse: The subscriber path deserializes thousands of updates per second. We reduced allocations with pool-based reuse of intermediate buffers.

This flattened the tail. Before, Lighter's end-to-end p99 (execution engine to client, measured on the \order_book\ WebSocket channel) would spike to 200–280 ms during allocation-heavy periods. After the deep copy elimination, immutable caches, and heap allocation work, p99 settled to a flat ~50–60 ms band with virtually no spikes:

End-to-End Latency: Execution Engine -> Client (p99)

GC pressure on the API layer also dropped measurably, but this wasn't just one change. We fundamentally reworked how API server caches operate. Previously, caches had TTL-based expiry and allocated new objects on every update. Each of those short-lived allocations became GC work. After the rework, the full exchange state is hot-started from a snapshot and kept in memory as long-lived, immutable structures that get swapped atomically. That removed the TTL churn and the per-update allocations, and GC pressure dropped with them. We monitor memory usage closely and the working set is bounded.

We ran the old-style apiserver and the reworked apiserver side by side on mainnet traffic. The reworked server's GC pause duration (p75) sat around ~3 ms, compared to ~5–6 ms on the old server. Roughly half the GC pause time:

GC Pause Duration: Standard vs. Snapshot API Server

The frequency tells an even clearer story. The old apiserver triggered GC cycles ~2.2× more often than the reworked one, directly reflecting fewer short-lived allocations and less GC pressure overall:

GC Cycle Frequency: Standard API Server vs. Snapshot API Server

GOGC Tuning on the Execution Engine

We applied similar GC thinking to the execution engine itself. Go's \GOGC\ parameter controls how aggressively the garbage collector runs. The default trades CPU time for memory efficiency, but for a latency-critical path, the trade-off was wrong.

After tuning \GOGC\, the execution engine's GC duration dropped from an average of ~30 µs with spikes hitting 100 µs, down to a stable ~10 µs band. A ~3× reduction with virtually no spikes:

Execution Engine GC Duration After GOGC Tuning

Three months of mainnet data confirms the improvement held: the after period is flat and predictable.

Transaction processing improved with it. Over 90 days of mainnet data covering every transaction type (create order, cancel, liquidate, deleverage, transfer, and more), p99 went from frequent 20–30 ms spikes before GOGC tuning to mostly under 1 ms after, with occasional 3–4 ms outliers. The improvement has held for over two months:

Transaction Processing Times - p99 (Last 90 days)

Snapshot Service — Less Data, Hot Deploys

After we eliminated the allocation overhead, we tackled the next bottleneck: the volume of data flowing through the internal message bus. We built a snapshot service that maintains full in-memory state snapshots. This had two major effects:

• Less data on the wire. With the snapshot service holding the full state, the execution engine no longer needs to push complete state through the message bus on every update. It writes less data, which means less network bandwidth consumed and less deserialization work on the receiving end.
• Zero warm-up deploys. On deploy, the API layer boots from a snapshot (accounts, order books, market info, API public keys) and is immediately ready to serve. There's no warm-up period while caches fill. After loading the snapshot, the API layer subscribes to the update stream and applies deltas in real-time.

On top of the snapshot base, in-memory caches are updated continuously:

• Account info: Lock-free reads via \sync. Map\, atomic pointer swaps for updates.
• Order books: Stored as immutable snapshots. Reads get a pointer, updates swap in a new version. No locks on the read path.
• API key cache: All keys fit in memory. Eliminated external lookups entirely.

The result: the entire exchange state lives in local memory, updated in real-time, and every deploy starts hot.

The knock-on effect on the execution engine was significant. Previously, the execution engine was writing cache keys across the network, updates that services would read from time to time. With everything living in memory and the snapshot service handling state distribution, those network writes became unnecessary. We deleted them. The result: block times p99 dropped from ~2.6 ms to ~1.2–1.8 ms, simply because the execution engine writes far less now:

Block Times - p99

CPU Planning

Once we optimized everything in userspace, we reached for the kernel.

We have many goroutines and they need to be scheduled on the CPU to run. Minimizing scheduling overhead is important for low-latency systems. By default, Go's runtime multiplexes goroutines across OS threads, and the OS can migrate threads across CPU cores freely. Both introduce unpredictable latency.

We eliminated this by stacking four mechanisms:

1. `runtime.LockOSThread()`: Locks the goroutine to a single OS thread, preventing Go's scheduler from migrating it.
2. CPU affinity via `sched_setaffinity`: Pins that OS thread to a specific CPU core (Linux). This prevents the kernel from migrating it between cores, avoiding L1/L2 cache invalidation.
3. High-priority scheduling via SCHED_FIFO: Elevates the thread's scheduling priority, ensuring the kernel favors it over other work.
4. Busy-wait spin loop: The hot path runs a \select\ with an empty \default\ case, so the goroutine never parks. Without it, Go moves the goroutine to a "runnable" state when no data is available, and rescheduling adds wake-up latency. With the spin loop, the goroutine stays running on its pinned core and picks up new updates with zero scheduling delay.

Measuring the impact correctly matters here. Apply time varies with traffic conditions, so absolute numbers shift with load. To isolate the effect of pinning, we ran two groups of API servers side by side under the same traffic. One group stayed unpinned as a control, and we switched the other to CPU pinning. The percentage difference between them, measured at the same time under identical load, tells the real story.

Before pinning, both groups track together. Same base latency, same spike behavior:

Hot Path Apply Time - Before CPU Planning

After enabling pinning on one group, it consistently sits below the unpinned baseline. Same traffic, lower latency. The pinned group's spikes are also capped lower, because thread migration jitter and L1/L2 cache invalidation are eliminated:

Hot Path Apply Time - After CPU Planning

NUMA-Aware Deployment

CPU pinning alone isn't enough if the pinned core's memory accesses cross NUMA boundaries. A NUMA node is a group of CPUs with their own local memory. Accessing memory from a remote NUMA node carries a 10× penalty compared to local access.

Lighter's API servers were originally running on larger machines with 2 NUMA nodes:

1$ lscpu | grep NUMA
2NUMA node(s):                            2
3NUMA node0 CPU(s):                       0-95
4NUMA node1 CPU(s):                       96-191
5
6$ cat /sys/devices/system/node/node0/distance
710 100

The distance matrix tells the story: local access costs 10, cross-node access costs 100. A 10× penalty. With N API servers sharing the machine, some inevitably had their pinned CPU on one NUMA node and their working memory (in-memory caches, update buffers) on the other. Every hot path iteration was paying the cross-node tax.

The fix was counterintuitive: we moved API servers to smaller machines with a single NUMA node. Half the spec, but all memory accesses are now guaranteed local:

1$ lscpu | grep NUMA
2NUMA node(s):                            1
3NUMA node0 CPU(s):                       0-95

This cut costs and improved latency at the same time. The hot path apply time dropped further to the ~100–250 µs range, with off-peak dipping to ~100 µs. Compare this to the ~200–520 µs range with CPU pinning alone on the 2-NUMA-node machines:

Hot Path Apply Time - NUMA Optimization

Custom Binary Serialization

Lighter's execution engine publishes state updates to the API layer through an internal message bus. Every system update flows through this path. The original serialization used a general-purpose encoding library, but reflection, type switches, and per-field allocations created unnecessary overhead on a hot path.

We replaced that with hand-rolled binary serialization: fixed-layout, zero-reflection encode/decode for every entity type in the system. Each type has a dedicated encoder/decoder that reads and writes fields at known byte offsets. No reflection or type switches, and minimal allocations. Every codec has round-trip and fuzz tests to catch regressions.

The result was significantly less serialization overhead on the path that feeds every downstream cache.

Independent benchmarking is available through a dashboard built and maintained by community member @UngusTrade , which compares live transaction latencies across perpetual trading venues: latency.perps.trading