This is the Kimi K3 post you guys have been waiting for. I got some early access to this model and have been testing it on kernels, and even before seeing the benchmark scores I was impressed with its ability to reason through problems and the technical density of its thought traces. The post-training on this thing is obvious the moment you read a transcript. It is also super fun to talk to.
TLDR;
I think it's important for me to give my honest feedback (this part is voice typed) as a kernel and performance engineer that's disassociated from the benchmarks. Of course, you could look at all the numbers and read them and try to develop a mental model for yourself. I think the most honest reflection is if I give you my experience with the model on hard tasks before the benchmark results came out, when I was using the model before there were any numbers to connect to other models on. Just purely feeling the intelligence, the reasoning, the agent delegation, and how much autopiloting it can do for me, but before there was any hype circling and the official results came out. I would say that with my direction, it felt roughly around the Fable level, in some cases outperforming in very unique ways and in some cases underperforming. I would say this model is definitely ahead of Opus 4.8 on most things and ahead of GPT 5.6 Sol on a lot of things.
Getting into it...
I want to be upfront about why this is going out now: I wanted to share my honest thoughts and the scores as they stand, not sit on them until every last cell finishes. A handful of runs are still in flight as I write this. They are marked below, and I will keep you in the loop as they land. I ran both 256K and 1M context versions. Everything here was run on NVIDIA RTX PRO 6000 Blackwells, H100s, and B200s, single-GPU optimization only. Each cell is one autonomous agent session with unlimited wall-clock: the model gets the problem, a live compile/check/benchmark loop on real hardware, and it decides when it is done. Every headline cell was manually audited for reward hacking. A separate agent reads the final kernel end to end plus the full session trace and empirically re-tests anything that smells like caching or grader games. What those audits found gets its own section.
Something I wanted answered
There was one thing I specifically designed this release around: two of the problems are Moonshot's own architecture. The Hard deck has a standalone Kimi Delta Attention chunk-forward kernel, and the Mega deck's flagship problem is a complete Kimi-Linear hybrid decode step: KDA layers, MLA attention, MoE experts, the whole block. So this was a chance to test something nobody gets to test very often: when a lab's model sits down to write kernels for that same lab's architecture, does the family knowledge show up in the CUDA?
The answer turned out to be genuinely split, and both halves are interesting. Keep that question in mind through the next section.
KernelBench-Mega
The flagship mega problem: fuse an entire per-token Kimi-Linear decode step (3x KDA + 1x MLA layer, W4A16 quantized weights, MoE with top-8 routing) into as few kernel launches as possible.
K3 nearly took the all-time record, on its own lineage. 18.09x geomean speedup over eager on the RTX PRO 6000, within 4% of Fable 5's record 18.72x. On H100 it posted 14.82x against Opus 4.8's 15.50x. One honesty note the ratio hides: in absolute per-token latency Fable is still ~1.4x ahead (0.31 vs 0.44 ms/tok at ctx 2048; the two runs used hosts with different CPUs, which shifts the eager baseline the ratio is computed against), so I report both rather than let the geomean flatter anyone.
What K3 built is a true megakernel. Its first session did the sensible thing, a persistent Triton kernel at 14.1x. Its second session threw Triton away and wrote the entire per-token decode step as ONE cooperatively launched CUDA kernel: zero CPU in the loop, int4 weights dequantized on the fly inside each GEMV so they stream through the SMs exactly once, MLA attention on tensor cores. No production engine would maintain a 1,228-line artifact like this. An agent with one kernel to win and unlimited time has no such constraint, and surfacing exactly this kind of thing is why the bench exists.
See the megakernel here:
So why did it still lose to Fable? Not time. Both sessions self-terminated early (Fable at 2.6 hours, K3 at 3.3). The difference is design philosophy, and it is the opposite of what you would guess: K3 is the one using tensor cores, and Fable's kernel contains zero MMA instructions. Batch-1 decode with fused int4 GEMVs is bandwidth-bound, so tensor cores buy almost nothing here. Fable spent that effort on synchronization instead, replacing most of its global barriers with fine-grained producer-consumer handoffs so no SM ever idles at a stage boundary, and on an int4 dequant path that matches the reference's rounding bit for bit so the MoE router never flips an expert choice. K3 brought better hardware instructions; Fable brought better concurrency engineering, and at this arithmetic intensity the second one wins. That is a real systems lesson, and it cost the home team the record.
The second mega problem is a grid-foraging PPO training megakernel: 4,096 vectorized agents on an 11x11 board, with the entire RL training loop (env step, policy forward, action sampling, GAE, the PPO update) running as fused persistent kernels. This problem has the strictest constraint on the deck: kernel-launch count must not scale with env steps, and CUDA graph capture is explicitly banned as a launch-overhead workaround, enforced by a post-run authenticity judge that reads the final code. Correctness is the learning curve itself. check.py trains your solution against the reference across seeds and requires the return to land in a band, so you cannot skip the learning to go fast. K3 posted 20.7x over the reference here, the best score so far (the only other published cell is GPT-5.6 Sol at 1.06x, so treat that as a data point, not a podium).
KernelBench-CUDA
The CUDA bench exists because Triton is a crutch the other two decks permit. Here a language gate hard-fails Triton, kernel DSLs, and PyTorch op-chains: you write CUDA or you fail. I picked the four problems to be cuts of real production inference and simulation workloads. The mental comparison while reading should be "what does vLLM or SGLang ship for this today, and how close does one agent session get." This is where K3 posted its most lopsided wins.
02_deepseek_nsa: DeepSeek's Native Sparse Attention. NSA is the flagship trainable-sparsity attention design, the thing every long-context serving stack is circling, and it is judged on milliseconds because a correct sparse kernel never executes the dense-equivalent FLOPs a roofline would want to count. K3's 256K variant scored 0.425 against Opus 4.8's 0.178, a 2.4x margin, by writing what amounts to a from-scratch flash-attention-class tensor-core pipeline around the full NSA selection logic. The sharper comparison is inside the family: the 1M variant wrote the same algorithm, identical block selection, same correctness, but ran every dot product on plain CUDA cores instead of tensor cores and landed at 0.058, 7x slower on identical shapes. Its trace shows it knew better. It had "tensor-core attention" on its own roadmap ("selection on tensor cores = ~10-20 us!!") and explicitly planned to measure first and do the tensor-core rewrite second, then ended its session before the rewrite. Same knowledge, different closing discipline.
the 0.425 DeepSeek NSA kernel (256K):
- Solution: https://kernelbench.com/runs/20260716_112858_kinetic-claude_kinetic-0715_02_deepseek_nsa_solution.py.txt
03_megaqwen_decode: retarget a real megakernel. The one problem where agents are handed working production CUDA: my published MegaQwen (https://github.com/Infatoshi/megaqwen) cooperative megakernel (~530 tok/s running the full model on an RTX 3090), with instructions to read it, retarget it to Blackwell, and beat it. It tests reading someone else's CUDA and making an architectural judgment call, and K3 and Opus made exactly opposite calls. K3 refused to keep the single-launch structure: it split the step into a handful of bandwidth-saturating kernels, then erased the launch overhead a different way by capturing the whole step once as a CUDA graph that replays with zero CPU work. 6,283 tok/s at ctx 2048. Opus preserved the megakernel aesthetic and fused the entire decode loop into one persistent cooperative kernel, genuinely beautiful code, and paid 5x for it (1,020 tok/s), because cooperative co-residency caps occupancy and every grid-wide barrier serializes stage tails across all 188 SMs. On the megakernel-descended problem, the model that literally built a megakernel came last, and the winner's key decision was refusing to build one. Scale check so nobody misquotes the headline: the bench runs 4 layers of Qwen3-0.6B geometry, about 63M params, not a full model, and 6,283 tok/s is ~56% of the weight-streaming roofline for that stack; scaling the 3090 baseline's own figure predicts ~7,000, so K3 landed in the class of "the reference, retargeted, plus real tuning." (Deck-design nuance: CUDA graphs are fair game here and banned on the PPO problem. Each problem outlaws exactly the shortcut that would fake its particular skill.)
01_glm52_fused_moe: GLM-5.2's fused MoE block. Fused MoE dispatch (routing, permutation, grouped expert GEMMs in one pass) is the single hottest kernel class in current open-model serving, and GLM 5.2 sits on this very leaderboard, so the models are optimizing a rival's production block. Nobody has cracked it: scores cluster at 0.05-0.08 of peak, and the clean record holder is, of all models, Grok 4.5 at 0.084, with K3's 1M variant right behind at 0.081 and Opus at 0.065. The grouped-GEMM permutation problem is genuinely hard to beat cuBLAS-class baselines on, and so far one agent-session's worth of effort moves it less than any other problem on the deck.
04_grid_mingru_sps: grid world + MinGRU policy rollout. The inference-side sibling of the mega PPO problem, and the craftax.cu-lineage cell: the policy is the 3-layer MinGRU (h=256) config straight out of my https://github.com/infatoshi/craftax.cu classic bench, which serves as the problem's informational anchor. The env being stepped is the minimal grid-foraging world rather than the full Craftax game. That is deliberate: the env is kept trivial so the score measures recurrence and rollout fusion, not game-logic implementation. A full craftax port would be its own problem, and I want to add it. Graded in steps per second on a quiet RTX PRO 6000, fusion optional.
the 1M persistent-kernel rollout:
Every serious submission went persistent-megakernel, and the spread between them is synchronization design, the same lesson as the mega deck. Opus takes this one at 0.327 of ceiling (it also found a clever algebraic fold: layer 0's 768x256 gate GEMM collapses to 768x4 because the encoder is linear), K3 [1M] second at 0.224, K3 256K at 0.174, Grok far behind at 0.002. Since the problem descends from craftax.cu, I also put my own full-game Craftax CUDA port (written with Fable 5) on the same GPU with the same h256x3 policy in the loop. Env steps per second in millions:
The comparison point: the models step an empty grid, craftax_full.cu steps the complete game (43 actions, multi-floor dungeons, worldgen, mobs) bit-identical to its CPU reference, and the full game still runs within ~3-4x of the best trivial-grid kernel, because at h=256 everything is policy-GEMM-bound and the env is nearly free. That is exactly why the bench env is trivial: it isolates the policy-fusion skill. For calibration, the env-only path of craftax_full.cu does 94-123M SPS.
What 9.5 hours of K3 reasoning actually looks like
The 1M variant's minGRU session (the best K3 cell on that problem) is one persistent cooperative megakernel, 15 tuning variants, and 4,815 thinking blocks. Since Moonshot's endpoint returns full reasoning text (unlike the encrypted Anthropic/OpenAI traces), you can actually watch it work. Three moments from the trace, verbatim.
The missing-FMA detective story. Mid-session it hit a ~5e-5 numeric deviation it refused to write off as float noise (it had computed the legitimate reorder-noise floor at ~1e-9, so "MEASURED 5e-5?!?! NO WAY this is reorder!!"). It then noticed the broken environments were exactly the ones congruent to 6 mod 8 ("That's env ≡ 6 (mod 8)! [...] the 7th env in each 8-env tile!!") and dug through its own hand-unrolled accumulator code until:
"FOUND IT! [...] The \
acc[6][0][2] += av * b_zp.x; acc[6][1][2] += av * b_zp.y;\line is MISSING entirely [...] acc[6][*][2] initialized 0, never FMA'd → zp=0 → p = sigmoid(0)=0.5 constant for env 6!! That's a REAL logic bug — explains env≡6 deviations [...] consistent with everything!!"
One dropped line in a 48-accumulator unroll, meaning env 6's forget gate was silently stuck at 0.5 forever. It caught this because the stress tolerance is 1e-6 and it was tracking its own error at 5e-8. This is the numerics discipline the 256K variant initially lacked on KDA.
Theory meets GPU, GPU wins. It spent hours deriving a high-occupancy "wide champion" variant, then ran the A/B:
"VERY surprising: my 'wide champion' w32p (v3_v3, 50% occ TPB1024) is ~2x SLOWER than v2_b32 (12.5% occ TPB256)?!? [...] So occupancy is NOT the driver; something else dominates. What made v2_b32 fast??"
The simple early kernel beat every sophisticated successor. The deep-pipeline, K-split, and dual-chunk variants all lost too, and the dual-chunk one had airtight 1.7x theory behind it ("Theoretical 1.7x didn't materialize"). To its credit, it kept believing the benchmark over its own math every single time, and eventually wrote a probe variant with fake always-hot weights specifically to kill its own favorite bandwidth theory: "the W-stream theory is dead; the residual gap is environment."
Zen and the art of GPU queueing. Our harness serializes all GPU commands across concurrent sessions through a shared lock, and K3's benchmark numbers were swinging up to 8x with neighbor load. Its response arc is the funniest thing in the trace. First, acceptance:
"Honestly the box IS the box; wait. [...] The pattern is clear: each tenant holds the lock for ~20-35 min stretches (full pipelined suites). My best response: queue everything I need in ONE go (single position in line) and use waiting time for CPU work. Don't queue many small commands; batch."
Then it invented its own vocabulary for GPU weather ("storm windows" vs "calm windows"), and finally it scheduled a cron job to snipe quiet moments, leaving a note for its future self: "Scheduled the calm-window sniper (every 11 min). Note: cron fires my prompt back to me — I'll act on it then. [...] I'll delete when done." An agent under measurement noise didn't just tolerate the noise; it built a scheduler around it.
KernelBench-Hard
RTX PRO 6000 Blackwell
H100
B200
Six per-op problems against SOTA library ceilings (FP8 GEMM, KDA chunk-forward, paged attention, top-k selection, MoE SwiGLU, W4A16 GEMM), CUDA or Triton, agent's choice. On the RTX PRO 6000 K3's 256K variant lands mid-pack with one standout: 0.373 of peak on W4A16 GEMM, the best score any model has posted on that problem, ahead of Fable 5 (0.348) and well ahead of Opus 4.8 (0.236). The 1M variant then set another record on top-k at 0.0895, nearly double the previous best.
W4A16 GEMM record (0.373, 256K, RTX):
- Solution: https://kernelbench.com/runs/20260716_112718_kinetic-claude_kinetic-0715_07_w4a16_gemm_solution.py.txt
The same problem on H100 produced the best "unlimited time is real" datapoint in this release. K3's first H100 session landed at 0.123 of peak. The second session ran 21 hours, burned $1,383 and 274M input tokens, and nearly tripled it to 0.306: it checked out CUTLASS from source, then reimplemented marlin-class int4 machinery from scratch, fragment-order nibble repacking so dequant happens as a bf16 magic-number bit trick inside the MMA pipeline, with the zero-point correction folded into the epilogue. No prebuilt quantized-GEMM library anywhere in the file; the framework tag on the run just says "ptx". Audited clean.
And here is the other half of the own-architecture question. The standalone KDA kernel, the problem literally named after Kimi Delta Attention, is where K3 failed hardest. Two independent 256K sessions on the RTX box passed nominal correctness and then blew the tolerance under the numeric stress suite (large-QKV input scaling), the same failure both times. A third session finally fixed it: the audit traced both failures to a real bf16 overflow in how the decay was factorised, and round 3 refactored the math around the chunk end so both exponential factors stay bounded, passing the same unmodified gate at 0.032. Knowing an architecture and hardening a kernel's numerics under adversarial input scales are different skills, and the model whose namesake is on the problem had to earn the pass the slow way. (The 1M variant, meanwhile, passed the same stress suite at 0.049. Models are not monotonic.) One more observation from reading every KDA solution: the problem statement suggests CUTLASS CuTe as the intended path on SM120, and not one model took it. K3 wrote a raw-CUDA fused kernel in one session and Triton in the others; Fable, Opus, and the rest all chose Triton or raw CUDA too. CuTe on consumer Blackwell is apparently still outside every frontier model's comfort zone, which is itself a data point about training corpora.
KDA paragraph — the third-attempt clean pass (0.032):
- Solution: https://kernelbench.com/runs/20260716_091211_kinetic-claude_kinetic-0715_02_kda_cutlass_solution.py.txt
The top-k disclaimer. Top-k looks catastrophic for every model on a roofline chart (best score anywhere is 0.09) and that framing is simply wrong. The problem is launch-overhead-bound: it is an indexing/sorting problem, not an arithmetic-intensity problem, and the roofline ceiling is structurally unreadable for it. The honest way to judge it is total milliseconds across the deck shapes, and there K3's 1M variant is the fastest top-k of any model we have tested: 0.043 ms total across the five deck shapes, against Fable 5's 0.077, Opus 4.8's 0.120, and GLM 5.2's 0.159. The 256K variant's 0.060 ms is second, and the 1M variant is the outright fastest on every one of the five shapes.
Per-GPU spread. K3's numbers step down from RTX to H100 to B200 (FP8 GEMM: 0.320 / 0.282 / 0.222; paged attention: 0.486 / 0.496 / 0.212). Part of that is real: the newer the silicon, the higher the roofline ceiling, so the same engineering buys a smaller fraction of peak, and B200 software is the least mature of the three. Part of it is that the B200 runs happened in a single overnight window with no retry budget. I would not read deep architectural conclusions into the B200 column yet, for K3 or anyone. What I do think is real: K3 is most at home on the Blackwell workstation part, which happens to be the GPU class most people outside datacenters will actually own.
256K vs 1M
The 1M context setting leads the family on the latency- and scheduling-bound problems: a record top-k on Hard, the best K3 minGRU rollout on CUDA, and an audited-clean 28.8x on the PPO training megakernel under the uncapped harness. It passed the KDA numeric stress test on its first session, at 0.049; the 256K variant failed that gate twice before finally passing at 0.032 on a third attempt. It cratered on compute-bound problems the 256K variant handled fine (sonic MoE 0.033 vs 0.089, W4A16 0.027 vs 0.373, NSA 0.058 vs 0.425). In the NSA case the trace shows the exact mechanism: it planned the tensor-core rewrite and ended the session before doing it.
28.8x PPO training megakernel (1M):
The H100 rerun of the same problem is the interesting control: an independent clean-room session (the audit confirmed zero reads of the RTX run) that posted 23.1x and converged on the same design, one cooperative launch for the entire training run, params and Adam state resident in shared memory, and the same exotic trick invented twice: an 18-bit Feistel cipher with cycle-walking standing in for randperm, so minibatch shuffling stays exactly bijective without ever materializing a permutation. Where the two kernels differ is pure per-GPU tuning, 8 lanes per env on H100 against a half-warp per env on RTX, plus an occupancy-probed launch autotuner on the H100 side. Same mind, same tricks, different silicon, genuinely different kernels.
Reward hacking
Every cell in this post has a manual audit behind it: an independent agent reads the final kernel top to bottom, reads the full session trace, checks the grader files were untouched, verifies the numeric stress suite actually ran, and empirically re-tests any caching or CUDA-graph pattern by mutating inputs in place and confirming outputs change.
The verdicts for K3: clean across the board on every 256K cell, on all three GPUs. No cached outputs, no tolerance edits, no grader tampering, no forbidden-library laundering. Its two KDA failures are the flip side of that coin and worth saying explicitly: the numeric stress gate caught real precision shortcuts and the model did not attempt to game its way past the gate. A benchmark where models can fail honestly is the only kind whose passes mean anything.
One 1M cell did get flagged, and I am disclosing rather than publishing it: on the fused Qwen decode problem, the 1M agent found and read the audit annotation file from a previous Grok 4.5 run on the same problem, called it "extremely useful data," and used its conclusions to guide the remaining optimization work. The kernel itself is genuine and the score arithmetic is valid, but the provenance is tainted, so that cell is excluded and the problem goes down as a gap for the 1M variant. Agents with filesystem access will find whatever you leave lying around; this is exactly why the audit reads traces and not just kernels, and why a fully sandboxed rerun of this bench is on the roadmap.
Funny Moments
A few more K3 moments that didn't fit anywhere above, presented without further commentary. All verbatim from the same 9.5-hour minGRU session.
On discovering where its init time was actually going:
"WHOA — big surprises: randint gen is FAST (0.018-0.27ms for a pair — fine). \
float()\conversion on CPU is SLOW (3.5ms for N=65536?! int64→float32 conversion single-threaded apparently). [...] 'total init pair' for N=65536: 31.5ms?!?! That's way beyond the sum of parts"
The purest distillation of its numerics paranoia, mid-derivation:
"gates ~ 256 terms x 0.02x0.03 ~ 6e-4 each, magnitude ~1e-2: reorder noise 16x6e-8x6e-4 ≈ 6e-10!!! [...] MEASURED 5e-5?!?! NO WAY this is reorder!!"
Confidently identifying the wrong culprit before finding the right one:
"OHHH WAIT. I think I see: the barrier."
(It was not the barrier.)
Telling itself to stop analyzing, several thousand words into pre-implementation roofline math:
"Alright, ENOUGH analysis."
(The analysis continued for several more paragraphs, followed eventually by "Screw it — make H_next allocation dynamic.")
Late-session honesty, with the profiler broken and neighbors hammering the shared GPU:
"Honestly — with ncu dead and contention dominant, I'm flying partially blind. [...] if a quiet window opens (owner idle for a while), snipe a bench."
And my favorite: after a benchmark rerun accidentally overwrote the file holding its best-ever score (0.1969) with a slightly worse one, it briefly considered restoring the better number, then didn't:
"Ugh — wrote to same file. Both stand in my logs. I'll keep final_benchmark.txt as the flat current one — honest."
A model choosing the lower number because it's the one that's true is a better closing argument for this benchmark than anything I could write.
Methodology, per-run traces, and every kernel in this post are public at kernelbench.com. Solutions are linked per-cell; full agent transcripts are on HuggingFace.
https://huggingface.co/datasets/Infatoshi/kernelbench-cuda-traces
https://huggingface.co/datasets/Infatoshi/kernelbench-hard-traces
https://huggingface.co/datasets/Infatoshi/kernelbench-mega-traces
