TL;DR: Jev is an innovation in inference engineering which can be applied to any LLM. We “Jevified” a Gemma-4-31B and the API is live with a customer (already!). In this post we show how you can do the same!
Simply put, Jev is a model that is amazing at answering multiple-choice questions at the cost of being lame at answering long-answer type questions.
The crazy part is that its engineering was always right in front of us.
I don’t mean it in a “I had dreamt about Jev a year ago” way. But that, you actually don’t need long training runs to get the efficiency unlock that @typesafeai got.
Of course, the base model still needs to be intelligent for this to work. But more intelligent models are larger and hence run slower.
That is what Autoloops’ Hanoi engine solved: our API for Jevified-Gemma-31B outputs probabilities in under 200ms, even for multimodal inputs.
Read on to find how you can get these performance gains purely through inference engineering and what made our customers shift to Jev.
Here’s our model tackling the classic problem of classifying hot dog v/s not-hot dog (multi-modal Jev!!).
What is Jev?
Inference broadly has two parts: prefill (where the prompt is read and represented in a latent space) and decode (converting that latent representation into output tokens). Most of the model’s intelligent output is available at the end of prefill; decode is just the mouthpiece.
Every token of LLM output is basically the highest-probability choice of words that the decoder thought would make the user happy. But it’s really damn expensive to output long answers one token at a time! That’s the “decode tax” we pay for the cute human-like prose output.
However, most developer pipelines are anyway using LLMs like a smart decision-tree. They only care about taking the right decision while being aware of the confidence score behind it, which is where Jev’s brilliance is.
Can any model be “Jevified”?
Technically, Jev is an inference method that reads directly from an LLM’s prefill output logits, converts them into probabilities over a fixed set of choices, and stops before autoregressive decoding begins.
Most of the speedup comes from 1) skipping the decode entirely and 2) caching the prefill pass of the prompt, and answering questions in parallel.The input prompt goes through a prefill pass and its KV-cache is created. Then, multiple questions can have their own prefill paths in parallel and the answer probabilities can be sampled from their outputs.
We first tried Jevifying a smaller model (Gemma-4B) but realised that the model was extremely confident about everything (probability was 99% when it should be 70%). We had to train a small LoRA adapter to teach it to talk like Jev (available here: https://huggingface.co/kushalpatil/jevify-gemma4-e4b
1# POST http://127.0.0.1:8000/v1/systemone2pip install "jevify[transformers] @ git+https://github.com/kushalpatil07/jevify"3jevify serve --model kushalpatil/jevify-gemma4-e4b
That is the part that Jev has fixed, training the model to output probability for each choice with high accuracy.
But model specific calibration doesn’t align with the bitter lesson; a better approach is to use a more intelligent model and make it work for your use case without training.
That is what we achieved by adding Jev support to our existing Gemma-4-31B hosted on the Hanoi engine. The model is intelligent enough to make the prefill probabilities sensible and our engine is fast enough to attain sub-200ms latencies on multimodal inputs.
Who can use Jev-style models, and what are they good for?
The sweet spot for Jev is anywhere software repeatedly needs to make a fuzzy decision from a finite set of actions.
For a browser-agent, there might be 30 clickable elements and a handful of possible actions: click, type, scroll, wait or finish. @browser_use's jev-ultrafast converts the page into an indexed list of DOM elements, then Jev chooses the action + element. (Browser Use — jev-ultrafast).
Bunch of other cool use cases:
- @cekuraAi has shifted multiple LLM workloads to Jev in their simulations.
- Judge agents without another LLM: @langchain now supports Jev as an evaluator for complete agent traces. LangChain's Jev-as-a-Judge experiment
- Route work to the right model: @higgsfield showed Jev looking at a creative brief and deciding which of its image/video models should handle it. Higgsfield model-routing demo
This is quickly becoming a model category.
- Laya takes the idea to its logical extreme: instead of adapting a decoder LLM, it uses a ~421M parameter ModernBERT encoder purpose-built to answer choice, score, and noul questions in a single non-autoregressive forward pass. Laya
- Kev goes the other direction: take existing Qwen3.5 base models and add a small decision architecture on top like we did with the LoRA adapters. Kev
Which brings me back to the question: do we actually need a special Jev model at all, or can we take models that are already great at seeing, hearing, coding or understanding a specific domain and Jevify them?
Want to be a Jeveloper?
Try out our Jevified-Gemma-4-31B API from autoloops.ai/jev!
