In Men in Black II, Agent J visits K at a seemingly ordinary post office. He opens a mail-sorting machine and reveals an alien inside, rapidly sorting the letters.
It is a ridiculous little scene, but it also gives us a useful way to think about AI.
Between receiving something and doing something with it, there is often a smaller decision to make: where does this belong?
Get that decision right early, and the rest of the system can follow the appropriate path.
I’ve been thinking about that little alien while building AI software.
An application receives a billing question, a bug report, a product photo and a request to book a demo. Each needs different handling. Yet we can end up asking the same large language model to inspect everything and work out what happens next. Not efficient nor cost effective at all
That is why, I think specialised models will play a much bigger role in AI’s future. Example; Jev is built for structured decisions that software can act on. Coinbase has adapted Qwen3.5-9B specifically for fraud detection in its Onramp service. Each focuses on a defined job, with outcomes that can be measured.
https://x.com/brian_armstrong/status/2109412046244913643
I am building the sorting layer
Familiar requests can go straight to a defined workflow. Requests that need more reasoning continue to another model. Ambiguous ones get flagged for clarification or review.
The potential saving comes from giving each request the amount of processing it needs. Then we have to measure whether the sorter gets enough decisions right to justify being there.
Our starting point was with CludeMem. We were trying to decide what an AI should remember, what it should retrieve, and what was unnecessarily consuming tokens. A lasting preference, a temporary instruction and an unresolved question deserve different treatment. Before making those decisions, we needed to classify the information.
That became the idea behind a broader classification layer. The same step could help sort a support request, route a document or choose which specialist should handle a task.
GIF
As more specialised models emerge, applications will need to decide which inputs deserve their attention. That is the part we’re building: classify what arrives, handle familiar cases locally, and pass the harder ones to the appropriate model or review process.
Enter Clude Cat (Cat for categorisation 😉)
Before storing a memory, retrieving a document, calling a tool or generating a response, there is usually a decision about what kind of information we are dealing with.
Those decisions are everywhere.
Imagine someone writes to a software company:
“Where can I download my invoice?”
An AI agent could read the message, inspect its tools, consider the customer’s intent and decide to look for an invoice.
A more direct workflow could look like this:
Message → billing-document request → account verification → invoice retrieval.
A language model could still write the reply. The initial routing decision might be handled locally.
Now change the message:
“I’ve been charged twice, my account is locked, and I want to cancel.”
Sending that confidently into a single invoice workflow would create more work. There are several problems to resolve, and the right response might require clarification or a person.
The sorter needs permission to say, “I can’t place this reliably.”
That is an essential part of the product. A system that confidently sends requests to the wrong destination can erase its speed advantage through the work needed to fix its mistakes.
For the customer, the intended benefit is less waiting and less repetition.
For the developer, it is an explicit decision they can inspect, test and change. They can see the selected category, the model version, the processing time and whether the system declined to make a decision.
Underneath, the text implementation starts with a small pretrained model called an encoder.
It converts a message into a list of numbers representing aspects of its meaning. Developers call that representation an embedding.
“Where is my invoice?” and “I need a copy of my bill” use different words. Their numerical representations can still be similar enough for the system to recognise the same kind of request.
A classification layer uses those representations to score categories the developer has defined.
We have implemented two approaches. One compares the incoming message with examples of each category. The other learns the boundaries between categories from labelled examples.
In the fitted approach, we keep the encoder fixed and train a smaller decision layer above it. We do not need to train a new foundation model for each application.
The developer still has to define what the categories mean and connect them to useful actions. The classifier cannot invent the company’s departments, refund policy or account permissions.
It also needs an acceptance policy.
The highest-scoring category might still be a poor match. Two categories might be almost tied. An unfamiliar request might resemble something familiar.
Our current local scores are not calibrated probabilities. A score of 0.92 does not automatically mean “92% likely to be correct.”
We have to test what happens when those scores control decisions.
The output therefore distinguishes between an accepted category, a request that needs review and a technical failure. Timeouts, cancellation and model errors also need predictable handling.
That is the engineering around the sorter. It determines whether another application can use the result safely and consistently.
We ran an experiment across 770 banking requests to see whether this approach could reduce generative-model work.
The first configuration sent every request to a locally running Qwen2.5 7B model for classification.
The second used our local classifier first and sent unresolved requests to Qwen.
The results were:
- Qwen calls fell from 770 to 227.
- Correct classifications increased from 535 to 642.
- Accuracy across all inputs rose from 69.5% to 83.4%.
- The sum of measured query times fell from 299.7 seconds to 87.7 seconds.
The local stage avoided 543 Qwen calls, a reduction of 70.5%.
Caching remained enabled, and the direct approach ran before the combined approach. The timing figures exclude setup and warmups. We measured classification, without generating final customer replies or paying a model provider.
Those numbers describe this configuration. They do not establish an advantage over every LLM or prove customer savings.
A later, separate experiment tested a fitted local classifier on 24 travel and workplace intents.
Out of 720 requests, it accepted 688 locally. Of those, 681 were correct and seven were wrong. It left 32 for review.
That gives us two different measurements:
- Coverage: it handled 95.6% of the requests locally.
- Accepted precision: it was correct on 99.0% of the decisions it accepted.
Both belong in the result. A classifier could achieve impressive precision by accepting only a handful of easy requests.
We also tested 100,000 unfamiliar inputs that should fall outside its supported categories. It was accurate more than 94% of the time.
Recognising what the system does not know remains part of the work.
Speed needs the same care. In a separate warm CPU test, the complete banking classifier took roughly 1.8 milliseconds per request at the median on the M4 Max. That included tokenisation, embedding and scoring.
It excluded downloading the model and starting it up. It is a classification measurement, not the time needed to resolve the user’s whole request. It also does not establish the same performance on a phone or an ordinary laptop.
Jev fits into this discussion because it already addresses a similar class of decisions.
TypeSafe documents Jev as a model for bounded judgments, including choosing between supplied options and routing requests to ordinary code, specialist models or humans. \Jev’s intent-routing documentation\
The role I see for Jev is an optional escalation path for requests the local classifier cannot resolve reliably.
The architecture we’re working toward is straightforward:
Incoming request → local classification → familiar requests take a defined path; uncertain requests move to Jev, another model or human review.
Without Jev, the same local classification stage can hand unresolved requests back to the application’s existing LLM, clarification step or review process.
The developer controls the categories, acceptance thresholds and fallback. They can introduce the local classifier into an existing workflow and measure which decisions it can take over.
Today, the Jev integration operates in shadow mode, comparing decisions across five broad topics without changing the application’s behaviour. The next step is to test whether selective escalation improves the complete workflow. The Qwen results above are not a Jev benchmark.
That is what we need to measure: fewer unnecessary model calls, faster handling of familiar requests, and an explicit path for cases that need more attention.
We have started extending the same approach to images.
Imagine a business receiving a receipt, a screenshot of an error and a product photograph. Each needs a different processing path.
A receipt could go to receipt extraction. A screenshot could enter a support workflow. A product photo could enter a catalogue workflow. An uncertain image could go to review.
The classifier selects the destination. Reading the purchase total, diagnosing the error and identifying product details remain separate tasks.
Our experimental image implementation uses a pretrained CLIP encoder to produce a numerical representation of the image.
A fitted research candidate correctly classified 19 of 20 food-product photographs in a small diagnostic test. It also sent all 30 bean-leaf images to review.
The mistake was a bottle’s nutrition panel and barcode being classified as a receipt.
You can see the problem. Both can contain dense text, numbers and rectangular layouts. We need examples that teach the boundary our application cares about.
That sample is too narrow for a production claim. The labels were dataset-derived rather than independently adjudicated, and there were no fresh receipt or screenshot examples in the test. The public demo uses a separate baseline, so the fitted candidate’s result should not be attributed to it.
Image processing was also heavier than text classification. Warm median processing took approximately 36 milliseconds in Node and 197 milliseconds in Chromium on the M4 Max. The browser first needs roughly 85 MiB of model assets, plus its runtime.
Local execution still has costs. Download size, memory, startup time and device performance all belong in the evaluation.
The product I have in mind fits into an existing application where it decides what happens next.
A developer defines categories, supplies representative examples, connects the destinations and chooses what should happen when classification is uncertain.
The first deployment should observe the current workflow without changing its outcome. We can then compare proposed decisions with verified results before allowing the classifier to control routing.
We need to inspect accepted mistakes as carefully as rejected requests. A confidently wrong decision can otherwise disappear behind a reassuring score.
Training, calibration and testing also need separate data. Training teaches the categories. Calibration selects acceptance rules. A protected test checks the frozen result.
Once a test mistake influences the next model, that example becomes development data. The next evaluation needs a fresh test.
The little alien in the mail sorter gives people a way to picture the job. Building a useful version means finding out exactly which inputs it can handle, when it should ask for help, and what its mistakes cost.
The next step is a representative workflow with independent labels and a record of every outcome. We need to measure how many requests take the local route, how many reach another model, how many go wrong, and whether the complete application becomes faster or cheaper after accounting for those errors.
Time to scale up and give this local model some steroids.
