Jev vs Claude: I tested Jev before trusting it

A few weeks ago, I discovered Jev, a decision engine from TypeSafe AI, and wired it into a web application. It's a fantastic new tool for the web stack. Before showing it at a workshop at Republikken in Copenhagen, I wanted to get past the launch hype and test it against its competitors.

Machine learning specialists joked about Jev. "Jev in 25 lines of Python", as a post by NobodyWho put it. Or "it is called a classifier, nihil novi sub sole". Others said open-source models do the same job or better (Laya).

So I tested it myself, shared the results with the room, and I am sharing them here.

Why I tested it

As a fractional CTO, I build web applications. A new tool arises that could improve the product? I try it and if it makes sense, I recommend it to my clients.

Jev just created its category: decision AI. You give it a question and the shape of the answer: it gives you the answer with a confidence percentage. Because it's not a chat it can afford to "think" in one pass. You cannot ask it to justify itself. But it works out of the box. And it's pretty good for about any kind of decision because it's being trained for a variety of use cases so you don't have to.

Don't ask me how it works though.

Because I am not a machine learning engineer. I learned what a logit is a few days before this workshop.

But also it is a black box. It's not open source and we're not even sure where the data goes.

The NobodyWho post, "Jev in 25 lines of Python", simplifies it, with humor, to the extreme. Load Qwen3-0.6B, a small open model, on your machine. Ask the question with lettered options. Read the logits of the next token for A and B, and normalise them. You get a number between 0 and 1. "That's Jev."

The open-source camp has alternatives too. During the workshop, a participant pointed me to Laya, an open-weights engine that runs on a laptop and accepts Jev's request format. Released 3 days after Jev! If an open model does the same job a few days later, why pay for the original?

Wasn't sure I fully understood either approach, even with Claude's help.

But when clients ask "why don't we use this instead?", I owe them more than a LinkedIn thread. So I did what I usually do for clients, out of sight:

I set up a protocol that can justify a rational decision.

2 Use cases

Chosen from the workshop sign-up form:

• Inbox: Is this message unsolicited?
• Trips: Is this a business trip?

In each case, the goal is to let the machine handle the clear-cut cases and reserve human review for the ambiguous ones.

The method

I could not use real emails or real trips, so Claude Sonnet 5.5 wrote 340 emails and 350 trips.

• Each item was written to match a label decided in advance. The answer key stays in a separate file, and each item gets an anonymous id.
• Items vary in difficulty, easy, medium or hard, and in language English, Danish or French.
• 3 models from 2 families (Gemma 4 running locally, Claude Sonnet and Claude Opus) audited every item without seeing its label.
• Easy and medium items they disagreed on were dropped. Hard ones stayed, waiting for a human review I have not done yet. That left about 325 items per dataset.
• Every scorer (a model) then received the same question, word for word, and the item alone, anonymized.

The public repo

Steal the method, not just the data: synthetic items written from a known answer, three difficulty levels, blind auditors, anonymous ids. Point it at your own emails, trips or tickets. Everything is here: hmatringe/workshops_material_20261008_republikken_meet_jev

No time to read the code? Ask Claude:

Read this repo:
https://github.com/hmatringe/workshops_material_20261008_republikken_meet_jev
notably README.txt and benchmark/README.txt.
then tell me how to run this benchmark on my own data,
with my own yes or no question, and which API keys I need.

The data snapshot

Sort the columns. On a large screen, hover or click a scorer to see its reliability diagram.

Inbox Trips

Hover a scorer to preview its reliability diagram, click to keep it. Click a column header to sort.

Scorer	Accuracy	Median ms	Total time	Cost USD

Accuracy uses a 50% threshold. Median ms is the time per item. Total time and cost are for the whole dataset.

The full benchmark

Every metric, 16 reliability diagrams → The page I presented in the workshop. Sort any column, click a header for its definition.

The lessons

Claude against Jev

• Claude is the most accurate. 97 to 98.5% on both datasets. Jev is 6 to 7 points behind on the emails, 2 to 3 points on the trips.
• Jev loses on the tricky cases, not the obvious ones. On easy and medium emails, Jev is right 95% of the time. On hard ones, like a phishing email imitating a bank, it drops to 84%. Claude stays at 97 to 98%.
• Jev plays in another league on speed and cost. It handles the 325 emails in 11 seconds for half a cent. Claude Sonnet takes 9 minutes and 55 cents. At 100,000 emails, that is about $2 against $170.
• The most expensive model is not the best. Claude Fable costs twice as much as Opus and about four times as much as Sonnet. It is never more accurate than Opus, and on the emails it is behind Sonnet too.
• Both know when they don't know. Only 2 of Jev's 44 mistakes came with 90% confidence or more. Claude: none. So let them decide only when they're sure: on the trips, Jev then decides 9 out of 10 on its own, right 98.6% of the time, and asks the driver about the tenth. The local models can't say the same.

Jev in 25 lines of Python

• It gives you Jev's interface, not its judgment. On Qwen3-0.6B, the model from the NobodyWho post, it is a coin toss: 54% on the emails, 49% on the trips. It calls 315 of the 325 emails "not spam", when half of them are spam. Swapping the answer options changed nothing: it is not a quirk of the letters A and B.
• A bigger model buys judgment, not honesty. On Qwen3-4B, the same script is right four times out of five. But it is almost always sure of itself: of its 59 mistakes on the emails, 50 came with a confidence of 90% or more. A confident wrong answer is the dangerous kind, because nothing tells you to check it.
• How you ask matters, and no way of asking wins everywhere. The same Qwen3-4B, asked to state its own confidence instead of reading its raw word probabilities, does better on the emails (86% against 82%) and much worse on the trips (66% against 81%).
• Local and free is not instant. On my laptop, Qwen3-4B needed 4 to 10 minutes per dataset, about one second per item. Jev needs 29 milliseconds.

The open-source alternative

• Copying Jev's request format does not copy the engine. Laya is free and fast: 22 to 28 seconds for a whole dataset on a laptop. But it reached 70.5% on the emails and 57% on the trips, where it calls 139 of the 158 business trips personal.
• A caveat: I tested one checkpoint, the multilingual one, with my own integration. If I set it up badly, tell me and I will rerun it.

Found along the way

• The difficulty levels held. Most scorers drop on hard items, Jev and the local models most of all. That is the sign the synthetic data tests something real.
• Models do not read the same clues. During the audit, Gemma 4 needed the driver's note to call a trip business. A company name in the destination address was not enough. Claude made that inference.
• The local models rarely doubt. As an auditor, Gemma 4 reported an average confidence of 100% over the 340 emails, the disputed ones included. Claude Sonnet and Opus lowered theirs on the disputed emails.

Limits

• The data is synthetic. Realistic, not real.
• The setup favours Claude. Claude wrote the items and two of the three auditors were Claude models.
• 32 hard items await my review. They hold 11 of Jev's 27 email mistakes, and all of Claude's trip mistakes. Read the gap between Jev and Claude as a few points either way.
• One run per scorer. Claude was called one email at a time: parallel calls would cut its time, not its cost.

Which one should you pick?

It depends on what you care about most.

Run it on your data, then tell me where I am wrong. The shape of your data will move these numbers in ways I cannot predict, so this table may not hold once you test a few hundred of your own items. Measure before you pick. And if your numbers disagree with mine, or if I set up one of the scorers badly, I will rerun it and update this post, with credit.