Classifying email with a decision model: Jev, and a four-way bake-off

InboxVelocity, my upcoming AI email project, reads every message that lands in my inboxes and decides what it is: a newsletter, a receipt, something from a person who wants an answer, something with a deadline. For most of its life that decision was one gpt-5-mini call per email, returning everything at once: category, an urgency score, a requires-action flag, the kind of document attached, a summary, key points and suggested actions.

Most of that is classification, and a model that writes prose is overqualified for "is this a receipt." So I tried TypeSafe's Jev, a decision model, as the classifier. It now decides what every email is, and gpt-5-mini only writes text for mail I'll actually read.

This post covers what Jev is, how I evaluated it and tuned the questions, how it's wired in, and a bake-off I ran on October 6 against three newer challengers: Cloudflare's Clef and Clef-flash, and OpenAI's Decisions API. The short version is that Jev still wins on my workload. The longer version is mostly about why that result might not hold on yours.

What Jev is

Jev doesn't generate text. You send it a "state" (the facts it should look at) and one or more typed questions: pick one of these options, give a score on this scale, or answer yes or no. It returns a probability for each option. It's priced on input tokens only, at $0.042 per million, and output is free. Everything in this post ran on jev-1.13.0.¹

TypeSafe publishes a "jaggedness" page for the model, which is the most useful vendor document I read during this project. It says plainly that score levels are weak in numerical calibration, that the model reads dates as text rather than as quantities you can compare, that you shouldn't hide several judgments inside one question, and that accuracy drops as the state fills with material unrelated to the decision. Every one of those turned out to matter.

How I evaluated it: replaying already-triaged mail

My first spec was over-built: a live shadow mode running Jev beside gpt-5-mini on new mail, snapshots, a comparison endpoint, accuracy tiers and confidence intervals. I replaced it with something much smaller. Replay mail the app had already triaged through Jev, and score Jev's answers against what gpt-5-mini had stored.

The replay tool is one admin endpoint. It reads triaged mail, sends each email's state and the questions to the model, runs the answers through the same post-processing production uses, and writes a report. It only reads mail data, it has a one-dollar spend guard per run, and nothing in production read a replay answer during the evaluation.

Three choices made it trustworthy enough to act on:

• I used a fixed split: a stable hash of the email id puts 70% of mail in a tune set and 30% in a held-out set. I edited prompts only by reading misses on the tune set. Each held-out run happened once, on frozen prompts.
• The best reference I had was the stored gpt-5-mini answer, replaced by my correction on the 16 emails where I'd corrected it.
• Results are weighted. The sample is stratified by category, capped at 60 per category, so a rare category doesn't vanish. The headline number is weighted back to the real mix of mail.

Three of the questions I wanted to ask (is this written by a human, does it need a reply, does it give me a to-do) had no stored answer, because gpt-5-mini had never been asked. For those I had Sonnet subagents label mail from written definitions: 300 tune emails, then 150 held-out emails never used for tuning.

A replay harness is fast to iterate with. Each run takes minutes and costs cents, and it ended up doing far more than the original evaluation, as the bake-off below shows.

Tuning: literal questions converge, judgment questions plateau

Here is every question, how many prompt versions it took, and where it ended:

The pattern held across every task: the more literal the question, the faster it converged.

"Written by a human" worked on the first try. Everyone, including the model, already agrees on what it means.

Document kind overshot and came back. Version 1 said a shipping notice "is not the order document itself and is NONE," which missed real order confirmations. Version 2 said a shipping update that restates the order with prices counts as an order confirmation, and the model started calling every tracking email a document. Version 3 added "an order number, item names or a tracking number without prices are not enough," and that held on the held-out set.

Adding examples made requires-action worse. Version 2 added a list of automated mail that looks like it needs action and lowered the threshold to 0.35. Precision fell apart. Version 3 removed that list, added what does not need action ("a friend, follow or connection request... ask nothing, and neither does a digest or summary"), and went back to 0.5. That version passed: 93.0% on held-out, precision 85.4%, recall 92.9%.

Has-to-do needed a definition from me. The Sonnet judges kept flagging meeting invites, statements and autopay renewals as borderline. I hadn't decided what a to-do was. Once I wrote it down (something I have to do outside of reading or replying; invitations and autopay don't count) and put that definition in the prompt, it reached 95.3% on held-out.

Category went in a circle and urgency never got close. Category took five versions, and the six-point drop from tune to held-out is overfitting, so I stopped at about 90 on tune. Urgency started at 67.5% and peaked at 72.9% against a 90% bar.

The reference was the noisy part

That scorecard said Jev was fine for yes/no flags and not good enough for category or urgency. Reading the urgency misses changed my mind. The stored gpt-5-mini answers gave identical kinds of automated notices an urgency of 10 to 15 in some emails and 20 to 25 in others. No prompt can match a reference that disagrees with itself.

So I got two more references. First, Sonnet, using the app's own gpt-5-mini triage prompt as the rubric. On 210 tune emails, gpt-5-mini agreed with Sonnet 78.6% of the time and Jev 81.4%. On the 44 emails where Jev and gpt-5-mini disagreed, Sonnet sided with Jev 22 times, gpt-5-mini 14 times, and neither 8 times. On the held-out set it was close to even: gpt-5-mini 84.7%, Jev 84.0%.

Second, me. I labeled 40 tune-set emails blind. On the 26 where the two models disagreed on category, gpt-5-mini matched me 14 times and Jev 11. On urgency, across 39 of those emails, gpt-5-mini hit my exact band more often (22 against 17), but Jev was within one band more often (37 against 33). My labels also showed that Sonnet rates urgency about two thirds of a band lower than I do, so Sonnet is a usable category judge for my mail and a bad urgency judge.

Put together: on category Jev was about as good as gpt-5-mini, and on urgency it was different but not worse for how I use it. Agreement with the model you're replacing is not accuracy. Check the reference against something independent, including a small sample of your own labels. Forty emails were enough to change the decision.

Jev's confidence scores and the fallback cascade

Jev returns a probability for every option, and for category it carried real information. Answers at 0.95 confidence or higher covered 52% of mail and agreed with the reference 99.1% of the time; at 0.70 or higher they covered 75% at 96.4%. For urgency it barely meant anything: the top bucket agreed 79.7% of the time.

That gave me a cascade: accept Jev's category when its confidence is at least 0.8, otherwise run the full gpt-5-mini call. On the held-out set with Sonnet as the reference, Jev decided 102 of 150 emails and was right 90.2% of the time on those, against 88.2% for gpt-5-mini on the same emails. The combined result came to about 86%, against 84.7% for gpt-5-mini alone.

Splitting urgency into literal signals

The jaggedness page says to break multi-judgment questions into literal yes/no questions and combine the answers in code. Urgency is the most multi-judgment question I have, so I tried seven literal signals (bulk mail, a stated deadline, time pressure, a close contact, a reported problem, a security alert, scheduling) and fit a 12-parameter linear combiner on the tune set.

Seven extra questions and a fitted combiner didn't beat one question by enough to matter, so I dropped it. The prompt style stuck, though. The deadline signal says, "Only look for the stated time in the words of the email. Do not work out whether the time is near, far or already past." That respects the "dates are text" warning, and it's how I'd write any date-adjacent question for this model now.

How InboxVelocity uses Jev

Jev doesn't write summaries, key points, suggested actions or deadlines, so swapping one model for the other wasn't an option. The call got split instead:

• One Jev call per email asks seven questions against one shared state: category, urgency, requires-action, document kind, human-written, needs-reply and has-to-do.
• Live mail below 0.8 category confidence runs the old gpt-5-mini call, and so does live mail where Jev errors or times out. Jev's answer is still recorded next to it.
• gpt-5-mini writes text only for mail I'll see: on arrival if Jev flagged it or nothing is going to file it away, otherwise the first time I open it.
• Backfill never calls gpt-5-mini. Classifying a whole mailbox at signup is pure classification, and that's where a decision model fits best.

Before switching production, I reran the replay on the previous week's mail with a simple rule: requires-action, document kind and category each no more than three points below the held-out numbers. They came in at 92.1%, 100% and 89.7%, and production switched to Jev on September 29. Reverting is one environment variable.

Why it fits this job:

• The questions are decisions: category, flags and document kind are choices from a fixed list. That's exactly the shape the API is built around, and the answers come back typed, so there is no output to parse or repair.
• Literal questions are reliable: four of the seven questions converged in one to three versions and scored 93% or better on held-out mail.
• The confidence is usable on category, so the expensive model only sees the mail Jev is unsure about. In the first week of production that was about half of my mail, mostly work mail, which is where Jev is least sure.
• It's fast and cheap per call: about 150 ms from my server, and about $0.00025 per email for four questions in the replay.

Where it's weaker is what the jaggedness page predicts: a fine-grained 0 to 100 urgency score, and a nine-way category on mail that is genuinely ambiguous.

The bake-off: three challengers, a week after the switch

On October 1, Cloudflare launched Clef and Clef-flash, open-source decision models on Workers AI that serve the same API as Jev. Their launch post reports median latency of 38.8 ms for Clef-flash and 209.3 ms for Clef against 524.1 ms for Jev, benchmark scores on tool calling and intent classification, a 64k context window against the 32k per question Jev allows, and fine-tuning first through Cloudflare's team and later self-serve.

Around the same time, OpenAI opened its Decisions API as a public beta. It asks the same three kinds of question (a yes/no probability, a choice and a score) with one model, gpt-6-luna, at $0.10 per million input tokens with no output charge.

Both were good reasons to ask whether I'd picked the wrong model. Because the replay harness existed, Clef took one new parameter and a few lines to unwrap Workers AI's response envelope. The Decisions API isn't wire-compatible: its yes/no question is called a predicate, questions go as a list instead of a map, and answers come back as a list. So it took a small translation layer in both directions, one pull request in an evening.

I ran all four back to back from my production server on October 6, on the same 230 held-out emails (213 in the main sample plus a separate document-kind draw), with the same frozen prompts. Agreement is with the stored reference, weighted to the real mix of mail. Two things to hold in mind while reading it, both covered in more detail below: every prompt was tuned against Jev's misses and the challengers got those prompts frozen, and the reference is gpt-5-mini's stored answers, which I've already shown are noisy on urgency.

None of the 3,004 calls failed (751 per model, one per question). Where Jev and a challenger disagreed on category, Jev matched the reference 38 times to gpt-6-luna's 4, 19 to Clef's 5, and 39 to Clef-flash's 8.

How they rank, on this workload:

1. Jev. Best or tied on every accuracy row, cheapest per email, and fast.
2. gpt-6-luna. Tied Jev on urgency, within two points on requires-action, and the fastest of the four. Its whole category gap is one mistake made repeatedly: it called 21 work emails and 19 transactional ones newsletters. A check with two made-up emails, a colleague's request and a shipping notice, came back WORK at 0.99 and TRANSACTIONAL at 0.96, so the translation layer isn't the cause; real automated work and vendor mail looks like bulk mail to it. Its confidence is usable (answers at 0.8 or above covered just over half the mail at 92% agreement), but its yes/no probabilities sit almost entirely at 0 or 1, so there's no threshold to tune on requires-action. It costs about twice what Jev does per email.
3. Clef. Second on category, but low recall on requires-action and well behind on urgency. It puts transactional mail in NEWSLETTER, and its confidence scale doesn't transfer: two thirds of its category answers sit below 0.5 (Clef-flash: 206 of 213), so a gate like mine would send most mail to the fallback. It's also the most expensive of the four per email.
4. Clef-flash. It labels ordinary mail as spam (17 transactional emails, 11 newsletters, 4 each of work and financial mail) and catches fewer than half the emails that need action.

Clef also improved between October 1 and October 6: its requires-action recall went from 68% to 76%, and Clef-flash's median latency fell from 526 ms to 334 ms. I found no new Clef weights published in that time, so the speedup is most likely Workers AI capacity settling. The two runs used different samples, and Jev itself moved by up to 3.6 points between them, so I'm not reading anything into the smaller changes. Clef-flash is still about nine times slower from my server than Cloudflare's published median.

Your mileage will vary

I'd weight this ranking lightly outside my own inbox, for reasons that are specific and worth checking against your workload:

• The prompts were tuned on Jev: every question went through up to five rounds of wording changes against Jev's misses. The challengers got those prompts frozen, with no tuning at all. A model can lose a bake-off on someone else's prompts and win on its own.
• The prompt shape wasn't identical. OpenAI's yes/no question has no field for "what true means" and "what false means," so those descriptions went into the instructions instead. That's a different prompt from the one Jev and Clef saw.
• This is one workload on one mailbox: email triage on one person's mail, with a lot of newsletters and vendor notices next to work and personal mail. Cloudflare's benchmarks (intent classification, tool calling) are different tasks, and by Cloudflare's own figures Clef does well on them. Your categories, your mix of mail and your definitions will move these numbers.
• The reference is noisy. Agreement is measured against gpt-5-mini's stored answers, which I've already shown disagree with themselves on urgency. Small gaps between models are inside that noise.
• There was one run per model: each figure is one held-out run of 230 emails. The October 1 run shows how much the numbers move between samples.
• Latency is from my server and includes the network trip from where my code runs. Cloudflare's published latency comes from Cloudflare's network, and yours will depend on where you are.
• Some of these are new: the Decisions API is a public beta with one model, and Clef had launched five days before the run. Both are likely to change.

If I were starting over today, I'd pick a model by running the same replay on my own mail rather than from this table. The ranking took one evening to produce because the harness already existed, and that's the part I'd recommend copying.

The prompt work I'd try first is on gpt-6-luna, because its misses all run one way and the place to start is obvious: the NEWSLETTER and WORK definitions. Live triage stays on Jev for now, and I'll rerun the same four models when Clef gets new weights or self-serve fine-tuning, or the Decisions API leaves beta.

What I'd tell someone doing this

• Build the replay harness first. It decided the switch, checked the previous week's mail before the flip, answered the Clef question on launch day, and took OpenAI's new API in an evening.
• Ask literal questions. "Written by a human" worked on the first try. "Urgency 0 to 100" never got close. If a question needs judgment, write the judgment down.
• Your definitions are the prompt. Has-to-do went from 81.7% to 95.3% when I stopped tuning wording and wrote down what I meant.
• Prune instead of adding examples. Lists of what counts made precision worse. Lists of what doesn't count did better.
• Check your reference before you tune to it. Get a second judge, and label 40 emails yourself, starting with the disagreements.
• Gate on confidence only where it means something, and measure it per question and per model. Jev's category confidence is a usable signal; Clef's scale is different; gpt-6-luna's yes/no answers barely have one.
• "API compatible" means you can swap the endpoint. You can still get different answers, a different confidence scale, and different latency from where your code runs. Clef is wire-compatible with Jev and answered differently; OpenAI's API isn't compatible at all and came closest.

Notes

¹ TypeSafe released jev-1.14.0 on October 14, 2026, after these runs, as its preview model at the same price. jev-1.13.0 is still the default. I haven't rerun the replay on 1.14.