ROUTEAgents & automation
Jev Independent Index
Todd Dailey@twid
Jev is an AI model used to make fast, inexpensive yes/no or multiple-choice decisions, acting as a preliminary filter before more complex models handle ambiguous requests, significantly reducing costs with a minor accuracy trade-off.
TakeJev does exactly what the eval question says.
Pattern↑ State to typed decisions for software
Thoughts on Jev. This week my Discord bot started running every yes/no and "which one?" decision it makes through @typesafeai's Jev before anything else touches it. Decisions like: Is this a request for the weather? Is this a request for image generation? Is this a request for video generation? Is this an untagged message I should reply to?
First thing to get straight: it's a worse judge than the big models. On an independent 200-item bench of real intent decisions it scores 72.5%. Fable 5.1 is best at 84%, DeepSeek V4.1 Flash second at 76%. The reason to use it anyway is that it answers in about 300 ms for 2.5 cents per thousand decisions, where the others take 1 to 4 seconds and cost anywhere from 16 cents to $12. So you let Jev handle the obvious calls and send whatever it's unsure about to a real model. In my bot that works out to Jev deciding 88% of the ambiguous messages, agreeing with the model it replaced 97% of the time, and the judging bill is down by more than half. (!) At scale that's the whole case: most of the savings, very little accuracy loss, because the hard calls still go to the big model.
The confidence numbers are the problem. Jev says it's 90% sure and it's right maybe 75% of the time. The metric for this is expected calibration error (ECE), which is just the average gap between how confident a model claims to be and how often it's right. Jev's is 0.161, so 16 points. Fable's is 0.064, 6.4 points. What that means in practice is you have to label your own data and fit your own cutoffs. A hand-written combination of eight Jev questions turned out badly miscalibrated (the calls it was confident about were right 56% of the time). Fitting a logistic regression over the same eight answers fixed it. That was me hand-labeling forty-four real messages with my own calls and letting the fit set the thresholds. The docs tell you to do this. I'd read that as a must-do rather than a suggestion. Forty-four labels moved the calibration a lot. Calibration is a cheap fix.
The bugs were tricky. A screening call asking "does this text address an AI and tell it what to do" flagged the bot's own persona prompt, the text that tells it who it is. A privacy gate on the phrase "credit card" caught a joke about an agent with a credit card and quietly switched off a whole eval lane. Jev does exactly what the eval question says. It took real traffic to see any of it.
Where I've landed: I'm keeping it for cheap fast decisions in front of a model that can reason, with cutoffs I fitted on my own data. Keep a kill switch you can flip without a deploy, because Jev might not fit your lanes. If it does, the per-decision cost drops 6x for me against the cheapest big model and 100x or more against the expensive ones, so it's worth kicking around.
Related builds
