CHOOSEBusiness workflows
Jev Model Quality Benchmark
Asta@asta_td
0:49This project evaluates the accuracy of the Jev model in classifying and routing simulated emergency dispatch incidents, testing its ability to identify incident types, assess urgency, select response units, and make disposition decisions.
TakeConclusion Jev is fast and cheap, and its operational model (typed decisions rather than text generation) makes it a really interesting option to try for high-speed classification or routing tasks.
Pattern↑ Jev AI decision models
Jev has just launched, and its approach to handling a large volume of independent, parallel decisions is quite impressive. But does "fast and cheap" also mean accurate? I built a small demo to test the model's quality.
Setup: Emergency Control Room
An app simulating my city's emergency dispatch center. Each incident is sent to Jev to:
- Identify the incident type
- Assess urgency
- Select the appropriate response unit
- Make a decision: handle immediately, escalate for review, or reject as out-of-scope
A policy engine within the app uses these results to prioritize tasks and display them on the dispatch map.
Benchmark: 300 synthetic events:
- 17.6s for the entire batch
- Average 340ms/event; p95 427ms
- 300/300 outputs in the correct format (zero schema errors)
- 97.2% correct incident type
- 85.7% correct response unit
- 65.9% correct urgency level
- 57% of events correct across all 4 criteria
Total cost: $0.0124 (~$0.00004/event)
The biggest weakness I see with Jev is its inability to say "I don't know."
My schema includes a "disposition" field with three peer options: CLASSIFIED, NEEDS_REVIEW, and OUT_OF_SCOPE. The OUT_OF_SCOPE label was available, yet out of 24 events intentionally designed to be out-of-scope, Jev selected OUT_OF_SCOPE correctly only once. The remaining 23 cases were forced into known incident categories, even though none of them actually fit.
It’s not that Jev can't abstain - it is capable of doing so, and has performed correctly in other benchmarks comparing scenarios with and without an abstain label. The issue lies in the selection tendency: when faced with unfamiliar input, the model exhibits a bias toward the nearest familiar label rather than opting for the "reject" option, even though that option is clearly defined in the schema.
Conclusion
Jev is fast and cheap, and its operational model (typed decisions rather than text generation) makes it a really interesting option to try for high-speed classification or routing tasks. However, the results aren't quite good enough yet. I look forward to future updates from Jev. It is definitely worth trying and anticipating.