Jev Independent Index

Sep 20

Jev is an AI model effective for classifying inputs into predefined categories when the criteria are explicitly defined within the input data, offering significant cost and latency advantages over existing models.

Pattern↑ Jev AI decision models

With the numbers from a day of benchmarking @typesafeai's Jev against our production judges... What Jev is good for Picking a label from a fixed list, when the list is defined by you and the evidence is in the input. That is the shape where it matched or beat the models we run today, at a fraction of the cost and latency. - Classifying sales-call transcripts into one of 32 call types: 35 of 36 correct on a synthetic set with 20 adversarial cases, versus 34 of 36 for our current small-model classifier. Zero label changes across three repeat runs. About 100 times cheaper per call. - Typing the relationship between two knowledge-graph nodes from a fixed vocabulary: passed all four of our hard quality gates on a 47-case gold corpus, one case behind production, at roughly 65 times lower cost and 15 times lower latency. - Categorizing Slack channels into four buckets: 16 of 16 on the held-out set, tied with a frontier model. - Yes/no questions about evidence that is visible in the input: in our tests, "does this item contain a visible defect" and "is this candidate a real customer signal" both worked. It is also fast and stable. Median latency was 100 to 200 milliseconds per call. Across roughly 3,700 calls we saw nine server errors and no throttling. Confident answers were identical run to run; only low-confidence answers flipped. The confidence score is the most useful thing about it. When it was wrong, its confidence was usually low. A frontier model told us it was 95 percent confident on both of its misses. What Jev is not good for Judging quality, or anything where the right answer depends on the model's own sense of what counts as good. - Judging whether an AI agent's output is good enough to publish, against a written rubric: 45 of 50 on our labeled fixtures versus 48 of 50 for a frontier model alone. On a held-out set weighted toward data-heavy outputs it dropped to 18 of 27 with nine false rejects, while production got 24 of 27 with none. Six rewrites of the question changed nothing. On 71 real production rejections it would have let 27 through. - Judging whether two decisions contradict each other: on 36 human-labeled pairs it found 2 of the 12 real contradictions. Ten of them scored a probability of 0.13 or lower. - Code review. A staged Jev-only reviewer we tested on 14 real merged PRs, with human review findings as ground truth, found none of the 13 human-identified problems on the PRs it could process and produced 24 low-severity "compatibility" and "test gap" notes across PRs that humans approved cleanly. The pattern across all of it: Jev works when a definition outside the model fixes the answer, such as a taxonomy or a checkable condition. It fails when the answer is a matter of judgment, because its prior about what is trivial, safe, or contradictory does not match ours and no phrasing moved it.

Original post