Jev: A System One Model for Classification and Scoring

Sep 20

Jev is a "System One" model that returns classifications, scores, and probabilities, unlike text-generating LLMs, and can evaluate multiple dimensions of AI agent sessions in parallel at a lower cost.

SignalIt returns classifications, scores and probabilities.

Pattern↑ Jev AI decision models

Jev is insane 😱 We have been testing it for the last 24 hours. On one of our early eval sets, it reached close to 99% accuracy at roughly 1/100th the cost of our current LLM-based approach. Jev is a new “System One” model. It does not generate text like an LLM. It returns classifications, scores and probabilities. One use case we are testing is measuring AI fluency. At Larridin, we evaluate agent sessions across dimensions such as context management, prompt clarity, multi-turn thinking, agent steering, thought-partner use and advanced connector use. We currently run much of this through LLMs, even though the output is ultimately a set of classifications and scores. With an LLM, each dimension either becomes a separate call, or everything gets packed into one large prompt. Separate calls are slow and expensive. Large prompts can produce inconsistent scoring across dimensions. Jev can evaluate all these questions in parallel against the same session. Adding more questions does not add proportional latency. This lets us use a much richer fluency rubric without multiplying the cost or response time. Twenty-four hours is nowhere near enough testing to draw broad conclusions. We still need to evaluate it against more data and more edge cases. But we have already identified several other places in our stack where it might fit. I suspect there are a lot of expensive LLM calls in production today that should simply be classifiers.

Original post