VERIFYAgents & automation
Jev Decision Model
Jev is a typed decision model that returns a probability for a given state (question, answer, evidence) instead of generating prose or structured data, making it faster and cheaper than traditional AI judges.
TakeAnd a rule I wrote down before writing code: the cheap judge never decides whether the expensive one runs until it has been calibrated on labelled cases — a cheap judge that filters its own mistakes out of view is worse than no judge.
Most "AI judges" write an essay to answer yes or no.
Last night I wired up one that just answers.
TypeSafe's Jev is a different kind of model. You give it some state — a question, an answer, the evidence — and a typed question: "Is every claim supported by the evidence?" It returns a probability. No prose, no JSON to parse, no retry when the JSON is broken. It cannot write a sentence; that is the trade.
My first real calls, on a small ledger example:
• grounded answer → P(yes) = 0.97
• fabricated date, method and fee → 0.01
• right month, wrong payment method → 0.02
Each in about 300 ms, at $0.042 per million input tokens, output free. A frontier judge takes seconds and a hundred times the cost per call.
Now the part I care about more than the speed.
Is it more accurate than a generative judge? No. On TypeSafe's own published evaluations it is mid-pack: level with Sonnet 5, five to six points under Opus 5, well above Haiku 4.5. It reads instructions literally, cannot count or compare dates, and — their docs say this — adversarial content in the state can move its answer. Nothing published shows the probabilities are calibrated.
So I did not add it to AgentEval as "a faster judge". I added a third evaluator kind
✅ code proves (schema, tool called, latency) ✅ a decision model decides (a probability on a narrow claim) ✅ an LLM judge reasons (the why)
Every result says which one produced it. The probability survives into the persisted result, so a threshold can be revisited later. And a rule I wrote down before writing code: the cheap judge never decides whether the expensive one runs until it has been calibrated on labelled cases — a cheap judge that filters its own mistakes out of view is worse than no judge.
What's next is the fun part: AgentEval evaluating the evaluator. 178 labelled agentic cases and 263 labelled compliance cases, Jev beside a generative judge, per category: agreement, false-pass rate, calibration, latency, cost. The numbers decide where it is allowed to vote.
I also proposed the same abstraction to Microsoft Agent Framework and implemented it there — IDecisionClient, a loop evaluator, a TypeSafe provider — PR open, link in the comments.
Would you trust a probability from a model that cannot explain itself? Where would you let it decide, and where never?
#AIevaluation #DotNet #AgenticAI #LLMOps #MicrosoftAgentFramework #ResponsibleAI
