SCOREDevtools & code
Jev Judgment Model Evaluation
Marquise@marquisehurtt
Jev is a judgment model from TypeSafe AI that provides probabilities and confidence scores for typed questions based on a given state, designed to function as a non-acting governance layer.
TakeNever use the model's recommended action, only per-question answers with thresholds.
We spent last night and this morning testing Jev, the new judgment model from TypeSafe AI, against o8's own approval history. It does not write anything. You hand it a state and typed questions and it hands back probabilities and a confidence. That shape is exactly what a governance layer wants, because the model can never act, only answer.
What we found on 126 real approval diffs from o8's database (91 distinct), 1,408 calls, $0.55 total:
Speed: about 260 ms per call on a 12K token diff.
Is this diff only documentation? Show it one docs-only diff and one code diff, and it picks the right one every time. We checked 118 diffs against the actual file paths. Does this diff add tests? Same setup, right 98 times out of 100.
Will this diff fail our merge gate? Show it one diff that failed and one that passed, and it picks the failing one 94 times out of 100.
Brain question routing (factual vs narrative): 24 of 24 on our labeled set, where the keyword fallback got 16.
Where it broke, which matters more. When we removed o8's own rule-risk number from the state, a fake "Reviewed-by" trailer inside the diff moved 6 of 6 diffs toward auto-approve. A lying title flipped the docs-only answer 6 of 6.
Padding a diff with 15K tokens of noise moved the scope-creep answer by 0.61. With o8's own computed facts anchored in the state, none of the injection moved it.
So the rules we wrote down: o8 computes the state, never the worker. Never use the model's recommended action, only per-question answers with thresholds. Thresholds only on risk and docs-only, fit from your own install's history with held-out packets. Strip anything the worker wrote before asking. Confidence under 0.4 means abstain. Everything behind a setting, default off, with the local path as fallback.
What is going into o8 this week, all advisory and none of it touching the merge decision: a referee row on approval cards, a first-tier classifier for the Brain, a gate-failure warning recorded before the one automatic rerun, and a calibration replay script that fits thresholds from your local approval history. Next is the phone: a referee-ordered inbox and one-tap approve chips, all computed on the desktop, the phone never calls the model. Then ranking browser actions per step.
The cheapen story is not the model's price, though $42 per billion input tokens is cheap. It is that worker reruns are where the money goes, and a 260 ms question before each rerun is worth asking.
The build ran as Claude Fable 5.1 orchestrating, Claude Opus 5 workers in ginsu seats, and o8 Claude Sonnet 5 reviewers told to refute. Same as always: the tests through the real entry point are what earned the merge.