Jev Model Performance Evaluation

Sep 20

This project evaluates Jev, a probability-based model, on two real-world tasks: analyzing SEC filings and checking extracted facts, highlighting its strengths in narrow, yes-or-no questions and its limitations with complex document interpretation.

SignalUsed as a first-pass filter, or a second opinion that sends disagreements to a bigger model, it earns its place, and we kept it for exactly that in the criteria pipeline.

Pattern↑ Jev AI decision models

First findings with Jev (@typesafeai's System One model) I put Jev through two real jobs from our pipeline, and found its strength's and weaknesses. Just as background for some: Jev is a different kind of model. You hand it a piece of text and a question with a fixed set of answers. It hands back a probability for each answer. It does not write text like ChatGPT or Claude. That is why it is much faster (about 1.2 seconds a call) and cheaper ($0.042 per million input tokens, output free). Test 1: reading SEC filings. Our production classifier reads a biotech's 8-K and answers three things: is this a trial readout, did it hit, and what kind of announcement is it. On our 120 hand-built adversarial test cases, Jev matched production exactly: 120/120 on readout, 115/120 on outcome, 120/120 on type, for under a cent. Then we ran it on 30,582 real filings and had a third model settle every disagreement. Jev called a readout wrongly on 3.9% of filings. Production got 0.9%. The failure is easy to describe: a quarterly results release or a "we will present a poster" notice is full of trial-result language, and Jev, reading in one pass, wrongfully calls it a result. With some optimizations we managed to cut Jev's errors to 1.6%. But still behind production. Cost was not the deciding factor. Jev came to $0.00016 per filing, production $0.00034. So it wasn't even that much cheaper. Test 2: Checking extracted facts. Our criteria pipeline turns trial eligibility text into facts ("age 18 or over", "no prior EGFR therapy"). Against an answer key of 994 facts with 55 wrong, Jev caught 75% of the wrong ones. An internal classifier caught 71%. Jev cost $0.05 per 100 studies; an internal classifier cost $0.08. But only 19% of Jev's flags were real, against 77% for our internal classifer. It go split by question: near-perfect on "is this line a requirement" (AUC 0.99) and "was a requirement missed" (0.95), mediocre on "was the limit or the AND/OR read right" (0.80). What the two tests agree on. Jev is dependable on narrow yes-or-no questions about short text, and it gives you a calibrated probability you can threshold. Sadly, it is not as dependable where the answer depends on what a whole document means. Used as a final classifier it loses to a cheap generating models. Used as a first-pass filter, or a second opinion that sends disagreements to a bigger model, it earns its place, and we kept it for exactly that in the criteria pipeline. Practical notes for anyone trying it. 1. Write the question you mean, literally; it answers the one you wrote. 2. Split any rule with two steps into two questions and do the second step in code. 3. Keep the input short and on topic; a 14,000-character document is a distractor. 3. Cost can still add up, 50,000 requests can cost $10, if you are running at scale, costs are still meaningful I think @typesafeai is making an interesting bet with the kind of model, sadly its intelligence is not yet up to par, it really does feel like its missing a "reasoning" capability as it gets caught in tricky decisions, I hope in the next version they can tackle that.

Original post