CHOOSEOther
Jev Decision Model Evaluation
Nikhil Deshpande@nikhdesh
Jev, a decision model from TypeSafe, was tested on standard AI problems, answering structured questions with probabilities and performing comparably to or better than other models on single-step tasks, but struggled with sequential reasoning without state information.
TakeThe simulation computes the legal moves, Jev picks from the list.
The same model tags English at the level of a trained statistical model and cannot get three missionaries across a river.
Both are true. Neither is about difficulty.
We put Jev, TypeSafe's new decision model, through seven problems from the standard AI syllabus. It is not a chatbot. It answers one structured question at a time and returns a probability for each option, in about a third of a second.
The rule throughout: code owns the world, the model only chooses. The simulation computes the legal moves, Jev picks from the list. The same state and the same options go to the textbook algorithm and to GPT-5.6 Luna.
Part-of-speech tagging. 49 possible tags, 2,593 hand-annotated words. Jev 92.9%. A hidden Markov model trained on that corpus, 92.3%. Jev saw every word cold.
Missionaries and cannibals. Eleven moves, first-year students solve it on paper. From the state alone Jev solves it 0 times out of 10. It locks into a two-state loop.
Then we handed it a list of visited states, computed in plain code. No strategy, no advice.
10 out of 10. Nothing about the model changed.
That is the finding I would take to work tomorrow. When a decision model fails a sequential task, the fix is usually a few lines of state, not a bigger model.
Then the part I did not expect.
We ran GPT-5.6 Luna through the same problems with reasoning effort set to none. Missionaries 0 out of 10. Tic-tac-toe 7 out of 20. The same failures as the small model.
Set reasoning on and it goes to 8 out of 10, and 18 out of 20.
So the frontier model's planning advantage is not size and it is not training. It is the reasoning loop. And that loop has a price.
Jev: 10,164 decisions, $0.39, 367ms median. Luna with reasoning off: $1.42 and 1,026ms. With it on: $1.16 for a fifth as many calls, at 1,563ms.
Two more worth having.
On the 8-puzzle, Jev ranks which board looks closest to solved and gets 16 of 30. One line of Manhattan distance gets 29. A perfect ranker gets 30 and an uninformed one gets 1, so the judgment is real, just worse than a formula from 1968. Where there is a formula, use the formula.
And we got things wrong. On the first pass, with small samples, "beats the textbook tagger" became "on par" at twice the data, and "never died in the Wumpus world" became died in 10 of 50. Every number on the site now carries a 95% interval.
Three of the seven are labelled stress tests, because TypeSafe's own docs say the model is not for multi-step dependent reasoning. Failures there corroborate the vendor rather than catch it out.
Judgment is cheap now. Planning still is not.
About 22,000 decisions, under $3. Play tic-tac-toe against it yourself: jev-ai101.nikhilrdeshpande.com
Related builds
