CHOOSEBusiness workflows
Jev vs. LLM Support Ticket Test
Crazy Citizen@CrazyCitizen01
This project tested Jev against a traditional LLM on ambiguous support tickets, finding Jev to be deterministic and calibrated while the LLM exhibited non-determinism.
Gave @typesafeai Jev a fair test vs an LLM: 10 ambiguous support tickets, same input, 10 runs each.
On "one user can't log in — urgent?" the LLM answered yes 5×, no 5×. Same prompt. That's the non-determinism you don't see in a demo.
Jev: 0 flips in 100 calls, and it said 0.55 on the truly borderline one instead of faking certainty.
Honest caveats: on easy tickets both were 100% accurate & consistent, and latency/cost aren't the "1000×" once you batch the LLM.
But for typed decisions you branch on in code, deterministic + calibrated is the real unlock. 👏