SCOREBusiness workflows
Jev System One Model Evaluation
Shirsha Ghosh@shirshagh
This project evaluates Jev, a System One model from TypeSafeAI, on its ability to score interactions by returning probabilities and confidence for typed questions in a single parallel pass, rather than generating text.
TakeYou send a state plus typed questions (yes/no, pick one, or a scale you define).
Tested Jev, the new System One model from @TypeSafeAI, on a pharma medical affairs problem.
Jev doesn't generate text. You send a state plus typed questions (yes/no, pick one, or a scale you define). It returns probabilities and confidence for every answer in one parallel pass.
Use case: scoring MSL and KOL interactions. After field medical meets a key opinion leader, someone has to judge the conversation: sentiment, depth of engagement, off-label questions, adverse events. Today that's manual and inconsistent.
Setup: a synthetic transcript of an ophthalmologist discussing a fictional retinitis pigmentosa gene therapy, scored against 15 typed questions.
Results:
• KOL sentiment: conditionally positive, 100% confidence. Correct.
• Scientific engagement: 4.2/5, 86% confidence.
• Insight value: 3.99/5, 98% confidence.
• Unmet need identified: 97% yes.
The best result was the one that failed.
"Did sentiment shift during the conversation?" came back 55% improved, 43% unchanged, 41% confidence.
The model wasn't wrong. My question was. I asked it to compare the start and end of a conversation in one hop with no baseline. Fix: two questions (opening stance, closing stance), compare in code.
Lessons:
1. Calibrated confidence is the feature. Low confidence means fix the question or route to a human.
2. Describe concrete situations in rubrics, not degrees like "moderate."
3. Split multi-hop questions and compose in code.
4. Jev scores and routes. An LLM still writes the summary.
For regulated workflows, a model that says "I'm not sure" as a number you can threshold on beats one that confidently writes a paragraph.
