ROUTEAgents & automation
Jev and Laya Pre-Request Gates
Ran Aroussi@aroussi
This project evaluates Jev and Laya, AI models designed for pre-request classification, comparing their performance against existing systems in speed and accuracy for tasks like credential detection and prompt injection.
I tested both Jev and Laya against @muxi_ai's pre-request gates and we're not switching.
Laya is the open-weights version of Jev, @typesafeai's System One model, a classifier that returns typed decisions instead of text. The pitch is speed and no hallucinations, because nothing gets generated.
MUXI already runs 11 of these decisions before any request reaches an agent. Is this actionable, does it need clarification, is it a recall question, is the user trying to stop. They run on a 95 MB ONNX embedding model with hand-picked examples per gate. 100% on all 11 eval sets, about 50 ms for all.
I pointed Laya at the two decisions we still hand to gpt-4o-mini. Same fixtures, same labels, incumbent vs Jev vs Laya.
Credential detection: 96.7% / 52.2% / 54.4%
Complexity within one point: 80.0% / 27.5% / 10.0%
Routing accuracy: 71.4% / 81.1% / 33.3%.
Prompt-injection recall: 10/10 / 3/10 / 10/10
False positives on safe messages: 2.3% / 14.0% / 93.0%
p50 latency: 866 ms / 127 ms / 69 ms
Jev/Laya is seven to twelve times faster. they also misses 7 of 10 attacks, or flags 93% of normal requests as attacks, depending on which checkpoint you pick. Routing is the security check in MUXI. Fast and wrong on that gate is not a trade.
The reframe I took away is that Jev's pitch is right and its model is beside the point.
Decide before you generate, with typed answers, is the correct design. We already had it.
A small embedding model with a dozen examples per gate beat a 300 to 400M parameter decision model on our data, because our gates are generic intent questions and that is exactly the regime embeddings are good at.
What we're doing instead is keeping the Jev-shaped interface, typed questions in and typed answers out, and running it over our own model.
All 11 gates now share one embed per request. And the complexity call to the LLM only fires when a request actually needs decomposing.
muxi.org
Related builds
