ROUTEAI infra
Jev Routing Experiment
Sinda | SKatalyst AI@SKatalystAI
An independent routing experiment across 13 models and 5,835 held-out queries found that Jev's mechanism for routing queries to models is based on their historical performance on similar tasks, rather than predicting task difficulty.
Found something about Jev that surprised me:
in an independent routing experiment across 13 models and 5,835 held-out queries, Jev + retrieval reached 62.4% accuracy, beating the best single model at 60.3%.
But when they removed Jev, retrieval alone still scored 62.4%. The routing gain wasn’t coming from predicting whether a task was easy or hard. It came from knowing which models had historically performed well on similar tasks.
Even crazier, an oracle choosing the cheapest model that could correctly answer each query would reach 82.6% accuracy at $4.83 per 1,000 queries, compared with 62.4% at $26.73 for the tested router.
So if removing Jev produced the exact same 62.4%, what exactly is Jev adding to the routing decision?
Related builds
