ROUTEAI infra
Intelligent Model Routing Benchmark
Mony Mony@monymonykay
1:24This project benchmarks three approaches for intelligent model routing: LLM, Decision Model (Jev @typesafeai), and Semantic Similarity, finding the decision model to be the most promising.
Many customers are asking for Intelligent Model Routing.
I benchmarked 3 approaches: 1/ LLM, 2/ Decision Model (Jev @typesafeai ), and 3/ Semantic Similarity. I used AgentCore Gateway as my router. Using a decision model was the most promising.
TypeSafe Jev scored 94.9% on the same test and cut model spend 56%.
Semantic search with ~200 examples had the lowest routing accuracy as it tell you what a prompt is about, not how hard it is. Route on difficulty, not similarity.
Video below shows the Prompt and which model it routed to base on configurations defined in the decision model. Full writeup and the numbers in the comments.