Feishu Life-Assistant Intent Routing
Donald Lampard@DonaldLamp84do
This project evaluates the performance of a classifier-based routing system (TypeSafe Jev) against a generative model (glm-4-flash) for user intents within the Feishu life-assistant.
TakeHard limit: Jev routes.
299 real user intents from our Feishu life-assistant router. Same gold labels. Same hard suite (19 skills, ambiguity cases).
glm-4-flash (generative, production): 271/299 = 90.6%
TypeSafe Jev (classifier / System One): 262/299 = 87.6%
I’d still put the classifier on the critical path first.
Of the 29 cases only glm got: 23 were one fuzzy skill (trip-planner vs life-service briefs). Of the 20 only Jev got: 15 were glm JSON/parse failures that fall back to unknown in prod.
p50: 0.55s vs 3.6s. Format parse fails: 0 vs 16. 299×2 on Jev cost $0.086.
Hard limit: Jev routes. It does not extract params. Production still needs a generative step after the gate.
Measured with llm-fit on the same production intent path we already ship — not a public leaderboard.
@typesafeai @CompleteSkeptic
Related builds
