SCOREAI infra
VulcanBench Verdict Eval Suite
Morgan@morganlinton
VulcanBench Verdict is an evaluation suite designed for System One Models, utilizing Jev for rapid testing.
Pattern✦ Agent trace evaluation with typed judges
It has been a really interesting experience to build an eval suite for System One Models like Jev.
My first two attempts didn't work out. But I learned a lot in the process, and since Jev is so fast, and cheap, it's a lot easier to run quick tests than with an LLM.
I'm on attempt number three with this eval suite, and thought I'd share more about it.
The eval suite is called @VulcanBench Verdict, and here's the high-level on how it works:
