CHOOSEDevtools & code
Jev Whetstone Skill Selection Experiment
Ilia@iliaa
This project tested Jev's ability to select the correct Whetstone skill for requests, showing it excels with unfamiliar wording but makes occasional mistakes.
Got access to @typesafeai first experiment was testing whether Jev can choose the right Whetstone skill for a request.
- On newly written, tricky requests: Jev got 63 of 64 right. Which is much better than 32/64 regex rules got
- On older test cases: regex rules got all 132 right. Jev got 120 right.
Jev looks better at understanding unfamiliar wording, but it also makes mistakes fixed rules avoid.