SCOREAI infra
LLM Letter Counting Accuracy
Paolo Rosson@redp314
0:32This project evaluates the accuracy of a large language model (LLM) in counting letters within words, specifically highlighting its undercounting of doubled letters and improved performance when provided with individual letters.
can @typesafeai Jev count the r's in strawberry?
no.
47% says 3, 47% says 2. a coin flip, same as every LLM.
On 70% on 168 test words, it undercounts doubled letters like everyone
then I gave it the letters instead of the word: ["s","t","r","a","w","b","e","r","r","y"]
168/168. same model, same question, 260ms
@CompleteSkeptic is this expected?
Related builds
