CHOOSEAI infra
Jev Latency Analysis
henry@HenryThegreat12
This project analyzes the latency drivers of Jev, a model with no decode step, by running 8.8k calls to observe its performance characteristics.
1)
Ran 8.8k calls against @typesafeai's Jev to see what actually drives latency in a model with no decode step. 20 min, $1.50. Short version: it behaves like pure prefill.
latency ≈ 72 ms + 6 ms per 1k input tokens + 0.2 ms per question + 0 per option
