ROUTEAI infra
Kev-0.6B ANE Performance Measurement
Nick Lo@NickLo641579
0:14This project measures the performance of Kev-0.6B, a Jev-class decision model, running on the Apple Neural Engine, detailing its latency and power consumption.
Kev-0.6B, a Jev-class decision model, running on the Apple Neural Engine.
One forward pass, no token generation. 7.18 ms per decision, p99 7.53.
Placement isn't asserted yet, it's measured three ways: CPU-only returns zeros, Apple's xctrace Neural Engine instrument counts 303 dispatch intervals against 300 predicts, and the ANE power rail goes 0 → 7000 mW.
Alongside a 27B resident on the GPU, the ANE keeps 93% of its own throughput. The LLM's cost scales with decision rate: 98% retained at 5 decisions/s, 65% at saturation.
Looking into applying our mutable model, lora swapping ANE work to live schema switching.
Kev-0.6B is @jaredpalmer's open implementation (Apache-2.0) of the Jev decision paradigm from @CompleteSkeptic at TypeSafe, following the architecture Archer Hume inferred. We only ported it to the ANE and measured it:
Related builds
