docs / benchmarks
Benchmarks
Every figure we have published, with its date and conditions. Newest first. Results that went against us stay in.
Answer-key correction. Reading the 294 held-out phrasings turned up mistakes in the answer key. The 3,000-phrasing figures came from the same checks, so all of those rows will be re-scored here. The v4 to v5 comparison is fair, since both models were scored against the same key.
2026-09-16 Request to command on the RK3588, run 11
| Measure | Result |
|---|---|
| Main sequence, text input22 calls: routing, confirms, batches, refusals, setup | 2.8 s meanmax 4.95 s |
| "turn on the living room tv"decode 21 tok, 1754 ms. Median of 3. Fastest 1.86 s, prompt cached (88.7 ms) | 2.16 s |
| "lock the front door"decode 19 tok, 1580 ms. Stops at confirm. Median of 3 | 1.9 s |
| "turn off all the lights"batch, decode 43 tok, 3684 ms. Median of 3 | 4.0 s |
| New device list, first call106 of 334 prompt tokens not cached, 2202 ms | 3.98 s |
Model v5, llama.cpp server, 4 threads. Model, gate and dispatcher on the board. IR transmitter simulated.
Against us: two misses in this run. "learn my garage remote" got the refusal. The opposite of "lock the front door" came out as lock, and the confirm step read "lock" back, so nothing moved without a yes. Both are in the v6 plan.
2026-09-15 to 09-16 Unseen devices and 3,000 phrasings, v4
Eight device types were kept out of every training file through v4 (checked across all 49,036 lines) and a home was built from them. Run on 15 Sep. Pass mark fixed before the run. v5 has since trained on most of these types, so it needs a new unseen set.
| Unseen device types, v4 | Result |
|---|---|
| Route to the right device and commandaquarium heater, 3D printer, sauna, kiln, pool pump and more | 14/14 |
| Ask when two devices fittwo 3D printers, two greenhouse vents | 3/3 |
| Act on a group | 2/2 |
| Answer a capability question | 3/3 |
| Add, learn or list devices | 3/3 |
| Decline something that isn't thereevery miss was here | 2/5 |
| Totalpass mark 26, RK3588 | 27/30 |
Then 3,000 generated phrasings across 60 kinds of request: terse, rambling, misspelled, off the wall. All asked on the RK3588. The raw score said 57% and was wrong in both directions, so every failure and 300 sampled passes were re-judged against the real device list. Both figures are estimated from that sample.
| Model v4, 3,000 phrasings | Result |
|---|---|
| Acceptableestimated | ~56% |
| Would have operated something it should not haveestimated | 5 to 7% |
Verified mistakes became training data for v5, trained on one desktop GPU. 294 verified phrasings were held out of that training. On those:
| Held-out, RK3588 | v4 | v5 |
|---|---|---|
| Operated something it should not have3 of the 11 stop at the confirm step | 33 | 11 |
| Missed a refusalof 34 | 31 | 3 |
| Decode, averaged over the run294 questions, 19 min | 9.78 tok/s | |
| Hottest coreopen board, no enclosure, throttled on 147 of 294 | 88.6 °C |
Against us: sustained load throttles the chip, 10.5 down to 9.3 tok/s over the run. Cooling is now a design requirement. A command to every device took up to 46 s, because the model writes each one out. The software can expand a group itself, so v6 won't have to.
2026-09-04 to 09-07 Full loop on the RK3588, v3
Model, dispatcher and gate together on the board for the first time, through a scripted sequence of text commands. Nine runs.
| Measure | Result |
|---|---|
| Decode in the running loop4 threads, runs 8 and 9 | 11.5 tok/s |
| Prompt, new textper token | 21 ms |
| Prompt, a conversation turn9 to 16 new tokens, the rest cached. Run 9 | 0.29 to 0.47 s |
| Text command to action, run 97 Sep, 18 commands. Slowest: a refusal, 5.06 s | 2.87 s mean |
Transmitter simulated. Earlier runs averaged 3.0 to 4.4 s. Decoding took most of the time. When the history changed, reading the prompt again cost up to 3 s. Each run recorded the raw model output. Gate verdicts were recorded from run 4 on.
2026-09-01 llama-bench on the RK3588
| Configuration | Prefill | Decode |
|---|---|---|
| 4 threads, pinned to the A76 cores | 54.2 tok/s | 12.7 tok/s |
| 8 threads, all cores | 50.4 tok/s | 10.1 tok/s |
Trained 3B router, Q4_0, 256-token prompt, 128 generated, heatsinked, CPU only. 53.6 °C after both runs. 4.9x the RK3566. Clears our 3.5 tok/s floor by 3.6x. Newer llama.cpp build than the RK3566 runs.
Eight threads are slower than four. The workload is bandwidth-bound, so the A55 cores add contention, not throughput. The plan is the model on the four A76 cores and the A55 cores for audio.
2026-08-24 Routing eval, trained
30 commands against 15 devices built to confuse: three TVs, three light groups, two locks.
| Behaviour | Before | After |
|---|---|---|
| Routeright device and command | 18/19 | 19/19 |
| Askthree TVs, "turn on the TV" | 0/4 | 4/4 |
| Refusedevice not on the list | 4/7 | 7/7 |
| Total | 22/30 | 30/30 |
Same model before and after training on registry data. About 1 s per command on CPU. The test devices shared vocabulary with training, so this proved the model reads the list it is handed, not that it handles new kinds of device. That test is v4's 27/30 above, on 15 Sep. Training data: 253 hand-written examples plus 4,000 generated ones, each with a different random device list. GPU spend for all fine-tuning to that date: about $15.
2026-08 Evals, fine-tuned 3B
| Eval | Stock 3B | Fine-tuned |
|---|---|---|
| Tool use | 7/25 | 16/25 |
| Tool callright tool and arguments | 7/13 | 10/13 |
| Routing, unseen device listbefore routing training | - | 18/19 |
2026-08 Decode on the RK3566 prototype
| Model | Prefill | Decode |
|---|---|---|
| Qwen2.5-0.5Bfan, 30 °C idle | 70.21 | 14.09 |
| Llama-3.2-1B | 25.81 | 6.30 |
| Llama-3.2-3B | 9.36 | 2.62 |
tok/s. llama-bench, Q4_0, 4x Cortex-A55 at 1.8 GHz, 4 threads, 256-token prompt, 128 generated, active cooling.
2.6x the parameters ran 2.4x slower. Bandwidth-bound. That picked the RK3588. See how it works.
Speech and thermals, RK3566
| Metric | Result |
|---|---|
| Speech to texton device, RK3566 prototype, Aug 2026. Not yet on the RK3588 | 2x realtime |
| Sustained inference temperatureRK3566 prototype, Aug 2026, no throttling | 71 °C |
Against us
- The 3B missed the floor on the RK3566. 2.62 tok/s against a 3.5 floor. Running speech alongside cost the 1B another 38% (6.30 to 3.90), so a contended 3B would land near 1.6. The chip was dropped.
- A fine-tune made a model worse. Tool-use training cut the 1B's grounded-answer score from 16/20 to 13/20. Part of why the product uses the 3B.
- The router overfit on a second pass. Training loss near zero, validation loss rising. The single-pass model shipped.
On measurement
Five times, the instrument broke, not the model: a scorer blind to the word "not", a parser that reported 4/30 for a model doing 22/30, a bench against a server still loading, a server started with more threads than cores, and a training file cut to exactly 2 MiB on upload. The rule since: read the raw output before any score, and treat a sudden collapse as a suspect instrument until proven otherwise.
Tooling: llama.cpp and llama-bench on unmodified retail boards. The firmware goes open source at release, so these runs can be checked on the same hardware.