Most teams deploy LLM inference servers on Arm64 cloud instances with stock configurations: all cores, default batch sizes, and whatever quantization ships with the model. A new open-source tool called armtune demonstrates that this approach leaves substantial performance on the table—and that the "correct" settings are host-specific in ways that challenge conventional wisdom.
The Core Claim
armtune, submitted to the Arm AI Optimization Challenge 2026 (Cloud AI track), automates the search for optimal llama-server configurations on Arm64 hardware. Rather than relying on static heuristics, it profiles the host's ISA features, core topology, and memory constraints; searches the configuration space under live load; and emits a tuned launch configuration plus a reproducible evidence report.
The headline finding: on Apple M2 (4 performance + 4 efficiency cores), running -t 4 beats -t 8 by 1.35x on goodput while using half the cores. Time-to-first-token (TTFT) improves 2.4x and inter-token latency (ITL) 2.2x. The response curve peaks exactly at the performance-core count.
Why Measurement Discipline Was the Hard Part
The project's substantial engineering effort went not into the search algorithm but into measurement validity. The development host—a fanless MacBook Air with 8 GB RAM—was challenging by design: swap was 88% full at idle, and thermal throttling was inevitable under sustained load.
To distinguish genuine configuration effects from noise, armtune implements several validity gates:
- Trial rejection: Any measurement window showing swap-in above 2,048 pages, swap-out above 8,192 pages, CPU throttling, or load-generator CPU consumption above 0.5 core-seconds per second is discarded and retried (up to 3 attempts). Rejected trials contribute nothing to reported figures.
- Drift canary: A reference configuration is re-measured periodically. If its performance degrades beyond tolerance—indicating thermal soak, memory pressure, or background interference—the session is flagged and results quarantined.
- Interleaved ordering: Configurations are tested A,B,A,B rather than A,A,A,B,B,B, ensuring that temporal drift affects both arms equally.
- Warmup discard: A full warmup window runs before measurement to isolate cold-start effects from steady-state performance.
The result: 0.8% relative spread on repeated measurements of the same configuration, compared to approximately 15% from llama-bench at the same sample count. At ±15% spread, a 3% difference between builds is unmeasurable—so an early hypothesis about KleidiAI kernel selection was withdrawn as unproven rather than reported as a finding.
The Mechanism: Static Partitioning Meets Heterogeneous Cores
The M2 result has a specific technical explanation. The underlying inference engine, llama.cpp, partitions GEMM work statically across its threadpool and synchronizes at barriers after each operation. When a thread is scheduled onto an efficiency core, it becomes a straggler that performance cores wait on at every barrier. Adding efficiency cores adds stragglers, not throughput.
This is the opposite of the correct answer on homogeneous Neoverse cores, where -t = ncores normally wins. The optimum is a property of the host, not of the software. armtune's argument is that one hardcoded default cannot be correct across Arm64 parts—a claim demonstrated rather than asserted.
Search Strategy and Scope
The full configuration space spans 10,368 combinations (threads, batch threads, batch sizes, slots, KV cache precision, flash attention, thread polling, memory locking, and build variants). Exhaustive search at ~60 seconds per trial would require 7 days—on a thermally limited machine.
armtune uses coordinate descent with two-phase screening: short trials (20 seconds) rank candidates; long trials (120 seconds, n=5) confirm only the top 5 finalists. Total wall time: approximately 60–90 minutes, though not contiguous due to mandatory cooldown periods. The search is acknowledged to be a local optimizer—claims are "best found by this procedure," never "global optimum."
The Quality Guard: Preventing Silent Accuracy Regressions
A tuner optimizing only for speed will recommend settings that quietly destroy model quality. armtune includes a perplexity gate that verifies every numerically-affecting configuration change against the baseline.
Measured on Qwen2.5-0.5B: KV cache quantization to q4_0 raises perplexity from 19.54 to 295.28—a factor of 15—while looking excellent on every speed and memory metric. Without the guard, this would be recommended. The default search space now excludes q4_0 for cache_type_k based on this evidence.
The guard is honest about its limitations: at the default 12 chunks, combined standard error is ~8%, sufficient to catch catastrophes but not subtleties. Resolving a true 1% regression would need ~700 chunks (~13 minutes per configuration).
Technical Approach and Outputs
armtune provides six subcommands:
profile: Detect ISA features, core topology, cache, and memoryspace: Show the pruned search spacebench: Measure one configuration with validity gatestune: Execute the full searchquality: Run perplexity guard on candidate configurationsreport: Render self-contained HTML reports
Both baseline and KleidiAI-optimized llama.cpp builds are CPU-only by design—Metal and Accelerate disabled—matching the target environment of Arm64 cloud instances. Outputs include a recommended llama-server launch configuration, JSON with full provenance (host profile, ISA features, commit hash, build flags, model SHA256, every trial including rejected ones), and a self-contained HTML report with per-knob response curves.
Limitations and Honest Reporting
The project's documentation is unusually explicit about what it does not claim:
- KleidiAI kernel selection showed a 7% apparent slowdown, but this was smaller than within-variant noise and is reported as unproven
- The coordinate descent strategy cannot find optima requiring simultaneous changes in multiple knobs
- Perplexity resolution limits mean the quality guard catches catastrophes, not subtle degradations
- Energy metrics via
powermetricsare supplementary and macOS-specific; AWS does not expose RAPL counters on Graviton - No SVE/SVE2 support on the development host means results do not transfer to Graviton3/4 without re-measurement
Why This Matters
Arm64 cloud instances (Graviton, Axion, Ampere) are increasingly common for cost-efficient inference. Yet standard deployment practice uses stock configurations that ignore host-specific topology. armtune demonstrates that this invisible performance loss is both real and measurable—and that addressing it requires moving beyond static heuristics toward honest, validity-gated benchmarking.
The tool's broader significance is methodological: it treats measurement infrastructure as a first-class engineering problem, worth substantial effort to get right, and reports uncertainty explicitly rather than burying it in averages. For infrastructure teams running inference at scale, the approach—automated host profiling, contamination-aware trial management, and quality-guarded configuration search—offers a template for reliable optimization.
