ggml-metal-tuning
Offline kernel tuner for the Metal backend.
It sweeps a kernel's config grid on the machine it runs on and prints pasteable table rows for ggml/src/ggml-metal/ggml-metal-tuning.cpp.
This is not a test: it never reports pass/fail on performance. A non-zero exit code means bad arguments or a wrong environment (no Metal device, missing proc bridges), never a perf result.
| tuner | tunes | table |
|---|---|---|
fa-vec |
flash-attn vec (Q, NE) per (dtype, head size, KV depth, batch width) |
fa_vec_tuned_table |
Adding a device to the FA-vec table
Build on the target machine:
cmake -B build -DGGML_METAL=ON
cmake --build build --target ggml-metal-tuning -j
cmake --build build --target test-backend-ops -j
Sweep the grid (6 dtypes x 10 head sizes x 4 KV depths x 9 batch widths; a few hours):
./build/bin/ggml-metal-tuning fa-vec > fa_vec_rows.txt 2> fa_vec_sweep.log
fa_vec_rows.txt is the finished table block, ready to paste: the min-max-regret target, the aggregate benefit gate, the short-KV drop and the pointwise compression are already applied.
fa_vec_sweep.log holds the per-cell timings, bucket coverage, noise floor and any cooldown activity.
Post both: the log is what makes the rows reviewable.
Long sweeps can be split.
--dtype f16,q4_0 and --dk 128,192 restrict the grid, and the emitted rows for one (dtype, head size) do not depend on the others.
Then validate the numerics, where Metal is compared against the CPU reference:
./build/bin/test-backend-ops test -o FLASH_ATTN_EXT -b MTL0
This forces every legal (Q, NE) on dk=128 and dk=576.
The tuner itself does no numerical checks, so the other head sizes have no automated numerical coverage.
If the device is not in enum ggml_metal_device_id yet, register it in ggml/src/ggml-metal/ggml-metal-device.{h,m} first.
The tuner emits whatever token the runtime reports for the machine, so an unregistered device emits GGML_METAL_DEVICE_GENERIC and its rows would apply to every unknown device.
Thermal throttling
Long sweeps heat the GPU, and a throttled measurement is indistinguishable from a slow kernel.
The tuner re-measures a fixed baseline config every four candidates as an anchor.
When the anchor drifts more than --cool-drift (10% by default) from the coolest anchor seen in that cell, the tuner:
- discards every candidate measured since the last clean anchor,
- sleeps with exponential backoff until the anchor comes back within
--cool-eps(3%), - re-measures the discarded candidates.
If it cannot cool down within --cool-max-wait seconds, or a cell needs more than --cool-max-retry rounds, that cell is dropped from the table and reported on stderr.
--no-cooldown only warns on drift and keeps the measurement.
Use it to reproduce a sweep taken without cooling.