LLM-Inference/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-08-12 22:29:39 +04:00
Code Issues Packages Projects Releases Wiki Activity
4,676 Commits 993 Branches 1 Tag
9da563d09d95e1ed029090e53464282ba9e4c2c5
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
mb8565GitHubClaude Opus 4.8mgkwillKawrakow
 9da563d09d deepseek2 : GLM-DSA sparse attention (lightning indexer), --dsa off by default (#2045) 
* Add GLM-5.2/DeepSeek-V3.2 DSA lightning indexer (batch-local, single-seq prefill)

Implements the sparse top-k "lightning indexer" attention for LLM_ARCH_GLM_DSA
in build_deepseek2_layer_attention (ik's deepseek2 graph).

What it does (per layer, gated on model.arch==GLM_DSA && indexer_attn_q_b):
- indexer_q = indexer_attn_q_b(q_lora latent), split rope(64)/nope(64), NEOX-rope
  the pe part, concat. indexer_k = indexer_attn_k(attn_norm out), LayerNorm w/ bias,
  same rope/concat (single key head, MQA).
- scores = relu(indexer_k . indexer_q), scaled per-head weights (indexer_proj),
  summed over heads, + base causal mask, then ggml_top_k(min(top_k, n_tokens)).
- sparse mask: ggml_fill(-inf) -> ggml_set_rows(0) at top_k positions -> + causal,
  used in the soft_max_ext attention path (-mla 1 -fa 0) instead of KQ_mask.

Simplifications (intentional, proven sound):
- Batch-local: no indexer KV-cache. Indexer keys are the current batch tokens.
- Walsh-Hadamard transform omitted: orthonormal rotation, (Hq).(Hk)==q.k, no score change.

Validation (GLM-5.2-UD-IQ2_M, 3x P100, -mla 1 -fa 0):
- Compiles clean (CUDA sm_60); loads and runs.
- c512 -b512 (n_seq=1) PPL = 2.7760, byte-identical to dense baseline (indexer
  disabled) = 2.7760, all 8 chunks match -> indexer is an exact no-op when
  top_k>=n_tokens. Proves correctness-preservation.
- 3105-token prompt completion (top_k=2048 < 3105 -> indexer ACTIVELY masks):
  prompt-eval produces coherent, accurate continuation, identical to dense for the
  prompt+early-gen tokens. No NaN/crash. Confirms the masking path works in prefill.

Known limitations (documented follow-ups, NOT handled):
- Single-sequence prefill only. Multi-sequence batches (n_seq>1, e.g. perplexity
  default n_batch>n_ctx) and kv_head>0 (decode) break the batch-local key->slot
  mapping. n_seq>1 -> NaN (use n_batch==n_ctx). Decode (kv_head>0): each generated
  token sees only itself as an indexer key, so generation degenerates into repetition
  after the prompt (dense A/B stays coherent) -- this is the decode-cache stub, the
  documented next step.
- Flash-attn path (-fa 1, F16 mask) still uses dense KQ_mask (soft_max path only).
- Decode indexer KV-cache + Hadamard cached-K storage not implemented.

Runtime gate: DSA_INDEXER_DISABLE=1 falls back to dense attention (for A/B).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-5.2 DSA indexer: decode-correct via persistent indexer-K cache

Make the lightning-indexer correct for DECODE (not just prefill). Previously the
indexer was batch-local, so a generated token only scored against itself and
generation degenerated. Now the indexer keys are cached across the full context.

Changes
- llama_kv_cache: add per-layer indexer-key cache `kr_l` [indexer_head_size, kv_size]
  (F16, MQA single head), allocated alongside the MLA latent cache for GLM_DSA.
- build_deepseek2_dsa_indexer: write the batch's (Hadamard-rotated) indexer keys to
  kr_l at kv_head, read back the full [128, n_kv] cached keys, and score the indexer
  queries against ALL past keys. Returns the full descending argsort of the scores.
- Walsh-Hadamard rotation of indexer q/k (cparams.dsa_indexer_hadamard, default on;
  filled in llama_set_inputs). Score-preserving; improves cached-K F16 precision.
- build_deepseek2_dsa_sparse_mask: rank-based full-coverage scatter (write a 0/-BIG
  penalty into EVERY key slot keyed by rank) instead of partial set_rows into a -inf
  fill — the CUDA in-place set_rows does not preserve an un-written base, which had
  corrupted decode when n_kv > top_k.
- Attention-sink force-inclusion (DSA_SINK, default 1): boost the first key(s) so the
  sink always survives top-k. The IQ2_M-quantized indexer under-ranks the sink, and
  masking it collapsed decode; with the boost, top_k=2048 over n_kv>2048 stays coherent.

ggml backend fixes (needed by the indexer)
- CUDA argsort: report unsupported when padded ncols > 1024 (one-thread-per-column
  bitonic launch limit) so the scheduler falls back to the CPU argsort. Fixes
  "invalid configuration argument" for top_k over a large n_kv.
- CUDA cpy/dup: support I32 -> I32 (top_k index copies / cross-backend moves).

Validation (GLM-5.2-UD-IQ2_M, 3xP100 + --cpu-moe, -mla 1 -fa 0)
- c512 PPL = 2.0743, byte-identical to dense (all 8 chunks): no-op path exact.
- Short-context decode (300 tok): coherent, identical to dense.
- Long-context decode (2521-tok prompt, n_kv>top_k, real masking of ~474 keys,
  120+ tok generated): coherent with the sink boost; dense A/B also coherent.

Gated behind arch==GLM_DSA + indexer tensors + kr_l cache; DSA_INDEXER_DISABLE=1
forces dense. Remaining: FA path still uses the dense KQ_mask; multi-sequence
(n_seq>1) batches; deepseek32 arch wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-5.2 DSA indexer: wire sparse mask into the flash-attention path (-fa 1)

The DSA sparse top-k mask is now applied on the -fa 1 path (our serving config),
not just -fa 0 soft_max. c512 PPL on -fa 1 = 2.0743, byte-identical to dense
(no regression, indexer no-op exact at n_kv <= top_k). Gated arch==GLM_DSA with
DSA_INDEXER_DISABLE escape; -fa 0 path unchanged.

Long-context -fa 1 decode coherence (n_kv > top_k, mask actually biting) validation
is still running at commit time; the FA mask reuses the same full-coverage scatter
proven coherent on the -fa 0 decode path, so it should hold, but confirm before
relying on long-context -fa 1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-5.2 DSA indexer: UPDATE 4 — MLA-FA fix merged, FA path validated, multi-seq characterized

Document the re-validation after cherry-picking the MLA-FA vec-decode fix (5f18dcc0):

- FA path is ALIVE. Long-ctx -fa 1 decode (2521-tok prompt > top_k, mask actively
  biting) is now COHERENT at -mla 1 and -mla 3, vs the pre-fix degeneration into
  "0.0.0.0..." repetition. Matches dense (DSA_INDEXER_DISABLE) and -fa 0 controls.
- c512 -fa 1 PPL: indexer-ON == dense == 2.0854, byte-identical all 8 chunks (exact
  no-op when n_kv <= top_k; no regression). The 2.0743->2.0854 shift is the MLA-FA
  fix changing V accumulation, not an indexer artifact (ON==dense proves it).
- Indexer is feature-complete + validated for single-seq prefill+decode on both
  -fa 0 and -fa 1, at -mla 1 and -mla 3 (the R740 serving target).

Remaining PR gaps, characterized honestly:
- Multi-seq (n_seq>1) with active mask is BROKEN (n_seq=2 c4096 PPL 62.6 vs dense
  multi-seq 2.54 and single-seq indexer 3.05). No NaN/crash anymore. Root cause:
  the indexer uses a single scalar kv_head/n_kv for the whole ubatch; multi-seq
  needs per-sequence cache writes + per-sequence top-k. Fix deferred (structural).
- deepseek32 arch: N/A in this fork. DSA lives entirely under LLM_ARCH_GLM_DSA;
  there is no LLM_ARCH_DEEPSEEK32 enum. Documented the steps to add one if a real
  deepseek32 GGUF is ever served.

Also commit DSA_REFERENCE.md (verbatim mainline deepseek32/glm-dsa source, the port
reference), trimmed of a stray agent-handoff footer.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-5.2 DSA indexer: per-sequence attention sink — fix multi-seq (n_seq>1)

UPDATE 5. The DSA lightning indexer was numerically broken for multi-sequence
batches once the top-k mask bites (n_kv > top_k): c4096 n_seq=2 PPL 62.6 vs
dense 2.54, while single-seq was fine. Root cause: the attention-sink
force-include boosted the GLOBAL key range [0, n_sink) by +1e20, which only
protects sequence 0's sink. With several sequences packed contiguously into one
ubatch (seq 0 at cells [0,n0), seq 1 at [n0,n1), ...), every non-first
sequence's sink lives at cell n0.. (not cell 0), got no boost, and was dropped
from top-k once the mask bites — collapsing that sequence (chunk[2]=61.2 while
chunk[1]=2.33).

The cache write and score/argsort were already per-sequence correct: tokens are
placed contiguously like the main K cache, and the base KQ_mask (filled from
kv_self.cells[i].has_seq_id) already drives cross-seq keys to -inf before
argsort. Only the sink was anchored at the wrong (global) cell.

Fix: replace the global arange sink boost with a per-graph input tensor
inp_dsa_sink {n_kv, n_tokens} (F32), filled on the CPU in llama_set_inputs from
kv_self.cells exactly like the KQ_mask:
  inp_dsa_sink[j,i] = 1e20 iff cell[i].pos in [0,n_sink) AND
                              cell[i].has_seq_id(seq_of_query_j), else 0
so each query force-includes only its OWN sequence's sink. For a single
contiguous sequence from pos 0 this is exactly the old "cell index < n_sink"
set with the same magnitude, so n_seq==1 is byte-identical.

Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, wikitext-2):
- c4096 n_seq=2 indexer chunk[2]: 61.2 -> 3.07 (== single-seq 3.05).
- c2048 topk=1024 (mask bites): n_seq=4 == n_seq=1 chunk-for-chunk
  (2.5005/2.6080/2.7759/3.1137 vs .../3.1138) -> multi-seq is numerically
  identical to processing each sequence alone.
- c512 n_seq=1 indexer ON == dense, all 4 chunks byte-identical (no regression).

n_seq=4 at full c4096 (n_kv=16384) OOMs the P100 compute buffer (capacity, not
correctness; n_seq=4 proven correct at c2048/n_kv=8192).

GLM-5.2 DSA indexer is now sequence-correct for n_seq>=1, prefill+decode,
soft_max+FA, -mla 1/-mla 3. Fully general and PR-ready.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-5.2 DSA indexer: UPDATE 6 — serving-correctness (kr_l maintained across shift/defrag/seq-ops; per-seq sink on first-present pos)

An adversarial review found the indexer was proven on the perplexity path but
not the serving path: the persistent indexer-K cache kr_l was written/read but
never *maintained* by the KV-cache mutators, and the attention sink anchored on
absolute pos<n_sink (wrong after multi-turn seq_rm). This closes those gaps and
pins down what is actually reachable on the MLA model.

kr_l maintenance:
- build_k_shift (llama-build-context.cpp): rotate the indexer keys by the same
  per-cell delta as the main K. The cached key is H*concat(RoPE(k_pe,pos),k_nope),
  so un-Hadamard (H sym/orthonormal => H*H=I) -> RoPE-delta the pe sub-block ->
  re-Hadamard. Exact because GLM-DSA has no rope-scaling metadata (ext_factor=0,
  attn_factor=1, freq_scale=1), so NEOX RoPE is pure/composable. Params mirror the
  forward indexer RoPE exactly (rope_factors=nullptr); no DEEPSEEK2 yarn-shift leak.
  Non-in-place (cont->rope->concat->re-Had->cpy), no aliasing. K-shift Hadamard
  input filled in llama_set_k_shift with the identical Sylvester construction.
- build_defrag: kr_l row-move mirrors the k_l move (defrag never changes pos, so
  no re-RoPE). max_moves divisor 6->9 *n_layer when the indexer cache is present.
- seq_rm/seq_cp/seq_keep are metadata-only (verified) so kr_l rows stay matched to
  cells; seq_add/seq_div set has_shift and route through K-shift. No seq-op change.

Per-seq sink (llama.cpp llama_set_inputs): anchor on each sequence's FIRST PRESENT
pos (min present pos over the scored n_kv span), not absolute pos<n_sink. After
multi-turn seq_rm drops a sequence's early tokens its earliest survivor has
pos>=n_sink; the absolute test would protect nothing. Fresh seq at pos 0 => min=0
=> byte-identical to the old behaviour.

Serving-shift finding (the whole point): a RoPE context-shift on this model is
REFUSED BY THE ENGINE. get_can_shift() returns false for all MLA models
(is_mla_model() includes GLM_DSA); llama_kv_cache_update returns 1 ->
"main : failed to eval". Reproduced AND isolated with a dense control
(DSA_INDEXER_DISABLE=1): dense fails identically at the same token. The failure is
pre-existing MLA engine behaviour, independent of the indexer. On the MLA path the
shift never happens, so the indexer's kr_l can never desync via K-shift; the
build_k_shift kr_l block is correct-and-dormant (documented loudly in code).

Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, GGML_CUDA_NO_PINNED=1,
numactl --interleave=all, wikitext-2):
- No regression: c512 n_seq=1 indexer ON == dense == 2.1957 +/- 0.12031,
  byte-identical all 4 chunks (2.2770/2.8741/2.3956/2.1957).
- Multi-seq: c4096 n_seq=2 chunk[1]=2.33 chunk[2]=3.07 healthy (== UPDATE 5;
  per-seq sink change did not regress).
- Serving shift: engine-refused for MLA, dense control fails identically.
- Independent adversarial review: GO, no correctness defect in the diff.
- Build clean (llama-cli, llama-perplexity, sm_60).

Comments updated (build_deepseek2.cpp): multi-seq+FA no longer limitations; sink
description matches per-seq min-pos anchoring; BIG=1e30 masks on both soft_max and
FA paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-5.2 DSA indexer: UPDATE 7 — FIX latent graph-reuse cache-fixup omission for the kr_l indexer cache

update_cache_copies() re-points the K/V cache writes to the current kv_head whenever a
compute graph is REUSED (can_reuse_graph reuses iff kv_self.n == prev->n_kv). The persistent
indexer-key cache write (kr_l) is a separate ggml_cpy whose destination view bakes kv_head at
graph-build time, and it was NEVER registered for that fixup. Under FA the cache pads to 256,
so consecutive single-token decode ubatches share the same padded n_kv and the graph IS reused;
without the fixup the kr_l write keeps landing in the first ubatch's slot and later ubatches
never populate their own recent index-key cells (those cells stay at the alloc-zeroed 0.0).
Structurally identical to the MiniMax MSA bug (fork commit 133d14c9).

Fix (mirrors the K/V cache_copies fixup, same shape as MSA 133d14c9):
- llama-context.h: new std::vector<CacheCopy> dsa_cache_copies.
- llama.cpp ctor: resize dsa_cache_copies to n_layer (null entries -> no-op when DSA off).
- build_deepseek2.cpp: register the kr_l ggml_cpy as dsa_cache_copies[il] = {kr_cpy, kr->nb[1]}.
- llama.cpp update_cache_copies(): re-point each registered cpy view_offs = kv_head*step and
  patch src[1]->data/data, exactly like K/V, with the c.cpy->view_src == kv_self.kr_l[il]
  (+ null/op) guard the MSA fix omitted. soft_max / non-DSA paths byte-identical.

Validation (GLM-5.2-UD-IQ2_M, 3x P100 -ngl 99 --cpu-moe -t 32, NO_PINNED, P2P-disable patch
re-applied to get a working multi-GPU baseline — see UPDATE 7.3; that patch was lost in the
upstream rebase and is required separately):
- c512 -fa1 -mla3 indexer ON: 2.1983 (== prior baseline; build healthy).
- Long-ctx FA decode, 2735-tok recall prompt, -mla3 -fa1 temp0, reuse ON (default): coherent,
  correct deep-context recall ("Dr. Mariana Velasquez ... Daniel Okonkwo") on BOTH the fixed and
  the unfixed binary.
- ub128 PPL -fa1 -mla3 reuse ON, unfixed: 1.7239/1.8211/2.1888/2.4517, healthy (no inflation).

Honest scope: the bug is real in code but LATENT for GLM-DSA at its configured top_k=2048
(permissive selection keeps the genuinely-attended recent blocks even when reuse leaves some
recent index-key cells stale), unlike MSA's tighter top-k where it inflated PPL ~2x. The fix is
correct and prevents the latent corruption from biting at any tighter top_k / longer ctx /
future serving config. The pre-P2P-patch "nan" seen at ub128 was P2P corruption, not this bug.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-DSA: convert sparse-attention control from env vars to CLI args (off by default)

Implements ikawrakow's direction from discussion #2040: the DSA sparse
indexer must be controllable via command-line argument (not environment
variables), and must be OFF by default for now.

Control surface, before -> after:
  DSA_INDEXER_DISABLE (env, inverted: on-by-default)  -> --dsa / -dsa
      (cparams.dsa, default false; opt-in, dense-by-default)
  DSA_TOPK_OVERRIDE   (env)                            -> --dsa-top-k N / -dsatk N
      (cparams.dsa_top_k, default -1 == model's configured indexer_top_k)
  DSA_HADAMARD_DISABLE, DSA_SINK (env)                 -> kept as DEBUG-ONLY env
      knobs (clearly commented; no CLI surface, not system on/off controls)

Plumbing mirrors existing boolean/int feature flags (-mla, -khad):
  include/llama.h        llama_context_params {bool dsa; int dsa_top_k;}
  src/llama.cpp          default_params (false / -1); cparams assignment
  src/llama-cparams.h    llama_cparams {bool dsa=false; int dsa_top_k=-1;}
  common/common.h        gpt_params {bool dsa=false; int dsa_top_k=-1;}
  common/common.cpp      arg parse + help text + cparams copy
  src/graphs/build_deepseek2.cpp  gate now checks cparams.dsa instead of
      getenv; top-k override reads cparams.dsa_top_k. Stays arch-gated to
      LLM_ARCH_GLM_DSA. When --dsa is off (default) the indexer function is
      never called -> existing dense MLA path, byte-identical to no-feature.

Validation (GLM-5.2-UD-IQ2_M, 3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1,
wikitext-2, 4 chunks @ c2560):
  --dsa OFF (default, dense):              PPL 2.4151  (graph nodes 4166)
  --dsa ON, default top_k=2048:            PPL 2.4697  (graph nodes 8846)
  --dsa ON, --dsa-top-k 1024:              PPL 3.5107
Off-by-default runs the dense path; ON activates the indexer (node count
jumps, PPL shifts as the top-k mask bites once n_kv > top_k). No env var
is consulted for the primary on/off or the top-k knob.

Graph-parallel (-sm graph) interaction (the item ikawrakow flagged):
Under -sm graph the MLA layers are TP-split (wo->extra) and route to
build_deepseek2_tp_attention(), which contains NO indexer code. So --dsa
is silently a NO-OP under -sm graph: it does not error or crash, it runs
dense. Empirically, --dsa --dsa-top-k 1024 under -sm graph gives
PPL 2.4308 (chunks 1.6967/1.7906/2.1664/2.4308) -- the dense baseline
(2.4151), NOT the DSA top_k=1024 numbers (3.5107). The 0.016 delta is
f16 TP-reduce numerics, not DSA. Conclusion: DSA "works under deepseek2"
only on the non-TP (layer) path; serving DSA with -sm graph would require
wiring the indexer into the TP attention path (or a dedicated DSA arch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-DSA: warn that --dsa is inactive under -sm graph/attn (TP path runs dense MLA)

The DSA lightning indexer is built only in the layer-mode (non-TP) attention
path. Under -sm graph / -sm attn the tensor-parallel attention path has no
indexer, so --dsa would silently run dense MLA. Emit a clear one-time
LLAMA_LOG_WARN at context creation instead of degrading silently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-DSA: drop in-tree dev reference docs from the PR branch

DSA_REFERENCE.md and the R740 progress note are development scratch, not
part of the submission. Remove them so the PR diff is code-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* GLM-DSA: fix CPU-only crashes in the sparse-attention path

PR #2045 adds GLM-DSA sparse attention but was validated on CUDA (--cpu-moe).
A CPU-only build (-ngl 0 --dsa) crashes in four spots where the CUDA backend
tolerates something the CPU backend does not. These make GLM-5.2 --dsa run
coherently on CPU; with --dsa off they are no-ops (DSA CPU path only).

1. set_rows into an F32 dest segfaults (ggml.c set_rows_f32):
   type_traits[F32].from_float is NULL, so the DSA sparse-mask scatter calls a
   NULL fn (segfault at ip=0). memcpy when the dest is F32. CUDA has a real F32
   set_rows path, so this only bit the CPU build.

2. ggml_add(F32 score, F16 mask) aborts on CPU (build_deepseek2_dsa_indexer and
   build_deepseek2_dsa_sparse_mask): under -fa 1 the dense KQ_mask is F16 and CPU
   add only accepts F32+F16 when src0 is F16. Cast the causal mask view to F32.
   CUDA's add accepts the mixed types.

3. dsa_fa_mask dim-1 concat must be F32 on CPU (build_deepseek2_dsa_fa_mask):
   CPU ggml_concat only supports F16 along dim 0; do the row (dim-1) concat in
   F32 then cast the result to F16. CUDA supports the F16 dim-1 concat.

4. indexer k_norm epsilon is 0 -> ggml_norm aborts (llama-hparams.cpp): the
   lightning-indexer k_norm is a non-RMS LayerNorm using f_norm_eps, but the
   GLM-DSA GGUF only carries the RMS eps so f_norm_eps stays 0
   (GGML_ASSERT(eps > 0)). Mirror the RMS eps. CUDA's norm doesn't assert on eps=0.

Validated: GLM-5.2 UD-Q4_K_M, single-socket Xeon w7-2475X, CPU-only (-ngl 0 --dsa)
- coherent at 49K+ ctx, correct 30K needle retrieval, prefill flat with length
(~32 tok/s, the O(L) DSA signature) vs the dense build's O(L^2) decline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* DSA: loop over attention heads + use builtin Hadamard

* DSA: ggml_blend

* DSA: remove a bunch of unnecessary ggml_cont

* DSA: fix CUDA blend - but something is still wrong

* DSA: use ggml_top_k instead of ggml_argsort when FA is ON

* CUDA: add CUB based argsort

* DSA: avoid graph leaves

* Various

* GLM-5.2 DSA: IndexShare (shared layers reuse full-layer top-k)

GLM-5.2's indexer_types marks 21 'full' layers that compute their own
lightning-indexer top-k and 57 'shared' layers that reuse the previous
full layer's top-k. This port computed an independent top-k on every
layer, which mis-selects keys on the 57 shared layers (the transformers
reference sets indexer=None on shared layers and reuses prev_topk).

Shared layers now reuse the most-recent full layer's selection. Full/
shared map derived from the config rule (full iff il<=1 or il%4==2),
which reproduces indexer_types exactly; loader can later override from
GGUF metadata. Built on #2063's tree; head-loop/ggml_hadamard/ggml_blend/
argsort/FA-mask unchanged.

4K PPL (unsloth IQ2_M, top_k 2048, CPU): DSA-on 3.1922 -> 2.7111, dense
2.6972 (~97% of the gap). top_k>=n_kv reproduces dense exactly. Single-
seq and 4x8 parallel decode coherent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Apply suggestion from @ikawrakow

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: mgkwill <168222+mgkwill@users.noreply.github.com>
Co-authored-by: Kawrakow <iwankawrakow@gmail.com>
2026-07-02 09:36:49 +02:00
.devops
Update nix flake: sync with upstream, fix for newer nixpkgs (#1371)
2026-03-24 08:17:52 +01:00
.github
Fix Build and Push Container Image (#1633)
2026-04-16 11:24:19 +02:00
ci
common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369)
2026-03-09 11:03:33 +01:00
cmake
Graph parallel: the next generation (#1080)
2025-12-24 08:31:48 +01:00
common
deepseek2 : GLM-DSA sparse attention (lightning indexer), --dsa off by default (#2045)
2026-07-02 09:36:49 +02:00
docker
Add Turing and Ampere (A100) GGML to docker build file (#1691)
2026-05-07 12:58:58 +03:00
docs
on-demand tensor reload (#1989)
2026-06-22 16:36:34 +02:00
examples
Normalize disabled context-shift overflow error (#2051)
2026-06-29 08:30:05 +02:00
ggml
deepseek2 : GLM-DSA sparse attention (lightning indexer), --dsa off by default (#2045)
2026-07-02 09:36:49 +02:00
gguf-py
DFlash: support MiMo-V2.5-Pro draft conversion and runtime (#2048)
2026-06-29 13:26:29 +02:00
github-data
Add GitHub data: filename sanitization (#640)
2025-07-23 13:31:53 +02:00
grammars
llama : add token matching support to llama-grammar (#1220)
2026-02-03 07:57:17 +02:00
include
deepseek2 : GLM-DSA sparse attention (lightning indexer), --dsa off by default (#2045)
2026-07-02 09:36:49 +02:00
media
README: add graphic for matrix multiplication (#6881)
2024-04-24 21:29:13 +02:00
models
Add native MiniMax-M3 tool call parser (#2008)
2026-06-23 09:36:02 +02:00
pocs
build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)
2024-06-13 00:41:52 +01:00
prompts
llama : add Qwen support (#4281)
2023-12-01 20:16:31 +02:00
requirements
Prune examples/llava. Dead code. (#2025)
2026-06-26 08:48:48 +02:00
scripts
Replace image resizers with avx2/neon simd impls from stb_img_resize2.h
2026-06-14 20:28:08 +01:00
spm-headers
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
src
deepseek2 : GLM-DSA sparse attention (lightning indexer), --dsa off by default (#2045)
2026-07-02 09:36:49 +02:00
tests
chat: Cohere2MoE/North Code: parse unopened thinking under --reasoning off (follow-up to #1968) (#2012)
2026-06-24 09:04:41 +02:00
vendor
Replace image resizers with avx2/neon simd impls from stb_img_resize2.h
2026-06-14 20:28:08 +01:00
.dockerignore
Added workflow to build container images (#1279)
2026-04-10 08:06:47 +02:00
.ecrc
Nomic Vulkan backend (#4456)
2024-01-29 15:50:50 -05:00
.editorconfig
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
.flake8
Initial dflash implementation
2026-05-28 18:57:58 -03:00
.gitignore
webui: update llamacpp webui (#1903)
2026-06-04 15:41:23 +02:00
.gitmodules
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
.mailmap
Add .mailmap
2025-07-22 14:53:50 +03:00
.pre-commit-config.yaml
Update autofix and presets (#1867)
2026-05-24 07:30:44 +03:00
AUTHORS
Update AUTHORS
2026-06-18 08:11:41 +00:00
CMakeLists.txt
server: stop processing the prompt when client disconnects (#1134)
2026-01-13 07:56:59 +02:00
CMakePresets.json
Update autofix and presets (#1867)
2026-05-24 07:30:44 +03:00
CONTRIBUTING.md
Merge mainline - Aug 12 2024 (#17)
2024-08-12 15:14:32 +02:00
convert_hf_to_gguf_update.py
convert : recognize gpt-oss (o200k_harmony) tokenizer (#2060)
2026-06-30 09:02:53 +02:00
convert_hf_to_gguf.py
convert : map self-contained DFlash draft embed_tokens (#2062)
2026-07-01 15:45:02 +02:00
convert_imatrix_gguf_to_dat.py
Vibe coded script + constants from mainline + pip requirements (#1405)
2026-03-11 15:41:17 +01:00
convert_llama_ggml_to_gguf.py
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
convert_lora_to_gguf.py
Fix missing rope_freqs with convert_hf_to_gguf (#402)
2025-05-09 09:17:41 -05:00
docker-bake.hcl
Fix Build and Push Container Image (#1633)
2026-04-16 11:24:19 +02:00
docker-bake.override.hcl
Update docker readme and local build (#1710)
2026-04-30 08:16:12 +02:00
flake.lock
Update nix flake: sync with upstream, fix for newer nixpkgs (#1371)
2026-03-24 08:17:52 +01:00
flake.nix
Update nix flake: sync with upstream, fix for newer nixpkgs (#1371)
2026-03-24 08:17:52 +01:00
LICENSE
Use links for ggml/llama.cpp authors (#318)
2025-04-07 17:25:06 +02:00
llama-mmap.h
on-demand tensor reload (#1989)
2026-06-22 16:36:34 +02:00
mypy.ini
convert : partially revert PR #4818 (#5041)
2024-01-20 18:14:18 -05:00
Package.swift
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
poetry.lock
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
pyproject.toml
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
pyrightconfig.json
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
README.md
Update docs (#1956)
2026-06-12 08:24:22 +02:00
requirements.txt
Vibe coded script + constants from mainline + pip requirements (#1405)
2026-03-11 15:41:17 +01:00
test-function-calls.md
Function calling support for Kimi-K2 (#628)
2025-07-23 18:11:42 +02:00

README.md

ik_llama.cpp: llama.cpp fork with better CPU performance

License: MIT

TL;DR

This repository is a fork of llama.cpp with better CPU and hybrid GPU/CPU performance, new SOTA quantization types, first-class Bitnet support, better DeepSeek performance via MLA, FlashMLA, fused MoE operations and tensor overrides for hybrid GPU/CPU inference, row-interleaved quant packing, etc.

Important

If you are running hybrid CPU/GPU inference for MoE models with all or some experts left on the CPU, do not use -rtr unless you know what you are doing. The -rtr option causes all tensors left in RAM to be repacked to row-interleaved format while loading the model. As not all quantization types have a CUDA implementation, this will result in matrix multiplications with these tensors to be always done on the CPU, even when it would have been much better to offload the computation to the GPU, typically resulting in much lower prompt processing speed. Most notably, k-quants (K2_K, Q3_K, Q4_K, Q5_K, Q6_K) do not have CUDA row-interleaved implementation.

Note

The only fully functional and performant compute backends are CPU (AVX2 or better, ARM_NEON or better) and CUDA (Turing or newer). Please do not enter issues related to ROCm, Vulkan, Metal, old Nvidia GPUs, AVX CPUs, etc. They will not get resolved unless you roll up your sleeves and help bring your favorite backend up to speed. With the current regular contributors this project simply does not have the bandwidth to work on all backends available in llama.cpp.

Important

Do not use quantized models from Unsloth that have _XL in their name. These are likely to not work with ik_llama.cpp.

The above has caused some stir, so to clarify: the Unsloth _XL models that are likely to not work are those that contain f16 tensors (which is never a good idea in the first place). All others are fine.

Note

Some users have reported issues with graph parallel (a.k.a. split mode graph) and partial GPU offload (using --cpu-moe or --n-cpu-moe or tensor overrides). If you are using/want to use split mode graph and observe gibberish/incoherent responses, try adding -cuda graphs=0 to your command line.

Quickstart

Prerequisites

git clone https://github.com/ikawrakow/ik_llama.cpp

cd ik_llama.cpp

On Debian/Ubuntu Linux, install the required packages (if using another Linux distro, you need to find the corresponding packages and adapt):

apt-get update && apt-get install build-essential git libcurl4-openssl-dev curl libgomp1 cmake

Build for CPU

cmake -B build -DGGML_NATIVE=ON

cmake --build build --config Release -j$(nproc)

For AVX-512-capable CPUs (AMD Zen4 / Intel Sapphire Rapids+), see docs/build.md section "CPU build flags for AVX-512" for the additional flags that activate the IQK quantized GEMM kernels (the HAVE_FANCY_SIMD path). Without those flags, a vanilla Release build silently falls back to the AVX2 path on this hardware.

Build for GPU

Install Nvidia Drivers and CUDA Toolkit.

cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON

cmake --build build --config Release -j$(nproc)

Step-by-step instructions for a case of a successful Windows build

https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md

Run

Download .gguf model files (e.g. bartowski/Qwen_Qwen3-0.6B-IQ4_NL.gguf) to your favorite directory (e.g. /my_local_files/gguf).

Start the server with one of the commands (CPU or GPU):

./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096

./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096 -ngl 999

That's all! Open http://127.0.0.1:8080 in Browser start chatting.

Step by step guide for ik_llama.cpp in podman/docker container including llama-swap

Common parameters and options

Latest News

Model Support

LlaMA-3-Nemotron PR 377, Qwen3 PR 355, GLM-4 PR 344, Command-A PR 341, bitnet-b1.58-2B-4T PR 337, LLaMA-4 PR 321, Gemma3 PR 276, DeepSeek-V3 PR 176, Kimi-2 PR 609, dots.llm1 PR 573, Hunyuan PR 565, GLM-4.5 PR 668 (4.5/4.6/4.7/AIR), Ernie 4.5 MOE and 0.3B PR 759, grok-2 PR 782, Ling/Ring (Bailing-MoE2) PR 833, Qwen3-VL PR 883, SmolLM3 PR 934, GigaChat3 PR 995, ministral3 PR 1030, Mimo-V2-Flash PR 1096, GLM-4.7-Flash PR 1168, Seed-OSS PR 1218, Step-3.5-Flash PR 1231, GLM-5 PR 1268, Qwen3-Next PR 1266, Qwen3.5-MoE PR 1288 and dense Qwen-3.5 1326, Mistral 4 PR 1450, Bonsai 1-bit PR 1570, Gemma4 PR 1581, Mimo-2.5 PR 1723, JetBrains Mellum2 PR 1919, Poolside Laguna XS.2 PR 1911, Cohere2-MoE North Mini Code PR 1945

Quantization

Quantization additions

Trellis quants (IQ1_KT, IQ2_KT, IQ3_KT, IQ4_KT)

Information and the original CUDA implementation in PR 113. Additional implementations: Metal PR 475, Neon PR 471, CPU PR 441. IQ1_KT was added more recently in PR 616. Note: these are base on a novel, integer-base trellis, which allows to achieve reasonable CPU performance, see PR 529 and PRs quoted there for details.

IQK quants

Information can be found in Discussion 8.

Initial implementations (Zen4, AVX2, NEON): IQ5_KS_R4 PR 426, IQ5_KS PR 422, IQ4_KS_R4 PR 150, IQ5_K_R4 PR 149, IQ2_K_R4 PR 146, IQ3_K_R4 PR 145, IQ4_K_R4 PR 138, IQ4_KSS PR 89, IQ2_KS PR 85, IQ4_KS PR 83, IQ6_K PR 14, IQ2_K, IQ3_K and IQ5_K PR 7, IQ4_K PR 6

Cuda implementations: IQ4_KS_R4 and IQ5_KS_R4 PR 493, IQ1_S_R4 PR 492, IQ1_M_R4 PR 494. IQ4_KS_R4 and IQ5_KS_R4 PR 462, IQ2_K_R4, IQ3_K_R4, IQ4_K_R4, IQ5_K_R4 PR 461, IQ4_K, IQ5_K, IQ6_K PR 417, IQ2_KS, IQ2_K, IQ3_K PR 418

IQ2_KL is a more recent addition in PR 602

Hadamard transforms for K-cache

CPU PR 1033 and CUDA PR 1034

Hadamard transforms for V-cache

PR 1527

MXFP4 as used in gpt-oss models

Implemented for Zen4, AVX2, ARM_NEON, Metal, CUDA PR 682

Quantization improvements

  • IQ1_M PR 327, IQ2_XS PR 312, Q2_K, Q4_K, Q5_K, Q4_1, Q5_1 PR 302, Q4_0, Q5_0, Q6_0, Q3_K, Q6_K, IQ4_XS, IQ4_NL PR 295
  • Low perplexity Q4_0 KV cache PR 1547 PR 1556
  • MTP: option to use re-quantized output tensor --mtp-requantize-output-tensor new_type PR 1809

Quantization performance improvements

  • Much faster CPU prompt processing for all non-interleaved quants. Initial idea in PR 515 and PR 531, with many follow up PRs to apply to all quantization types for the 3 supported CPU platforms.
  • All quantization types now have quantized matrix multiplication CUDA kernels, see PR 557 and several others
  • Faster CPU prompt processing for Trellis quants and MoE models. PR 488
  • Trellis quants: faster CPU prompt processing PR 482.
  • Minor (~2%) iq2_ks TG performance improvement on CUDA PR 468
  • Faster IQ3_KT and IQ4_KT PR 453
  • Zen4: Faster PP for IQ2_KS, IQ4_KS, IQ5_KS PR 428
  • Fast GEMM/GEMV for IQ1_S PR 212
  • AVX-VNNI optimizations PR 1446 PR 1455 PR 1467 PR 1474 PR 1482

Features

  • New split mode "graph" for multi GPU setups PR 1022
  • Fused delta-net for Qwen3-Next and Qwen3.5-MoE PR 1315 PR 1333 PR 1362 PR 1373
  • Hadamard transforms for K-cache and V-cache PR 1033 PR 1034 PR 1527
  • Auto-fit offloaded tensors to available VRAM (MoE and dense models) PR 1501 PR 1504, allows per GPU fit margin PR 1872
  • Checkpoints for recurrent models PR 1310 PR 1398
  • MTP decoding support for popular models like GLM-4.x MoE 1270, Qwen 3.5/3.6 1698 1745, Gemma 4 1744, GLM 5 1890
  • Self speculative decoding, ngram PR 1261, suffix PR 1646
  • String ban function for all completions PR 1185 PR 1243
  • Expiring Logit Bias PR 1731
  • OpenAI /v1/responses API endpoint PR 1184
  • Function call support PR 628
  • jinja template support PR 677
  • Webui: New Features for Conversations, Settings, and Chat Messages PR 618, MCP PR 1904
  • Dynamic control vector management endpoints PR 1223
  • Legacy quants conversion schemes in convert_hf_to_gguf.py PR 449, Q6_0 in PR 483
  • Adaptive-P Sampler PR 1100 implemented as designed by it's author; supported on Webui
  • Multi-modal Vision support in llama-mtmd-cli PR 798 and in llama-server PR 901
  • mikupad as an alternative WebUI PR 558
  • June 8 2025: Webui updated (legacy still available when --path ./examples/server/public_legacy is passed) PR 481
  • June 8 2025: RPC improvements PR 480
  • June 7 2025: Add an endpoint that lists all the saved prompt caches to server PR 502
  • June 6 2025: Make prompt cache saving and restoring MLA aware PR 497
  • June 3 2025: Added samplers, XTC PR 486, top-n σ PR 489.
  • May 22 2025: Refactor iqk_mul_mat.cpp which speeds up compilation time significantly. PR 435
  • May 17 2025: Option to enable or disable the CPU FA kernels PR 429.
  • May 12 2025: User can now control if/which operations with tensors held in RAM are offloaded to the GPU. See PR 405
  • May 12 2025: Compatibility issues with mainline llama.cpp GGUFs for DeepSeek models with MLA enabled were resolved in PR 394. The lower prompt processing performance resulting from using llama.cpp-style MLA GGUFs was recovered in PR 409.
  • April 21 2025: ik_llama.cpp builds and runs successfully on Android (using termux), see PR 336
  • March 1 2025: Smart Expert Reduction for faster DeepSeek inference PR 239
  • Feb 25 2025: Tensor overrides for better control where model weights are stored (GPU or CPU) PR 232
  • Feb 23 2025: sweep-bench - better performance benchmarking PR 225
  • Feb 19 2025: Q8_KV - new type for 8-bit KV-cache quantization PR 208
  • March 7 2025: Custom quantization mixes using regular expressions PR 244

Performance improvements

  • Better GPU offload strategy for MoE models when using hybrid HPU/CPU inference, see PR 520
  • Much faster rng sampling PR 1187
  • May 13 2025: Better CPU FA performance for DeepSeek-Lite. PR 410
  • May 11 2025: Slightly faster flash attention for DeepSeek models on CUDA, along with extending compatibility to Touring or newer GPUs. PR 408
  • May 4 2025: Significant token generation performance improvement on CUDA with Flash Attention for GQA models. For details and benchmarks. PR 370
  • April 17 2025: Better CPU Flash Attention token generation performance. PR 332
  • April 3 2025: Much faster MoE implementation on Metal. PR 307
  • March 25 2025: Better MoE performance on CUDA PR 283
  • March 23 2025: Better batched processing speed for DeepSeek models PR 282
  • March 18 2025: Reduce compute buffer size PR 237
  • March 10 2025: Better TG performance for MoE models on CUDA PR 248
  • Feb 23 2025: Fused FFN ops for faster MoE inference PR 229

Flash-MLA

  • May 7 2025: 🚀 FlashMLA-3 for DeepSeek models on CUDA. PR 386. Caveat: Ampere or newer Nvidia GPU required
  • March 21 2025: 🚀 FlashMLA-3: fastest CPU-only inference for DeepSeek models PR 273
  • March 17 2025: 🚀 FlashMLA-2 performance improvements PR 253
  • March 12 2025: Allow Q8_0 KV cache with FlashMLA-2 on CUDA PR 265
  • March 9 2025: 🚀 FlashMLA on CUDA PR 247
  • March 8 2025: 🚀 Faster FlashMLA CPU implementation PR 243
  • March 3 2025: 🚀 Introducing FlashMLA - MLA with Flash Attention PR 240
  • Feb 27 2025: MLA without transposed cache PR 235
  • Feb 13 2025: Allow Q8_0 quantized cache with MLA PR 206
  • Feb 11 2025: 🚀 Flash Attention support for DeepSeek models PR 200
  • Feb 9 2025: 🚀 MLA for DeepSeek models PR 188

Fixes

  • Fix bug in MMVQ kernel PR 446
  • Fix AVX2 implementation of IQ4_K, IQ4_KS, IQ5_K, IQ6_K PR 427
  • Fix standard attention on the CPU PR 421
  • Fix imatrix calculation for MLA models PR 411
  • Fix new CUDA FA on Touring PR 413
  • Fix SER. CPU: PR 415 CUDA: PR 416

Resources

There is no single point of reference describing all new ik_llama.cpp features. Pull requests often contain detailed information, so browsing the PRs is often the best way to learn about new features and how to use them. In addition

  • The Wiki page has performance comparisons to mainline llama.cpp
  • This guide is a good place to start if you came here because of DeepSeek models
  • This discussion is about running DeepSeek-V3/R1 on a 16 x 3090 setup
  • This discussion describes the new quantization types available in ik_llama.cpp

Testing

Function Calls Tests

To run the function calls test suite:

cd build
cmake --build . --target test-function-calls
./bin/test-function-calls

The test suite covers parser functionality, streaming, error handling, content cleaning, and server integration. All tests should pass to ensure production readiness.

Contributing

Contributions in form of pull requests, issue submissions (bug reports, feature requests), or general discussions, are welcome.

License

Development documentation

Seminal papers and background on the models

If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:

Completions

Command-line completion is available for some environments.

Bash Completion

$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

Optionally this can be added to your .bashrc or .bash_profile to load it automatically. For example:

$ echo "source ~/.llama-completion.bash" >> ~/.bashrc

Dependencies

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • subprocess.h - Single-header process launching solution for C and C++ - Public domain
Reference in New Issue View Git Blame Copy Permalink
S
Description
No description provided
Readme MIT
146 MiB
Languages
C++ 59.7%
C 14.2%
Cuda 13%
Python 5.3%
Metal 2.5%
Other 5.3%