* CUDA indexer topk: this is better for PP
* Don't overstep
* Cleanup
* Allow Q8_0 cache in the CUDA DSA implementation
* DS4: do not cast caches to f32
* Fix massive inefficiency in CUDA Q->f32/f16 and f32/f16->Q copies
* Re-enable -ictk | --indexer-cache-type-k
* CUDA indexer topk: this is better for PP
* Don't overstep
* Cleanup
* Allow Q8_0 cache in the CUDA DSA implementation
* DS4: do not cast caches to f32
Compute it before the reuse check, pass it to can_reuse_graph(), and reuse the same
value when the graph is rebuilt, so it is computed exactly once whether the existing
graph is kept or rebuilt.
Verified at np=6, same binary, only IK_LEGACY_GRAPH_REUSE differing: legacy 6/12
replies kept their own codeword, this branch 12/12. The value at the call site matched
the value at the store site on 50 of 50 rebuilds sampled at np=2 and np=6.
* Fix wrong output for hybrid/recurrent models at -np > 1 (graph reuse key)
Hybrid and recurrent architectures return silently wrong output when more than
one sequence is resident. No crash and no warning: every slot keeps producing
fluent text, it is just no longer conditioned on that slot's prompt, and slots
degenerate into repetition loops. Reported in #1932.
Three things have to line up, and on these architectures they do:
1. can_reuse_graph() keys reuse on the ubatch SHAPE. Two consecutive decode
steps for different sequences match on every field it checks.
2. update_cache_copies() re-points the baked view_offs for K/V, but only for
attention layers -- it skips recurrent ones via
is_attn_layer = !hparams.is_recurrent(il).
3. The delta-net bakes the recurrent state row into the graph as a
compile-time view offset, not as an input tensor.
So a graph built for sequence A is reused to decode sequence B and nothing
re-points the recurrent state: every sequence reads and writes sequence A's
state. On Qwen3.6-35B-A3B only 10 of 40 layers carry a KV cache, so the 30
layers that silently share state are three quarters of the network.
Fix: extend the reuse key with a fingerprint of the ubatch's sequence
composition -- which sequences, in what order, and which start at position 0.
That last term matters because a state reset is baked into the graph as a node;
it mirrors exactly the condition the builder itself uses (batch.pos[i] == 0 in
build_layer_attn_linear). Gated on llm_arch_is_hybrid() ||
llm_arch_is_recurrent() on both the compare and the compute side, so
architectures that never consult the fingerprint do not pay to build it.
Evidence, RTX 3090 / sm_86, Qwen3.6-35B-A3B-UD-IQ4_XS, 6 concurrent requests
each carrying a unique codeword, 600-token generations, two rounds:
before (IK_LEGACY_GRAPH_REUSE=1): 7/12 replies degenerate
after: 0/12
np=1 is unaffected. Throughput is 132.9 tok/s against 132.4 before, and the
fingerprint is a pure additional invalidation -- it can only ever add reuse
misses, never remove them -- so counting the misses it causes on its own bounds
its cost exactly. Over 13000 can_reuse_graph() calls at np=1:
calls=13000 hit=12949 miss_other=51 miss_fingerprint_only=0
Zero, so np=1 graph reuse is bit-identical to before this patch. That covers
MTP, which runs at n_parallel == 1: its draft/verify ubatch alternation was
already keyed by the existing n_tokens / mtp_op_type / mtp_step_idx /
mtp_n_heads checks, and all the fingerprint adds beyond those is per-token
seq_ids and the pos == 0 flags, both constant at np=1 during decode. prev and
prev_mtp are populated through the same reference binding, so the MTP cache
carries the fingerprint too.
IK_LEGACY_GRAPH_REUSE restores the previous behaviour so the before/after above
can be reproduced from a single build.
* Address review: fold the arch and legacy checks into the fingerprint
llama_graph_bakes_seq_state moves above the fingerprint; the fingerprint takes the arch
and returns 0 for architectures that do not bake sequence state, and when
IK_LEGACY_GRAPH_REUSE is set. Both call sites become a plain call and the two guards
live in one place.
Returning 0 keeps behaviour identical for everything else: stored and computed values
are both 0, so the comparison always matches and reuse proceeds as before this patch.
Re-verified at np=6, same binary, only the env var differing:
legacy 6/12 sequences degenerated -> DIRTY
fixed 0/12 -> clean
* openpangu: opt-in compacted sliding-window KV cache (--swa-compress)
* openpangu: shrink the compacted window and drop the zero fill
* openpangu: correct the --swa-compress state I/O refusal message
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* Fix DSV4 tool calls and reasoning
There are multiple changes. The most important one is the wiring, to
avoid falling back to the autoparser. With autoparser, all arguments
will be forced by the generated grammar to have the `string="true"`
attribute, which then breaks prompt caching, as it would diverge from
what is rendered by the template. Parallel tool calls also doesn't work
when falling back to autoparser.
Other changes:
* Order tool results by tool call order.
* Consume `</think>` instead of `<think></think>` when thinking is
disabled.
* Use "preserved thinking" mode when any tool is defined, otherwise use
"interleaved thinking" mode, e.g. for multi-turns chat. Set template
arg `drop_thinking` to false to force "preserved thinking" mode even
when no tool is defined.
* Add a message to system prompt when reasoning effort is set to max.
The changes were made by following:
1. The Technical Report: https://arxiv.org/abs/2606.19348
2. Reference implementatin: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py
3. VLLM implementation: https://github.com/vllm-project/vllm/blob/main/vllm/tokenizers/deepseek_v4_encoding.py
For the last bullet point about max reasoning effort, the reference
implementation diverges from the other 2, so we follow the techinical
report and the VLLM implementation, for now. This needs more testing.
* Add back trailing newline
* Update the reasoning effort instruction to follow the reference impl
Using the prompt counting test from @coder543, 0731 does have a special
instruction for "high" and another special instruction for "max".
This will break preview, but assuming most people will use the 0731
release, it should be fine.
[1] https://www.reddit.com/r/DeepSeek/comments/1vdqjwr/openrouter_reasoning_effort_levels_are_broken_for/
* ggml : assert the delta-net value head dim equals the key head dim
The CPU forward sizes its result from the value head dim (src[2]->ne[0]) but
indexes it with the key head dim (src[0]->ne[0]). A model where the two differ
was mis-indexed silently. The CUDA op has asserted this all along.
* ggml : fuse the delta-net recurrent state copy into the op
The problem: the delta-net op produces the new recurrent state into the tail of
its result. Then llama copies that tail into the KV slot the state was read
from. The copy buys nothing - the kernel could write the slot itself.
The change: the slot's two halves are written by two narrow CPY nodes instead of
one CONCAT. That isolates the state write in an ordinary node, which the
scheduler places by the rules it already has. A backend that recognises the
pattern lets the kernel write the slot directly and skips that node. One that
does not implement the fusion runs the copy as before. No public header changes.
Notes: this leaves ggml_concat_inplace(), added in #1777 for exactly this site,
without a caller.
* common: fix coding mistakes (typos in identifiers, flags and log strings)
Fix misspelled identifiers and user-facing strings across common, server
and model loading:
- allow_ruless -> allow_rules (misspelled identifier used in the allowlist
CLI parsing and the server slot/context code)
- get_formated_timings/get_formated_generation -> get_formatted_*
- 'termionated' -> 'terminated' in the fit-margin assert message
- 'defaulr' -> 'default' in the YAML dump
- 'overriden' -> 'overridden' in tensor buffer type override logs
- 'becausee' -> 'because' in the output-tensor split log
- 'etected NaNs' -> 'detected NaNs' in the imatrix error message
* common: fix comment typos across src, common, include and examples
Fix misspelled words in code comments:
- llama.h: 'typy' -> 'type', 'transfrom' -> 'transform', 'ecoder' ->
'encoder', 'indicies' -> 'indices', 'Intializes' -> 'Initializes'
- common.h: 'embendings' -> 'embeddings', 'pr' -> 'or' in the
fused-indexer-topk comment
- chat.cpp: 'overridde' -> 'override'
- ngram-map: 'occurences' -> 'occurrences', 'stastistics' -> 'statistics'
- speculative.cpp: 'dont'/'inehit' -> 'don't'/'inherit'
- llama-mmap.cpp: 'dont't' -> 'don't'
- llama-model.h: 'hcurrently andle' -> 'currently handle'
- build_gemma3/4.cpp: 'emdeddings' -> 'embeddings'
- examples: 'quantizuation', 'logprobe', 'throught', 'retrun', 'swich',
'convinient', 'temporally' (-> 'temporary'), 'temproal', 'preceed'
* common: remove duplicate definitions and duplicate help entries
- clip-impl.h: drop the second, identical #define TN_FFN_GATE
- common.cpp: remove the duplicate '-t, --threads N' help entry that was
misplaced in the export-lora section (already listed in the general
section)
- common.cpp: merge the two 'embedding' help groups into a single group
so the embedding options are listed together
- llama.cpp: remove the redundant LLAMA_MAX_LAYERS define (llama-hparams.h
already defines the same value and is included by llama.cpp)
* common: fix remaining typos (accomodate, recommanded, occurences, occassionally)
- accomodate -> accommodate in src/llama.cpp comment
- recommanded -> recommended in quantize.cpp user-facing output
- occurences -> occurrences in test-chat.cpp JSON string
- occassionally -> occasionally in vendor/stb/stb_image_resize2.h comment
Note: tokenizer.ggml.seperator_token_id kept as-is to match GGUF spec
* common: remove duplicate help entries
- remove the duplicate '--reasoning-budget N' help entry that was repeated
in the main section (introduced in e0596bf614 'Autoparser - complete
refactoring of parser architecture (PR 1376)')
- remove the second '--parallel-tool-calls' help entry that advertised the
'-ptc' short flag, which belongs to '--print-token-count' (introduced in
e0596bf614 'Autoparser - complete refactoring of parser architecture
(PR 1376)'); the '-ptc' alias was non-functional for '--parallel-tool-calls'
because the parser only binds it to '--print-token-count'
The canonical help entries are kept:
- '--reasoning-budget N' is listed once
- '--parallel-tool-calls' is listed once (without the conflicting '-ptc' alias)
* common: remove duplicate LOG_ENABLE define
- the '#undef LOG_ENABLE / #define LOG_ENABLE() // dummy stub' pair was
repeated verbatim inside the LOG_DISABLE_LOGS section
- remove the second occurrence (introduced in a2588b53e1 'main : log
file (PR 2748)')
* llama-bench: align MLA and attention-max-batch flags with common tools
llama-bench used '--mla-attn' and '--attn-max-batch' while the common
CLI parsing (common/common.cpp) uses '--mla-use' and
'--attention-max-batch' for the same features. This made the flags
inconsistent across tools.
- update the help text to advertise the canonical names
'--mla-use' and '--attention-max-batch'
- keep the old '--mla-attn' and '--attn-max-batch' spellings working
as aliases so existing scripts are not broken
The divergent names were introduced in 3e536b95b0 'Add optional MLA
(PR 188)'.
* fix typos in comments and user-facing strings
- ngram-map.cpp: 'Do we haven a existing' -> 'Do we have an existing'
(introduced in 1cb7e1bf39 'spec : add self speculative decoding,
ngram and refactor (PR 1261)')
- build_mamba.cpp: 'weigth' -> 'weight' (introduced in 8befd92ea5
'Refactor model compute graphs (PR 1651)')
- gguf-split.cpp: 'one of splits have 0 tensors' -> 'one of the splits
has 0 tensors' (introduced in 75b580db0a 'split: allow
--split-max-size option (PR 6343)')
- gguf-split.cpp: 'merged from %d split' -> 'merged from %d splits'
(introduced in 1b5523dc79 'gguf-split: split and merge gguf per
batch of tensors (PR 6135)')
- convert-llama2c-to-ggml.cpp: missing opening quote in the help line,
'(default %s\\')' -> '(default '%s\\')' (introduced in bb9ebb4394
'Adding support for llama2.c models (PR 2559)')
* harmonize British and American spelling to American English
The codebase uses American English (e.g. --embd-normalize, --color),
but a few strings/comments still used British spellings. Unify them:
- 'normalisation' -> 'normalization' in common.h, common.cpp help text
and code comment, and llama-build-context.cpp comment
- 'colorise' -> 'colorize' in the --color help text (common.cpp)
- 'behaviour' -> 'behavior' in a chat.cpp warning and a llama.cpp comment
- also fix 'openai' -> 'OpenAI' capitalization in the embedding help
text and common.h comment (embedding output format is OpenAI-style)
* common: fix help text formatting inconsistencies
- '-smf16'/'--split-mode-f16' and '-smf32'/'--split-mode-f32' help
entries displayed hardcoded 'true'/'false' as the default value;
show the actual state derived from params.reduce_type instead
- '-no-mmad' help entry had 'fused_mmad?' without a space before the
ternary operator
- '--reasoning-tokens' help continuation lines used tab characters for
indentation while the sibling '--reasoning-format' entry uses spaces;
convert to consistent space indentation
* common: revert smf16/smf32 help text default display change
Revert the '-smf16'/'--split-mode-f16' and '-smf32'/'--split-mode-f32'
help entries back to their original hardcoded 'true'/'false' default
display. The change to derive the default from params.reduce_type was
not desired; the split-mode options are legacy and the hardcoded
defaults reflect their intended meaning.
The other formatting fixes in the same area (fused_mmad ternary
spacing and the reasoning-tokens tab-to-space indentation) are kept.
* llama-bench: fix help text column alignment
The --mla-use and --attention-max-batch help lines introduced by the
flag alignment landed one column off from the sibling entries
((default: at column 51 instead of 50). Adjust the padding so all
help lines align.
* common: fix help text defaults for graph-reduce-type and log-format
Mismatch 1: -grt, --graph-reduce-type help shows default "f32", but actual default (common.h:463) is "f16" and llama.cpp uses GGML_TYPE_F16.
Mismatch 2: --log-format help shows default "json", but actual default (common.h:536 log_json=false) is text.
* common: add -ptcall short flag for --parallel-tool-calls
* typo
common_speculative_on_target_batch stored this batch's last hidden into target_hidden_by_seq before reading the map back for the shifted warmup conditioning, so row 0 was conditioned on this batch's last hidden (a future state) instead of the previous call's, and the position-0 zeros fallback was unreachable. Snapshot the previous value before the store; other readers are unaffected. Warmup-only; affects draft acceptance, not correctness.
read_kv_cache_data gated the restore on kv_self.v_trans != (v_state == 1).
v_state == 2 records that the writer had no V cache, while v_trans tracks
flash attention rather than V allocation, so with -fa 0 on a K-only or MLA
cache the two disagree and the restore is refused.
Compare V cache presence in both directions, and transposition only when a
V cache exists on both sides. The write path and serialized layout are
unchanged.
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* Short aliases for context checkpoints
* common: fix typos and add/document obvious short aliases
Typos fixed in --help output:
- Swapped long names for -ctv-last/-ctk-last: --cache-type-k-last and
--cache-type-v-last now match their K/V semantics
- --scheduler-async (was --scheduler_async) and fix stray ')' in its help text
- Remove stray trailing commas in --merge-qkv, --merge-up-gate-experts,
--k-cache-hadamard, --v-cache-hadamard, --split-mode-f16, --split-mode-f32,
--split-mode-graph-scheduling
- 'top-n-sigma parmeter' -> 'parameter'
- 'embendings' -> 'embeddings' (2x)
Short aliases:
- New: -okv alias for --override-kv
- Document previously undocumented aliases in --help: -rtr, -cmoe, -ncmoe,
-thp (also adds the previously missing --transparent-huge-pages entry),
-ofreq, -to, -spf
* common: standardize arg parser to short-alias-first ordering
gpt_params_find_arg now consistently lists the short alias before the
full --long-name argument, matching the dominant convention (103 short-first
lines). Reordered 19 options that had the long name first: -gan, -gaw, -dt,
-mea, -ps, -mtprot, -mg, -sm, -ts, -ot, -gfm, -cmoe, -ncmoe, -dr, -op,
-no-ooae, -to, -sps, -wb. Pure style change, no functional impact.
* common: systematize cache-type help order, fix -cram-n-min help
- Reorder the mixed KV cache-type help entries to k-first, k-last,
v-first, v-last for logical grouping (short alias, then full name,
matching the parser convention)
- -cram-n-min now shows its N argument in --help (was missing)
llama_sampler_init_grammar_impl built the trigger_words pattern in a block-scoped std::string, stored a pointer into it (trigger_pattern_c), pointed trigger_patterns at that pointer, then let the block drop both locals before llama_grammar_init_impl dereferenced trigger_patterns -- a read of a dangling pointer into freed std::string storage (UB). Hoist trigger_pattern and trigger_pattern_c to function scope so both outlive the call.
