mirror of
https://github.com/ikawrakow/ik_llama.cpp.git
synced 2026-08-12 22:29:39 +04:00
main
510
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Samuel Oliveira AlvesandGitHub
|
7ebbb906d2 |
Initial implementation of DSpark (#2280)
* Implement initial arch for DSpark * feat: Add Dspark architecture support * avoid to many splits in graph and improve rope logic |
||
|
|
e21eed5f58 |
deepseek4: compacted sliding-window KV cache (--swa-compress) (#2266)
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com> |
||
|
|
d44e2cbe57 |
openpangu: per-sequence state save/restore with --swa-compress (#2261)
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com> |
||
|
|
cf1aa57e1a |
openpangu: opt-in compacted sliding-window KV cache (--swa-compress) (#2253)
* openpangu: opt-in compacted sliding-window KV cache (--swa-compress) * openpangu: shrink the compacted window and drop the zero fill * openpangu: correct the --swa-compress state I/O refusal message --------- Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com> |
||
|
NexesenexandGitHub
|
f8b5664c18 |
Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249)
* ggml: fix typos in comments across ggml/src Corrects misspellings found in source comments (no code/logic changes): CUDA: - quantize.cu / quantize_id.cu: 'memroy' -> 'memory', stray 'Exchange' word in reduction comment - fattn-mma-f16.cuh / fattn-new-mma.cu: 'synchonization' -> 'synchronization' - fattn-new-mma.cu / fattn-vec-common.cuh: 'at lease' -> 'at least' - fattn-vec-f32.cuh: 'Currenlty'/'dose' -> 'Currently'/'does' - mmq_id_common.cuh: 'alays' -> 'always' - softmax.cu: 'noncontigous' -> 'noncontiguous' CPU / quantization: - ggml-quants.c: 'At leat' -> 'At least' - ggml.c: 'repeatition' -> 'repetition' CANN: - aclnn_ops.cpp: 'alloced' -> 'allocated', 'contigous' -> 'contiguous' - kernels/dup.cpp: 'contigous' -> 'contiguous' IQK: - iqk_gemm_1bit.cpp: 'explicitely' -> 'explicitly' - iqk_gemm_ktquants.cpp: 'nn AVX2' -> 'in AVX2' Vulkan: - ggml-vulkan.cpp: duplicated 'in in' -> 'in' * ggml: fix less-common typos in comments (spellchecker pass) Second sweep using a spell-check pass with edit-distance filtering to catch typos missed by the common-misspelling list. Comment/comment-context fixes only, no code changes: - aclnn_ops.cpp: 'performace' -> 'performance' - acl_tensor.cpp: 'shoule in' -> 'should be in' - common.h: 'opertors'/'available' -> 'operators'/'available' - ggml-cuda.cu: 'resepctive' -> 'respective', 'resinding' -> 'residing' - conv2d.cu: 'ouptut_chanles' -> 'output_channels' - scale.cu: 'Whehn' -> 'When' - mmq_id_common.cuh: 'renameing' -> 'renaming' - solve_tri.cu: 'supress' -> 'suppress' - ggml-quants.c: 'ptoducts' -> 'products', 'quckly' -> 'quickly', 'Acummulate' -> 'Accumulate' - ggml-sycl.cpp: 'solutino'/'walkaroud' -> 'solution'/'workaround' - ggml-vulkan.cpp: 'aross' -> 'across' - ggml.c unified base: signficantly -> significantly (recorded in iqk too) Also fixed duplicate word 'get get' in the quckly comments (line 14272/14488). * Fix typo: correct hard-to-count words in comments/docs across common/examples Spellcheckedtypos across common/, examples/, tests/ and include/ (from typos2.txt): preserve->preserving, replacement->replacemnt, enhance->enchance, imatrix/ima->imatrix, correct->corerct, parameter->parmeter, utilizing->utilitizing, backward->backwrad, manipulate->manupulate, together->togather, incomplete->parial, sentence->dentence, retrieval->retie, prepared->prepa, partial->parial, randomly->Randonly. Comment/prose only, no code changes. * fixing typos (public_simplechat example) * fixing typos (examples subdirs) |
||
|
|
87eeec9f74 |
openpangu: support server context checkpoints and prompt reuse (#2245)
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com> |
||
|
NexesenexandGitHub
|
c3b075f069 |
Chores : tidy up more typos project wide (ggml directory excluded), new -ptcall alias (#2237)
* common: fix coding mistakes (typos in identifiers, flags and log strings) Fix misspelled identifiers and user-facing strings across common, server and model loading: - allow_ruless -> allow_rules (misspelled identifier used in the allowlist CLI parsing and the server slot/context code) - get_formated_timings/get_formated_generation -> get_formatted_* - 'termionated' -> 'terminated' in the fit-margin assert message - 'defaulr' -> 'default' in the YAML dump - 'overriden' -> 'overridden' in tensor buffer type override logs - 'becausee' -> 'because' in the output-tensor split log - 'etected NaNs' -> 'detected NaNs' in the imatrix error message * common: fix comment typos across src, common, include and examples Fix misspelled words in code comments: - llama.h: 'typy' -> 'type', 'transfrom' -> 'transform', 'ecoder' -> 'encoder', 'indicies' -> 'indices', 'Intializes' -> 'Initializes' - common.h: 'embendings' -> 'embeddings', 'pr' -> 'or' in the fused-indexer-topk comment - chat.cpp: 'overridde' -> 'override' - ngram-map: 'occurences' -> 'occurrences', 'stastistics' -> 'statistics' - speculative.cpp: 'dont'/'inehit' -> 'don't'/'inherit' - llama-mmap.cpp: 'dont't' -> 'don't' - llama-model.h: 'hcurrently andle' -> 'currently handle' - build_gemma3/4.cpp: 'emdeddings' -> 'embeddings' - examples: 'quantizuation', 'logprobe', 'throught', 'retrun', 'swich', 'convinient', 'temporally' (-> 'temporary'), 'temproal', 'preceed' * common: remove duplicate definitions and duplicate help entries - clip-impl.h: drop the second, identical #define TN_FFN_GATE - common.cpp: remove the duplicate '-t, --threads N' help entry that was misplaced in the export-lora section (already listed in the general section) - common.cpp: merge the two 'embedding' help groups into a single group so the embedding options are listed together - llama.cpp: remove the redundant LLAMA_MAX_LAYERS define (llama-hparams.h already defines the same value and is included by llama.cpp) * common: fix remaining typos (accomodate, recommanded, occurences, occassionally) - accomodate -> accommodate in src/llama.cpp comment - recommanded -> recommended in quantize.cpp user-facing output - occurences -> occurrences in test-chat.cpp JSON string - occassionally -> occasionally in vendor/stb/stb_image_resize2.h comment Note: tokenizer.ggml.seperator_token_id kept as-is to match GGUF spec * common: remove duplicate help entries - remove the duplicate '--reasoning-budget N' help entry that was repeated in the main section (introduced in |
||
|
NexesenexandGitHub
|
52836917a1 |
DeepSeek V4 - Checkpoints support (#2195)
* DSV4 checkpoints WIP
* DSV4 checkpoints: fix per-sequence state save/restore writing all streams
Critical bug: llama_state_seq_get_data() and llama_state_seq_set_data()
were serializing/deserializing ALL DSV4 compressed cache streams instead
of only the stream for the target sequence. This corrupted other active
sequences' compressed indexer state during checkpoint restore.
Fix:
- Add dsv4_stream_offset_size() helper to compute per-stream byte
offset and size for any DSV4 cache tensor (CSA K, LID K, HCA K,
and all state tensors)
- write_kv_cache_data: emit dsv4_single_stream flag + stream_idx
so per-sequence saves only write that stream's portion
- write_dsv4_cache: accept stream_idx parameter (-1 = full tensor,
>= 0 = single stream at computed offset)
- read_kv_cache_data: read the new format, validate consistency
(per-stream data needs dest seq_id, full data needs seq_id=-1),
and restore only the destination stream's bytes into the correct
tensor offset
Format change (WIP, backward compat not required):
[has_dsv4_cache] [n_layer] [single_stream] [stream_idx] [n_stream]
[per-layer: layer_type + stream tensor data]
* DSV4 checkpoints: fix checkpoint search using pos_max instead of pos_min
Bug: checkpoint search condition cur.pos_min < n_past || cur.pos_min == 0
always matched all DSV4 checkpoints because they all have pos_min=0 (the KV
cache starts at 0 and never evicts). The reverse-iterator always picked the
NEWEST checkpoint regardless of how far past the intended rewind/divergence
point it extended, causing n_past to be set to the checkpoint's pos_max
(e.g. 9500) instead of the rewind point (e.g. 5000). This made the system
skip reprocessing tokens between the rewind point and the checkpoint's
pos_max.
Fix in both batch_pending_prompt and apply_checkpoint:
- Condition changed to cur.pos_max < n_past — only checkpoints that end
BEFORE the divergence/rewind point are eligible
- Post-restore n_past uses it->pos_max directly instead of the incorrect
max(pos_min + 1, pos_max) which always returned pos_max for DSV4 anyway
Now when rewinding to position 5000 with checkpoints at pos_max=8000+:
no checkpoint matches, falls through to full reprocessing (correct).
When rewinding to position 8000 with a checkpoint at pos_max=7500: only
500 tokens need reprocessing (optimal).
* DSV4 checkpoints: throttle creation via interval gating on all paths
When ctx_checkpoints_interval > 0, DSV4 checkpoints (~145 MiB) were still
created at every transition point (PP done, TG start, release) because
direct create_checkpoint() calls bypassed the interval gate.
Fix:
- Modified create_checkpoint_at_interval() to handle interval <= 0 as
'always create' (preserving recurrent model behavior for small state)
- Replaced all 4 external direct create_checkpoint() calls with
create_checkpoint_at_interval() so the interval gate is respected
- Unified the prompt-loading branch that had split
create_checkpoint / at_interval calls
Now with --ctx-checkpoints-interval N, all checkpoint creation is
throttled to at most 1 per N positions regardless of the transition
phase.
* DSV4 checkpoints: clarify divergence log message for models with state checkpoints
The message 'does not support partial KV reuse' was misleading for DSV4,
which now supports checkpoint-based state restoration. Split the fallback
message: models with state checkpoints (DSV4, recurrent, hybrid) now print
'no checkpoint before divergence point' instead of 'does not support
partial KV reuse', explaining that the restore failed due to missing
checkpoints at the right position, not due to lack of support.
* DSV4 checkpoints: don't erase the just-restored checkpoint
The erasure condition pos_max > pos_min_thold was equivalent to
pos_max >= pos_next, erasing any checkpoint whose data touched or
went past the current write position. The checkpoint just restored
from (pos_max == pos_next) was immediately erased, wasting a ~145 MiB
checkpoint that was perfectly valid.
Fix:
- Changed erasure condition to pos_max > pos_next (strictly greater
than the next write position). Checkpoints at exactly the current
position (pos_max == pos_next, e.g. the one we just restored from)
are kept.
- Preserved cache-aligned pos_next through the restore block so the
erasure compares against cache positions, not prompt positions
(pre-existing bug where the prompt-tokens call at line 3677
overwrote pos_next with a prompt position).
* DSV4 checkpoints: update interval gate position after restore
After a checkpoint restore, slot.checkpoint_pos was still 0 (from
slot.release()), so the interval gate in create_checkpoint_at_interval
always passed (0 + 2048 <= pos + 1), creating a new ~145 MiB checkpoint
immediately after every restore — even just 5 tokens past the restored
checkpoint's position.
Fix: set slot.checkpoint_pos = it->pos_max in both restore paths
(apply_checkpoint generic restore and batch_pending_prompt DSV4
restore). This tells the gate that a checkpoint already exists at the
restored position, and no new one is needed until another interval
(2048 tokens) has elapsed.
* Missing info
* DSV4 checkpoints: document float-reduction-order reproducibility after restore
After a checkpoint restore, the PP batch loop processes remaining tokens
sequentially from n_past_prompt in chunks of n_batch. Because the loop
is stateless with no carry-over from earlier batches, the chunk boundaries
at and after the restore point are identical to a full-reprocess control
arm. This ensures float-reduction-order reproducibility between arms
when performing correctness validation.
Addresses joelfarthing's finding on openPangu, where mismatched chunk
boundaries between restore+reprocess and full-reprocess controls caused
bit-level differences that masked actual restore bugs.
* DSV4 checkpoints: verify position after restore
After both restore paths (apply_checkpoint and DSV4 batch_pending_prompt),
verify that llama_kv_cache_seq_pos_max() matches the checkpoint's pos_max.
A size-matched but misplaced restore can silently corrupt the KV cache;
on mismatch, force a full reset.
The DSV4 path pre-sets restored = true on byte-level success, then the
position check can revert it to false. Only if (restored) proceeds with
the restored state, matching the apply_checkpoint pattern.
* DSV4 checkpoints: correct misleading comment about chunk-boundary reproducibility
* DSV4 checkpoints: add FNV-1a checksum integrity check for checkpoint data
Sanity check (pos_max): catches misplaced restores (wrong stream offset,
partial overwrite) where the byte count matches but the cache position
doesn't.
Correctness check (FNV-1a hash of serialized data): catches in-memory
corruption of the checkpoint data vector between creation and restore.
Both checks are applied in the standard (apply_checkpoint) and DSV4
(batch_pending_prompt) restore paths. A mismatch in either causes the
restore to be treated as failed, falling back to full reprocess.
File format bumped to CKPT v2 (magic 0x434b5054, version 2) with a
data_hash field per checkpoint. Old-format files (LLAMA_STATE_SEQ_MAGIC)
are still loaded: the hash is computed on load so validation works
uniformly.
* Defer checkpoint hash computation to offload creation path
The FNV-1a hash (~5-15ms per 150 MiB checkpoint) is no longer computed
during checkpoint creation. Instead, data_hash is set to 0 and
hash_computed to false. The hash is computed lazily on first access via
ensure_checkpoint_hash(), called from:
- apply_checkpoint (before the integrity check during restore)
- save_checkpoints_to_file (before writing to disk)
This removes the hash computation from the time-critical checkpoint
creation path, reducing the pause between batches.
* Reuse pre-allocated scratch buffer for checkpoint serialization
Adds a reusable std::vector<uint8_t> scratch buffer to server_context,
eliminating the per-checkpoint zero-init allocation (~150 MiB memset)
from ckpt.data.resize(). The scratch is grown on demand and handed
off to the checkpoint via swap() — a zero-copy move.
Also removes default arguments from server_prompt_checkpoint_update()
since all callers already pass every parameter explicitly.
* Compute checkpoint data hash incrementally during serialization
Instead of a second pass over the serialized buffer (costly for 145 MiB DSV4
checkpoints) or deferring to save/restore time (breaks in-memory verification),
compute the FNV-1a hash as a streaming operation during llama_state_seq_get_data.
llama_data_write_buffer gains an optional fnv_hash pointer and updates it
during write() and write_tensor_data() — the hash is computed from bytes as
they land in the output buffer, with zero extra memory reads.
The server then obtains the hash at creation time by passing &ckpt.data_hash
(pre-initialized to the FNV-1a offset basis) to llama_state_seq_get_data.
This replaces the deferred hash approach (ensure_checkpoint_hash / hash_computed)
and restores in-memory round-trip verification.
* DSV4 checkpoints: add round-trip serialization verification
After restore, re-serialize the KV cache and compare byte-for-byte against
the original checkpoint data. This directly catches serialization bugs
that produce internally-consistent but wrong values (Joel's 59/64 case:
correct position, corrupted tensor data).
The check is added to both restore paths (standard apply_checkpoint and
DSV4 batch_pending_prompt) and runs after the pos_max sanity check and
FNV-1a hash integrity check. Cost: one extra llama_state_seq_get_size +
llama_state_seq_get_data + memcmp of the checkpoint data.
* Remove dead _ckpt_max_size member
_ckpt_max_size was set by server_prompt_checkpoint_update but never read.
Removed the member, the function parameter, and the call site.
* Remove no-op resize after swap in server_prompt_checkpoint_update
After swap, ckpt.data holds the scratch buffer which was already
resized to checkpoint_size. Since n == checkpoint_size (asserted),
the resize is a no-op.
* remove dead (void)has_hash cast
The variable is actually used later (for old-format file detection),
so the unused-variable suppression cast is misleading.
* add missing const qualifiers on to_json() methods
Both server_prompt_checkpoint::to_json() and server_prompt::to_json()
were missing const, preventing use on const references.
* remove duplicate n_kept_prompt assignment in server_prompt::from_json()
n_kept_prompt was assigned twice with the same value, clobbering the
slot where n_discarded_prompt should have been read.
* remove redundant params_base parameter from create_checkpoint_at_interval()
The parameter is already accessible as a member of server_context.
All callers were passing this->params_base, so the indirection was
unnecessary.
* factor duplicated restore verification into verify_restored_checkpoint() helper
The 3-step verification (pos_max sanity, FNV-1a hash, and round-trip
memcmp) was duplicated verbatim across apply_checkpoint() and
batch_pending_prompt(). Extract it into a shared static helper with
a label parameter for context-specific log messages.
Also eliminates the pos_next save/restore dance in apply_checkpoint
by using a local variable for the prompt-limit computation, and
removes a stray commented-out debug printf.
* fix two comments: fnv1a_hash comment was misleading, erasure comment imprecise
- fnv1a_hash() is used for all checkpoint verification (not just
backward-compat file loading) — broadened the description.
- 'may contain stale per-position state' → 'its per-position state
is stale' — the erasure is unconditional when pos_max > pos_next,
so the staleness is definite, not possible.
* remove FNV-1a hash and file-format bump (perf, Joelfarthing's review feedback)
The streaming FNV-1a hash added 65 ms to checkpoint creation and 107 ms
to restore (75 MiB checkpoints; roughly double at DSV4's 145 MiB). The
pos_max sanity check alone is sufficient for catching the real failure
modes (wrong stream offset, partial overwrite), and the initial byte-
count check from llama_state_seq_set_data catches outright corruption.
Removed:
- Streaming hash from llama_data_write_buffer (fnv_hash, fnv_update)
- hash_out parameter from llama_state_seq_get_data / llama.h API
- data_hash field from server_prompt_checkpoint struct
- FNV-1a computation during checkpoint creation and verification
- CKPT v2 file format (revert to LLAMA_STATE_SEQ_MAGIC/version)
- fnv1a_hash() helper function
Kept:
- pos_max sanity check in verify_restored_checkpoint (cheap, catches
misplaced restores)
- Scratch buffer reuse via swap() in server_prompt_checkpoint_update
(pure perf win, independent of hash)
* fix: restore off-by-one in n_past calculation after checkpoint restore
size_up_to_pos(pos_max) returns the number of cached tokens at positions
STRICTLY LESS THAN pos_max (non-mtmd: min(pos_max, size)). Since the
checkpoint encodes state for positions [pos_min, pos_max], the next
position to process is pos_max + 1, not pos_max.
This matters for DSV4 whose accumulator state is not position-indexed:
reprocessing the token at pos_max would double-count it in the compressed
indexer. For recurrent models the old pos_min+1 workaround happened to
give the right answer (since pos_min == pos_max there), but using
pos_max + 1 is correct for both.
Fixes both restore paths (apply_checkpoint and DSV4 in batch_pending_prompt).
* fix: only write/read DSV4 cache section for DSV4 models
The has_dsv4_cache uint32 was emitted unconditionally, changing the
serialized state layout for every model architecture without bumping
LLAMA_STATE_SEQ_VERSION. Old state/session files (which end before
this field) would fail with 'unexpectedly reached end of file' when
read by the new code.
Fix: guard the entire DSV4 section on both write and read sides with
ctx->model.arch == LLM_ARCH_DEEPSEEK4. Non-DSV4 models see the
identical layout they always had.
* fix: validate stream_idx < n_stream in dsv4_stream_offset_size
stream_idx was only checked >= 0 via GGML_ASSERT, but never checked
against n_stream. An invalid seq_id could compute an out-of-range
tensor offset or size, leading to memory corruption.
Now asserts 0 <= stream_idx < n_stream.
* fix: scratch buffer reuse — copy instead of swap
swap(scratch) moved the written data into ckpt.data but left scratch
empty. The next call's resize would then re-allocate from scratch,
defeating the purpose.
Now copies the data (ckpt.data = scratch) so scratch retains its size
and capacity across calls. resize becomes an in-place extension when
needed rather than a fresh allocation.
* fix: restore interval<=0 = disable semantics, split unconditional paths
The interval gate was inverted: interval <= 0 opened the gate, so every
call to create_checkpoint_at_interval created a checkpoint (PP, TG,
release, speculative). This changed the documented behavior ('<=0
disable' per --help) and created extra checkpoints on every decoded
token for recurrent models.
Fix:
- create_checkpoint_at_interval returns immediately when interval <= 0
(restoring the no-op semantics from the original code)
- Unconditional paths (release, PP end, PP start with slot.do_checkpoint)
call create_checkpoint(slot) directly, matching the original layout
- Interval-gated paths (TG tokens, PP start without slot.do_checkpoint,
speculative decoding) stay behind create_checkpoint_at_interval
* fix: gate ALL checkpoint creation by interval, not just TG paths
Three call sites bypassed the interval gate by calling create_checkpoint(slot)
directly instead of create_checkpoint_at_interval(slot):
- PP batch-boundary (was creating mid-PP checkpoints at unpredictable positions)
- PP end (created a checkpoint at every end-of-prompt, even if within the interval)
- release (created a checkpoint at every release, even if just 5 tokens later)
This caused checkpoints 5 and 6 in the log to be created only 5 tokens apart
(pos_max=8488 and pos_max=8493), and checkpoint 7 at release 455 tokens later,
all with interval=2048.
The original code had all checkpoint creation gated by interval (single
create_checkpoint_at_interval function called everywhere). The 'unconditional'
paths were introduced by our earlier fix that split create_checkpoint_at_interval
into a no-op for interval<=0 — but the split was too aggressive, making release,
PP-end, and PP-batch-boundary always fire.
Fix: route all checkpoint creation through create_checkpoint_at_interval, which
already handles do_checkpoint (early return) and interval <= 0 (no-op) correctly.
create_checkpoint is now an internal helper called only from
create_checkpoint_at_interval.
Result: with interval=2048 and an 8494-token prompt, checkpoints are created at
2048, 4096, 6144, 8192 only — the original semantics.
* fix: off-by-one in checkpoint gate condition, use -1 sentinel
The gate condition 'checkpoint_pos + interval <= 1 + pos' opened one
position early for non-first intervals. With checkpoint_pos=6143,
interval=2048: 6143+2048=8191, and pos=8190 gives 8191 <= 1+8190=8191
→ TRUE, creating a checkpoint at pos_max=8190 instead of 8191.
Root cause: checkpoint_pos=0 served dual duty ('no checkpoint yet' and
'checkpoint at position 0'). The '1 +' in the condition compensated
for this at startup but overcompensated later.
Fix:
- Change checkpoint_pos from size_t to llama_pos, initialized to -1
- Drop the '1 +' — condition is now checkpoint_pos + interval <= pos
With checkpoint_pos=-1: -1+2048=2047 <= 2047 → first checkpoint after
2048 tokens (correct).
With checkpoint_pos=6143: 6143+2048=8191 <= 8190 → FALSE (no early
open), 8191 <= 8191 → TRUE (opens at correct position).
Also fixes the mixed signed/unsigned comparison that existed with
size_t checkpoint_pos vs llama_pos pos.
* perf: serialize directly into ckpt.data, drop scratch buffer
The ckpt.data = scratch copy added ~10ms to checkpoint creation
(memcpy of 145 MiB). The scratch buffer was originally introduced to
avoid per-checkpoint resize allocation, but the lazy-zero paging of
modern OSes makes the resize essentially free.
Drop the _ckpt_scratch member entirely. Serialize directly into
ckpt.data after resize — same allocation cost, no extra copy.
* fix: replace GGML_ASSERT with runtime check in dsv4_stream_offset_size
GGML_ASSERT is compiled out in release builds (NDEBUG). An invalid
non-negative seq_id from the public state API would then compute
out-of-range tensor offsets and sizes, leading to memory corruption.
Replace with a runtime conditional that logs the error and sets
safe fallback values (offset=0, size=0). The caller that reads/writes
0 bytes will fail downstream in a defined way.
* fix: restore tolerance mechanism, slot.do_checkpoint bypasses interval gate
Samuel reviewed that we removed the slot.do_checkpoint branch from PP
batch-boundary, but batch_pending_prompt still sets slot.do_checkpoint
when the tolerance threshold is reached. Nowhere checks it, so the
tolerance checkpoint for short prompts (shorter than interval) is dead.
Fix: create_checkpoint_at_interval now checks slot.do_checkpoint — if
true, the interval gate is bypassed. After a successful creation the
flag is cleared so normal interval gating resumes for subsequent
checkpoints. Also handles interval <= 0 + slot.do_checkpoint correctly:
the early-return for disabled interval is itself gated by
!slot.do_checkpoint.
* revert: erasure condition back to cur.pos_max > pos_min_thold
The change from pos_min_thold to pos_next affected all models, not just
DSV4. Revert to the original condition (cur.pos_max >= pos_next after
integer simplification) which correctly erases checkpoints at or past
the write position.
* restore unconditional release checkpoint per firecoperana review
Release is a lifecycle boundary. The interval gate is for throttling
mid-processing checkpoints; the release should always capture the final
state (when do_checkpoint is enabled).
* restore original PP batch-boundary branching per firecoperana review
The explicit slot.do_checkpoint branch in the PP batch-boundary is
restored. The tolerance bypass is removed from create_checkpoint_at_interval
since it was only ever intended for the PP batch-boundary path (the
original code checked slot.do_checkpoint exclusively there). This keeps
the tolerance mechanism from leaking into TG, PP-end, and other paths.
* restore original PP-end checkpoint condition per firecoperana review
The original created an unconditional checkpoint at PP end when tolerance
is disabled (<=0). When tolerance > 0, the tolerance mechanism in the PP
loop handles the end-of-prompt capture at the tolerance point, so no
additional PP-end checkpoint is needed.
* consolidate DSV4 restore path into apply_checkpoint per firecoperana review
The DSV4-specific restore path in batch_pending_prompt duplicated the core
logic of apply_checkpoint (search, restore, verify) with different search
conditions and missing erasure. Consolidate by:
- Adding is_state_ckpt_model flag to apply_checkpoint
- Bypassing the pos_min >= pos_min_thold guard for state-checkpoint models
(DSV4 always has pos_min=0 from no eviction, so the guard blocked entry)
- Using pos_next instead of pos_min_thold for the search condition when
is_state_ckpt_model (allows finding checkpoints at pos_max == n_past - 1)
- Differentiating the reset log message per model type
- Recomputing n_past_offset and n_discarded_prompt after apply_checkpoint
(previously handled in the DSV4-specific path)
* conditional pos_next formula
State-checkpoint models (DSV4, recurrent) use pos_max + 1 — correct
for DSV4's multi-position checkpoints where pos_min=0, po neviction
max(pos_min+1, pos_max) = pos_max, which undercounts by 1.
For recurrent models pos_min==pos_max so both formulas agree.
Non-state-checkpoint models keep the original
max(pos_min + 1, pos_max) formula unchanged.
* remove redundant n_past_offset / n_discarded_prompt after apply_checkpoint
Both n_past and n_past_prompt are shifted by the same delta from the
restored checkpoint, so the difference (n_past_offset) is unchanged.
n_discarded_prompt is not used in the critical path.
* remove redundant speculative-decoding checkpoint per firecoperana review
speculative_decoding_accept is called from within the TG generation loop
which already creates interval-gated checkpoints at n_decoded > 1 (line
4779). The inner call would double-create.
* narrow DSV4-specific search and pos_next formula to DSV4 only per firecoperana review
is_state_ckpt_model includes recurrent models (e.g. Qwen 3.6) where
pos_max < pos_next search semantics may not be appropriate. Only
DSV4 needs pos_max+1 formula and pos_next-based search threshold.
* revert divergence-reset guard to original per firecoperana review
Unnecessary wrapping of the OpenPangu-only divergence path inside
!llama_model_supports_state_checkpoints. The condition is already
specific enough (!llama_model_supports_partial_kv_reuse is
OpenPangu-only), and OpenPangu does not support state checkpoints,
so the original code was functionally identical.
* narrow guard bypass to DSV4 only per firecoperana review
Recurrent state-checkpoint models don't need the pos_min >=
pos_min_thold guard bypass — only DSV4 (which always has pos_min=0
due to no KV cache eviction) requires it.
* narrow reset log message to DSV4 only per firecoperana review
Replace remaining is_state_ckpt_model with is_dsv4 in the
do_reset log branch; remove the now-unused variable.
* cleanup: revert unnecessary newlines, spacing, and comment changes
* fix: restore partial KV reuse for DSV4 in llama_model_supports_partial_kv_reuse
DSV4 has private per-position state but uses state checkpoints to
restore after a mid-sequence divergence. The function was returning
false, causing batch_pending_prompt to reset n_past=0 before
apply_checkpoint could restore from a checkpoint, which broke the
entire checkpoint mechanism.
* Remove bloat
* Reinstate deleted comment
* replace strcmp(arch_string) with llama_model_is_deepseek4()
SamuelOliveirads added the helper upstream — cleaner and avoids
the fragile string comparison.
* inline llama_model_supports_state_checkpoints into call site
Replaced with the inline expression
llama_model_has_recurrent(model) || llama_model_is_deepseek4(model)
and removed the now-unused function from llama.h and llama-model.cpp.
* fix: GCC 13.3 variadic macro trailing comma in SLT_WRN
SLT_WRN expands to LOG_WRN with __VA_ARGS__ at the end. When no extra
args follow the format string, the dangling comma causes GCC 13.3 to
error with 'expected primary-expression before')' token. Use '%s'
pattern consistent with all other zero-arg SLT_WRN callers.
* dsv4_stream_offset_size: bool return, GGML_ASSERT on write, graceful abort on read
dsv4_stream_offset_size silently returned offset=0, size=0 for invalid
stream indices. Now returns bool — writer hard-aborts via GGML_ASSERT
(prevents writing corrupt checkpoints), reader aborts the restore via
return false (handles corrupt checkpoints gracefully).
|
||
|
Samuel Oliveira AlvesandGitHub
|
dd837ff21a |
DeepSeek V4 spec checkpoints (#2205)
* add DSV4 speculative checkpoints * Fix DSV4 checkpoint cleanup indentation |
||
|
pbrejtfusandGitHub
|
0a4e10c7fb |
server : remove usage field from intermediate streaming chunks (#2189)
While using the server with the Mistral Vibe agent, I ran into an issue where every single prompt triggered an immediate context auto-compaction, showing that the model's context window was fully maxed out. The root cause is a mismatch with the OpenAI API spec. In streaming mode (stream: true), every intermediate SSE chunk incorrectly includes a usage object containing prompt_tokens and completion_tokens. According to the OpenAI specification, usage should only appear once in the final chunk (alongside an empty choices array). Upstream llama.cpp handles this correctly — if you look at examples/server/server-task.cpp, server_task_result_cmpl_partial::to_json_oaicompat_chat() builds delta chunks without any usage field. However, in this fork, both to_json_oaicompat_partial() and to_json_oaicompat_chat_partial() embed a usage object in every single intermediate chunk. Since standard agents aggregate prompt_tokens from streaming chunks, they end up multiplying the real token count by the number of chunks. For example: Prompt: 9,000 tokens x 80 chunks = 720,000 "tokens" reported This triggers fake "100% context" errors and forces an auto-compaction on every request. What changed: I removed the usage block from the partial chunk methods. The _final methods (to_json_oaicompat_chat_stream(), to_json_oaicompat_final()) already send usage correctly in the last chunk with empty choices, so I left those untouched. (Debugged with some assistance from Mistral Vibe Qwen3.6-27B) |
||
|
dmaivelandGitHub
|
b20bff2ae0 | Add /models and /responses endpoints (#2187) | ||
|
|
e5357286c0 |
hotswap: keep load-time-derived and transformed tensors coherent after reloads (follow-up to #2131) (#2163)
* hotswap: keep load-time-derived and transformed tensors coherent after reloads - Re-derive the MLA combined attn_kv_b (computed_wkv_b) in place when a layer's attn_k_b/attn_v_b are hot-swapped: the mla>1 prompt-processing path consumes the derived tensor, so swaps of the source tensors previously had no effect (KLD stayed exactly 0 in per-tensor benchmarks, e.g. GLM-5.2 attn_k_b/v_b). - Refuse (loudly) hot-swaps that cannot be correct: views into -mqkv/-muge merged tensors, khad-folded MLA weights, in-place-scaled ffn_gate_inp_s, BitNet fused scales, OpenPangu parameter-sink sources, and same-dtype swaps of mmap-backed tensors. A refused reload produces no 'reloaded tensor' line, so benchmark drivers quarantine the round instead of recording wrong data. - Propagate reloaded data to same-name duplicate instances (tied lm head copy of token_embd, per-layer rope_freqs/rope_factors copies, expert-bias dups), warning when a duplicate cannot be refreshed. - Warn that derived state stays stale where a refresh is not possible: pre-transposed wk_b_pp under -sm graph/attn, requantized MTP head (output_extra.weight), and k_b/v_b derived from a reloaded attn_kv_b. - Warn at registration time when -rtr is enabled (restores cannot reproduce the run-time-repacked state; F16 -> BF16_R16 is lossy). - server: only attempt the /health hot-swap reload when no slot is processing, and clear the KV cache + cached prompts after a successful reload (they were computed with the previous weights). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Deduplicate llm_compute_wkv_b --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
7945404458 |
DS4: slowly approaching a meaningful performance (#2165)
* initial map to load deepseek 4 arch
* wip
* wip: match graph build and attn logic for dpv4
* wip: Enhance DeepSeek-V4 architecture with new tensor types and sqrtsoftplus gating function
* Update DeepSeek-V4 to support raw key indexing with read/write indices
* fix mismatch in attn_raw
* Enable FA with CSA/HCA
* Fix logit mismatch with FA path
* Clean traces and logs for debug
* Refactor DSV4 tensor handling for MTP execution and improve raw context management
* Refactor DeepSeek4 tensor operations: replace manual weighted sum and post-processing with new helper functions
* Share mHC pre-projection and fix packed DSV4 writes
* DSV4: add shared top-k selection and improve mask handling
* Fix DSV4 c2048 view stride and duplicate loader instantiation
* Reuse shared RMS normalization in DSV4 graph
* Replace DSV4 indexer rotation with shared Hadamard
* Share CSA visibility mask with DSV4 LID
* dsv4: document dependency ordering and reset state
* Remove DSV4 zero-dependency graph shim
* Fix DSV4 packed stream execution
* Remove DSV4 l_out backend override
* Enable DSV4 quantized K-only cache
* Revert "Enable DSV4 quantized K-only cache"
This reverts commit
|
||
|
Samuel Oliveira AlvesandGitHub
|
3c6cbf6e2a | feat: allow dflash to work with spec auto tune (#2112) | ||
|
|
b90939934a |
model: add openPangu-2.0-Flash (92B-A6B) with MLA-latent cache, DSA/SWA, mHC, and multi-head MTP (#2065)
* openpangu: Stage-1 converter probe for openPangu-2.0-Flash
Add OpenPanguV2ForCausalLM conversion support (converter-only; runtime graph
is Stage-2). Registers a new LLM_ARCH_OPENPANGU on the Python/gguf-py side:
- gguf-py/constants.py: MODEL_ARCH.OPENPANGU + name, indexer KV keys, 22 new
tensor enums (DSA indexer x4, MoME convs x3, param-sink x2, mHC/Hyper-
Connections x12, block-post-norm), and the full MODEL_TENSORS list reusing
the deepseek MLA + MoE + NextN bricks.
- tensor_mapping.py: arch-specific block mappings that disambiguate the
sandwich norms (post_attention/pre_mlp/post_mlp) and pin every Pangu-only
tensor; non-block global mHC merge module.
- convert_hf_to_gguf.py: OpenPanguV2Model (subclasses DeepseekV2Model) with
set_gguf_parameters (MLA/MoE/indexer/mHC/param-sink/DSA+SWA metadata),
modify_tensors (expert merge, kv_b split, no MTP skip), and the
OpenPanguV2Tokenizer pre-tokenizer hash.
Validated offline against the real 50-shard safetensors index: all 37,587
tensors map to a GGUF target (0 unmapped), and set_gguf_parameters reads only
hparams present in config.json. No weights downloaded; no GPU. Pinned on the
ik/dsa_loop_hadamard_blend DSA substrate.
* openpangu: Stage-2 arch scaffold (LLM_ARCH_OPENPANGU) — loadable, compiles
New arch on main (DSA-decoupled). Declares openPangu-2.0-Flash to the runtime so
the model loads into memory; the compute graph is the next step.
- llama-arch.{h,cpp}: LLM_ARCH_OPENPANGU + name; 3 KV keys (mhc_num_stream,
mhc_recur_norm, param_sink_number); 18 tensor enums (mHC x12, MoME conv x3,
param-sink x2, block-post-norm).
- llama-model.cpp: OPENPANGU tensor-name block, strings matched to the converter.
- llama-model.h: layer + model struct fields (mHC / conv / sink / block-post / merge).
- llama-hparams.{h,cpp}: reader (MLA + MoE + sigmoid gate + indexer + mHC +
param-sink + NextN); n_layer_kv_from_start = n_layer - nextn (MTP skipped).
- llama-load-tensors.cpp: create_openpangu_tensors (GLM-DSA MLA/MoE base + Pangu
tensors; indexer loaded-but-unused for dense fallback); dispatch + is_mla_attn.
Builds clean (CPU-only libllama). Dense-fallback design: no DSA indexer / SWA
windowing / MTP for first generation (exact <=512 tokens). Graph is Stage-2b.
* openpangu: fix compresskv_conv dim (kv_lora_rank, not +rope); pin attention order in spec
* openpangu: end-to-end runtime — build_openpangu graph runs, generates (garbled)
First full forward pass of openPangu-2.0-Flash on ik_llama. Pipeline works end to
end: new LLM_ARCH_OPENPANGU loads the Q4 GGUF, the graph executes, and llama-cli
generates 40 tokens (EXIT=0). Output is currently garbled (tensor-layout bug to
debug), but the structure is proven.
graphs/build_openpangu.cpp: dense decompressed-MHA attention + 4-stream mHC
(Hyper-Connections) with 20-iter Sinkhorn + MoE(sigmoid+shared) + sandwich norms
+ entry stream-repeat/tail-merge + inp_out_ids selection.
Bring-up fixes to load+run:
- llama-vocab.cpp: register 'openpangu' pre-tokenizer (QWEN2 family)
- llama.cpp: OPENPANGU -> LLAMA_ROPE_TYPE_NORM (was defaulting to NONE=-1)
- llama-load-tensors.cpp: full wkv_b load; k_b/v_b as flattened 2D; block_post_norm
dim = S*H (10240); conv weights 2D {3,C}; mHC alpha/beta/gamma + param_sink +
merge params use bare (no-.weight) tensor names
- llama-model.cpp: OPENPANGU is NOT is_mla_attn (decompressed MHA, standard KV cache)
- graph loop bounded to base layers (skip NextN/MTP)
v0 deferrals (need conv-state cache / manual attention path, all documented):
MoME convs (passthrough), o_conv, param_sink. Next: fix the layout bug to coherence.
* openpangu: COHERENT generation — NEOX rope, Sinkhorn orientation, MoME convs, param_sink
Four correctness fixes on top of the end-to-end scaffold, verified checkpoint-by-
checkpoint against a Python golden reference running on the GGUF's own dequantized
weights (block-0 activations now match to rounding at full fidelity):
- rope: NORM -> NEOX. Pangu config rope_interleave=false; the Infer source maps it
as is_neox_style = not rope_interleave (rotary_mode='half').
- mHC Sinkhorn: the flat h_res block is torch-[r,c] row-major, so a bare ggml
reshape lands column-fastest; the doubly-stochastic iteration ran transposed
(Sinkhorn is not transpose-symmetric). One transpose at input fixes the whole
chain including the mhc_post application.
- MoME convs (qa/compresskv/o): were passthrough stubs. Implemented as
out = x + causal_conv1d(x) (every Infer call site uses residual_connection=1;
tap stats confirm the perturbation form). Taps cast f16->f32 for ggml_mul.
Batch-local v0: exact for fresh-sequence prefill; decode steps miss the
t-1/t-2 taps until a conv-state cache exists.
- param_sink: 128 learned latent-KV entries prepended per layer via a manual
attention path (kv_store + explicit soft_max over [sinks ++ cache]); huge
effect at short context. o_conv now applied pre-o_proj on the same path.
flash_attn forced off for OPENPANGU (FA kernel cannot see the sinks).
- converter: add_bos_token=true (HF prepends <|pangu_text_start|> via the
post-processor; the key was absent so ik dropped BOS).
Greedy Q4_K_M smoke, chat template + <think>: coherent CoT reasoning and a
correct answer. Layer-0 instrumentation (opg0_* names) kept for now.
* openpangu: MoME conv-state cache — decode steps get real t-1/t-2 taps
Allocate a per-layer cache_s_l tensor for OPENPANGU base layers holding the last
two pre-conv latents of the three MoME sites, packed
[qa 2*1024 | compresskv 2*512 | o 2*6144] f32 (~60KB/layer). The conv helper
reads the [C,2] history window (zeros at sequence start, kv_head==0), builds
xx = [hist ++ x], and writes the last two columns back each ubatch — the concat
naturally handles both prefill chaining and the T==1 shift. Read precedes write
in graph order; the fixed-offset copy is graph-reuse safe.
Verified: prefill anchors unchanged (bit-path identical, zero-history branch);
-ub 1 token-by-token run matches the golden reference at t4 (qlora_conv 0.084,
R_block 0.008 rel; attn_out 0.15 on one channel = f16 KV-cache rounding, washes
out by post-norm); final logits differ from full-batch only by a common-mode
shift that softmax cancels. Chat-template greedy smoke: think-block repetition
is gone — clean structured CoT and correct answer.
v0 limits documented in the helper: one state slot (single sequence); cache
rewinds leave the state stale.
* openpangu: NextN/MTP speculative decoding — 1.7-1.8x TG on CPU
Wire the three NextN layers (46-48) into ik's MTP speculative framework
(--spec-type mtp). v0 drafts with head 1 (layer 46), self-chained by the
framework.
- llama.cpp: add OPENPANGU to the cparams.mtp arch allowlist (it was silently
zeroed, which left the target context without a logits buffer once the server
enabled embeddings -> GGML_ASSERT(lctx.logits) in speculative_is_compat).
- load-tensors: MTP layers carry no mHC tensors (tail_use_mhc=false in the
reference) — create them only for base layers. nextn.* tensors were already
wired by the Stage-1 probe.
- build_openpangu: extract the attention sublayer into
build_openpangu_attention (shared base/MTP); add build_openpangu_mtp:
eh_proj(cat(enorm(embed), hnorm(prev_hidden))) -> one plain-residual Pangu
block (sandwich norms, convs+param_sink, MoE+shexp, no mHC/block_post_norm)
-> shared_head norm+head. MTP branch returns the draft graph when
mtp_op_type != NONE; main graph keeps all-token outputs under cparams.mtp.
MTP convs run batch-local (no conv-state slot) — affects acceptance only.
A/B (Q4_K_M, CPU, greedy, 192-token chat CoT completion, warm back-to-back,
medians of 3, bracketed B/A/B):
no-spec: 2.44 t/s (2.34-2.86)
--spec-type mtp:n_max=3: 4.23 / 4.49 t/s (brackets) => ~1.7-1.8x
Draft acceptance 34% on CoT prose (46% on repetitive text); spec and no-spec
greedy outputs are byte-identical. Headroom: conv-state for MTP drafts, n_max
tuning, true 3-head chaining (spec_step_idx).
* server: include draft_n/draft_n_accepted in /completion timings
get_formated_timings() (the /completion path) omitted the speculative
counters that get_timings() (the OAI path) already reports; add them,
guarded by n_draft_total > 0 like the OAI path.
* openpangu: position-indexed MoME conv-state ring — rollback-safe spec decoding + MTP draft chaining
The v0 single-slot conv state held the last-2 pre-conv latents of the most
recent batch, so any speculative draft rejection left latents of REJECTED
positions in the state and every later decode ran with wrong t-1/t-2 taps
(3 conv sites x 46 layers). At 192-token greedy runs every spec config
diverged from no-spec, each differently (rejection-pattern dependent).
Replace it with a per-layer ring cache_s_l [n_lora_q+n_lora_kv+n_head*v_dim, 16]:
column pos%16 holds position pos's pre-conv latents ([qa|ckv|o] packed).
Invariant: reads target only positions before the first batch token, which
are committed, and committed latents depend only on the committed prefix -
rollback-safe by construction, no checkpointing. Writes cover the last
min(T,16) batch positions in <=2 contiguous cpy segments; the copy sources
are views of the [hist ++ x] concat so the history read is an ancestor of
every write (read-before-write by graph dependency).
The ring is also allocated for the NextN/MTP layers, so the draft head
chains real conv taps across WARMUP -> sequential DRAFT_GEN steps (was
batch-local zero-history per draft token).
graph_reuse is forced off for the arch: ring view offsets are position-
baked and the reuse patcher only updates the standard K/V-store copies.
Measured cost on the CPU server path: none visible. ggml_set_rows driven
by an input index tensor is the future reuse-safe shape.
Verified (Q4_K_M, CPU, greedy 192-tok chat-CoT, warm single process):
- no-spec output byte-identical to pre-ring build
- spec output byte-identical to no-spec below the n_predict cap, for all
of n_max in {1,2,3,4,6} x p_min in {0,0.3,0.6} (old build: all diverged)
- acceptance n3-p0: 33.9% -> 60.9%; n3-p0.3: 58.1% -> 68.9%
- TG medians: no-spec 3.19-3.32 t/s; mtp:n_max=3,p_min=0.3 6.97 t/s (~2.1x)
* openpangu: DSA lightning indexer + SWA schedule — long-context correctness past the dense fallback
The dense fallback was exact only <=512 tokens (SWA window). This wires the real
DSA/SWA hybrid schedule, self-contained from GGUF keys the converter already
writes (openpangu.swa_layers + sliding_window_list; absent keys keep the old
dense fallback):
- SWA layers (30 base @512): the generic inp_KQ_mask_swa path, per-layer mask
choice in the builder. The NextN/MTP layers are SWA @2048 in the checkpoint
schedule; MTP graphs run in their own context, so the mask fill picks
hparams.n_swa_mtp when built with an MTP op type.
- DSA layers (16, every 3rd): lightning indexer implemented in-graph from the
Infer reference semantics (jointfix _pangu_torch_calib): q_idx = wq_b on the
post-conv post-norm q-lora latent (24x128), k_idx = rms-normed wk(x) shared
across heads, both NEOX-roped on the FIRST n_rot channels; score =
sum_g w_g * relu(q_g . k) in f32, causal-masked, exact top-k via
argsort + ggml_set_rows scatter into a -1e30 base -> additive selection mask
on the existing manual soft_max seam. Selection engages only when the causal
window exceeds index_top_k (2048); below that the layer is exactly dense.
- Indexer keys cached per position (cache_idx_l, f32 [128, kv_size], DSA layers
only) with the same committed-position invariant as the conv-state ring, so
speculative rollbacks stay safe.
- param sinks remain outside both the window and the selection budget, matching
the reference.
Verified (Q4_K_M, CPU):
- <=512 tokens: byte-identical to the dense build (96/160-token greedy)
- indexer scores vs a GGUF-dequant golden reference at 2101 tokens: 1e-3 rel
(f16 weight rounding); top-3 selection indices exact on all compared queries
- >512 coherence clean; 3.4K-token needle retrieval through active selection
(needle outside every SWA window, ~1300 positions pruned) answers exactly
* openpangu: MLA-latent KV cache — attention absorbed into the 512-latent, 14x smaller cache, ~2.2x TG
Store per position only [ckv_norm 512 | roped k_pe 64] (f32, k_l) plus the
transposed 512-latent (f32, v_l, v_trans layout); per-head K/V are never
materialized. q_nope is absorbed through attn_k_b (loaded 2D from the
converter split for base layers; derived at load via llm_prepare_mla for the
NextN layers - now guarded for layers without attention weights, e.g. the
idle NextN heads 2/3). The value side is the latent itself, up-projected
through attn_v_b after the weighted sum, matching the Infer _forward_dsa
reference. param sinks are native latent-space entries, which removes the
per-step full-cache concat+cast that dominated long-context decode.
llama_state row sizes now come from llama_kv_k_row_embd/llama_kv_v_row_embd
(arch-aware), fixing an out-of-bounds crash in the server prompt-cache save
path (hparams-derived 9216-wide rows vs actual 576-wide latent rows).
Verified (Q4_K_M, CPU): layer-0 attention output matches an f32 golden
reference computed from the same GGUF weight encodings (~1e-2 on O(1)
values); MTP spec output byte-identical to no-spec; 3.4K needle retrieval
through active DSA selection exact under greedy. Output differs from the
materialized build at the token level because attn_k_b/attn_v_b are
independently quantized tensors - both are legitimate Q4-fidelity encodings.
Perf (CPU, warm): no-spec TG 3.2-3.3 -> 6.9-7.1 t/s; mtp:n_max=3,p_min=0.3
-> 11.1 t/s (byte-exact, 67% acceptance); prefill 30.5 t/s at 3.4K; KV self
size at 4K ctx: 5.5 GiB -> 391 MiB. Not yet supported on the latent cache:
K-shift/defrag (context shifting) - unreached in current usage.
* openpangu: fence unsupported serving modes, truth-pass comments, drop dead weight/keys
Post-audit hardening. The cache's position-indexed side state (MoME conv ring,
DSA indexer keys) made several generic serving paths silently unsound; they are
now fenced loudly instead of documented as unsupported:
- s_l_position_ring flag on llama_kv_cache: the qnext-state predicate no longer
claims the conv ring, so per-seq state save, seq_cp and the s_copy graph skip it
- state save/restore refused for the arch at every llama_state_* entry (the ring
and idx_l are not in the state format; restoring without them diverges silently)
- K-shift/self-extend assert, defrag skips with a warning, server ctx_shift off
via new llama_model_supports_ctx_shift()
- single sequence enforced at context creation (n_seq_max > 1 refused)
- server prompt-cache reuse limited to pure extension via new
llama_model_supports_partial_kv_reuse(): mid-cache divergence reprocesses from
scratch (the 16-column ring cannot rewind); multi-turn continuation stays fast
- MTP draft length clamped to 13 via new llama_model_max_draft_tokens() so a
rejected draft can never overwrite the ring columns the next decode reads
- K/V cache types forced to f32 for the arch so the KV size log reports the truth
- cache_size(): real latent-cache branch (was falling through to the ~14x larger
materialized estimate used for offload planning)
- unused fused wkv_b no longer loaded (TENSOR_SKIP; the graph runs entirely on the
pre-split k_b/v_b), llm_prepare_mla openPangu special-case removed (it was a no-op)
- stale v0 comments rewritten to describe the shipped graph; converter stops
writing dead keys (dsa_layers, block_post_layernorm_idx) and the tokenizer
pre-hash is registered in convert_hf_to_gguf_update.py
Gates on this build: greedy spec output byte-identical to no-spec (EOS-terminated,
sha-equal); 3.4K needle retrieved exactly; -np 2 / state save / n_max=20 / stale
prefix reuse all refused or clamped with clear messages.
* openpangu: assert kv_head == first batch position at graph build
The ring, indexer and latent stores are addressed by absolute position through
kv_head; the fences make append-only decode the only reachable mode, but the
invariant was unchecked. Assert it at both graph entries (base and MTP) so any
future cache plumbing that breaks it fails at build instead of corrupting
output. Worst-case measurement builds pass pos = null and are exempt.
* openpangu: cont h_pre before the mHC broadcast mul (CUDA binbcast misreads strided views)
h_pre is a row-slice view of the fused mixes tensor. The CPU mul handles the
strides; the CUDA broadcast path reads the view as if contiguous, so token 0
mixes correctly and every later token gets h_post/h_res rows instead. First
divergent node in the whole graph (oracle rel 0.36 at opg0_attn_mhcpre_x,
fixed to 7.5e-5). Sibling views h_post/h_res were already cont-wrapped, which
is why only h_pre was exposed.
* openpangu: keep DSA zero-trick sources finite (CUDA clamp propagates the 0*(-inf) NaN)
The selection-mask base and zeros were built by scaling the MASKED scores by
zero, but post-mask sc contains -inf and 0 * -inf = NaN. The CPU clamp launders
NaN back to -1e30 (fminf/fmaxf ignore NaN); the CUDA clamp propagates it, so
every DSA layer emitted NaN masks at n_kv > top_k and logits collapsed
(observed: eval-callback CLAMP sum -1.3e36 on CPU vs nan on CUDA, 11748 NaNs
downstream). Scale the pre-mask finite scores instead, which is correct on any
backend regardless of clamp NaN semantics. Also defensively cont the strided
KQ_mask slice feeding the score add (same strided-view kernel class as the mHC
h_pre fix; unproven here but cheap). Gates after fix: 2600-token probe coherent,
3.4K needle exact ('7391') with and without MTP speculation, PP ~120 t/s.
* openpangu: f16 latent KV cache option (explicit -ctk/-ctv f16 halves cache memory, f32 stays default)
Track explicit cache-type requests through CLI/env; openPangu resolves no-request
to f32 (unchanged), accepts explicit f32/f16, warns and falls back to f32 for
BF16/quantized. Sink and cached-token KQ paths stay separate until after KQ so
the latent cache is read directly without the f32-only concat; value is the sum
of the sink and cache matmuls. Ring and DSA indexer caches stay f32; cache_size()
follows the resolved types.
* openpangu: enable graph reuse
* openpangu: wire multi-head MTP drafting
* openpangu: add MTP heads override
* openpangu: keep MTP update logits last
* openpangu: scope MTP warmup heads
* speculative: apply per-request MTP heads before warmup
* openpangu: fix multi-head MTP warmup computing on unwritten inputs
Each chained head called the build_inp_* helpers itself, so the warmup and
update graphs held one inp_tokens/inp_pos/inp_out_ids/KQ_mask tensor per
head while llama_set_inputs only fills the tensors the lctx pointers
reference, i.e. the last head's copies. Every head but the last read
unwritten compute-buffer memory: with heads=3 active even head 1's ring,
latent cache, and cached one-token draft were computed from garbage, which
is why depth-1 acceptance measured 4% against 98% for the heads=1 control.
Create the batch inputs once in build_openpangu and pass them to every
build_openpangu_mtp call, and fix the two chaining errors that were hiding
behind the garbage inputs:
- Shift the chained hidden: head k+1's row at position p consumes head k's
output row at p-1, the same convention head 1 uses for the target's
conditioned hidden rows. The predecessor of a batch's first row lives in
the previous warmup/update, carried across decodes through a new
inp_mtp_carry input backed by lctx.mtp_carry (written back per ubatch,
zeroed when a prompt warmup restarts from position 0).
- Fill head 3's cache row at draft step 2: each draft step runs one head,
so head 3's own decode at step 3 attended over a never-written row at
the step-2 position. Pre-write it from the committed carry.
Also include the active head count in the graph-reuse key next to the
existing step index (reuse stays forced off for this arch).
* speculative: default MTP drafting to a single head
A stage without an explicit heads= override previously resolved to 0,
meaning all model heads, so multi-head drafting was silently on by
default for models that carry more than one NextN layer. Keep it opt-in
(heads=N or heads=0 for all) until multi-head measures a win over the
single-head config; single-head models are unaffected either way.
* speculative: fence MTP head upshift over a warmed prefix
Deeper NextN heads only hold valid cache rows for spans that were warmed
with them. A request drafting with more MTP heads than the cached prefix
was warmed with (e.g. a heads=1 conversation continued with heads=3, a
pure extension the divergence fence deliberately allows) would read
never-written deeper-head rows: verification keeps the output correct,
but acceptance quietly collapses and any measurement taken there is
misleading.
Track the minimum head count the committed context has been warmed with
since position 0 and have the server reprocess from scratch when a
request asks for more. Also announce the model's NextN head count and
the single-head default once at MTP context setup.
* openpangu: skip dead MTP chain compute and stall-free carry readback
The update chain's last head and the draft-time row fill only matter for
their latent-cache and conv-ring writes; their FFN, norms, and shared
head fed nothing. Add a cache-writes-only mode to the MTP block builder
that returns after the attention block, and use it at both sites.
The multi-head carry readback previously synchronized the scheduler
after every warmup/update decode, a hard stall on CUDA. Issue the
device-to-host copy async on the backend stream instead (stream order
protects the source buffer from later graphs) and synchronize lazily
when the host buffer is next consumed or resized.
* openpangu: stop emitting fused kv_b tensor
* openpangu: default latent cache to f16
* openpangu: refuse unsupported latent cache types
* Window OpenPangu SWA cache reads
* Gather OpenPangu DSA decode reads
Gather DSA decode attention over the selected latent rows for OpenPangu base-model decode and verify graphs. The gathered branch now uses ggml_top_k order directly, runs maskless softmax over sinks plus selected rows for T <= 14, and derives values from the gathered k_l rows instead of the transposed latent cache.
* Chunk OpenPangu indexer prefill scoring
* Chunk OpenPangu prefill attention
* Gather OpenPangu sparse prefill attention
* Drop OpenPangu value cache
* Add OpenPangu indexer cache type flag
* Add OpenPangu q8_0 latent cache type
Store the OpenPangu MLA latent K cache as q8_0 via -ctk q8_0 (about 0.53x of
f16); the default stays f16 so behavior is unchanged without the flag. Latent V
stays f16/f32.
The q8 latent cache is a storage format only: it is dequanted to F32 before all
compute. K reads go through openpangu_build_k_latent_for_read, V derivation
through openpangu_build_v_latent_from_k (full 576-wide row to F32, then slice),
and the DSA gather paths already dequant via get_rows. Feeding a q8 latent view
directly into the KQ mul_mat corrupts large-context prefill, so that path is
removed for quantized caches. The cache write stages ckv and kpe through F32 and
writes one full 576-wide q8 row per token.
Verified on a small discriminator model: the default f16 path is byte-identical
to the prior code; the first-DSA-layer attention envelope is within 0.6% of the
f16 cache (linf_rel 0.0057); top-k selection is bit-identical between cache
types; the q8 latent cache is 0.531x the f16 size at 8K and 32K context; and
generation stays coherent on both the dense and DSA-gather paths at all tested
context lengths.
* Remove OpenPangu debug trace env knobs and redundant DSA_TOPK override
Drop the five LLAMA_OPENPANGU_*_TRACE debug-logging knobs (DSA_GATHER_TRACE,
IDX_CHUNK_TRACE, ATT_CHUNK_TRACE, PREFILL_GATHER_TRACE, SWA_WINDOW_TRACE) and the
LLAMA_OPENPANGU_DSA_TOPK override, which duplicated the -dsatk / --dsa-top-k CLI
flag; top-k now comes solely from cparams.dsa_top_k. The five perf-tuning knobs
(DSA_GATHER, IDX_CHUNK, ATT_CHUNK, ATT_KQ_MAX_MIB, PREFILL_GATHER) are retained
pending the perf battery. No change to default behavior.
* Subchunk OpenPangu DSA prefill gather to fit CUDA grid limit
The prefill gathered-attention ggml_get_rows produced dst rows = topk *
token_chunk (2048 * 256 = 524288) mapped to the CUDA grid.y dimension, which
caps at 65535, crashing with GET_ROWS invalid argument at long context (N_KV
around 10.5K with the natural topk of 2048). Split the prefill gather into token
subchunks so topk * subchunk_tokens stays within the grid limit, and guard the
decode gather with the same fit check (falling back to the dense masked path if
a pathological topk would not fit). The subchunking is over the token dimension
only, so per-token attention is unchanged and the result is numerically
identical. Verified: the GPU sweep runs past the old crash boundary to 22K+ with
zero CUDA errors; CPU and -ctk q8_0 paths unaffected.
* openpangu: fix scheduler node budget for chunked DSA prefill; drop unused attn_kv_b; remove env tunables
- Size the scheduler graph node budget for the chunked DSA prefill so 32K/ub2048 no
longer trips the hash-set reservation assert; derive the extra budget from the
builder's chunk/top-k/window structure with a fixed safety margin.
- Remove LLAMA_OPENPANGU_* environment tunables from both the node-budget estimator
and build_openpangu.cpp; use fixed constants in both so they stay in sync.
- Converter: emit only the split attn_k_b/attn_v_b projections and drop the unused
fused attn_kv_b tensor.
* openpangu: restore DeepSeek converter kv_b; drop trace env + dead code; fix dense-fallback node budget
- convert_hf_to_gguf.py: restore fused attn_kv_b in DeepseekV2Model (shared
parent); openPangu subclass keeps split-only k_b/v_b. Stops newly-converted
DeepSeek GGUFs from failing to load.
- src/llama.cpp: remove LLAMA_GRAPH_REUSE_TRACE getenv, hit/miss counters, and
the unconditional destructor log (no getenv or behavior change for any arch);
node-budget estimator now covers the dense-fallback (n_swa==0) attention-chunk
loop while skipping absent idx/top-k terms, preserving a strict overcount;
remove unreachable openPangu split-cache block.
- src/llama-context.h: drop now-dead graph_reuse_hits/misses members.
- include/llama.h: move type_k/type_v/idx_type_k *_explicit bools to struct end
to avoid a mid-struct ABI shift for out-of-tree consumers.
- src/graphs/build_openpangu.cpp: replace vestigial env-struct singletons with
the OPENPANGU_* constants; drop a redundant Sinkhorn permute round-trip
(one transpose; greedy output verified byte-identical).
Decode output unchanged (byte-identical greedy generation verified); shared-file
changes are openPangu-gated or restore the pre-PR baseline.
* openpangu: chat-parser support (reasoning split + thinking toggle)
Two openPangu-only fixes, both gated on the arch-unique token
<|pangu_text_start|> so no other model's parsing changes.
- chat-diff-analyzer: add a workarounds entry that force-sets TAG_BASED
reasoning with an empty start and a </think> end. openPangu prefills
<think> in the generation prompt, so the output is delimited only by
</think>; the differential detector otherwise learns start="<think>"
from the assistant-history form and fails to split, leaking reasoning
into content. Same shape as the existing Laguna prefill patch.
- chat.cpp: bridge enable_thinking to the template's `thinking` variable.
openPangu's template gates reasoning on `thinking` rather than the
ecosystem-standard `enable_thinking`, so the standard toggle was inert.
An explicit `thinking` chat_template_kwarg still overrides via the
extra_context merge.
Blast radius: test-chat-auto-parser 437/437 unchanged; the sole
test-chat-template diff is a pre-existing GLM trailing-newline.
* openpangu: use ggml_cast for latent dequant reads
Replace ggml_cpy(view, ggml_new_tensor_2d(F32, ...)) with ggml_cast in the MLA
latent V-from-K and K-read helpers. ggml_cast emits the identical GGML_OP_CPY
node into a fresh f32 tensor, so behavior is unchanged; it is the idiomatic
form. Per review.
* openpangu: narrow SWA reuse-key fields to 32-bit
The openpangu_swa_window_view reuse key stored n_kv/n_tokens/window/pad as
int64_t, but these are bounded well under 2^31 (window/pad are uint32_t at
source; n_kv/n_tokens <= context length). Narrow to int32_t/uint32_t and drop
the widening casts. w_view/win_off stay int64_t: they feed ggml view
dims/offsets. Per review.
* openpangu: precompute param_sink derived tensors at load
The per-layer attention-sink block (sink_blk [576,NS]) and its transposed
latent (s_lat_t [NS,512]) are pure functions of the layer weights, yet were
rebuilt every eval across all 49 layers (RMS-norm + cast + concat + transpose).
Compute them once at load, mirroring the wk_b derived-weight precompute, and
read the stored tensors in build_openpangu_attention. Numerically identical;
removes per-token work at decode.
* openpangu: replace conv position-ring with ggml_ssm_conv + spec-rollback checkpoint
Migrate the MoME depthwise causal conv (three sites per attention sublayer:
qa-lora, compressed-kv, attn-out) from the bespoke 16-column position-indexed
ring onto the core ggml_ssm_conv op with a recurrent conv-state slot.
Cache: s_l becomes [2*conv_col_ne, qnext_state_slots], holding the (d_conv-1)=2
history taps per channel for the three sites (float offsets 0 / 2*n_lora_q /
2*(n_lora_q+n_lora_kv)). Drops the conv_hist_idx / conv_write_idx graph inputs
and their fill in llama_set_inputs; adds one single-sequence sq input for
ggml_ssm_conv shared across the three sites and the MTP head.
Speculative rollback: the position ring self-healed rejected draft columns by
absolute position; a recurrent slot does not, since seq_rm is a no-op for
recurrent state. openPangu is admitted at the three spec-checkpoint save/init
gates so the whole-slot shadow checkpoint (gpu-fallback) snapshots the conv
slot before drafting and restores it before the accepted-token replay. The
restore path is already keyed on ckpt.valid, so no gate change is needed there.
Per-step checkpoint mode is declined for openPangu, which has no SSM recurrent
term, so auto mode resolves to the whole-slot shadow.
Gated: non-spec needle unchanged; MTP-spec needle correct with healthy draft
acceptance (rollback verified via the acceptance canary).
* openpangu: single ggml_concat copy for the latent cache store
The non-quantized latent store split the [ckv | roped k_pe] row into two views
and two cache copies, with a base_offset field on the CacheCopy struct to place
the second one. Match the quantized path: concat the two parts and do one copy
into the cache row. This drops the second cache-copy slot (OPENPANGU_COPY_K_KPE)
and removes base_offset from CacheCopy entirely.
Cache contents are unchanged: the concat writes the same [ckv 512 | k_pe 64]
bytes to the same row. Gated on the needle for both the f16 latent path (the one
that changed) and the q8 latent path, plus coherence.
* openpangu: reuse the shared kr_l indexer cache instead of a separate idx_l
The DSA lightning indexer stored its per-position keys in an openPangu-only idx_l
cache, parallel to the kr_l indexer cache GLM-DSA already uses. Both have the same
storage contract: [indexer_head_size, kv_size], idx_type_k dtype, one row per KV
cell, written at kv_head and read [dim, n_kv] from zero. openPangu now allocates
its indexer keys into kr_l and shares the dsa_cache_copies graph-reuse fixup.
The fixup patch is factored into a helper that both the generic path and the
openPangu update_cache_copies branch call, so the openPangu indexer copy is
repointed to the current kv_head on graph reuse like every other cache write.
This drops the idx_l vector, its allocation and memory accounting, and the
openPangu third cache-copy slot (now one latent copy per layer).
Per-arch allocation predicates stay separate (GLM uses indexer_is_full, openPangu
uses the window==0 DSA schedule); only the kr_l storage and the copy fixup are
shared. openPangu keeps its no-shift/no-defrag/no-state-I/O behavior, and the GLM
Hadamard/k-shift logic stays GLM-gated.
Gated: needle correct on f16 and q8 latent caches and under MTP speculation
(acceptance unchanged at 0.67), plus coherence.
* openpangu: discard pos-0 graphs from reuse; retire stale conv-state comments
The ggml_ssm_conv refactor bakes the pos-0 conv-state reset into graph
topology (a scale-by-zero node on the state view). A graph built at pos 0
could be reused at pos > 0 when the batch shape and padded n_kv match (a
1-token prompt followed by TG is the concrete case), zeroing the conv
history on every reused decode. Admit openPangu at the existing
reset_previous gate so pos-0 graphs are discarded from reuse, the same
guard the qnext recurrent state relies on.
Also retire the internal phase-plan comments the conv refactor left
behind: they claimed the spec-checkpoint wiring had not landed in the
commit that landed it, and misdescribed the s_l slot as awaiting rollback
support.
Gated: needle 8457 on f16 and q8 latent, MTP-spec needle (drafts fully
accepted), coherence.
* openpangu: drop the _explicit cache-type plumbing; validate unconditionally
Review follow-up (item 1 of the second review). The explicit/default
distinction carried less than claimed: the latent K/V fallback was f16,
which is already the -ctk/-ctv and API default, so distinguishing unset
from set-to-the-default bought nothing, and the two bools were behaviorally
redundant. The only load-bearing use was the indexer cache, where openPangu
defaulted to f32 while -ictk defaults to f16. Gating the f16 indexer
directly (needle on f16 and q8 latent paths, MTP speculation, coherence)
shows no quality difference, so openPangu now takes the standard f16
indexer default and the f32 special case is gone. Default indexer cache
memory halves (64 -> 32 MiB at c 8192).
Removes type_k_explicit/type_v_explicit/idx_type_k_explicit from llama.h,
the cparams/mparams plumbing, and common; the resolve helpers become plain
unconditional validators, so -ctk q8_0 is honored and an unsupported type
errors out at load instead of silently coercing.
Gated: needle 8457 on the new f16-indexer default, on q8 latent with MTP
speculation, and with -ictk f32 explicitly honored (64 MiB f32 buffer in
the load log); -ctk q4_0 and -ictk q4_1 refused with a clear error.
* openpangu: keep MTP draft decodes position-contiguous under speculation
The MTP framework's one-token draft shortcut caches a prediction one row
past the accepted prefix during the accepted-token update, then skips
re-decoding the last sampled token at the next draft round. A
mask-addressed cache tolerates the resulting position gap; openPangu's
position-addressed append-only cache (cell == position) does not: after a
rollback the next draft decode lands one cell behind its position, and
after a full acceptance the cache head sits one row ahead of the next
draft base, either way tripping the kv_head == pos[0] invariant and
aborting the server. The checkpoint admission in the conv refactor made
this the standard openPangu speculative flow; the needle-first gates
never generated enough draft rounds against a short prompt to reach it.
Decline the shortcut re-seed for openPangu in mtp_accept_batch (restoring
the drafting behavior all measured acceptance numbers were taken on) and
trim rows at or beyond the draft base in mtp_speculative_gen_draft, so
every draft decode stays position-contiguous with the cache head.
Gated: the crashing flow (short prompt, 512-token spec generation, then a
second request) completes with acceptance 0.60 prose / 0.87 code,
matching the pre-checkpoint baseline profile; needle 8457 plus coherence
on f16+spec and q8+spec.
* openpangu: remove stale ring limits and fix MTP graph reuse
* cli: preserve speculative carry on fallback
Decode an already-emitted pending token when a draft cannot be used instead of sampling unchanged logits and duplicating output. Document single-head MTP as the default and multi-head modes as experimental.
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
|
||
|
|
606d9db301 |
server: do not recover prompt below cache-ram-similarity and other cleanup (#2105)
* server: clean up * server: do not recover prompt below cache-ram-similarity --------- Co-authored-by: firecoperana <firecoperana> |
||
|
Jun YamogandGitHub
|
33dabeae21 | Normalize disabled context-shift overflow error (#2051) | ||
|
FarmadupeandGitHub
|
af62a37acd |
Prune examples/llava. Dead code. (#2025)
examples/llava has been replaced by mtmd since late 2025, and has been out-of-build in ik_llama.cpp since examples/CMakeLists.txt removed it in #798. Repointed descriptions from llava to mtmd where they remained. |
||
|
FarmadupeandGitHub
|
d3e86a5431 |
Free raw multimedia data from server_tokens after encoding, as it will never be read again (#2029)
Data server_tokens.map_idx_to_media.tokens_image.batch_f32 is read exactly once, by mtmd_encode, however it was retained as long as the input image was present in the sequence. Add a manual free function to clear out this data after encoding. Solves: * Memory wasted in struct server_tokens * The same wasted memory in the ram cache * Long copy durations cloning this data to/from ram cache * Accounting failures in ram cache (`batch_f32` can be larger than a sequence's entire KV) * The above accounting failures leading to terminal memory leaks in pathological cases * Remove JSON serialization for `batch_32` which was unused, and had no foreseeable usecase |
||
|
|
befbc0945b |
server: variance based checkpoint eviction (#2020)
Co-authored-by: firecoperana <firecoperana> |
||
|
magikRUKKOLAandGitHub
|
72440a19fc |
on-demand tensor reload (#1989)
* host-swap tensor loop the host-swap functionality is only triggered when the certain env. variables are declared * target_include_directories tweak * hot-swap tensor support two intrusions: 1.) at the model loading to collect the snapshot 2.) the modification of the `/health` HTTP endpoint to be able to trigger the hot-swap via sending the `llama-server` the HTTP-request. *both a braced by the specific env. variables * hot-swap tensor support; graph invalidation ggml_backend_cuda_invalidate_graphs export * hot-swap tensor support graph invalidation implementation; extended debug output (commented out) * llama_reload_changed_tensors export * tensor hot-swap on-demand reload cpu-only/hybrid/gpu-only with split mode layer/graph full support implementation * docs * reuse the gguf parsing from llama.cpp gguf_init_from_file, gguf_find_tensor, ggml_get_tensor * remove the manual scheduling for hybrid inference * update docs * tensor shape validation * update docs * update docs accidentally wiped the previous changes; so recovered them * revert the GGML_CUDA_MAX_DEVICES to 16 * update llama_reload_changed_tensor update llama_reload_changed_tensor, revert CMakeLists.txt * update llama_reload_changed_tensor * GGML_MAX_SRC GGML_MAX_SRC compile-time definition support * GGML_MAX_SRC GGML_MAX_SRC compile-time definition support * GGML_MAX_SRC GGML_MAX_SRC compile-time definition support * llama_reload_changed_tensor update llama_reload_changed_tensor definition * refactory move the tensor-reloading implementation to llama-reload.cpp, llama-reload-info.h; some bugfixes and code reduction * revert added back the missing newline * update docs * reload_info constructor * bugfix: cpu-only TODO: improve the working environment by compiling for multiple hardware configurations; possibly make a test pipeline * cpu-only bugfix set the fix again after unsuccessful sync with main * windows os compilation fix #include <string> * fix windows os build error C2039: 'string': is not a member of 'std' * remove dead file * implement perplexity in server * Revert "implement perplexity in server" |
||
|
KawrakowandGitHub
|
71af16a6b7 | Fix DFlash oerformance with split mode graph (#1980) | ||
|
Jun YamogandGitHub
|
064d23a6f8 |
Codex CLI Responses Compatibility (#1964)
* responses: skip known unsupported Responses tool types from Codex CLI - Skip namespace, web_search, image_generation tools instead of HTTP 500 - Reject unknown non-function tool types with controlled error - Preserve function tool conversion logic unchanged Fixes Codex CLI 0.133.0 compatibility where it sends mixed tool types. * responses: harden codex compatibility coverage * responses: expose Codex model catalog metadata |
||
|
SamuelOliveirads
|
6cae8c7ba2 | clean logs | ||
|
SamuelOliveirads
|
0d75eee35a | remove duplicated code and unnecesary refactor | ||
|
SamuelOliveirads
|
3a1d46c4d1 |
Merge remote-tracking branch 'origin/main' into feat/dflash-implementation
# Conflicts: # common/common.cpp # common/speculative.cpp # convert_hf_to_gguf.py # examples/server/server-context.cpp # examples/server/server-context.h # src/llama-arch.cpp # src/llama-arch.h # src/llama-model.cpp # src/llama.cpp |
||
|
Samuel Oliveira AlvesandGitHub
|
8a38025174 |
Refactor: Move spec outside server (#1949)
* Refactor speculative decoding: move logic outside of server * remove duplicated tokens in mtp kv cache * narrow to only discard draft cells in MTP * revert mtp_speculative_gen_draft |
||
|
Simon LundellandGitHub
|
b1eb8bb0a1 |
server: gate llama_decode_stop() to the active decode (fix queued-cancel cascade) (#1941)
With --parallel 1, a client disconnect/timeout on a *queued* request aborts the
*active* decode of a different client (llama_decode: failed to decode, ret = -3 /
"Decode process is cancelled by user"), releasing the slot with the request
unfinished. To the active client the stream silently stalls and never returns,
while the server reports healthy — easy to misdiagnose as a network/proxy wedge.
Root cause: llama_decode_stop() signals a process-global stop flag that the
active decode loop polls. examples/server/server.cpp calls it *ungated* from the
request reader's connection-closed paths, so any reader closing (including a
queued, not-yet-running task's) trips the global flag against whatever decode is
currently active. Adjacent to #1576/#1673 ("clear sticky stop flag" +
hybrid/recurrent ret=-3), which did not gate these call sites against non-active
readers, so the queued-cancel-kills-active cascade still fires on current main.
Fix (minimal gate): add server_response_reader::any_task_on_slot() and gate the
three llama_decode_stop() sites on it, so the global stop is signalled only when
one of THIS reader's tasks is on a slot (the active decode). A queued task's
disconnect then only drops that queued task. Verified in production under heavy
concurrent, frequently-cancelled load (hundreds of queued-task cancels, zero
active-decode kills). Stdlib-only reproducer in the PR description.
Caveat: any_task_on_slot() reads the slots vector from the reader thread — the
same race class as the existing process-global flag; can be tightened to a
per-context/per-task cancellation if preferred.
|
||
|
KawrakowandGitHub
|
e6f8112f3b | Adjust CUDA FA kernel parameters for head size 512 on Turing (#1942) | ||
|
|
2a1148384c |
server: fix double submits of infill (#1944)
Co-authored-by: firecoperana <firecoperana> |
||
|
FarmadupeandGitHub
|
1520eda980 | prompt cache: Fix assertion that prompt cache does ot rewind to middle of image (#1913) | ||
|
Samuel Oliveira AlvesandGitHub
|
007d640098 |
Standardize speculative decoding arguments on the server (#1908)
* refactor spec args * add shell-safe quoting of string-valued stage keys in speculative decoding |
||
|
|
6c0180d702 |
server: enable mcp proxy (#1904)
* update http lib * Add cors proxy --------- Co-authored-by: firecoperana <firecoperana> |
||
|
|
074fc7dafd |
webui: update llamacpp webui (#1903)
update config ui: fix audio and video modality detection (#23756) When model props are fetched asynchronously from the server, modelPropsVersion is incremented to trigger reactivity, but only the vision effect was listening to it. webui: update ignore files ui: handle audio/vnd.wave as audio WAV file (#23754) Firefox on Linux uses this MIME type ui: exclude generated build dirs from prettier and eslint so lint errors stop being masked (#23910) webui: add custom CSS injection via config (#23904) * webui: add custom CSS injection via config register a customCSS setting in the Developer section under Custom JSON, syncable so it rides the existing ui-config pass through. inject the value into a single style element in the head, reactive on the setting. lets an operator theme a prebuilt binary through --ui-config without rebuilding, and lets a user set it from the settings panel. move the textContent write into a use: action on the head style node. the action is the idiomatic way to touch a node, so the no-dom-manipulating lint rule is satisfied without a disable. value stays text through textContent, never parsed as HTML. * Update tools/ui/src/lib/constants/settings-keys.ts Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> * ui: address review from @allozaur, rename custom config key to customJson with migration rename the custom config key to customJson across the type, the chat request builder, the settings save check and the custom tools reader, keeping the custom API param name unchanged. add a non destructive migration that copies the legacy custom key to customJson at startup. only render the head style tag when custom CSS is set. --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> server: real-time reasoning interruption via control endpoint (#23971) Builds on the manual reasoning budget trigger from #23949. Adds a CONTROL task that mirrors the CANCEL path on the live slot and calls common_sampler_reasoning_budget_force to end thinking mid-generation. POST /v1/chat/completions/control with { id_slot, action }, opt-in reasoning_control arms the budget sampler on demand. Router and single model. Minimal WebUI button as a skeleton for further UI work. * ui: track reasoning phase via explicit streaming state Add isReasoning to the chat store, mirroring the isLoading pattern: per conversation map, private setter, public accessor and reactive export. Set from the stream callbacks, true on reasoning chunks, false on the first content chunk, reset on stream end and resynced on conversation switch. The skip button now keys off isReasoning so it shows only during the thinking phase, not the whole generation. * ui: extract control endpoint and action into constants Move the chat completion routes, the slots route and the reasoning control action out of chat.service into api-endpoints and a dedicated control-actions module. No behavior change, drops the magic strings so the control protocol has a single source of truth. * server: target reasoning control by completion id Address @ngxson review on the control endpoint. Switch from id_slot to the chat completion id to avoid a TOCTOU: the slot can be reassigned between the lookup and the control request, so matching the live completion (oaicompat_cmpl_id) is safe and a finished one simply matches nothing. Rename the action to reasoning_end, guard it on the reasoning_control flag of the target slot, and reduce the response to {success} with an optional message. * ui: target reasoning control by completion id Keep the streamed completion id on the message and post it back to the control endpoint instead of probing /slots. Drops the slot discovery and the TOCTOU that came with it. Action renamed to reasoning_end, response read as {success}. * server: address review from @ngxson Move the control fields into task_params and drop the redundant comments on the control path. * server: document the reasoning control endpoint * Update tools/ui/src/lib/types/database.d.ts Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> * ui: rename cmplId to completionId Per @allozaur review, clearer name for the streamed completion id. * ui: wire completion id capture through the agentic flow The webui streams through the agentic flow, which relayed onModel but not onCompletionId, so the completion id never reached the message and the control request was never sent. Relay it through the flow and its callbacks type, declare id on the chunk type, and log an explicit error when the button fires without a usable id. * ui: target reasoning control model from the message The model is a property of the completion, so read it from the streaming message like the id, not from the model dropdown which is unrelated UI state. Makes the request self-consistent by construction instead of just unlikely to drift. --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> ui: Add Thinking mode toggle with reasoning effort levels + improvements for Chat Form Add Action UI (#23434) Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> * fix: Model tags ui: simplify network error handling (#23431) Previously error to string conversion was split in two different files, with one converting errors into strings, and another function analyzing those strings to generate yet another string. Now the the error handling for network fetches has been centralised and uses directly HTTP error codes whereas possible to generate the human-readable error strings. It also fixes an issue where all JSON errors reported from the backend, such as "Invalid API key", would get turned incorrectly in to "Failed to connect to server" due to poor matching logic in the now-gone getErrorMessage function. update html ui: Mermaid Diagrams in chat + interactive preview (#24032) webui: fix tool selector toggle/counter, key tools by stable identity (#24065) * webui: fix tool selector toggle/counter, key tools by stable identity Key the disabled set, counts and toggles by a stable per-tool key instead of bare function name, deduped from one canonical list. Per-tool checkboxes become presentational (single row handler, no nested button), category checkboxes drop the tristate (n/total carries partial). One getEnabledToolsForLLM keeps normalized MCP schemas and dedupes by name. * ui: use SvelteSet and SvelteMap for local tool collections to satisfy svelte/prefer-svelte-reactivity Co-authored-by: firecoperana <firecoperana> |
||
|
SamuelOliveirads
|
1250f522ed | add qwen, gemma and kimi dflash support | ||
|
SamuelOliveirads
|
1369e68471 | fix graph mask, swa layers and tokens positions | ||
|
SamuelOliveirads
|
532499836e | improve DFlash caching and profiling capabilities | ||
|
Samuel Oliveira AlvesandGitHub
|
3f40e73c36 | expand np guardrail for all mtp types (#1901) | ||
|
SamuelOliveirads
|
9f5f70cf7e | implement target position tracking and context management | ||
|
SamuelOliveirads
|
82cff238fe | Initial dflash implementation | ||
|
dungquixote42andGitHub
|
642c038ccd |
Extend expiring logit bias to other sampling parameters (#1770)
* initial commit * fix underflow bug, add debug prints, update macro/variable names * fix phrases-sharing-1-flag bug, replace macros with struct member function * cleanup * fix file parsing * string_split_open_close() -> string_extract(), improve escape handling * support multiple nested entries * make persistent entries global, simplify file parsing * cosmetic changes * add support for jumping to exitword * update variable names * fix bad search bug * better debug prints, reorg * replace lambda with string_is_found(), add string_unescape() for debug * add support for inline comments * add missing debug print macro * fix type promotion bug * actually fix type promotion bug |
||
|
Samuel Oliveira AlvesandGitHub
|
d51036a0c4 | fix: reset KV cache and prompt state in server_slot and server_context (#1860) | ||
|
Samuel Oliveira AlvesandGitHub
|
11a1fea9e2 |
Move embedding management to speculative (#1825)
* refactor speculative decoding with companion context and draft result structures * feat: add common speculative feature handling in server context * refactor: move embedings outside server * feat: harden draft input hidden state in llama context * remove unused functions * refactor: streamline speculative feature handling and remove unused code * remove redundant code * remove more unused variables * refactor: implement speculative feature handling |
||
|
Samuel Oliveira AlvesandGitHub
|
77413bc900 | Add Hadamard parameters to draft model loading (#1840) | ||
|
|
104846ddee |
spec : disacard last drafted token with low prob (#1820)
* spec : disacard last drafted token with low prob * Apply suggestion from @ikawrakow Co-authored-by: Kawrakow <iwankawrakow@gmail.com> --------- Co-authored-by: firecoperana <firecoperana> Co-authored-by: Kawrakow <iwankawrakow@gmail.com> |
||
|
|
f645ed1e2d |
AutoParser: improve reasoning budget and handling of space/newline in tool calls (#1819)
common/chat, server: refactor, move all conversion functions to common, add tests (#20690) jinja : remove unused header (#22310) common : fix jinja warnings with clang 21 (#22313) Signed-off-by: Adrien Gallouët <angt@huggingface.co> chat: fix handling of space in reasoning markers (#22353) * chat: fix handling of space in reasoning markers common : re-arm reasoning budget after DONE on new <think> (#22323) common : determine generation prompt using longest common prefix (#22657) common/autoparser: fixes for newline handling / forced tool calls (#22654) * chat/autoparser: the fixes * Move optspace() to chat-peg-parser, comment out server tests invalidated due to content now allowed with forced tool calls. * Trim whitespace on apply instead common/chat : preserve media markers for typed-content templates (#22634) common : revert reasoning budget +inf logit bias (#22740) common : do not wrap raw strings in schema parser for tagged parsers (#22827) common : enable streaming JSON argument values (#23173) * common : remove atomic from json arguments * common : remove parsing logic on JSON arguments common : do not pass prompt tokens to reasoning budget sampler (#22488) reasoning-budget: clone should do a deep-copy (#23095) Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> |
||
|
gapeleonandGitHub
|
c35189d83c |
fix(server): reset chat parser on slot reuse to prevent crash (#1763) (#1794)
If a slot is reused for a standard completion (`/v1/completions`) after being used for a chat completion (`/v1/chat/completions`), the previous chat's PEG parser would remain active in the slot's parameters. This caused standard text completions to throw on the raw text. |
||
|
Samuel Oliveira AlvesandGitHub
|
f4f4b3ff26 |
Allow dual speculative decoding (#1789)
* wip: test logic to use multiple specs * feat: introduce composite speculative decoding stages * handle MTP context and draft invalidation * fix: allow gemma mtp for speculative stages * fix: normalize spec stage keys * refactor: remove enable_mtp flag and improve speculative stage handling * fix: update cached text tokens handling for stage chains * feat: implement sync for external MTP after non-MTP accept |
||
|
Samuel Oliveira AlvesandGitHub
|
35fbe08d6e | disable MTP for parallel slots (#1804) | ||
|
ubergarmandGitHub
|
ca52a825db |
feat: add --threads-mtmd for independent multimodal thread count (#1797)
Add `-tm` / `--threads-mtmd` to control CPU thread count used during
multimodal image/audio processing (mmproj encoding), separate from the
main LLM thread count.
This allows running the LLM on GPU with minimal CPU threads (e.g. `-t 1`)
to reduce sync overhead, while using many threads (e.g. `-tm 16`) for
CPU-bound mmproj encoding with `--no-mmproj-offload`.
Fallback chain when `-tm` is not specified:
1. `--threads-batch` (-tb) — multimodal encoding is a batch/prefill-like
operation, so it makes sense to track with batch thread count
2. `--threads` (-t) — final default
Works with both mtmd-cli and llama-server.
AI: ubergarm/Qwen3.6-27B-GGUF MTP IQ4_KS 15.113 GiB (4.752 BPW) + pi.dev
|