LLM-Inference/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-08-12 22:29:39 +04:00
Code Issues Packages Projects Releases Wiki Activity

server: fix ret=-3 on hybrid/recurrent prompt cache, and clear sticky stop flag (#1673)

Browse Source 
Two related issues that manifest as 'llama_decode ret=-3' on hybrid
architectures (e.g. Qwen3.5/3.6 MoE, Qwen3-Next), matching the symptom
reported in #1576.

1) server_context::apply_checkpoint() was written around transformer KV
   semantics (pos_min / pos_max per-token window). For hybrid and pure
   recurrent models the per-token pos_min threshold does not apply: the
   recurrent state is a single snapshot, and the server-side checkpoint
   is a whole-prefix record. The old selector 'cur.pos_min < pos_min_thold'
   can succeed on a checkpoint whose pos_max is past the current n_past,
   and — more commonly — fall through to do_reset = true, which zeros
   slot.n_past / slot.n_past_prompt. Zeroing in-place while the recurrent
   state in the context is still populated makes the next decode batch
   disagree with the live state, returning ret=-3.

   This change gates the checkpoint path on
   llama_model_has_recurrent(llama_get_model(slot.ctx)):
   - selector uses pos_max <= slot.n_past && pos_max < pos_next
     (whole-prefix match, leaves at least one token to decode);
   - on miss, slot state is preserved rather than zeroed, letting
     update_slots() continue from the already-valid n_past_prompt;
   - the erase loop drops any checkpoint whose pos_max > pos_next,
     matching the rewind semantics for recurrent state.

   Transformer behavior is unchanged.

2) stop_internal_decode is a file-static global in src/llama.cpp, set by
   llama_decode_stop() (called on client disconnect) and polled inside
   the decode loop to bail out with ret=-3. The flag is only cleared on
   one conditional path in server_slot::release(), so a stop signal that
   arrives after the interrupted llama_decode() has already returned
   bleeds into the NEXT decode call and causes an immediate ret=-3 with
   no work performed. Clear it at the top of the public llama_decode()
   entry so the signal is scoped to the in-flight decode it was meant
   for.

Build-verified: llama-server with GGML_CUDA=ON, -DCMAKE_CUDA_ARCHITECTURES=86
(sm_86), IQK flash-attn + matmul enabled. No new APIs introduced —
llama_model_has_recurrent is already public and already used elsewhere in
server-context.cpp.

Closes #1576
This commit is contained in:
markaalonzo
2026-04-23 09:19:17 +02:00
committed by GitHub
parent e5355e9895
commit 48819dadaf
2 changed files with 31 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/server/server-context.cpp
+26 -7
View File
@@ -3179,11 +3179,14 @@ void server_context::create_checkpoint_at_interval(server_slot & slot, const gp
void server_context::apply_checkpoint(server_slot & slot) {
llama_pos pos_next = slot.cache_tokens.pos_next(slot.n_past);
const bool has_recurrent = llama_model_has_recurrent(llama_get_model(slot.ctx));
// For hybrid/recurrent models, pos_min semantics don't apply: the recurrent state is a single
// snapshot, not a per-token window. Use pos_max against n_past to match whole-prefix checkpoints.
const auto pos_min_thold = std::max(0, pos_next - 1);
if (slot.n_past > 0 && slot.n_past < slot.cache_tokens.n_tokens()) {
int32_t pos_min = llama_kv_cache_seq_pos_min(slot.ctx, slot.id);
if (pos_min > pos_min_thold) {
if (has_recurrent || pos_min > pos_min_thold) {
SLT_WRN(slot, "n_past = %d, slot.prompt.tokens.size() = %d, seq_id = %d, pos_min = %d\n", slot.n_past, (int)slot.cache_tokens.size(), slot.id, pos_min);
// search for a context checkpoint
@@ -3192,6 +3195,11 @@ void server_context::apply_checkpoint(server_slot & slot) {
slot.server_cached_prompt.checkpoints.rend(),
[&](const auto & cur) {
// guarantee that a checkpoint will result in at least one token being processed [TAG_PROMPT_LOGITS]
if (has_recurrent) {
// recurrent/hybrid: only whole-prefix checkpoints are valid; pick the latest one
// that covers no more than the current n_past and still leaves tokens to decode.
return cur.pos_max <= slot.n_past && cur.pos_max < pos_next;
}
return cur.pos_min < pos_min_thold;
}
);
@@ -3218,19 +3226,30 @@ void server_context::apply_checkpoint(server_slot & slot) {
}
if (do_reset) {
SLT_WRN(slot, "forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see %s)\n",
"https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055");
slot.n_past = 0;
slot.n_past_prompt = 0;
if (has_recurrent) {
// Hybrid/recurrent: do NOT zero n_past. The prompt prefix is already in cache_tokens
// and update_slots() reprocesses from slot.n_past_prompt; dropping to 0 forces a full
// recompute on every turn and — combined with cached state — trips llama_decode ret=-3.
SLT_WRN(slot, "no usable hybrid/recurrent checkpoint; preserving slot state (n_past = %d, n_past_prompt = %d)\n",
(int)slot.n_past, (int)slot.n_past_prompt);
} else {
SLT_WRN(slot, "forcing full prompt re-processing due to lack of cache data (likely due to SWA, see %s)\n",
"https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055");
slot.n_past = 0;
slot.n_past_prompt = 0;
}
}
}
}
{
// erase any checkpoints with pos_min > pos_min_thold
// erase checkpoints that are no longer consistent with the current decode position.
// Transformer: anything with pos_min beyond the threshold is stale.
// Recurrent/hybrid: anything with pos_max past pos_next refers to future tokens we've rewound past.
for (auto it = slot.server_cached_prompt.checkpoints.begin(); it != slot.server_cached_prompt.checkpoints.end();) {
const auto & cur = *it;
if (cur.pos_min > pos_min_thold) {
const bool stale = has_recurrent ? (cur.pos_max > pos_next) : (cur.pos_min > pos_min_thold);
if (stale) {
SLT_WRN(slot, "erased invalidated context checkpoint (pos_min = %d, pos_max = %d, size = %.3f MiB)\n", cur.pos_min, cur.pos_max, (float)cur.data.size() / 1024 / 1024);
it = slot.server_cached_prompt.checkpoints.erase(it);
} else {