mirror of
https://github.com/ikawrakow/ik_llama.cpp.git
synced 2026-08-12 22:29:39 +04:00
server : remove usage field from intermediate streaming chunks (#2189)
While using the server with the Mistral Vibe agent, I ran into an issue where every single prompt triggered an immediate context auto-compaction, showing that the model's context window was fully maxed out. The root cause is a mismatch with the OpenAI API spec. In streaming mode (stream: true), every intermediate SSE chunk incorrectly includes a usage object containing prompt_tokens and completion_tokens. According to the OpenAI specification, usage should only appear once in the final chunk (alongside an empty choices array). Upstream llama.cpp handles this correctly — if you look at examples/server/server-task.cpp, server_task_result_cmpl_partial::to_json_oaicompat_chat() builds delta chunks without any usage field. However, in this fork, both to_json_oaicompat_partial() and to_json_oaicompat_chat_partial() embed a usage object in every single intermediate chunk. Since standard agents aggregate prompt_tokens from streaming chunks, they end up multiplying the real token count by the number of chunks. For example: Prompt: 9,000 tokens x 80 chunks = 720,000 "tokens" reported This triggers fake "100% context" errors and forces an auto-compaction on every request. What changed: I removed the usage block from the partial chunk methods. The _final methods (to_json_oaicompat_chat_stream(), to_json_oaicompat_final()) already send usage correctly in the last chunk with empty choices, so I left those untouched. (Debugged with some assistance from Mistral Vibe Qwen3.6-27B)
This commit is contained in:
@@ -120,11 +120,6 @@ json server_task_result_cmpl_partial::to_json_oaicompat_partial() {
|
||||
{"created", t},
|
||||
{"model", oaicompat_model},
|
||||
{"object", "text_completion"},
|
||||
{"usage", json {
|
||||
{"completion_tokens", n_decoded},
|
||||
{"prompt_tokens", n_prompt_tokens},
|
||||
{"total_tokens", n_decoded + n_prompt_tokens}
|
||||
}},
|
||||
{"id", oaicompat_cmpl_id}
|
||||
};
|
||||
|
||||
@@ -207,11 +202,6 @@ json server_task_result_cmpl_partial::to_json_oaicompat_chat_partial() {
|
||||
{"id", oaicompat_cmpl_id},
|
||||
{"model", oaicompat_model},
|
||||
{"object", "chat.completion.chunk"},
|
||||
{"usage", json {
|
||||
{"completion_tokens", n_decoded},
|
||||
{"prompt_tokens", n_prompt_tokens},
|
||||
{"total_tokens", n_decoded + n_prompt_tokens},
|
||||
}},
|
||||
});
|
||||
};
|
||||
// We have to send an initial update to conform to openai behavior
|
||||
|
||||
Reference in New Issue
Block a user