From 0a4e10c7fb65d2dd5a4afb78339c7d373a8cdfaa Mon Sep 17 00:00:00 2001
From: pbrejtfus <99141595+pbrejtfus@users.noreply.github.com>
Date: Sun, 26 Jul 2026 18:20:57 +0200
Subject: [PATCH] server : remove usage field from intermediate streaming
 chunks (#2189)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

While using the server with the Mistral Vibe agent, I ran into an issue where every single prompt triggered an immediate context auto-compaction, showing that the model's context window was fully maxed out.

The root cause is a mismatch with the OpenAI API spec. In streaming mode (stream: true), every intermediate SSE chunk incorrectly includes a usage object containing prompt_tokens and completion_tokens.

According to the OpenAI specification, usage should only appear once in the final chunk (alongside an empty choices array). Upstream llama.cpp handles this correctly — if you look at examples/server/server-task.cpp, server_task_result_cmpl_partial::to_json_oaicompat_chat() builds delta chunks without any usage field.

However, in this fork, both to_json_oaicompat_partial() and to_json_oaicompat_chat_partial() embed a usage object in every single intermediate chunk. Since standard agents aggregate prompt_tokens from streaming chunks, they end up multiplying the real token count by the number of chunks.

For example:
Prompt: 9,000 tokens x 80 chunks = 720,000 "tokens" reported

This triggers fake "100% context" errors and forces an auto-compaction on every request.

What changed:
I removed the usage block from the partial chunk methods. The _final methods (to_json_oaicompat_chat_stream(), to_json_oaicompat_final()) already send usage correctly in the last chunk with empty choices, so I left those untouched.

(Debugged with some assistance from Mistral Vibe Qwen3.6-27B)
---
 examples/server/server-task.cpp | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/examples/server/server-task.cpp b/examples/server/server-task.cpp
index c35ce7107..95c1a05f2 100644
--- a/examples/server/server-task.cpp
+++ b/examples/server/server-task.cpp
@@ -120,11 +120,6 @@ json server_task_result_cmpl_partial::to_json_oaicompat_partial() {
         {"created",            t},
         {"model",              oaicompat_model},
         {"object",             "text_completion"},
-        {"usage", json {
-            {"completion_tokens", n_decoded},
-            {"prompt_tokens",     n_prompt_tokens},
-            {"total_tokens",      n_decoded + n_prompt_tokens}
-        }},
         {"id",                 oaicompat_cmpl_id}
     };
 
@@ -207,11 +202,6 @@ json server_task_result_cmpl_partial::to_json_oaicompat_chat_partial() {
             {"id", oaicompat_cmpl_id},
             {"model", oaicompat_model},
             {"object", "chat.completion.chunk"},
-            {"usage", json {
-                {"completion_tokens", n_decoded},
-                {"prompt_tokens",     n_prompt_tokens},
-                {"total_tokens",      n_decoded + n_prompt_tokens},
-            }},
             });
     };
     // We have to send an initial update to conform to openai behavior