From ae86645c28024aafd3ebb9102467bb79f101da64 Mon Sep 17 00:00:00 2001 From: Oliver Simons Date: Fri, 7 Aug 2026 20:59:36 +0200 Subject: [PATCH] Consistify polling frequency of llama-bench with args llama-bench differs from llama-server in 2 ways: 1. It attaches a persistent threadpool across cgraph invocations 2. It has a polling scheme of 50 (as opposed to 0 of args) Together, this causes a significant perf regression in the CUDA backend on Windows when the CPU backend is built with OpenMP disabled. Why is the CPU backend affecting the CUDA backend? Typically, we have a LUT on the CPU backend where we map Token_ID -> Embedding to save on VRAM budget. --- tools/llama-bench/llama-bench.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/llama-bench/llama-bench.cpp b/tools/llama-bench/llama-bench.cpp index c17a27b540..9862125378 100644 --- a/tools/llama-bench/llama-bench.cpp +++ b/tools/llama-bench/llama-bench.cpp @@ -380,7 +380,7 @@ static const cmd_params cmd_params_defaults = { /* n_threads */ { common_cpu_get_num_math() }, /* cpu_mask */ { "0x0" }, /* cpu_strict */ { false }, - /* poll */ { 50 }, + /* poll */ { 0 }, /* n_gpu_layers */ { -1 }, /* n_cpu_moe */ { 0 }, /* split_mode */ { LLAMA_SPLIT_MODE_LAYER },