mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-08-12 22:31:11 +04:00
Consistify polling frequency of llama-bench with args
llama-bench differs from llama-server in 2 ways: 1. It attaches a persistent threadpool across cgraph invocations 2. It has a polling scheme of 50 (as opposed to 0 of args) Together, this causes a significant perf regression in the CUDA backend on Windows when the CPU backend is built with OpenMP disabled. Why is the CPU backend affecting the CUDA backend? Typically, we have a LUT on the CPU backend where we map Token_ID -> Embedding to save on VRAM budget.
This commit is contained in:
@@ -380,7 +380,7 @@ static const cmd_params cmd_params_defaults = {
|
||||
/* n_threads */ { common_cpu_get_num_math() },
|
||||
/* cpu_mask */ { "0x0" },
|
||||
/* cpu_strict */ { false },
|
||||
/* poll */ { 50 },
|
||||
/* poll */ { 0 },
|
||||
/* n_gpu_layers */ { -1 },
|
||||
/* n_cpu_moe */ { 0 },
|
||||
/* split_mode */ { LLAMA_SPLIT_MODE_LAYER },
|
||||
|
||||
Reference in New Issue
Block a user