mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-08-12 22:29:39 +04:00

Files

T

History

NexesenexandGitHub c3b075f069 Chores : tidy up more typos project wide (ggml directory excluded), new -ptcall alias (#2237 )

* common: fix coding mistakes (typos in identifiers, flags and log strings)

Fix misspelled identifiers and user-facing strings across common, server
and model loading:

- allow_ruless -> allow_rules (misspelled identifier used in the allowlist
  CLI parsing and the server slot/context code)
- get_formated_timings/get_formated_generation -> get_formatted_*
- 'termionated' -> 'terminated' in the fit-margin assert message
- 'defaulr' -> 'default' in the YAML dump
- 'overriden' -> 'overridden' in tensor buffer type override logs
- 'becausee' -> 'because' in the output-tensor split log
- 'etected NaNs' -> 'detected NaNs' in the imatrix error message

* common: fix comment typos across src, common, include and examples

Fix misspelled words in code comments:

- llama.h: 'typy' -> 'type', 'transfrom' -> 'transform', 'ecoder' ->
  'encoder', 'indicies' -> 'indices', 'Intializes' -> 'Initializes'
- common.h: 'embendings' -> 'embeddings', 'pr' -> 'or' in the
  fused-indexer-topk comment
- chat.cpp: 'overridde' -> 'override'
- ngram-map: 'occurences' -> 'occurrences', 'stastistics' -> 'statistics'
- speculative.cpp: 'dont'/'inehit' -> 'don't'/'inherit'
- llama-mmap.cpp: 'dont't' -> 'don't'
- llama-model.h: 'hcurrently andle' -> 'currently handle'
- build_gemma3/4.cpp: 'emdeddings' -> 'embeddings'
- examples: 'quantizuation', 'logprobe', 'throught', 'retrun', 'swich',
  'convinient', 'temporally' (-> 'temporary'), 'temproal', 'preceed'

* common: remove duplicate definitions and duplicate help entries

- clip-impl.h: drop the second, identical #define TN_FFN_GATE
- common.cpp: remove the duplicate '-t, --threads N' help entry that was
  misplaced in the export-lora section (already listed in the general
  section)
- common.cpp: merge the two 'embedding' help groups into a single group
  so the embedding options are listed together
- llama.cpp: remove the redundant LLAMA_MAX_LAYERS define (llama-hparams.h
  already defines the same value and is included by llama.cpp)

* common: fix remaining typos (accomodate, recommanded, occurences, occassionally)

- accomodate -> accommodate in src/llama.cpp comment
- recommanded -> recommended in quantize.cpp user-facing output
- occurences -> occurrences in test-chat.cpp JSON string
- occassionally -> occasionally in vendor/stb/stb_image_resize2.h comment

Note: tokenizer.ggml.seperator_token_id kept as-is to match GGUF spec

* common: remove duplicate help entries

- remove the duplicate '--reasoning-budget N' help entry that was repeated
  in the main section (introduced in e0596bf614 'Autoparser - complete
  refactoring of parser architecture (PR 1376)')
- remove the second '--parallel-tool-calls' help entry that advertised the
  '-ptc' short flag, which belongs to '--print-token-count' (introduced in
  e0596bf614 'Autoparser - complete refactoring of parser architecture
  (PR 1376)'); the '-ptc' alias was non-functional for '--parallel-tool-calls'
  because the parser only binds it to '--print-token-count'

The canonical help entries are kept:
- '--reasoning-budget N' is listed once
- '--parallel-tool-calls' is listed once (without the conflicting '-ptc' alias)

* common: remove duplicate LOG_ENABLE define

- the '#undef LOG_ENABLE / #define LOG_ENABLE() // dummy stub' pair was
  repeated verbatim inside the LOG_DISABLE_LOGS section
- remove the second occurrence (introduced in a2588b53e1 'main : log
  file (PR 2748)')

* llama-bench: align MLA and attention-max-batch flags with common tools

llama-bench used '--mla-attn' and '--attn-max-batch' while the common
CLI parsing (common/common.cpp) uses '--mla-use' and
'--attention-max-batch' for the same features. This made the flags
inconsistent across tools.

- update the help text to advertise the canonical names
  '--mla-use' and '--attention-max-batch'
- keep the old '--mla-attn' and '--attn-max-batch' spellings working
  as aliases so existing scripts are not broken

The divergent names were introduced in 3e536b95b0 'Add optional MLA
(PR 188)'.

* fix typos in comments and user-facing strings

- ngram-map.cpp: 'Do we haven a existing' -> 'Do we have an existing'
  (introduced in 1cb7e1bf39 'spec : add self speculative decoding,
  ngram and refactor (PR 1261)')
- build_mamba.cpp: 'weigth' -> 'weight' (introduced in 8befd92ea5
  'Refactor model compute graphs (PR 1651)')
- gguf-split.cpp: 'one of splits have 0 tensors' -> 'one of the splits
  has 0 tensors' (introduced in 75b580db0a 'split: allow
  --split-max-size option (PR 6343)')
- gguf-split.cpp: 'merged from %d split' -> 'merged from %d splits'
  (introduced in 1b5523dc79 'gguf-split: split and merge gguf per
  batch of tensors (PR 6135)')
- convert-llama2c-to-ggml.cpp: missing opening quote in the help line,
  '(default %s\\')' -> '(default '%s\\')' (introduced in bb9ebb4394
  'Adding support for llama2.c models (PR 2559)')

* harmonize British and American spelling to American English

The codebase uses American English (e.g. --embd-normalize, --color),
but a few strings/comments still used British spellings. Unify them:

- 'normalisation' -> 'normalization' in common.h, common.cpp help text
  and code comment, and llama-build-context.cpp comment
- 'colorise' -> 'colorize' in the --color help text (common.cpp)
- 'behaviour' -> 'behavior' in a chat.cpp warning and a llama.cpp comment
- also fix 'openai' -> 'OpenAI' capitalization in the embedding help
  text and common.h comment (embedding output format is OpenAI-style)

* common: fix help text formatting inconsistencies

- '-smf16'/'--split-mode-f16' and '-smf32'/'--split-mode-f32' help
  entries displayed hardcoded 'true'/'false' as the default value;
  show the actual state derived from params.reduce_type instead
- '-no-mmad' help entry had 'fused_mmad?' without a space before the
  ternary operator
- '--reasoning-tokens' help continuation lines used tab characters for
  indentation while the sibling '--reasoning-format' entry uses spaces;
  convert to consistent space indentation

* common: revert smf16/smf32 help text default display change

Revert the '-smf16'/'--split-mode-f16' and '-smf32'/'--split-mode-f32'
help entries back to their original hardcoded 'true'/'false' default
display. The change to derive the default from params.reduce_type was
not desired; the split-mode options are legacy and the hardcoded
defaults reflect their intended meaning.

The other formatting fixes in the same area (fused_mmad ternary
spacing and the reasoning-tokens tab-to-space indentation) are kept.

* llama-bench: fix help text column alignment

The --mla-use and --attention-max-batch help lines introduced by the
flag alignment landed one column off from the sibling entries
((default: at column 51 instead of 50). Adjust the padding so all
help lines align.

* common: fix help text defaults for graph-reduce-type and log-format

Mismatch 1: -grt, --graph-reduce-type help shows default "f32", but actual default (common.h:463) is "f16" and llama.cpp uses GGML_TYPE_F16.

Mismatch 2: --log-format help shows default "json", but actual default (common.h:536 log_json=false) is text.

* common: add -ptcall short flag for --parallel-tool-calls

* typo

2026-08-03 08:01:18 +03:00

CMakeLists.txt

build : link against build info instead of compiling against it (#3879 )

2023-11-02 08:50:16 +02:00

llama-bench.cpp

Chores : tidy up more typos project wide (ggml directory excluded), new -ptcall alias (#2237 )

2026-08-03 08:01:18 +03:00

README.md

build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809 )

2024-06-13 00:41:52 +01:00

README.md

llama.cpp/examples/llama-bench

Performance testing tool for llama.cpp.

Syntax
Examples
Output formats
1. Markdown
2. CSV
3. JSON
4. SQL

Syntax

usage: ./llama-bench [options]

options:
  -h, --help
  -m, --model <filename>              (default: models/7B/ggml-model-q4_0.gguf)
  -p, --n-prompt <n>                  (default: 512)
  -n, --n-gen <n>                     (default: 128)
  -pg <pp,tg>                         (default: 512,128)
  -b, --batch-size <n>                (default: 2048)
  -ub, --ubatch-size <n>              (default: 512)
  -ctk, --cache-type-k <t>            (default: f16)
  -ctv, --cache-type-v <t>            (default: f16)
  -t, --threads <n>                   (default: 16)
  -ngl, --n-gpu-layers <n>            (default: 99)
  -sm, --split-mode <none|layer|row>  (default: layer)
  -mg, --main-gpu <i>                 (default: 0)
  -nkvo, --no-kv-offload <0|1>        (default: 0)
  -fa, --flash-attn <0|1>             (default: 0)
  -mmp, --mmap <0|1>                  (default: 1)
  --numa <distribute|isolate|numactl> (default: disabled)
  -embd, --embeddings <0|1>           (default: 0)
  -ts, --tensor-split <ts0/ts1/..>    (default: 0)
  -r, --repetitions <n>               (default: 5)
  -o, --output <csv|json|md|sql>      (default: md)
  -v, --verbose                       (default: 0)

Multiple values can be given for each parameter by separating them with ',' or by specifying the parameter multiple times.

llama-bench can perform three types of tests:

Prompt processing (pp): processing a prompt in batches (-p)
Text generation (tg): generating a sequence of tokens (-n)
Prompt processing + text generation (pg): processing a prompt followed by generating a sequence of tokens (-pg)

With the exception of -r, -o and -v, all options can be specified multiple times to run multiple tests. Each pp and tg test is run with all combinations of the specified options. To specify multiple values for an option, the values can be separated by commas (e.g. -n 16,32), or the option can be specified multiple times (e.g. -n 16 -n 32).

Each test is repeated the number of times given by -r, and the results are averaged. The results are given in average tokens per second (t/s) and standard deviation. Some output formats (e.g. json) also include the individual results of each repetition.

For a description of the other options, see the main example.

Note:

When using SYCL backend, there would be hang issue in some cases. Please set --mmp 0.

Examples

Text generation with different models

$ ./llama-bench -m models/7B/ggml-model-q4_0.gguf -m models/13B/ggml-model-q4_0.gguf -p 0 -n 128,256,512

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	132.19 ± 0.55
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 256	129.37 ± 0.54
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 512	123.83 ± 0.25
llama 13B mostly Q4_0	6.86 GiB	13.02 B	CUDA	99	tg 128	82.17 ± 0.31
llama 13B mostly Q4_0	6.86 GiB	13.02 B	CUDA	99	tg 256	80.74 ± 0.23
llama 13B mostly Q4_0	6.86 GiB	13.02 B	CUDA	99	tg 512	78.08 ± 0.07

Prompt processing with different batch sizes

$ ./llama-bench -n 0 -p 1024 -b 128,256,512,1024

model	size	params	backend	ngl	n_batch	test	t/s
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	128	pp 1024	1436.51 ± 3.66
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	256	pp 1024	1932.43 ± 23.48
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	512	pp 1024	2254.45 ± 15.59
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	1024	pp 1024	2498.61 ± 13.58

Different numbers of threads

$ ./llama-bench -n 0 -n 16 -p 64 -t 1,2,4,8,16,32

model	size	params	backend	threads	test	t/s
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	1	pp 64	6.17 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	1	tg 16	4.05 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	2	pp 64	12.31 ± 0.13
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	2	tg 16	7.80 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	4	pp 64	23.18 ± 0.06
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	4	tg 16	12.22 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	8	pp 64	32.29 ± 1.21
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	8	tg 16	16.71 ± 0.66
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	16	pp 64	33.52 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	16	tg 16	15.32 ± 0.05
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	32	pp 64	59.00 ± 1.11
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	32	tg 16	16.41 ± 0.79

Different numbers of layers offloaded to the GPU

$ ./llama-bench -ngl 10,20,30,31,32,33,34,35

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	10	pp 512	373.36 ± 2.25
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	10	tg 128	13.45 ± 0.93
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	20	pp 512	472.65 ± 1.25
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	20	tg 128	21.36 ± 1.94
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	30	pp 512	631.87 ± 11.25
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	30	tg 128	40.04 ± 1.82
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	31	pp 512	657.89 ± 5.08
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	31	tg 128	48.19 ± 0.81
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	32	pp 512	688.26 ± 3.29
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	32	tg 128	54.78 ± 0.65
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	33	pp 512	704.27 ± 2.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	33	tg 128	60.62 ± 1.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	34	pp 512	881.34 ± 5.40
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	34	tg 128	71.76 ± 0.23
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	35	pp 512	2400.01 ± 7.72
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	35	tg 128	131.66 ± 0.49

Output formats

By default, llama-bench outputs the results in markdown format. The results can be output in other formats by using the -o option.

Markdown

$ ./llama-bench -o md

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	2368.80 ± 93.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	131.42 ± 0.59

CSV

$ ./llama-bench -o csv

build_commit,build_number,cuda,metal,gpu_blas,blas,cpu_info,gpu_info,model_filename,model_type,model_size,model_n_params,n_batch,n_threads,f16_kv,n_gpu_layers,main_gpu,mul_mat_q,tensor_split,n_prompt,n_gen,test_time,avg_ns,stddev_ns,avg_ts,stddev_ts
"3469684","1275","1","0","0","1","1","13th Gen Intel(R) Core(TM) i9-13900K","NVIDIA GeForce RTX 3090 Ti","models/7B/ggml-model-q4_0.gguf","llama 7B mostly Q4_0","3825065984","6738415616","512","16","1","99","0","1","0.00","512","0","2023-09-23T12:09:01Z","212155977","732372","2413.341687","8.305961"
"3469684","1275","1","0","0","1","1","13th Gen Intel(R) Core(TM) i9-13900K","NVIDIA GeForce RTX 3090 Ti","models/7B/ggml-model-q4_0.gguf","llama 7B mostly Q4_0","3825065984","6738415616","512","16","1","99","0","1","0.00","0","128","2023-09-23T12:09:02Z","969320879","2728399","132.052051","0.371342"

JSON

$ ./llama-bench -o json

[
  {
    "build_commit": "3469684",
    "build_number": 1275,
    "cuda": true,
    "metal": false,
    "gpu_blas": true,
    "blas": true,
    "cpu_info": "13th Gen Intel(R) Core(TM) i9-13900K",
    "gpu_info": "NVIDIA GeForce RTX 3090 Ti",
    "model_filename": "models/7B/ggml-model-q4_0.gguf",
    "model_type": "llama 7B mostly Q4_0",
    "model_size": 3825065984,
    "model_n_params": 6738415616,
    "n_batch": 512,
    "n_threads": 16,
    "f16_kv": true,
    "n_gpu_layers": 99,
    "main_gpu": 0,
    "mul_mat_q": true,
    "tensor_split": "0.00",
    "n_prompt": 512,
    "n_gen": 0,
    "test_time": "2023-09-23T12:09:57Z",
    "avg_ns": 212365953,
    "stddev_ns": 985423,
    "avg_ts": 2410.974041,
    "stddev_ts": 11.163766,
    "samples_ns": [ 213837238, 211635853, 212328053, 211329715, 212698907 ],
    "samples_ts": [ 2394.34, 2419.25, 2411.36, 2422.75, 2407.16 ]
  },
  {
    "build_commit": "3469684",
    "build_number": 1275,
    "cuda": true,
    "metal": false,
    "gpu_blas": true,
    "blas": true,
    "cpu_info": "13th Gen Intel(R) Core(TM) i9-13900K",
    "gpu_info": "NVIDIA GeForce RTX 3090 Ti",
    "model_filename": "models/7B/ggml-model-q4_0.gguf",
    "model_type": "llama 7B mostly Q4_0",
    "model_size": 3825065984,
    "model_n_params": 6738415616,
    "n_batch": 512,
    "n_threads": 16,
    "f16_kv": true,
    "n_gpu_layers": 99,
    "main_gpu": 0,
    "mul_mat_q": true,
    "tensor_split": "0.00",
    "n_prompt": 0,
    "n_gen": 128,
    "test_time": "2023-09-23T12:09:59Z",
    "avg_ns": 977425219,
    "stddev_ns": 9268593,
    "avg_ts": 130.965708,
    "stddev_ts": 1.238924,
    "samples_ns": [ 984472709, 974901233, 989474741, 970729355, 967548060 ],
    "samples_ts": [ 130.019, 131.295, 129.362, 131.86, 132.293 ]
  }
]

SQL

SQL output is suitable for importing into a SQLite database. The output can be piped into the sqlite3 command line tool to add the results to a database.

$ ./llama-bench -o sql

CREATE TABLE IF NOT EXISTS test (
  build_commit TEXT,
  build_number INTEGER,
  cuda INTEGER,
  metal INTEGER,
  gpu_blas INTEGER,
  blas INTEGER,
  cpu_info TEXT,
  gpu_info TEXT,
  model_filename TEXT,
  model_type TEXT,
  model_size INTEGER,
  model_n_params INTEGER,
  n_batch INTEGER,
  n_threads INTEGER,
  f16_kv INTEGER,
  n_gpu_layers INTEGER,
  main_gpu INTEGER,
  mul_mat_q INTEGER,
  tensor_split TEXT,
  n_prompt INTEGER,
  n_gen INTEGER,
  test_time TEXT,
  avg_ns INTEGER,
  stddev_ns INTEGER,
  avg_ts REAL,
  stddev_ts REAL
);

INSERT INTO test (build_commit, build_number, cuda, metal, gpu_blas, blas, cpu_info, gpu_info, model_filename, model_type, model_size, model_n_params, n_batch, n_threads, f16_kv, n_gpu_layers, main_gpu, mul_mat_q, tensor_split, n_prompt, n_gen, test_time, avg_ns, stddev_ns, avg_ts, stddev_ts) VALUES ('3469684', '1275', '1', '0', '0', '1', '1', '13th Gen Intel(R) Core(TM) i9-13900K', 'NVIDIA GeForce RTX 3090 Ti', 'models/7B/ggml-model-q4_0.gguf', 'llama 7B mostly Q4_0', '3825065984', '6738415616', '512', '16', '1', '99', '0', '1', '0.00', '512', '0', '2023-09-23T12:10:30Z', '212693772', '743623', '2407.240204', '8.409634');
INSERT INTO test (build_commit, build_number, cuda, metal, gpu_blas, blas, cpu_info, gpu_info, model_filename, model_type, model_size, model_n_params, n_batch, n_threads, f16_kv, n_gpu_layers, main_gpu, mul_mat_q, tensor_split, n_prompt, n_gen, test_time, avg_ns, stddev_ns, avg_ts, stddev_ts) VALUES ('3469684', '1275', '1', '0', '0', '1', '1', '13th Gen Intel(R) Core(TM) i9-13900K', 'NVIDIA GeForce RTX 3090 Ti', 'models/7B/ggml-model-q4_0.gguf', 'llama 7B mostly Q4_0', '3825065984', '6738415616', '512', '16', '1', '99', '0', '1', '0.00', '0', '128', '2023-09-23T12:10:31Z', '977925003', '4037361', '130.891159', '0.537692');

README.md

llama.cpp/examples/llama-bench

Table of contents

Syntax

Examples

Text generation with different models

Prompt processing with different batch sizes

Different numbers of threads

Different numbers of layers offloaded to the GPU

Output formats

Markdown

CSV

JSON

SQL