LLM-Inference/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-08-12 14:24:03 +04:00
Code Issues Packages Projects Releases Wiki Activity

Chores : tidy up more typos project wide (ggml directory excluded), new -ptcall alias (#2237)

Browse Source 
* common: fix coding mistakes (typos in identifiers, flags and log strings)

Fix misspelled identifiers and user-facing strings across common, server
and model loading:

- allow_ruless -> allow_rules (misspelled identifier used in the allowlist
  CLI parsing and the server slot/context code)
- get_formated_timings/get_formated_generation -> get_formatted_*
- 'termionated' -> 'terminated' in the fit-margin assert message
- 'defaulr' -> 'default' in the YAML dump
- 'overriden' -> 'overridden' in tensor buffer type override logs
- 'becausee' -> 'because' in the output-tensor split log
- 'etected NaNs' -> 'detected NaNs' in the imatrix error message

* common: fix comment typos across src, common, include and examples

Fix misspelled words in code comments:

- llama.h: 'typy' -> 'type', 'transfrom' -> 'transform', 'ecoder' ->
  'encoder', 'indicies' -> 'indices', 'Intializes' -> 'Initializes'
- common.h: 'embendings' -> 'embeddings', 'pr' -> 'or' in the
  fused-indexer-topk comment
- chat.cpp: 'overridde' -> 'override'
- ngram-map: 'occurences' -> 'occurrences', 'stastistics' -> 'statistics'
- speculative.cpp: 'dont'/'inehit' -> 'don't'/'inherit'
- llama-mmap.cpp: 'dont't' -> 'don't'
- llama-model.h: 'hcurrently andle' -> 'currently handle'
- build_gemma3/4.cpp: 'emdeddings' -> 'embeddings'
- examples: 'quantizuation', 'logprobe', 'throught', 'retrun', 'swich',
  'convinient', 'temporally' (-> 'temporary'), 'temproal', 'preceed'

* common: remove duplicate definitions and duplicate help entries

- clip-impl.h: drop the second, identical #define TN_FFN_GATE
- common.cpp: remove the duplicate '-t, --threads N' help entry that was
  misplaced in the export-lora section (already listed in the general
  section)
- common.cpp: merge the two 'embedding' help groups into a single group
  so the embedding options are listed together
- llama.cpp: remove the redundant LLAMA_MAX_LAYERS define (llama-hparams.h
  already defines the same value and is included by llama.cpp)

* common: fix remaining typos (accomodate, recommanded, occurences, occassionally)

- accomodate -> accommodate in src/llama.cpp comment
- recommanded -> recommended in quantize.cpp user-facing output
- occurences -> occurrences in test-chat.cpp JSON string
- occassionally -> occasionally in vendor/stb/stb_image_resize2.h comment

Note: tokenizer.ggml.seperator_token_id kept as-is to match GGUF spec

* common: remove duplicate help entries

- remove the duplicate '--reasoning-budget N' help entry that was repeated
  in the main section (introduced in e0596bf614 'Autoparser - complete
  refactoring of parser architecture (PR 1376)')
- remove the second '--parallel-tool-calls' help entry that advertised the
  '-ptc' short flag, which belongs to '--print-token-count' (introduced in
  e0596bf614 'Autoparser - complete refactoring of parser architecture
  (PR 1376)'); the '-ptc' alias was non-functional for '--parallel-tool-calls'
  because the parser only binds it to '--print-token-count'

The canonical help entries are kept:
- '--reasoning-budget N' is listed once
- '--parallel-tool-calls' is listed once (without the conflicting '-ptc' alias)

* common: remove duplicate LOG_ENABLE define

- the '#undef LOG_ENABLE / #define LOG_ENABLE() // dummy stub' pair was
  repeated verbatim inside the LOG_DISABLE_LOGS section
- remove the second occurrence (introduced in a2588b53e1 'main : log
  file (PR 2748)')

* llama-bench: align MLA and attention-max-batch flags with common tools

llama-bench used '--mla-attn' and '--attn-max-batch' while the common
CLI parsing (common/common.cpp) uses '--mla-use' and
'--attention-max-batch' for the same features. This made the flags
inconsistent across tools.

- update the help text to advertise the canonical names
  '--mla-use' and '--attention-max-batch'
- keep the old '--mla-attn' and '--attn-max-batch' spellings working
  as aliases so existing scripts are not broken

The divergent names were introduced in 3e536b95b0 'Add optional MLA
(PR 188)'.

* fix typos in comments and user-facing strings

- ngram-map.cpp: 'Do we haven a existing' -> 'Do we have an existing'
  (introduced in 1cb7e1bf39 'spec : add self speculative decoding,
  ngram and refactor (PR 1261)')
- build_mamba.cpp: 'weigth' -> 'weight' (introduced in 8befd92ea5
  'Refactor model compute graphs (PR 1651)')
- gguf-split.cpp: 'one of splits have 0 tensors' -> 'one of the splits
  has 0 tensors' (introduced in 75b580db0a 'split: allow
  --split-max-size option (PR 6343)')
- gguf-split.cpp: 'merged from %d split' -> 'merged from %d splits'
  (introduced in 1b5523dc79 'gguf-split: split and merge gguf per
  batch of tensors (PR 6135)')
- convert-llama2c-to-ggml.cpp: missing opening quote in the help line,
  '(default %s\\')' -> '(default '%s\\')' (introduced in bb9ebb4394
  'Adding support for llama2.c models (PR 2559)')

* harmonize British and American spelling to American English

The codebase uses American English (e.g. --embd-normalize, --color),
but a few strings/comments still used British spellings. Unify them:

- 'normalisation' -> 'normalization' in common.h, common.cpp help text
  and code comment, and llama-build-context.cpp comment
- 'colorise' -> 'colorize' in the --color help text (common.cpp)
- 'behaviour' -> 'behavior' in a chat.cpp warning and a llama.cpp comment
- also fix 'openai' -> 'OpenAI' capitalization in the embedding help
  text and common.h comment (embedding output format is OpenAI-style)

* common: fix help text formatting inconsistencies

- '-smf16'/'--split-mode-f16' and '-smf32'/'--split-mode-f32' help
  entries displayed hardcoded 'true'/'false' as the default value;
  show the actual state derived from params.reduce_type instead
- '-no-mmad' help entry had 'fused_mmad?' without a space before the
  ternary operator
- '--reasoning-tokens' help continuation lines used tab characters for
  indentation while the sibling '--reasoning-format' entry uses spaces;
  convert to consistent space indentation

* common: revert smf16/smf32 help text default display change

Revert the '-smf16'/'--split-mode-f16' and '-smf32'/'--split-mode-f32'
help entries back to their original hardcoded 'true'/'false' default
display. The change to derive the default from params.reduce_type was
not desired; the split-mode options are legacy and the hardcoded
defaults reflect their intended meaning.

The other formatting fixes in the same area (fused_mmad ternary
spacing and the reasoning-tokens tab-to-space indentation) are kept.

* llama-bench: fix help text column alignment

The --mla-use and --attention-max-batch help lines introduced by the
flag alignment landed one column off from the sibling entries
((default: at column 51 instead of 50). Adjust the padding so all
help lines align.

* common: fix help text defaults for graph-reduce-type and log-format

Mismatch 1: -grt, --graph-reduce-type help shows default "f32", but actual default (common.h:463) is "f16" and llama.cpp uses GGML_TYPE_F16.

Mismatch 2: --log-format help shows default "json", but actual default (common.h:536 log_json=false) is text.

* common: add -ptcall short flag for --parallel-tool-calls

* typo
This commit is contained in:
Nexesenex
2026-08-03 08:01:18 +03:00
committed by GitHub
parent cb9147fd0d
commit c3b075f069
35 changed files with 92 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common/chat.cpp
+2 -2
View File
@@ -243,7 +243,7 @@ using chat_template_caps = jinja::caps;
struct common_chat_templates {
bool add_bos;
bool add_eos;
bool has_explicit_template; // Model had builtin template or template overridde was specified.
bool has_explicit_template; // Model had builtin template or template override was specified.
std::unique_ptr<common_chat_template> template_default; // always set (defaults to chatml)
std::unique_ptr<common_chat_template> template_tool_use;
};
@@ -2672,7 +2672,7 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
}
if (caps.supports_tool_calls && !caps.supports_tools) {
LOG_WRN(
"Template supports tool calls but does not natively describe tools. The fallback behaviour used may "
"Template supports tool calls but does not natively describe tools. The fallback behavior used may "
"produce bad results, inspect prompt w/ --verbose & consider overriding the template.\n");
}
}
common/common.cpp
+21 -26
View File
@@ -1809,7 +1809,7 @@ bool gpt_params_find_arg(int argc, char ** argv, const std::string & arg, gpt_pa
return true;
}
if (arg == "-ictk" || arg == "--indexer-cache-type-k") {
LLAMA_LOG_WARN("================== Quantized inexer cache has been disabled for now => argument '%s' ignored\n", arg.c_str());
LLAMA_LOG_WARN("================== Quantized indexer cache has been disabled for now => argument '%s' ignored\n", arg.c_str());
++i;
//params.indexer_cache_type_k = argv[++i];
return true;
@@ -2312,10 +2312,10 @@ bool gpt_params_find_arg(int argc, char ** argv, const std::string & arg, gpt_pa
}
if (arg == "--allowlist-unicode-rule") {
CHECK_ARG
if (params.allow_ruless.size() == 0) {
params.allow_ruless.push_back({});
if (params.allow_rules.size() == 0) {
params.allow_rules.push_back({});
}
params.allow_ruless.back().push_back(argparse_allowlist_unicode_rule(argv[i]));
params.allow_rules.back().push_back(argparse_allowlist_unicode_rule(argv[i]));
return true;
}
if (arg == "--allowlist-pieces") {
@@ -2326,7 +2326,7 @@ bool gpt_params_find_arg(int argc, char ** argv, const std::string & arg, gpt_pa
if (arg == "--allowlist-keyword") {
CHECK_ARG
params.allow_kws.push_back(argv[i]);
params.allow_ruless.push_back({});
params.allow_rules.push_back({});
return true;
}
if (arg == "--allowlist-keyword-delay") {
@@ -2733,7 +2733,7 @@ bool gpt_params_find_arg(int argc, char ** argv, const std::string & arg, gpt_pa
params.prefill_assistant = false;
return true;
}
if (arg == "--parallel-tool-calls") {
if (arg == "-ptcall" || arg == "--parallel-tool-calls") {
params.parallel_tool_calls = true;
return true;
}
@@ -3019,7 +3019,7 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
options.push_back({ "*", " --verbose-prompt", "print a verbose prompt before generation (default: %s)", params.verbose_prompt ? "true" : "false" });
options.push_back({ "*", "-dr, --dry-run", "skip loading tensors in the files"});
options.push_back({ "*", " --no-display-prompt", "don't print prompt at generation (default: %s)", !params.display_prompt ? "true" : "false" });
options.push_back({ "*", "-co, --color", "colorise output to distinguish prompt and user input from generations (default: %s)", params.use_color ? "true" : "false" });
options.push_back({ "*", "-co, --color", "colorize output to distinguish prompt and user input from generations (default: %s)", params.use_color ? "true" : "false" });
options.push_back({ "*", "-s, --seed SEED", "RNG seed (default: %d, use random seed for < 0)", params.seed });
options.push_back({ "*", "-t, --threads N", "number of threads to use during generation (default: %d)", params.n_threads });
options.push_back({ "*", "-tb, --threads-batch N", "number of threads to use during batch and prompt processing (default: same as --threads)" });
@@ -3059,7 +3059,7 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
options.push_back({ "*", "-no-fmoe, --no-fused-moe", "disable fused MoE (default: %s)", params.fused_moe_up_gate ? "enabled" : "disabled" });
options.push_back({ "*", "-ger, --grouped-expert-routing", "enable grouped expert routing (default: %s)", params.grouped_expert_routing ? "enabled" : "disabled" });
options.push_back({ "*", "-no-fug, --no-fused-up-gate", "disable fused up-gate (default: %s)", params.fused_up_gate ? "enabled" : "disabled" });
options.push_back({ "*", "-no-mmad, --no-fused-mul-multiadd", "disable fused mul-multi_add (default: %s)", params.fused_mmad? "enabled" : "disabled" });
options.push_back({ "*", "-no-mmad, --no-fused-mul-multiadd", "disable fused mul-multi_add (default: %s)", params.fused_mmad ? "enabled" : "disabled" });
//options.push_back({ "*", "-rcache, --rope-cache", "enable RoPE cache (default: %s)", params.rope_cache ? "enabled" : "disabled" });
options.push_back({ "*", "-gr, --graph-reuse", "enable graph reuse (default: %s)", params.graph_reuse ? "enabled" : "disabled" });
options.push_back({ "*", "-no-gr, --no-graph-reuse", "disable graph reuse (default: %s)", !params.graph_reuse ? "enabled" : "disabled" });
@@ -3070,7 +3070,7 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
options.push_back({ "*", "-vhad, --v-cache-hadamard", "Use Hadamard transform for V-cache (default: %d)", params.v_cache_hadamard});
options.push_back({ "*", "-smf16, --split-mode-f16", "Use f16 for data exchange between GPUs (default: %d)", true});
options.push_back({ "*", "-smf32, --split-mode-f32", "Use f32 for data exchange between GPUs (default: %d)", false});
options.push_back({ "*", "-grt, --graph-reduce-type", "Type for data exchange between GPUs (default: %s)", "f32"});
options.push_back({ "*", "-grt, --graph-reduce-type", "Type for data exchange between GPUs (default: %s)", "f16"});
options.push_back({ "*", "-gap, --graph-attn-precision", "Flash-attn precision under -sm graph (default: %s)", "f16"});
options.push_back({ "*", "-smgs, --split-mode-graph-scheduling", "Force Split Mode Graph Scheduling (default: %d)", params.split_mode_graph_scheduling});
options.push_back({ "*", "-sas, --scheduler-async", "Async evaluation of compute graphs (default: %d)", params.scheduler_async});
@@ -3166,7 +3166,7 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
"if suffix/prefix are specified, template will be disabled\n"
"only commonly used templates are accepted:\n"
"https://github.com/ggerganov/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template" });
options.push_back({ "main", " --parallel-tool-calls", "enable parallel tool calls\n" });
options.push_back({ "main", "-ptcall, --parallel-tool-calls", "enable parallel tool calls\n" });
options.push_back({ "main", " --chat-template JINJA_TEMPLATE",
"use jinja template for chat (default: disabled)\n" });
options.push_back({ "main", " --chat-template-file file_with_JINJA_TEMPLATE",
@@ -3183,17 +3183,15 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
options.push_back({ "main", " --reasoning-budget N", "token budget for thinking: -1 for unrestricted, 0 for immediate end, N>0 for token budget (default: -1)" });
options.push_back({ "main", " --reasoning-tokens FORMAT", "exclude reasoning tokens to select the slot more accurately.\n"
"none: include all tokens\n"
"auto: exclude all tokens between <think> and </think>\n"
"Or comma separated start and end tokens such as [THINK],[/THINK]\n"
"(default: auto)" });
"none: include all tokens\n"
"auto: exclude all tokens between <think> and </think>\n"
"Or comma separated start and end tokens such as [THINK],[/THINK]\n"
"(default: auto)" });
options.push_back({ "main", " --reasoning-budget-message", "message injected before the end-of-thinking tag when reasoning budget is exhausted (default: none)" });
options.push_back({ "main", " --skip-chat-parsing", "force a pure content parser, even if a Jinja template is specified; model will output everything "
"in the content section, including any reasoning and/or tool calls (default: disabled)" });
options.push_back({ "main", " --reasoning-budget N", "token budget for thinking: -1 for unrestricted, 0 for immediate end, N>0 for token budget (default: -1)" });
options.push_back({ "main", " --no-prefill-assistant", "whether to prefill the assistant's response if the last message is an assistant message (default: prefill enabled)\n"
"when this flag is set, if the last message is an assistant message then it will be treated as a full message and not prefilled\n" });
options.push_back({ "main", " -ptc, --parallel-tool-calls", "enable parallel tool calls\n" });
options.push_back({ "grammar" });
options.push_back({ "*", " --grammar GRAMMAR", "BNF-like grammar to constrain generations (see samples in grammars/ dir) (default: '%s')", sparams.grammar.grammar.c_str() });
options.push_back({ "*", " --grammar-file FNAME", "file to read grammar from" });
@@ -3206,6 +3204,9 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
"pooling type for embeddings, use model default if unspecified" });
options.push_back({ "embedding", " --attention {causal,non-causal}",
"attention type for embeddings, use model default if unspecified" });
options.push_back({ "embedding", " --embd-normalize", "normalization for embeddings (default: %d) (-1=none, 0=max absolute int16, 1=taxicab, 2=euclidean, >2=p-norm)", params.embd_normalize });
options.push_back({ "embedding", " --embd-output-format", "empty = default, \"array\" = [[],[]...], \"json\" = OpenAI style, \"json+\" = same \"json\" + cosine similarity matrix" });
options.push_back({ "embedding", " --embd-separator", "separator of embeddings (default \\n) for example \"<#sep#>\"" });
options.push_back({ "context hacking" });
options.push_back({ "*", " --rope-scaling {none,linear,yarn}",
@@ -3391,11 +3392,6 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
options.push_back({ "bench", "-wb, --warmup-batch", "run a warmup batch before measurement" });
options.push_back({ "bench", " --output-format FORMAT", "output format: table, jsonl, or csv (default: table)" });
options.push_back({ "embedding" });
options.push_back({ "embedding", " --embd-normalize", "normalisation for embeddings (default: %d) (-1=none, 0=max absolute int16, 1=taxicab, 2=euclidean, >2=p-norm)", params.embd_normalize });
options.push_back({ "embedding", " --embd-output-format", "empty = default, \"array\" = [[],[]...], \"json\" = openai style, \"json+\" = same \"json\" + cosine similarity matrix" });
options.push_back({ "embedding", " --embd-separator", "separator of embeddings (default \\n) for example \"<#sep#>\"" });
options.push_back({ "server" });
options.push_back({ "server", " --host HOST", "ip address to listen (default: %s)", params.hostname.c_str() });
options.push_back({ "server", " --port PORT", "port to listen (default: %d)", params.port });
@@ -3417,7 +3413,7 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
options.push_back({ "server", "-spf, --system-prompt-file FNAME",
"set a file to load a system prompt (initial prompt of all slots), this is useful for chat applications" });
options.push_back({ "server", " --log-format {text,json}",
"log output format: json or text (default: json)" });
"log output format: json or text (default: text)" });
options.push_back({ "server", " --metrics", "enable prometheus compatible metrics endpoint (default: %s)", params.endpoint_metrics ? "enabled" : "disabled" });
options.push_back({ "server", " --no-slots", "disables slots monitoring endpoint (default: %s)", params.endpoint_slots ? "enabled" : "disabled" });
options.push_back({ "server", " --slot-save-path PATH", "path to save slot kv cache (default: disabled)" });
@@ -3457,7 +3453,6 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
options.push_back({ "export-lora", "-m, --model", "model path from which to load base model (default '%s')", params.model.c_str() });
options.push_back({ "export-lora", " --lora FNAME", "path to LoRA adapter (can be repeated to use multiple adapters)" });
options.push_back({ "export-lora", " --lora-scaled FNAME S", "path to LoRA adapter with user defined scaling S (can be repeated to use multiple adapters)" });
options.push_back({ "*", "-t, --threads N", "number of threads to use during computation (default: %d)", params.n_threads });
options.push_back({ "export-lora", "-o, --output FNAME", "output file (default: '%s')", params.lora_outfile.c_str() });
printf("usage: %s [options]\n", argv[0]);
@@ -4268,7 +4263,7 @@ struct llama_model_params common_model_params_to_llama(const gpt_params & params
}
if (!params.fit_margin_array.empty()) {
GGML_ASSERT(params.fit_margin_array.size() % 2 == 0 && "Fit margin array does not have even number of elements");
GGML_ASSERT(params.fit_margin_array[params.fit_margin_array.size()-2] == -1 && "Fit margin array is not correctly termionated");
GGML_ASSERT(params.fit_margin_array[params.fit_margin_array.size()-2] == -1 && "Fit margin array is not correctly terminated");
mparams.fit_margin_array = params.fit_margin_array.data();
}
@@ -4973,7 +4968,7 @@ void common_embd_normalize(const float * inp, float * out, int n, int embd_norm)
double sum = 0.0;
switch (embd_norm) {
case -1: // no normalisation
case -1: // no normalization
sum = 1.0;
break;
case 0: // max absolute
@@ -5384,7 +5379,7 @@ void yaml_dump_non_result_info(FILE * stream, const gpt_params & params, const l
//fprintf(stream, "split_mode_f16: %s # default: true\n", params.split_mode_f16 ? "true" : "false");
fprintf(stream, "reduce_type: %s # default f16\n", params.reduce_type.c_str());
fprintf(stream, "scheduler_async: %s # default: false\n", params.scheduler_async ? "true" : "false");
fprintf(stream, "ser: %d,%g # defaulr: -1,0\n", params.min_experts, params.thresh_experts);
fprintf(stream, "ser: %d,%g # default: -1,0\n", params.min_experts, params.thresh_experts);
fprintf(stream, "temp: %f # default: 0.8\n", sparams.temp);
const std::vector<float> tensor_split_vector(params.tensor_split, params.tensor_split + llama_max_devices());
common/common.h
+5 -5
View File
@@ -364,7 +364,7 @@ struct gpt_params {
,uint32_t // upper codepoint
,std::string // unicode script name
,float // bias
>>> allow_ruless;
>>> allow_rules;
std::vector<std::string> allow_pieces; // each token to allowlist
std::vector<std::string> allow_kws; // keywords
size_t allow_kw_delay; // minimum n_decoded before first keyword is active
@@ -421,7 +421,7 @@ struct gpt_params {
bool rope_cache = false; // if to use RoPE cache (for supported models)
bool graph_reuse = true; // if to reuse compute graphs
bool dsa = false; // enable GLM DSA sparse attention (off by default; opt-in via --dsa)
bool fused_idx_topk = true; // enable the fused indexer topk op (off by default; opt-in via -fidx pr --fused-indexer-topk)
bool fused_idx_topk = true; // enable the fused indexer topk op (off by default; opt-in via -fidx or --fused-indexer-topk)
int dsa_top_k = -1; // DSA top-k override (<0 => use the model's configured indexer_top_k)
int min_experts = -1;
float thresh_experts = 0;
@@ -486,9 +486,9 @@ struct gpt_params {
// embedding
bool embedding = false; // get only sentence embedding
int32_t embd_normalize = 2; // normalisation for embendings (-1=none, 0=max absolute int16, 1=taxicab, 2=euclidean, >2=p-norm)
std::string embd_out = ""; // empty = default, "array" = [[],[]...], "json" = openai style, "json+" = same "json" + cosine similarity matrix
std::string embd_sep = "\n"; // separator of embendings
int32_t embd_normalize = 2; // normalization for embeddings (-1=none, 0=max absolute int16, 1=taxicab, 2=euclidean, >2=p-norm)
std::string embd_out = ""; // empty = default, "array" = [[],[]...], "json" = OpenAI style, "json+" = same "json" + cosine similarity matrix
std::string embd_sep = "\n"; // separator of embeddings
// server params
int32_t port = 8080; // server listens on this network port
common/log.h
-3
View File
@@ -830,9 +830,6 @@ inline std::string LOG_BATCH_TOSTR_PRETTY(const C & ctx, const B & batch)
#undef LOG_ENABLE
#define LOG_ENABLE() // dummy stub
#undef LOG_ENABLE
#define LOG_ENABLE() // dummy stub
#undef LOG_SET_TARGET
#define LOG_SET_TARGET(...) // dummy stub
common/ngram-map.cpp
+2 -2
View File
@@ -416,7 +416,7 @@ void common_ngram_map_draft(common_ngram_map & map,
continue;
}
// Do we haven a existing value m-gram or a new one after the key at index i?
// Do we have an existing value m-gram or a new one after the key at index i?
size_t idx_begin_value_key = i + n;
int idx_value = -1;
for (int v = 0; v < COMMON_NGRAM_MAX_VALUES; ++v) {
@@ -461,7 +461,7 @@ void common_ngram_map_draft(common_ngram_map & map,
slot_max = v;
}
}
// What is sum of the other occurences?
// What is sum of the other occurrences?
uint32_t sum_occur = 0;
for (int v = 0; v < COMMON_NGRAM_MAX_VALUES; ++v) {
if (v == slot_max) {
common/ngram-map.h
+3 -3
View File
@@ -44,16 +44,16 @@ llama_tokens common_ngram_simple_draft(
// statistics of a m-gram after a known n-gram
struct common_ngram_map_value {
size_t value_idx = 0; // index of value m-gram in token-history (0 if unused)
uint16_t value_num = 0; // number of occurences of this value m-gram after the key n-gram (0 in an unused values-slot)
uint16_t value_num = 0; // number of occurrences of this value m-gram after the key n-gram (0 in an unused values-slot)
int16_t n_accepted = -1; // number of accepted tokens at last draft (-1 if unused)
};
// statistics of a n-gram
struct common_ngram_map_key {
size_t key_idx; // index of key n-gram in token-history
size_t stat_idx; // index of last token of stastistics computation (key_num, values)
size_t stat_idx; // index of last token of statistics computation (key_num, values)
uint16_t key_num; // number of occurences of this key n-gram in token-history
uint16_t key_num; // number of occurrences of this key n-gram in token-history
common_ngram_map_value values[COMMON_NGRAM_MAX_VALUES]; // some known values after the key
};
common/speculative.cpp
+1 -1
View File
@@ -1923,7 +1923,7 @@ bool common_speculative_load_draft_model(
free_command_line(argc, argv);
}
// We likely dont want to inehit offload policy for MTP
// We likely don't want to inherit offload policy for MTP
if (params.has_stage_type(COMMON_SPECULATIVE_TYPE_MTP)) {
params_dft.ncmoe = 0;
params_dft.tensor_buft_overrides.clear();
examples/benchmark/benchmark-matmult.cpp
+1 -1
View File
@@ -251,7 +251,7 @@ int main(int argc, char ** argv) {
#endif
// Check that the matrix multiplication result is in the right ballpark
// We cannot use the exact value from the F32 multiplication because the quantizuation will be slightly different
// We cannot use the exact value from the F32 multiplication because the quantization will be slightly different
float sum_of_Q4_result = tensor_sum_elements(gf31->nodes[0]);
float delta = std::abs(sum_of_Q4_result - sum_of_F32_reference);
float allowed_delta = (sum_of_F32_reference) / 1000 / 1000; // Let's accept an epsilon of 10^-6
examples/convert-llama2c-to-ggml/convert-llama2c-to-ggml.cpp
+1 -1
View File
@@ -803,7 +803,7 @@ static void print_usage(int /*argc*/, char ** argv, const struct train_params *
fprintf(stderr, " -h, --help show this help message and exit\n");
fprintf(stderr, " --copy-vocab-from-model FNAME path of gguf llama model or llama2.c vocabulary from which to copy vocab (default '%s')\n", params->fn_vocab_model);
fprintf(stderr, " --llama2c-model FNAME [REQUIRED] model path from which to load Karpathy's llama2.c model\n");
fprintf(stderr, " --llama2c-output-model FNAME model path to save the converted llama2.c model (default %s')\n", params->fn_llama2c_output_model);
fprintf(stderr, " --llama2c-output-model FNAME model path to save the converted llama2.c model (default '%s')\n", params->fn_llama2c_output_model);
fprintf(stderr, "\n");
}
examples/cvector-generator/cvector-generator.cpp
+1 -1
View File
@@ -106,7 +106,7 @@ struct callback_data {
auto diff_filtered = filter_nonzero_rows(v_pos[il]);
v_diff_filtered.push_back(diff_filtered);
}
return v_diff_filtered; // for convinient, we return the result std::vector
return v_diff_filtered; // for convenient, we return the result std::vector
}
// delete zero rows from a given 2D tensor
examples/cvector-generator/pca.hpp
+2 -2
View File
@@ -148,7 +148,7 @@ static struct ggml_cgraph * build_graph_piter(
/*.mem_buffer =*/ buf.data(),
/*.no_alloc =*/ true, // the tensors will be allocated later by ggml_allocr_alloc_graph()
};
// create a temporally context to build the graph
// create a temporary context to build the graph
struct ggml_context * ctx0 = ggml_init(params0);
struct ggml_cgraph * gf = ggml_new_graph(ctx0);
@@ -189,7 +189,7 @@ static struct ggml_cgraph * build_graph_piter(
ggml_build_forward_expand(gf, distance);
}
// delete the temporally context used to build the graph
// delete the temporary context used to build the graph
ggml_free(ctx0);
return gf;
}
examples/gguf-hash/gguf-hash.cpp
+1 -1
View File
@@ -678,7 +678,7 @@ int main(int argc, const char ** argv) {
params.manifest_is_usable = true;
}
// By default if no swich argument provided, assume xxh64
// By default if no switch argument provided, assume xxh64
if (!params.xxh64 && !params.sha1 && !params.uuid && !params.sha256) {
params.xxh64 = true;
}
examples/gguf-split/gguf-split.cpp
+2 -2
View File
@@ -233,7 +233,7 @@ struct split_strategy {
i_split++;
if (ctx_out != NULL) {
if (gguf_get_n_tensors(ctx_out) == 0 && !allow_no_tensors) {
fprintf(stderr, "error: one of splits have 0 tensors. Maybe size or tensors limit is too small\n");
fprintf(stderr, "error: one of the splits has 0 tensors. Maybe size or tensors limit is too small\n");
exit(EXIT_FAILURE);
}
ctx_outs.push_back(ctx_out);
@@ -585,7 +585,7 @@ static void gguf_merge(const split_params & split_params) {
gguf_free(ctx_out);
}
fprintf(stderr, "%s: %s merged from %d split with %d tensors.\n",
fprintf(stderr, "%s: %s merged from %d splits with %d tensors.\n",
__func__, split_params.output.c_str(), n_split, total_tensors);
}
examples/imatrix/imatrix.cpp
+1 -1
View File
@@ -498,7 +498,7 @@ bool IMatrixCollector::collect_imatrix(struct ggml_tensor * t, bool ask, void *
const float * x = (const float *)((const char *)data + i11*src1->nb[1] + i12*src1->nb[2]);
if (add_and_check_nans(src1->ne[0], x, e.values.data() + e_start, e.counts.data() + e_start)) {
fprintf(stderr, "etected NaNs in %s\n", wname.c_str());
fprintf(stderr, "detected NaNs in %s\n", wname.c_str());
exit(1);
}
//for (int j = 0; j < (int)src1->ne[0]; ++j) {
examples/llama-bench/llama-bench.cpp
+4 -4
View File
@@ -352,8 +352,8 @@ static void print_usage(int /* argc */, char ** argv) {
printf(" -mg, --main-gpu <i> (default: %s)\n", join(cmd_params_defaults.main_gpu, ",").c_str());
printf(" -nkvo, --no-kv-offload <0|1> (default: %s)\n", join(cmd_params_defaults.no_kv_offload, ",").c_str());
printf(" -fa, --flash-attn <0|1> (default: %s)\n", join(cmd_params_defaults.flash_attn, ",").c_str());
printf(" -mla, --mla-attn <0|1|2> (default: %s)\n", join(cmd_params_defaults.mla_attn, ",").c_str());
printf(" -amb, --attn-max-batch <i> (default: %s)\n", join(cmd_params_defaults.attn_max_batch, ",").c_str());
printf(" -mla, --mla-use <0|1|2> (default: %s)\n", join(cmd_params_defaults.mla_attn, ",").c_str());
printf(" -amb, --attention-max-batch <i> (default: %s)\n", join(cmd_params_defaults.attn_max_batch, ",").c_str());
printf(" -ser, --smart-expert-reduction <i,f>(default: %s)\n", join(cmd_params_defaults.attn_max_batch, ",").c_str());
printf(" -gr, --graph-reuse <0|1> (default: %s)\n", join(cmd_params_defaults.reuse, ",").c_str());
printf(" -mmp, --mmap <0|1> (default: %s)\n", join(cmd_params_defaults.use_mmap, ",").c_str());
@@ -702,14 +702,14 @@ static cmd_params parse_cmd_params(int argc, char ** argv) {
}
auto p = string_split<bool>(argv[i], split_delim);
params.flash_attn.insert(params.flash_attn.end(), p.begin(), p.end());
} else if (arg == "-mla" || arg == "--mla-attn") {
} else if (arg == "-mla" || arg == "--mla-use" || arg == "--mla-attn") {
if (++i >= argc) {
invalid_param = true;
break;
}
auto p = string_split<int>(argv[i], split_delim);
params.mla_attn.insert(params.mla_attn.end(), p.begin(), p.end());
} else if (arg == "-amb" || arg == "--attn-max-batch") {
} else if (arg == "-amb" || arg == "--attention-max-batch" || arg == "--attn-max-batch") {
if (++i >= argc) {
invalid_param = true;
break;
examples/mtmd/clip-impl.h
-1
View File
@@ -74,7 +74,6 @@
#define TN_ATTN_K_NORM "%s.blk.%d.attn_k_norm.%s"
#define TN_ATTN_Q_NORM "%s.blk.%d.attn_q_norm.%s"
#define TN_FFN_DOWN "%s.blk.%d.ffn_down.%s"
#define TN_FFN_GATE "%s.blk.%d.ffn_gate.%s"
#define TN_FFN_UP "%s.blk.%d.ffn_up.%s"
#define TN_FFN_GATE "%s.blk.%d.ffn_gate.%s"
#define TN_LN_1 "%s.blk.%d.ln1.%s" // layer norm
examples/mtmd/clip.cpp
+1 -1
View File
@@ -307,7 +307,7 @@ struct clip_model {
// embeddings
ggml_tensor * class_embedding = nullptr;
ggml_tensor * patch_embeddings_0 = nullptr;
ggml_tensor * patch_embeddings_1 = nullptr; // second Conv2D kernel when we decouple Conv3D along temproal dimension (Qwen2VL)
ggml_tensor * patch_embeddings_1 = nullptr; // second Conv2D kernel when we decouple Conv3D along temporal dimension (Qwen2VL)
ggml_tensor * patch_bias = nullptr;
ggml_tensor * position_embeddings = nullptr;
examples/mtmd/mtmd.cpp
+1 -1
View File
@@ -578,7 +578,7 @@ struct mtmd_tokenizer {
if (!ctx->tok_sli_img_start.empty()) {
add_text(ctx->tok_sli_img_start);
} else if (!ctx->sli_img_start_tmpl.empty()) {
// If using a template to preceed a slice image
// If using a template to precede a slice image
const size_t sz = std::snprintf(nullptr, 0, ctx->sli_img_start_tmpl.c_str(), y+1, x+1) + 1;
std::unique_ptr<char[]> buf(new char[sz]);
std::snprintf(buf.get(), sz, ctx->sli_img_start_tmpl.c_str(), y+1, x+1);
examples/perplexity/perplexity.cpp
+1 -1
View File
@@ -1103,7 +1103,7 @@ static void hellaswag_score(llama_context * ctx, const gpt_params & params) {
//printf("max logprob ending idx %lu, gold ending idx %lu\n", ending_logprob_max_idx, hs_cur.gold_ending_idx);
// If the gold ending got the maximum logprobe add one accuracy point
// If the gold ending got the maximum logprob add one accuracy point
if (ending_logprob_max_idx == hs_cur.gold_ending_idx) {
acc += 1.0;
}
examples/quantize-stats/quantize-stats.cpp
+1 -1
View File
@@ -1685,7 +1685,7 @@ int main(int argc, char ** argv) {
return 0;
}
// loop throught quantization types
// loop through quantization types
for (int i = 0; i < GGML_TYPE_COUNT; i++) {
const ggml_type type = (ggml_type) i;
if (!params.include_types.empty() && std::find(params.include_types.begin(), params.include_types.end(), i) == params.include_types.end()) {
examples/quantize/quantize.cpp
+1 -1
View File
@@ -187,7 +187,7 @@ static void usage(const char * executable) {
printf(" Advanced option to override model metadata by key in the quantized model. May be specified multiple times.\n\n");
printf("Note: --include-weights and --exclude-weights cannot be used together\n");
printf("Note: The token embeddings tensor is loaded in system RAM, even in case of full GPU/VRAM offload.\n");
printf("Note: The recommanded type for the output tensor is q6_K for the ffn types > iq3_xxs and < q8_0.\n\n");
printf("Note: The recommended type for the output tensor is q6_K for the ffn types > iq3_xxs and < q8_0.\n\n");
printf("Note for the Custom Quant Scheme FTYPE:\n");
printf(" Write the specific tensor legacy quants as qN_N, the K-Quants as qN_K, the IQ-Quants as iqN_xx.\n");
printf(" Usually, attn-q-type can be one type below the chosen ffn type, and attn-v-type should be one type above.\n");
examples/server/function_calls.hpp
+1 -1
View File
@@ -54,7 +54,7 @@ static std::string extract_content_from_mixed_input(const std::string& content,
}
}
// Is this the right thing to do? If we have an open thinking tag, we just retrun and do not try to
// Is this the right thing to do? If we have an open thinking tag, we just return and do not try to
// remove function calls.
if (is_thinking) {
return result;
examples/server/server-context.cpp
+14 -14
View File
@@ -367,7 +367,7 @@ void server_context::init() {
slots.push_back(std::move(slot));
}
default_generation_settings_for_props = get_formated_generation(slots.front());
default_generation_settings_for_props = get_formatted_generation(slots.front());
default_generation_settings_for_props["seed"] = -1;
// the update_slots() logic will always submit a maximum of n_batch or n_parallel tokens
@@ -511,7 +511,7 @@ void server_slot::reset() {
ban_regex.clear();
ban_regex_ci.clear();
allow_ruless.clear();
allow_rules.clear();
allow_pieces.clear();
allow_kws.clear();
allow_kw_delay = 0;
@@ -625,7 +625,7 @@ void server_slot::release() {
}
json server_slot::get_formated_timings() const {
json server_slot::get_formatted_timings() const {
json timings = json{
{"prompt_n", n_prompt_tokens_processed},
{"prompt_ms", t_prompt_processing},
@@ -1686,8 +1686,8 @@ bool server_context::launch_slot_with_task(server_slot& slot, server_task& task)
do // populate allowlist biases
{
// TODO: JSON parsing for rules and keywords
slot.allow_ruless = params_base.allow_ruless;
if (slot.allow_ruless.size() == 0) {
slot.allow_rules = params_base.allow_rules;
if (slot.allow_rules.size() == 0) {
slot.allow_biasess.clear();
break;
}
@@ -1716,11 +1716,11 @@ bool server_context::launch_slot_with_task(server_slot& slot, server_task& task)
}
}
auto n_rules = slot.allow_ruless.size();
auto n_rules = slot.allow_rules.size();
if (n_rules > slot.allow_kws.size() + 1) {
// one more rules than keyword, last rules do not expire
n_rules = slot.allow_kws.size() + 1;
slot.allow_ruless.resize(n_rules);
slot.allow_rules.resize(n_rules);
} else if (n_rules < slot.allow_kws.size()) {
// every rules expire
slot.allow_kws.resize(n_rules);
@@ -1728,8 +1728,8 @@ bool server_context::launch_slot_with_task(server_slot& slot, server_task& task)
slot.allow_biasess.resize(n_rules);
for (size_t i = 0; i < n_rules; ++i) {
const auto& rules = slot.allow_ruless[i];
if ((i < slot.allow_ruless_prev.size()) && (rules == slot.allow_ruless_prev[i])) {
const auto& rules = slot.allow_rules[i];
if ((i < slot.allow_rules_prev.size()) && (rules == slot.allow_rules_prev[i])) {
continue;
}
LLAMA_LOG_DEBUG("%s: allowlist %zu is new\n", __func__, i);
@@ -1780,7 +1780,7 @@ bool server_context::launch_slot_with_task(server_slot& slot, server_task& task)
}
}
} while (false);
slot.allow_ruless_prev = slot.allow_ruless;
slot.allow_rules_prev = slot.allow_rules;
if (llama_model_has_recurrent(llama_get_model(slot.ctx)) || llama_model_is_deepseek4(llama_get_model(slot.ctx))) {
params_base.can_ban_phrases = false;
@@ -2293,7 +2293,7 @@ void server_context::populate_token_probs(const server_slot& slot, completion_to
}
}
json server_context::get_formated_generation(const server_slot& slot) const {
json server_context::get_formatted_generation(const server_slot& slot) const {
const auto eos_bias = slot.sparams.logit_bias.find(llama_token_eos(model));
const bool ignore_eos = eos_bias != slot.sparams.logit_bias.end() && eos_bias->second < 0.0f && std::isinf(eos_bias->second);
@@ -2505,7 +2505,7 @@ void server_context::send_final_response(server_slot& slot) {
{"model", params_base.model_alias},
{"tokens_predicted", slot.n_decoded},
{"tokens_evaluated", slot.n_prompt_tokens},
{"generation_settings", get_formated_generation(slot)},
{"generation_settings", get_formatted_generation(slot)},
{"prompt", slot.prompt},
{"truncated", slot.truncated},
{"stopped_eos", slot.stopped_eos},
@@ -2513,7 +2513,7 @@ void server_context::send_final_response(server_slot& slot) {
{"stopped_limit", slot.stopped_limit},
{"stopping_word", slot.stopping_word},
{"tokens_cached", slot.n_past},
{"timings", slot.get_formated_timings()},
{"timings", slot.get_formatted_timings()},
//{"oaicompat_chat_format", slot.params.oaicompat_chat_format},
};
@@ -2851,7 +2851,7 @@ void server_context::process_single_task(server_task&& task) {
int n_processing_slots = 0;
for (server_slot& slot : slots) {
json slot_data = get_formated_generation(slot);
json slot_data = get_formatted_generation(slot);
slot_data["id"] = slot.id;
slot_data["id_task"] = slot.id_task;
slot_data["state"] = slot.state;
examples/server/server-context.h
+4 -4
View File
@@ -105,8 +105,8 @@ struct server_slot {
std::map<int32_t, std::set<llama_token>> positional_bans;
// allowlist
std::vector<std::vector<std::tuple<uint32_t, uint32_t, std::string, float>>> allow_ruless_prev;
std::vector<std::vector<std::tuple<uint32_t, uint32_t, std::string, float>>> allow_ruless;
std::vector<std::vector<std::tuple<uint32_t, uint32_t, std::string, float>>> allow_rules_prev;
std::vector<std::vector<std::tuple<uint32_t, uint32_t, std::string, float>>> allow_rules;
std::vector<std::string> allow_pieces;
std::vector<std::string> allow_kws;
size_t allow_kw_delay = 0;
@@ -199,7 +199,7 @@ struct server_slot {
void release();
json get_formated_timings() const;
json get_formatted_timings() const;
result_timings get_timings() const;
@@ -315,7 +315,7 @@ struct server_context {
void populate_token_probs(const server_slot& slot, completion_token_output& result, bool post_sampling, bool special, int idx);
json get_formated_generation(const server_slot& slot) const;
json get_formatted_generation(const server_slot& slot) const;
void send_error(const server_task& task, const std::string& error, const enum error_type type = ERROR_TYPE_SERVER);
include/llama.h
+8 -8
View File
@@ -500,8 +500,8 @@ extern "C" {
bool only_active_experts;
bool prefetch_experts; // if true, stream mmap'd MoE expert weights into the page cache (Linux only)
int prefetch_experts_threads; // number of expert prefetch workers (<=0 = auto)
bool k_cache_hadamard; // if true, apply Hadamard transfrom to K-cache
bool v_cache_hadamard; // if true, apply Hadamard transfrom to V-cache (needs FA)
bool k_cache_hadamard; // if true, apply Hadamard transform to K-cache
bool v_cache_hadamard; // if true, apply Hadamard transform to V-cache (needs FA)
bool split_mode_graph_scheduling; // if true, force split mode graph scheduling
//bool split_mode_f16; // if true, cast intermediate results to f16 before copying to other GPUs
bool scheduler_async; // if true, with split mode "graph" graph evaluation will be done using multiple threads
@@ -533,8 +533,8 @@ extern "C" {
enum ggml_type ffn_gate_type; // feedforward network gate type
enum ggml_type ffn_down_type; // feedforward network down type
enum ggml_type ffn_up_type; // feedforward network up type
enum ggml_type ffn_gate_inp_type; // routed experts probabilities typy (relevant for MoE models only)
enum ggml_type extra_output_type; // routed experts probabilities typy (relevant for MoE models only)
enum ggml_type ffn_gate_inp_type; // routed experts probabilities type (relevant for MoE models only)
enum ggml_type extra_output_type; // routed experts probabilities type (relevant for MoE models only)
bool allow_requantize; // allow quantizing non-f32/f16 tensors
bool quantize_output_tensor; // quantize output.weight
bool only_copy; // only copy tensors - ftype, allow_requantize and quantize_output_tensor are ignored
@@ -1076,7 +1076,7 @@ extern "C" {
// Frees a batch of tokens allocated with llama_batch_init()
LLAMA_API void llama_batch_free(struct llama_batch batch);
// Processes a batch of tokens with the ecoder part of the encoder-decoder model.
// Processes a batch of tokens with the encoder part of the encoder-decoder model.
// Stores the encoder output internally for later use by the decoder cross-attention layers.
// 0 - success
// < 0 - error
@@ -1128,7 +1128,7 @@ extern "C" {
// Logits for the ith token. For positive indices, Equivalent to:
// llama_get_logits(ctx) + ctx->output_ids[i]*n_vocab
// Negative indicies can be used to access logits in reverse order, -1 is the last logit.
// Negative indices can be used to access logits in reverse order, -1 is the last logit.
// returns NULL for invalid ids.
LLAMA_API float * llama_get_logits_ith(struct llama_context * ctx, int32_t i);
@@ -1146,7 +1146,7 @@ extern "C" {
// Get the embeddings for the ith token. For positive indices, Equivalent to:
// llama_get_embeddings(ctx) + ctx->output_ids[i]*n_embd
// Negative indicies can be used to access embeddings in reverse order, -1 is the last embedding.
// Negative indices can be used to access embeddings in reverse order, -1 is the last embedding.
// shape: [n_embd] (1-dimensional)
// returns NULL for invalid ids.
LLAMA_API float * llama_get_embeddings_ith(struct llama_context * ctx, int32_t i);
@@ -1449,7 +1449,7 @@ extern "C" {
LLAMA_API void llama_sampler_reset(struct llama_sampler* smpl);
/// @details Intializes a GBNF grammar, see grammars/README.md for details.
/// @details Initializes a GBNF grammar, see grammars/README.md for details.
/// @param vocab The vocabulary that this grammar will be used with.
/// @param grammar_str The production rules for the grammar, encoded as a string. Returns an empty grammar if empty. Returns NULL if parsing of grammar_str fails.
/// @param grammar_root The name of the start symbol for the grammar.
src/graphs/build_gemma3.cpp
+1 -1
View File
@@ -10,7 +10,7 @@ ggml_cgraph * llm_build_context::build_gemma3() {
inpL = llm_build_inp_embd(ctx0, lctx, hparams, batch, model.tok_embd, cb);
// important: do not normalize weights for raw embeddings input (i.e. encoded image emdeddings)
// important: do not normalize weights for raw embeddings input (i.e. encoded image embeddings)
if (batch.token) {
inpL = ggml_scale(ctx0, inpL, sqrtf(n_embd));
cb(inpL, "inp_scaled", -1);
src/graphs/build_gemma4.cpp
+1 -1
View File
@@ -903,7 +903,7 @@ ggml_cgraph * llm_build_context::build_gemma4() {
inpL = llm_build_inp_embd(ctx0, lctx, hparams, batch, model.tok_embd, cb);
cb(inpL, "tok_embd", -1);
// important: do not normalize weights for raw embeddings input (i.e. encoded image emdeddings)
// important: do not normalize weights for raw embeddings input (i.e. encoded image embeddings)
if (batch.token) {
inpL = ggml_scale(ctx0, inpL, sqrtf(n_embd));
cb(inpL, "inp_scaled", -1);
src/graphs/build_mamba.cpp
+1 -1
View File
@@ -55,7 +55,7 @@ ggml_cgraph * llm_build_context::build_mamba() {
// Custom operator which is needed only to ease simultaneous sequence processing.
// For a single sequence, the equivalent is to concatenate the columns of conv_states and x,
// then make a self-overlapping view of that over d_conv columns at each stride in the 3rd dimension,
// then element-wise multiply that with the conv1d weigth,
// then element-wise multiply that with the conv1d weight,
// then sum the elements of each row,
// (the last two steps are a dot product over rows (also doable with mul_mat))
// then permute away the ne[0] dimension,
src/llama-build-context.cpp
+1 -1
View File
@@ -2337,7 +2337,7 @@ std::tuple<ggml_tensor*, ggml_tensor*, ggml_tensor*> llm_build_context::llm_buil
auto [Q, K, V] = llm_build_mul_mat_qkv(gf, cur, wq, bq, wk, bk, wv, bv, attention_scale, il, add_graph_split);
auto Qcur = ggml_reshape_3d(ctx0, Q, n_embd_head_k, Q->ne[0]/n_embd_head_k, n_tokens);
// Command-R/R+ uses LayerNorm (not RMSNorm) for per-head Q/K normalisation
// Command-R/R+ uses LayerNorm (not RMSNorm) for per-head Q/K normalization
const auto qk_norm_type = (model.arch == LLM_ARCH_COMMAND_R) ? LLM_NORM : LLM_NORM_RMS;
if (q_norm) {
Qcur = llm_build_norm(ctx0, Qcur, hparams, q_norm, NULL, qk_norm_type, cb, il);
src/llama-load-tensors.cpp
+3 -3
View File
@@ -450,7 +450,7 @@ ggml_context * create_tensors_helper::get_context_for_tensor(ggml_context * ctx,
if (o.second == default_cpu_buft) has_buft_overrides = true;
const struct ggml_tensor * cur = ml.get_tensor_meta(name.c_str());
const size_t nbytes = cur ? ggml_nbytes(cur) : 0;
LLAMA_LOG_INFO("Tensor %s (size = %.2f MiB) buffer type overriden to %s\n", name.c_str(), nbytes/1024./1024., ggml_backend_buft_name(o.second));
LLAMA_LOG_INFO("Tensor %s (size = %.2f MiB) buffer type overridden to %s\n", name.c_str(), nbytes/1024./1024., ggml_backend_buft_name(o.second));
ctx = ctx_for_buft(o.second);
break;
}
@@ -461,7 +461,7 @@ ggml_context * create_tensors_helper::get_context_for_tensor(ggml_context * ctx,
// if (std::regex_search(name, pattern)) {
// const struct ggml_tensor * cur = ml.get_tensor_meta(name.c_str());
// const size_t nbytes = cur ? ggml_nbytes(cur) : 0;
// LLAMA_LOG_INFO("Tensor %s (size = %.2f MiB) buffer type overriden to %s\n", name.c_str(), nbytes/1024./1024., ggml_backend_buft_name(overrides->buft));
// LLAMA_LOG_INFO("Tensor %s (size = %.2f MiB) buffer type overridden to %s\n", name.c_str(), nbytes/1024./1024., ggml_backend_buft_name(overrides->buft));
// ctx = ctx_for_buft(overrides->buft);
// break;
// }
@@ -5392,7 +5392,7 @@ bool create_tensors_helper::create_tensors() {
if (model.output) {
if (auto it = split_tensors.find(model.output); it != split_tensors.end()) {
if (ggml_backend_buft_is_host(model.buft_output.buft_matrix)) {
LLAMA_LOG_INFO("%s: not splitting output tensor becausee buffer is host\n", __func__);
LLAMA_LOG_INFO("%s: not splitting output tensor because buffer is host\n", __func__);
} else {
auto ctx_split = ctx_map[model.buft_output.buft_matrix];
auto split = create_split(model.output->ne[1], 16, model.splits, mem_used);
src/llama-mmap.cpp
+1 -1
View File
@@ -574,7 +574,7 @@ struct llama_mlock::impl {
char* errmsg = std::strerror(errno);
bool suggest = (errno == ENOMEM);
#if defined(TARGET_OS_VISION) || defined(TARGET_OS_TV) || defined(_AIX)
// visionOS/tvOS dont't support RLIMIT_MEMLOCK
// visionOS/tvOS don't support RLIMIT_MEMLOCK
// Skip resource limit checks on visionOS/tvOS
suggest = false;
#else
src/llama-model.h
+1 -1
View File
@@ -577,7 +577,7 @@ struct llama_model {
static inline int hadamard_size(int head_size) {
if ((head_size & ~(head_size - 1)) == head_size) return head_size;
// Note: we do not include 32 as an option because the CUDA Hadamard implementation
// does not hcurrently andle a block size of 32.
// does not currently handle a block size of 32.
for (int i = 512; i >= 64; i >>= 1) {
if (head_size % i == 0) return i;
}
src/llama.cpp
+2 -5
View File
@@ -125,9 +125,6 @@ void llama_set_mtp_n_heads(struct llama_context * ctx, int32_t mtp_n_heads);
#pragma warning(disable: 4244 4267) // possible loss of data
#endif
// bump if necessary
#define LLAMA_MAX_LAYERS 512
//
// helpers
//
@@ -807,7 +804,7 @@ int llama_context::max_nodes(int n_tokens, int n_kv) const {
model.layers[0].wkv_b) {
// In this case we perform the attention computation iteratively, and this adds
// 10 nodes per layer per iteration. Although in many cases the 65536 nodes we
// estimate by default are enough to accomodate, to be safe we add the additional
// estimate by default are enough to accommodate, to be safe we add the additional
// number of nodes required for the iterative MLA evaluation.
int n_head = model.hparams.n_head();
auto wkv_b = model.layers[0].wkv_b;
@@ -4836,7 +4833,7 @@ static void llama_set_inputs(llama_context & lctx, const llama_batch & batch) {
// NOTHING for that sequence and let the (now-)sink token be masked out of top-k, collapsing
// it. Anchoring on per-sequence min(pos) keeps the sink protection following the sequence's
// actual first present cell. For a fresh sequence starting at pos 0, min(pos)==0 so the
// boosted set is identical to the old behaviour (n_seq==1 byte-identical).
// boosted set is identical to the old behavior (n_seq==1 byte-identical).
GGML_ASSERT(ggml_backend_buffer_is_host(lctx.inp_dsa_sink->buffer));
static const int n_sink = []{ const char * e = getenv("DSA_SINK"); return e ? atoi(e) : 1; }();
const int64_t n_kv = lctx.inp_dsa_sink->ne[0];
tests/test-chat.cpp
+1 -1
View File
@@ -4277,7 +4277,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
},
"replaceAll": {
"type": "boolean",
"description": "Whether to replace all occurences."
"description": "Whether to replace all occurrences."
}
},
"required": ["oldString", "newString"]
vendor/stb/stb_image_resize2.h
Vendored
+1 -1
View File
@@ -6772,7 +6772,7 @@ static void stbir__get_split_info( stbir__per_split_info* split_info, int splits
// simply bump up our previous thread split range to include it, and then start this threads
// range with the smaller sample. It just moves one scanline from one thread split to another,
// so that we end with the unusual one, instead of start with it. To do this, we check 2-4
// sample at each thread split start and then occassionally move them.
// sample at each thread split start and then occasionally move them.
if ( ( is_gather ) && ( i ) )
{