mirror of
https://github.com/ikawrakow/ik_llama.cpp.git
synced 2026-08-12 22:29:39 +04:00
NexesenexandGitHub
52836917a1
* DSV4 checkpoints WIP
* DSV4 checkpoints: fix per-sequence state save/restore writing all streams
Critical bug: llama_state_seq_get_data() and llama_state_seq_set_data()
were serializing/deserializing ALL DSV4 compressed cache streams instead
of only the stream for the target sequence. This corrupted other active
sequences' compressed indexer state during checkpoint restore.
Fix:
- Add dsv4_stream_offset_size() helper to compute per-stream byte
offset and size for any DSV4 cache tensor (CSA K, LID K, HCA K,
and all state tensors)
- write_kv_cache_data: emit dsv4_single_stream flag + stream_idx
so per-sequence saves only write that stream's portion
- write_dsv4_cache: accept stream_idx parameter (-1 = full tensor,
>= 0 = single stream at computed offset)
- read_kv_cache_data: read the new format, validate consistency
(per-stream data needs dest seq_id, full data needs seq_id=-1),
and restore only the destination stream's bytes into the correct
tensor offset
Format change (WIP, backward compat not required):
[has_dsv4_cache] [n_layer] [single_stream] [stream_idx] [n_stream]
[per-layer: layer_type + stream tensor data]
* DSV4 checkpoints: fix checkpoint search using pos_max instead of pos_min
Bug: checkpoint search condition cur.pos_min < n_past || cur.pos_min == 0
always matched all DSV4 checkpoints because they all have pos_min=0 (the KV
cache starts at 0 and never evicts). The reverse-iterator always picked the
NEWEST checkpoint regardless of how far past the intended rewind/divergence
point it extended, causing n_past to be set to the checkpoint's pos_max
(e.g. 9500) instead of the rewind point (e.g. 5000). This made the system
skip reprocessing tokens between the rewind point and the checkpoint's
pos_max.
Fix in both batch_pending_prompt and apply_checkpoint:
- Condition changed to cur.pos_max < n_past — only checkpoints that end
BEFORE the divergence/rewind point are eligible
- Post-restore n_past uses it->pos_max directly instead of the incorrect
max(pos_min + 1, pos_max) which always returned pos_max for DSV4 anyway
Now when rewinding to position 5000 with checkpoints at pos_max=8000+:
no checkpoint matches, falls through to full reprocessing (correct).
When rewinding to position 8000 with a checkpoint at pos_max=7500: only
500 tokens need reprocessing (optimal).
* DSV4 checkpoints: throttle creation via interval gating on all paths
When ctx_checkpoints_interval > 0, DSV4 checkpoints (~145 MiB) were still
created at every transition point (PP done, TG start, release) because
direct create_checkpoint() calls bypassed the interval gate.
Fix:
- Modified create_checkpoint_at_interval() to handle interval <= 0 as
'always create' (preserving recurrent model behavior for small state)
- Replaced all 4 external direct create_checkpoint() calls with
create_checkpoint_at_interval() so the interval gate is respected
- Unified the prompt-loading branch that had split
create_checkpoint / at_interval calls
Now with --ctx-checkpoints-interval N, all checkpoint creation is
throttled to at most 1 per N positions regardless of the transition
phase.
* DSV4 checkpoints: clarify divergence log message for models with state checkpoints
The message 'does not support partial KV reuse' was misleading for DSV4,
which now supports checkpoint-based state restoration. Split the fallback
message: models with state checkpoints (DSV4, recurrent, hybrid) now print
'no checkpoint before divergence point' instead of 'does not support
partial KV reuse', explaining that the restore failed due to missing
checkpoints at the right position, not due to lack of support.
* DSV4 checkpoints: don't erase the just-restored checkpoint
The erasure condition pos_max > pos_min_thold was equivalent to
pos_max >= pos_next, erasing any checkpoint whose data touched or
went past the current write position. The checkpoint just restored
from (pos_max == pos_next) was immediately erased, wasting a ~145 MiB
checkpoint that was perfectly valid.
Fix:
- Changed erasure condition to pos_max > pos_next (strictly greater
than the next write position). Checkpoints at exactly the current
position (pos_max == pos_next, e.g. the one we just restored from)
are kept.
- Preserved cache-aligned pos_next through the restore block so the
erasure compares against cache positions, not prompt positions
(pre-existing bug where the prompt-tokens call at line 3677
overwrote pos_next with a prompt position).
* DSV4 checkpoints: update interval gate position after restore
After a checkpoint restore, slot.checkpoint_pos was still 0 (from
slot.release()), so the interval gate in create_checkpoint_at_interval
always passed (0 + 2048 <= pos + 1), creating a new ~145 MiB checkpoint
immediately after every restore — even just 5 tokens past the restored
checkpoint's position.
Fix: set slot.checkpoint_pos = it->pos_max in both restore paths
(apply_checkpoint generic restore and batch_pending_prompt DSV4
restore). This tells the gate that a checkpoint already exists at the
restored position, and no new one is needed until another interval
(2048 tokens) has elapsed.
* Missing info
* DSV4 checkpoints: document float-reduction-order reproducibility after restore
After a checkpoint restore, the PP batch loop processes remaining tokens
sequentially from n_past_prompt in chunks of n_batch. Because the loop
is stateless with no carry-over from earlier batches, the chunk boundaries
at and after the restore point are identical to a full-reprocess control
arm. This ensures float-reduction-order reproducibility between arms
when performing correctness validation.
Addresses joelfarthing's finding on openPangu, where mismatched chunk
boundaries between restore+reprocess and full-reprocess controls caused
bit-level differences that masked actual restore bugs.
* DSV4 checkpoints: verify position after restore
After both restore paths (apply_checkpoint and DSV4 batch_pending_prompt),
verify that llama_kv_cache_seq_pos_max() matches the checkpoint's pos_max.
A size-matched but misplaced restore can silently corrupt the KV cache;
on mismatch, force a full reset.
The DSV4 path pre-sets restored = true on byte-level success, then the
position check can revert it to false. Only if (restored) proceeds with
the restored state, matching the apply_checkpoint pattern.
* DSV4 checkpoints: correct misleading comment about chunk-boundary reproducibility
* DSV4 checkpoints: add FNV-1a checksum integrity check for checkpoint data
Sanity check (pos_max): catches misplaced restores (wrong stream offset,
partial overwrite) where the byte count matches but the cache position
doesn't.
Correctness check (FNV-1a hash of serialized data): catches in-memory
corruption of the checkpoint data vector between creation and restore.
Both checks are applied in the standard (apply_checkpoint) and DSV4
(batch_pending_prompt) restore paths. A mismatch in either causes the
restore to be treated as failed, falling back to full reprocess.
File format bumped to CKPT v2 (magic 0x434b5054, version 2) with a
data_hash field per checkpoint. Old-format files (LLAMA_STATE_SEQ_MAGIC)
are still loaded: the hash is computed on load so validation works
uniformly.
* Defer checkpoint hash computation to offload creation path
The FNV-1a hash (~5-15ms per 150 MiB checkpoint) is no longer computed
during checkpoint creation. Instead, data_hash is set to 0 and
hash_computed to false. The hash is computed lazily on first access via
ensure_checkpoint_hash(), called from:
- apply_checkpoint (before the integrity check during restore)
- save_checkpoints_to_file (before writing to disk)
This removes the hash computation from the time-critical checkpoint
creation path, reducing the pause between batches.
* Reuse pre-allocated scratch buffer for checkpoint serialization
Adds a reusable std::vector<uint8_t> scratch buffer to server_context,
eliminating the per-checkpoint zero-init allocation (~150 MiB memset)
from ckpt.data.resize(). The scratch is grown on demand and handed
off to the checkpoint via swap() — a zero-copy move.
Also removes default arguments from server_prompt_checkpoint_update()
since all callers already pass every parameter explicitly.
* Compute checkpoint data hash incrementally during serialization
Instead of a second pass over the serialized buffer (costly for 145 MiB DSV4
checkpoints) or deferring to save/restore time (breaks in-memory verification),
compute the FNV-1a hash as a streaming operation during llama_state_seq_get_data.
llama_data_write_buffer gains an optional fnv_hash pointer and updates it
during write() and write_tensor_data() — the hash is computed from bytes as
they land in the output buffer, with zero extra memory reads.
The server then obtains the hash at creation time by passing &ckpt.data_hash
(pre-initialized to the FNV-1a offset basis) to llama_state_seq_get_data.
This replaces the deferred hash approach (ensure_checkpoint_hash / hash_computed)
and restores in-memory round-trip verification.
* DSV4 checkpoints: add round-trip serialization verification
After restore, re-serialize the KV cache and compare byte-for-byte against
the original checkpoint data. This directly catches serialization bugs
that produce internally-consistent but wrong values (Joel's 59/64 case:
correct position, corrupted tensor data).
The check is added to both restore paths (standard apply_checkpoint and
DSV4 batch_pending_prompt) and runs after the pos_max sanity check and
FNV-1a hash integrity check. Cost: one extra llama_state_seq_get_size +
llama_state_seq_get_data + memcmp of the checkpoint data.
* Remove dead _ckpt_max_size member
_ckpt_max_size was set by server_prompt_checkpoint_update but never read.
Removed the member, the function parameter, and the call site.
* Remove no-op resize after swap in server_prompt_checkpoint_update
After swap, ckpt.data holds the scratch buffer which was already
resized to checkpoint_size. Since n == checkpoint_size (asserted),
the resize is a no-op.
* remove dead (void)has_hash cast
The variable is actually used later (for old-format file detection),
so the unused-variable suppression cast is misleading.
* add missing const qualifiers on to_json() methods
Both server_prompt_checkpoint::to_json() and server_prompt::to_json()
were missing const, preventing use on const references.
* remove duplicate n_kept_prompt assignment in server_prompt::from_json()
n_kept_prompt was assigned twice with the same value, clobbering the
slot where n_discarded_prompt should have been read.
* remove redundant params_base parameter from create_checkpoint_at_interval()
The parameter is already accessible as a member of server_context.
All callers were passing this->params_base, so the indirection was
unnecessary.
* factor duplicated restore verification into verify_restored_checkpoint() helper
The 3-step verification (pos_max sanity, FNV-1a hash, and round-trip
memcmp) was duplicated verbatim across apply_checkpoint() and
batch_pending_prompt(). Extract it into a shared static helper with
a label parameter for context-specific log messages.
Also eliminates the pos_next save/restore dance in apply_checkpoint
by using a local variable for the prompt-limit computation, and
removes a stray commented-out debug printf.
* fix two comments: fnv1a_hash comment was misleading, erasure comment imprecise
- fnv1a_hash() is used for all checkpoint verification (not just
backward-compat file loading) — broadened the description.
- 'may contain stale per-position state' → 'its per-position state
is stale' — the erasure is unconditional when pos_max > pos_next,
so the staleness is definite, not possible.
* remove FNV-1a hash and file-format bump (perf, Joelfarthing's review feedback)
The streaming FNV-1a hash added 65 ms to checkpoint creation and 107 ms
to restore (75 MiB checkpoints; roughly double at DSV4's 145 MiB). The
pos_max sanity check alone is sufficient for catching the real failure
modes (wrong stream offset, partial overwrite), and the initial byte-
count check from llama_state_seq_set_data catches outright corruption.
Removed:
- Streaming hash from llama_data_write_buffer (fnv_hash, fnv_update)
- hash_out parameter from llama_state_seq_get_data / llama.h API
- data_hash field from server_prompt_checkpoint struct
- FNV-1a computation during checkpoint creation and verification
- CKPT v2 file format (revert to LLAMA_STATE_SEQ_MAGIC/version)
- fnv1a_hash() helper function
Kept:
- pos_max sanity check in verify_restored_checkpoint (cheap, catches
misplaced restores)
- Scratch buffer reuse via swap() in server_prompt_checkpoint_update
(pure perf win, independent of hash)
* fix: restore off-by-one in n_past calculation after checkpoint restore
size_up_to_pos(pos_max) returns the number of cached tokens at positions
STRICTLY LESS THAN pos_max (non-mtmd: min(pos_max, size)). Since the
checkpoint encodes state for positions [pos_min, pos_max], the next
position to process is pos_max + 1, not pos_max.
This matters for DSV4 whose accumulator state is not position-indexed:
reprocessing the token at pos_max would double-count it in the compressed
indexer. For recurrent models the old pos_min+1 workaround happened to
give the right answer (since pos_min == pos_max there), but using
pos_max + 1 is correct for both.
Fixes both restore paths (apply_checkpoint and DSV4 in batch_pending_prompt).
* fix: only write/read DSV4 cache section for DSV4 models
The has_dsv4_cache uint32 was emitted unconditionally, changing the
serialized state layout for every model architecture without bumping
LLAMA_STATE_SEQ_VERSION. Old state/session files (which end before
this field) would fail with 'unexpectedly reached end of file' when
read by the new code.
Fix: guard the entire DSV4 section on both write and read sides with
ctx->model.arch == LLM_ARCH_DEEPSEEK4. Non-DSV4 models see the
identical layout they always had.
* fix: validate stream_idx < n_stream in dsv4_stream_offset_size
stream_idx was only checked >= 0 via GGML_ASSERT, but never checked
against n_stream. An invalid seq_id could compute an out-of-range
tensor offset or size, leading to memory corruption.
Now asserts 0 <= stream_idx < n_stream.
* fix: scratch buffer reuse — copy instead of swap
swap(scratch) moved the written data into ckpt.data but left scratch
empty. The next call's resize would then re-allocate from scratch,
defeating the purpose.
Now copies the data (ckpt.data = scratch) so scratch retains its size
and capacity across calls. resize becomes an in-place extension when
needed rather than a fresh allocation.
* fix: restore interval<=0 = disable semantics, split unconditional paths
The interval gate was inverted: interval <= 0 opened the gate, so every
call to create_checkpoint_at_interval created a checkpoint (PP, TG,
release, speculative). This changed the documented behavior ('<=0
disable' per --help) and created extra checkpoints on every decoded
token for recurrent models.
Fix:
- create_checkpoint_at_interval returns immediately when interval <= 0
(restoring the no-op semantics from the original code)
- Unconditional paths (release, PP end, PP start with slot.do_checkpoint)
call create_checkpoint(slot) directly, matching the original layout
- Interval-gated paths (TG tokens, PP start without slot.do_checkpoint,
speculative decoding) stay behind create_checkpoint_at_interval
* fix: gate ALL checkpoint creation by interval, not just TG paths
Three call sites bypassed the interval gate by calling create_checkpoint(slot)
directly instead of create_checkpoint_at_interval(slot):
- PP batch-boundary (was creating mid-PP checkpoints at unpredictable positions)
- PP end (created a checkpoint at every end-of-prompt, even if within the interval)
- release (created a checkpoint at every release, even if just 5 tokens later)
This caused checkpoints 5 and 6 in the log to be created only 5 tokens apart
(pos_max=8488 and pos_max=8493), and checkpoint 7 at release 455 tokens later,
all with interval=2048.
The original code had all checkpoint creation gated by interval (single
create_checkpoint_at_interval function called everywhere). The 'unconditional'
paths were introduced by our earlier fix that split create_checkpoint_at_interval
into a no-op for interval<=0 — but the split was too aggressive, making release,
PP-end, and PP-batch-boundary always fire.
Fix: route all checkpoint creation through create_checkpoint_at_interval, which
already handles do_checkpoint (early return) and interval <= 0 (no-op) correctly.
create_checkpoint is now an internal helper called only from
create_checkpoint_at_interval.
Result: with interval=2048 and an 8494-token prompt, checkpoints are created at
2048, 4096, 6144, 8192 only — the original semantics.
* fix: off-by-one in checkpoint gate condition, use -1 sentinel
The gate condition 'checkpoint_pos + interval <= 1 + pos' opened one
position early for non-first intervals. With checkpoint_pos=6143,
interval=2048: 6143+2048=8191, and pos=8190 gives 8191 <= 1+8190=8191
→ TRUE, creating a checkpoint at pos_max=8190 instead of 8191.
Root cause: checkpoint_pos=0 served dual duty ('no checkpoint yet' and
'checkpoint at position 0'). The '1 +' in the condition compensated
for this at startup but overcompensated later.
Fix:
- Change checkpoint_pos from size_t to llama_pos, initialized to -1
- Drop the '1 +' — condition is now checkpoint_pos + interval <= pos
With checkpoint_pos=-1: -1+2048=2047 <= 2047 → first checkpoint after
2048 tokens (correct).
With checkpoint_pos=6143: 6143+2048=8191 <= 8190 → FALSE (no early
open), 8191 <= 8191 → TRUE (opens at correct position).
Also fixes the mixed signed/unsigned comparison that existed with
size_t checkpoint_pos vs llama_pos pos.
* perf: serialize directly into ckpt.data, drop scratch buffer
The ckpt.data = scratch copy added ~10ms to checkpoint creation
(memcpy of 145 MiB). The scratch buffer was originally introduced to
avoid per-checkpoint resize allocation, but the lazy-zero paging of
modern OSes makes the resize essentially free.
Drop the _ckpt_scratch member entirely. Serialize directly into
ckpt.data after resize — same allocation cost, no extra copy.
* fix: replace GGML_ASSERT with runtime check in dsv4_stream_offset_size
GGML_ASSERT is compiled out in release builds (NDEBUG). An invalid
non-negative seq_id from the public state API would then compute
out-of-range tensor offsets and sizes, leading to memory corruption.
Replace with a runtime conditional that logs the error and sets
safe fallback values (offset=0, size=0). The caller that reads/writes
0 bytes will fail downstream in a defined way.
* fix: restore tolerance mechanism, slot.do_checkpoint bypasses interval gate
Samuel reviewed that we removed the slot.do_checkpoint branch from PP
batch-boundary, but batch_pending_prompt still sets slot.do_checkpoint
when the tolerance threshold is reached. Nowhere checks it, so the
tolerance checkpoint for short prompts (shorter than interval) is dead.
Fix: create_checkpoint_at_interval now checks slot.do_checkpoint — if
true, the interval gate is bypassed. After a successful creation the
flag is cleared so normal interval gating resumes for subsequent
checkpoints. Also handles interval <= 0 + slot.do_checkpoint correctly:
the early-return for disabled interval is itself gated by
!slot.do_checkpoint.
* revert: erasure condition back to cur.pos_max > pos_min_thold
The change from pos_min_thold to pos_next affected all models, not just
DSV4. Revert to the original condition (cur.pos_max >= pos_next after
integer simplification) which correctly erases checkpoints at or past
the write position.
* restore unconditional release checkpoint per firecoperana review
Release is a lifecycle boundary. The interval gate is for throttling
mid-processing checkpoints; the release should always capture the final
state (when do_checkpoint is enabled).
* restore original PP batch-boundary branching per firecoperana review
The explicit slot.do_checkpoint branch in the PP batch-boundary is
restored. The tolerance bypass is removed from create_checkpoint_at_interval
since it was only ever intended for the PP batch-boundary path (the
original code checked slot.do_checkpoint exclusively there). This keeps
the tolerance mechanism from leaking into TG, PP-end, and other paths.
* restore original PP-end checkpoint condition per firecoperana review
The original created an unconditional checkpoint at PP end when tolerance
is disabled (<=0). When tolerance > 0, the tolerance mechanism in the PP
loop handles the end-of-prompt capture at the tolerance point, so no
additional PP-end checkpoint is needed.
* consolidate DSV4 restore path into apply_checkpoint per firecoperana review
The DSV4-specific restore path in batch_pending_prompt duplicated the core
logic of apply_checkpoint (search, restore, verify) with different search
conditions and missing erasure. Consolidate by:
- Adding is_state_ckpt_model flag to apply_checkpoint
- Bypassing the pos_min >= pos_min_thold guard for state-checkpoint models
(DSV4 always has pos_min=0 from no eviction, so the guard blocked entry)
- Using pos_next instead of pos_min_thold for the search condition when
is_state_ckpt_model (allows finding checkpoints at pos_max == n_past - 1)
- Differentiating the reset log message per model type
- Recomputing n_past_offset and n_discarded_prompt after apply_checkpoint
(previously handled in the DSV4-specific path)
* conditional pos_next formula
State-checkpoint models (DSV4, recurrent) use pos_max + 1 — correct
for DSV4's multi-position checkpoints where pos_min=0, po neviction
max(pos_min+1, pos_max) = pos_max, which undercounts by 1.
For recurrent models pos_min==pos_max so both formulas agree.
Non-state-checkpoint models keep the original
max(pos_min + 1, pos_max) formula unchanged.
* remove redundant n_past_offset / n_discarded_prompt after apply_checkpoint
Both n_past and n_past_prompt are shifted by the same delta from the
restored checkpoint, so the difference (n_past_offset) is unchanged.
n_discarded_prompt is not used in the critical path.
* remove redundant speculative-decoding checkpoint per firecoperana review
speculative_decoding_accept is called from within the TG generation loop
which already creates interval-gated checkpoints at n_decoded > 1 (line
4779). The inner call would double-create.
* narrow DSV4-specific search and pos_next formula to DSV4 only per firecoperana review
is_state_ckpt_model includes recurrent models (e.g. Qwen 3.6) where
pos_max < pos_next search semantics may not be appropriate. Only
DSV4 needs pos_max+1 formula and pos_next-based search threshold.
* revert divergence-reset guard to original per firecoperana review
Unnecessary wrapping of the OpenPangu-only divergence path inside
!llama_model_supports_state_checkpoints. The condition is already
specific enough (!llama_model_supports_partial_kv_reuse is
OpenPangu-only), and OpenPangu does not support state checkpoints,
so the original code was functionally identical.
* narrow guard bypass to DSV4 only per firecoperana review
Recurrent state-checkpoint models don't need the pos_min >=
pos_min_thold guard bypass — only DSV4 (which always has pos_min=0
due to no KV cache eviction) requires it.
* narrow reset log message to DSV4 only per firecoperana review
Replace remaining is_state_ckpt_model with is_dsv4 in the
do_reset log branch; remove the now-unused variable.
* cleanup: revert unnecessary newlines, spacing, and comment changes
* fix: restore partial KV reuse for DSV4 in llama_model_supports_partial_kv_reuse
DSV4 has private per-position state but uses state checkpoints to
restore after a mid-sequence divergence. The function was returning
false, causing batch_pending_prompt to reset n_past=0 before
apply_checkpoint could restore from a checkpoint, which broke the
entire checkpoint mechanism.
* Remove bloat
* Reinstate deleted comment
* replace strcmp(arch_string) with llama_model_is_deepseek4()
SamuelOliveirads added the helper upstream — cleaner and avoids
the fragile string comparison.
* inline llama_model_supports_state_checkpoints into call site
Replaced with the inline expression
llama_model_has_recurrent(model) || llama_model_is_deepseek4(model)
and removed the now-unused function from llama.h and llama-model.cpp.
* fix: GCC 13.3 variadic macro trailing comma in SLT_WRN
SLT_WRN expands to LOG_WRN with __VA_ARGS__ at the end. When no extra
args follow the format string, the dangling comma causes GCC 13.3 to
error with 'expected primary-expression before')' token. Use '%s'
pattern consistent with all other zero-arg SLT_WRN callers.
* dsv4_stream_offset_size: bool return, GGML_ASSERT on write, graceful abort on read
dsv4_stream_offset_size silently returned offset=0, size=0 for invalid
stream indices. Now returns bool — writer hard-aborts via GGML_ASSERT
(prevents writing corrupt checkpoints), reader aborts the restore via
return false (handles corrupt checkpoints gracefully).
LLaMA.cpp HTTP Server
Fast, lightweight, pure C/C++ HTTP server based on httplib, nlohmann::json and llama.cpp.
Set of LLM REST APIs and a simple web front end to interact with llama.cpp.
Features:
- LLM inference of F16 and quantized models on GPU and CPU
- OpenAI API compatible chat completions, responses, and embeddings routes
- Parallel decoding with multi-user support
- Continuous batching
- Multimodal (wip)
- Monitoring endpoints
- Schema-constrained JSON response format
- Prefilling of assistant messages similar to the Claude API
- Function calling / tool use for ~any model
- Speculative decoding
- Easy-to-use web UI
The project is under active development, and we are looking for feedback and contributors.
Usage
usage: ./llama-server [options]
general:
-h, --help, --usage print usage and exit
--version show version and build info
-v, --verbose print verbose information
--verbosity N set specific verbosity level (default: 0)
--verbose-prompt print a verbose prompt before generation (default: false)
--no-display-prompt don't print prompt at generation (default: false)
-co, --color colorise output to distinguish prompt and user input from generations (default: false)
-s, --seed SEED RNG seed (default: -1, use random seed for < 0)
-t, --threads N number of threads to use during generation (default: 8)
-tb, --threads-batch N number of threads to use during batch and prompt processing (default: same as --threads)
-td, --threads-draft N number of threads to use during generation (default: same as --threads)
-tbd, --threads-batch-draft N number of threads to use during batch and prompt processing (default: same as --threads-draft)
--draft N number of tokens to draft for speculative decoding (default: 5)
-ps, --p-split N speculative decoding split probability (default: 0.1)
-lcs, --lookup-cache-static FNAME
path to static lookup cache to use for lookup decoding (not updated by generation)
-lcd, --lookup-cache-dynamic FNAME
path to dynamic lookup cache to use for lookup decoding (updated by generation)
-c, --ctx-size N size of the prompt context (default: 0, 0 = loaded from model)
-n, --predict N number of tokens to predict (default: -1, -1 = infinity, -2 = until context filled)
-b, --batch-size N logical maximum batch size (default: 2048)
-ub, --ubatch-size N physical maximum batch size (default: 512)
--keep N number of tokens to keep from the initial prompt (default: 0, -1 = all)
--chunks N max number of chunks to process (default: -1, -1 = all)
-fa, --flash-attn enable Flash Attention (default: disabled)
-p, --prompt PROMPT prompt to start generation with
in conversation mode, this will be used as system prompt
(default: '')
-f, --file FNAME a file containing the prompt (default: none)
--in-file FNAME an input file (repeat to specify multiple files)
-bf, --binary-file FNAME binary file containing the prompt (default: none)
-e, --escape process escapes sequences (\n, \r, \t, \', \", \\) (default: true)
--no-escape do not process escape sequences
-ptc, --print-token-count N print token count every N tokens (default: -1)
--prompt-cache FNAME file to cache prompt state for faster startup (default: none)
--prompt-cache-all if specified, saves user input and generations to cache as well
not supported with --interactive or other interactive options
--prompt-cache-ro if specified, uses the prompt cache but does not update it
-r, --reverse-prompt PROMPT halt generation at PROMPT, return control in interactive mode
can be specified more than once for multiple prompts
-sp, --special special tokens output enabled (default: false)
-cnv, --conversation run in conversation mode, does not print special tokens and suffix/prefix
if suffix/prefix are not specified, default chat template will be used
(default: false)
-i, --interactive run in interactive mode (default: false)
-if, --interactive-first run in interactive mode and wait for input right away (default: false)
-mli, --multiline-input allows you to write or paste multiple lines without ending each in '\'
--in-prefix-bos prefix BOS to user inputs, preceding the `--in-prefix` string
--in-prefix STRING string to prefix user inputs with (default: empty)
--in-suffix STRING string to suffix after user inputs with (default: empty)
--spm-infill use Suffix/Prefix/Middle pattern for infill (instead of Prefix/Suffix/Middle) as some models prefer this. (default: disabled)
sampling:
--samplers SAMPLERS samplers that will be used for generation in the order, separated by ';'
(default: top_k;tfs_z;typical_p;top_p;min_p;temperature)
--sampling-seq SEQUENCE simplified sequence for samplers that will be used (default: kfypmt)
--ignore-eos ignore end of stream token and continue generating (implies --logit-bias EOS-inf)
--penalize-nl penalize newline tokens (default: false)
--temp N temperature (default: 0.8)
--top-k N top-k sampling (default: 40, 0 = disabled)
--top-p N top-p sampling (default: 0.9, 1.0 = disabled)
--min-p N min-p sampling (default: 0.1, 0.0 = disabled)
--tfs N tail free sampling, parameter z (default: 1.0, 1.0 = disabled)
--typical N locally typical sampling, parameter p (default: 1.0, 1.0 = disabled)
--repeat-last-n N last n tokens to consider for penalize (default: 64, 0 = disabled, -1 = ctx_size)
--repeat-penalty N penalize repeat sequence of tokens (default: 1.0, 1.0 = disabled)
--presence-penalty N repeat alpha presence penalty (default: 0.0, 0.0 = disabled)
--frequency-penalty N repeat alpha frequency penalty (default: 0.0, 0.0 = disabled)
--dynatemp-range N dynamic temperature range (default: 0.0, 0.0 = disabled)
--dynatemp-exp N dynamic temperature exponent (default: 1.0)
--mirostat N use Mirostat sampling.
Top K, Nucleus, Tail Free and Locally Typical samplers are ignored if used.
(default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0)
--mirostat-lr N Mirostat learning rate, parameter eta (default: 0.1)
--mirostat-ent N Mirostat target entropy, parameter tau (default: 5.0)
--xtc-probability p xtc probability (default: 0.0 => disabled)
--xtc-threshold t xtc threshold (default: 1.0 => disabled)
--top-n-sigma t top-n-sigma parmeter (default: 0.0 => disabled)
-l TOKEN_ID(+/-)BIAS modifies the likelihood of token appearing in the completion,
i.e. `--logit-bias 15043+1` to increase likelihood of token ' Hello',
or `--logit-bias 15043-1` to decrease likelihood of token ' Hello'
--cfg-negative-prompt PROMPT
negative prompt to use for guidance (default: '')
--cfg-negative-prompt-file FNAME
negative prompt file to use for guidance
--cfg-scale N strength of guidance (default: 1.0, 1.0 = disable)
--chat-template JINJA_TEMPLATE
set custom jinja chat template (default: template taken from model's metadata)
if suffix/prefix are specified, template will be disabled
only commonly used templates are accepted:
https://github.com/ggerganov/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template
grammar:
--grammar GRAMMAR BNF-like grammar to constrain generations (see samples in grammars/ dir) (default: '')
--grammar-file FNAME file to read grammar from
-j, --json-schema SCHEMA JSON schema to constrain generations (https://json-schema.org/), e.g. `{}` for any JSON object
For schemas w/ external $refs, use --grammar + example/json_schema_to_grammar.py instead
embedding:
--pooling {none,mean,cls,last}
pooling type for embeddings, use model default if unspecified
--attention {causal,non-causal}
attention type for embeddings, use model default if unspecified
context hacking:
--rope-scaling {none,linear,yarn}
RoPE frequency scaling method, defaults to linear unless specified by the model
--rope-scale N RoPE context scaling factor, expands context by a factor of N
--rope-freq-base N RoPE base frequency, used by NTK-aware scaling (default: loaded from model)
--rope-freq-scale N RoPE frequency scaling factor, expands context by a factor of 1/N
--yarn-orig-ctx N YaRN: original context size of model (default: 0 = model training context size)
--yarn-ext-factor N YaRN: extrapolation mix factor (default: -1.0, 0.0 = full interpolation)
--yarn-attn-factor N YaRN: scale sqrt(t) or attention magnitude (default: 1.0)
--yarn-beta-slow N YaRN: high correction dim or alpha (default: 1.0)
--yarn-beta-fast N YaRN: low correction dim or beta (default: 32.0)
-gan, --grp-attn-n N group-attention factor (default: 1)
-gaw, --grp-attn-w N group-attention width (default: 512.0)
-dkvc, --dump-kv-cache verbose print of the KV cache
-nkvo, --no-kv-offload disable KV offload
-ctk, --cache-type-k TYPE KV cache data type for K (default: f16)
-ctv, --cache-type-v TYPE KV cache data type for V (default: f16)
perplexity:
--all-logits return logits for all tokens in the batch (default: false)
--hellaswag compute HellaSwag score over random tasks from datafile supplied with -f
--hellaswag-tasks N number of tasks to use when computing the HellaSwag score (default: 400)
--winogrande compute Winogrande score over random tasks from datafile supplied with -f
--winogrande-tasks N number of tasks to use when computing the Winogrande score (default: 0)
--multiple-choice compute multiple choice score over random tasks from datafile supplied with -f
--multiple-choice-tasks N
number of tasks to use when computing the multiple choice score (default: 0)
--kl-divergence computes KL-divergence to logits provided via --kl-divergence-base
--ppl-stride N stride for perplexity calculation (default: 0)
--ppl-output-type {0,1} output type for perplexity calculation (default: 0)
parallel:
-dt, --defrag-thold N KV cache defragmentation threshold (default: -1.0, < 0 - disabled)
-np, --parallel N number of parallel sequences to decode (default: 1)
-ns, --sequences N number of sequences to decode (default: 1)
-cb, --cont-batching enable continuous batching (a.k.a dynamic batching) (default: enabled)
multi-modality:
--mmproj FILE path to a multimodal projector file. see examples/mtmd/README.md
--image FILE path to an image file. use with multimodal models. Specify multiple times for batching
backend:
--rpc SERVERS comma separated list of RPC servers
--mlock force system to keep model in RAM rather than swapping or compressing
--no-mmap do not memory-map model (slower load but may reduce pageouts if not using mlock)
--numa TYPE attempt optimizations that help on some NUMA systems
- distribute: spread execution evenly over all nodes
- isolate: only spawn threads on CPUs on the node that execution started on
- numactl: use the CPU map provided by numactl
if run without this previously, it is recommended to drop the system page cache before using this
see https://github.com/ggerganov/llama.cpp/issues/1437
model:
--check-tensors check model tensor data for invalid values (default: false)
--override-kv KEY=TYPE:VALUE
advanced option to override model metadata by key. may be specified multiple times.
types: int, float, bool, str. example: --override-kv tokenizer.ggml.add_bos_token=bool:false
--lora FNAME apply LoRA adapter (implies --no-mmap)
--lora-scaled FNAME S apply LoRA adapter with user defined scaling S (implies --no-mmap)
--lora-base FNAME optional model to use as a base for the layers modified by the LoRA adapter
--control-vector FNAME add a control vector
note: this argument can be repeated to add multiple control vectors
--control-vector-scaled FNAME SCALE
add a control vector with user defined scaling SCALE
note: this argument can be repeated to add multiple scaled control vectors
--control-vector-layer-range START END
layer range to apply the control vector(s) to, start and end inclusive
-m, --model FNAME model path (default: models/$filename with filename from --hf-file
or --model-url if set, otherwise models/7B/ggml-model-f16.gguf)
-md, --model-draft FNAME draft model for speculative decoding (default: unused)
--spec-type SPEC[:k=v,...]
canonical speculative stage entry; repeat for a supported two-stage chain
examples: --spec-type mtp:n_max=1,p_min=0.0
--spec-type ngram-mod:n_max=64,n_min=2,ngram_size_n=8 --spec-type mtp:n_max=1,p_min=0.0
-mu, --model-url MODEL_URL model download url (default: unused)
-hfr, --hf-repo REPO Hugging Face model repository (default: unused)
-hff, --hf-file FILE Hugging Face model file (default: unused)
-hft, --hf-token TOKEN Hugging Face access token (default: value from HF_TOKEN environment variable)
server:
--host HOST ip address to listen (default: 127.0.0.1)
--port PORT port to listen (default: 8080)
--path PATH path to serve static files from (default: )
--embedding(s) restrict to only support embedding use case; use only with dedicated embedding models (default: disabled)
--api-key KEY API key to use for authentication (default: none)
--api-key-file FNAME path to file containing API keys (default: none)
--ssl-key-file FNAME path to file a PEM-encoded SSL private key
--ssl-cert-file FNAME path to file a PEM-encoded SSL certificate
--timeout N server read/write timeout in seconds (default: 600)
--threads-http N number of threads used to process HTTP requests (default: -1)
--system-prompt-file FNAME
set a file to load a system prompt (initial prompt of all slots), this is useful for chat applications
--log-format {text,json}
log output format: json or text (default: json)
--metrics enable prometheus compatible metrics endpoint (default: disabled)
--no-slots disables slots monitoring endpoint (default: enabled)
--slot-save-path PATH path to save slot kv cache (default: disabled)
--chat-template JINJA_TEMPLATE
set custom jinja chat template (default: template taken from model's metadata)
only commonly used templates are accepted:
https://github.com/ggerganov/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template
-sps, --slot-prompt-similarity SIMILARITY
how much the prompt of a request must match the prompt of a slot in order to use that slot (default: 0.50, 0.0 = disabled)
--lora-init-without-apply
load LoRA adapters without applying them (apply later via POST /lora-adapters) (default: disabled)
logging:
--simple-io use basic IO for better compatibility in subprocesses and limited consoles
-ld, --logdir LOGDIR path under which to save YAML logs (no logging if unset)
--log-test Run simple logging test
--log-disable Disable trace logs
--log-enable Enable trace logs
--log-file FNAME Specify a log filename (without extension)
--log-new Create a separate new log file on start. Each log file will have unique name: "<name>.<ID>.log"
--log-append Don't truncate the old log file.
Available environment variables (if specified, these variables will override parameters specified in arguments):
LLAMA_CACHE: cache directory, used by--hf-repoHF_TOKEN: Hugging Face access token, used when accessing a gated model with--hf-repoLLAMA_ARG_MODEL: equivalent to-mLLAMA_ARG_MODEL_URL: equivalent to-muLLAMA_ARG_MODEL_ALIAS: equivalent to-aLLAMA_ARG_HF_REPO: equivalent to--hf-repoLLAMA_ARG_HF_FILE: equivalent to--hf-fileLLAMA_ARG_THREADS: equivalent to-tLLAMA_ARG_CTX_SIZE: equivalent to-cLLAMA_ARG_N_PARALLEL: equivalent to-npLLAMA_ARG_BATCH: equivalent to-bLLAMA_ARG_UBATCH: equivalent to-ubLLAMA_ARG_N_GPU_LAYERS: equivalent to-nglLLAMA_ARG_THREADS_HTTP: equivalent to--threads-httpLLAMA_ARG_CHAT_TEMPLATE: equivalent to--chat-templateLLAMA_ARG_N_PREDICT: equivalent to-nLLAMA_ARG_ENDPOINT_METRICS: if set to1, it will enable metrics endpoint (equivalent to--metrics)LLAMA_ARG_ENDPOINT_SLOTS: if set to0, it will disable slots endpoint (equivalent to--no-slots). This feature is enabled by default.LLAMA_ARG_EMBEDDINGS: if set to1, it will enable embeddings endpoint (equivalent to--embeddings)LLAMA_ARG_FLASH_ATTN: if set to1, it will enable flash attention (equivalent to-fa)LLAMA_ARG_CONT_BATCHING: if set to0, it will disable continuous batching (equivalent to--no-cont-batching). This feature is enabled by default.LLAMA_ARG_DEFRAG_THOLD: equivalent to-dtLLAMA_ARG_HOST: equivalent to--hostLLAMA_ARG_PORT: equivalent to--port
Example usage of docker compose with environment variables:
services:
llamacpp-server:
image: ghcr.io/ggerganov/llama.cpp:server
ports:
- 8080:8080
volumes:
- ./models:/models
environment:
# alternatively, you can use "LLAMA_ARG_MODEL_URL" to download the model
LLAMA_ARG_MODEL: /models/my_model.gguf
LLAMA_ARG_CTX_SIZE: 4096
LLAMA_ARG_N_PARALLEL: 2
LLAMA_ARG_ENDPOINT_METRICS: 1 # to disable, either remove or set to 0
LLAMA_ARG_PORT: 8080
Build
llama-server is built alongside everything else from the root of the project
-
Using
make:make llama-server -
Using
CMake:cmake -B build cmake --build build --config Release -t llama-serverBinary is at
./build/bin/llama-server
Build with SSL
llama-server can also be built with SSL support using OpenSSL 3
-
Using
make:# NOTE: For non-system openssl, use the following: # CXXFLAGS="-I /path/to/openssl/include" # LDFLAGS="-L /path/to/openssl/lib" make LLAMA_SERVER_SSL=true llama-server -
Using
CMake:cmake -B build -DLLAMA_SERVER_SSL=ON cmake --build build --config Release -t llama-server
Web UI
The project includes a web-based user interface that enables interaction with the model through the /chat/completions endpoint.
The web UI is developed using:
vueframework for frontend developmenttailwindcssanddaisyuifor stylingvitefor build tooling
A pre-built version is available as a single HTML file under /public directory.
To build or to run the dev server (with hot reload):
# make sure you have nodejs installed
cd examples/server/webui
npm i
# to run the dev server
npm run dev
# to build the public/index.html
npm run build
NOTE: if you are using the vite dev server, you can change the API base URL to llama.cpp. To do that, run this code snippet in browser's console:
localStorage.setItem('base', 'http://localhost:8080')
Quick Start
To get started right away, run the following command, making sure to use the correct path for the model you have:
Unix-based systems (Linux, macOS, etc.)
./llama-server -m models/7B/ggml-model.gguf -c 2048
Windows
llama-server.exe -m models\7B\ggml-model.gguf -c 2048
The above command will start a server that by default listens on 127.0.0.1:8080.
You can consume the endpoints with Postman or NodeJS with axios library. You can visit the web front end at the same url.
Docker
docker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggerganov/llama.cpp:server -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080
# or, with CUDA:
docker run -p 8080:8080 -v /path/to/models:/models --gpus all ghcr.io/ggerganov/llama.cpp:server-cuda -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080 --n-gpu-layers 99
Testing with CURL
Using curl. On Windows, curl.exe should be available in the base OS.
curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'
Advanced testing
We implemented a server test framework using human-readable scenario.
Before submitting an issue, please try to reproduce it with this format.
Node JS Test
You need to have Node.js installed.
mkdir llama-client
cd llama-client
Create a index.js file and put this inside:
const prompt = `Building a website can be done in 10 simple steps:`;
async function Test() {
let response = await fetch("http://127.0.0.1:8080/completion", {
method: 'POST',
body: JSON.stringify({
prompt,
n_predict: 512,
})
})
console.log((await response.json()).content)
}
Test()
And run it:
node index.js
API Endpoints
GET /health: Returns the current state of the server
- 503 ->
{"status": "loading model"}if the model is still being loaded. - 500 ->
{"status": "error"}if the model failed to load. - 200 ->
{"status": "ok", "slots_idle": 1, "slots_processing": 2 }if the model is successfully loaded and the server is ready for further requests mentioned below. - 200 ->
{"status": "no slot available", "slots_idle": 0, "slots_processing": 32}if no slots are currently available. - 503 ->
{"status": "no slot available", "slots_idle": 0, "slots_processing": 32}if the query parameterfail_on_no_slotis provided and no slots are currently available.
If the query parameter include_slots is passed, slots field will contain internal slots data except if --slots-endpoint-disable is set.
POST /completion: Given a prompt, it returns the predicted completion.
*Options:*
`prompt`: Provide the prompt for this completion as a string or as an array of strings or numbers representing tokens. Internally, if `cache_prompt` is `true`, the prompt is compared to the previous completion and only the "unseen" suffix is evaluated. A `BOS` token is inserted at the start, if all of the following conditions are true:
- The prompt is a string or an array with the first element given as a string
- The model's `tokenizer.ggml.add_bos_token` metadata is `true`
- The system prompt is empty
`temperature`: Adjust the randomness of the generated text. Default: `0.8`
`dynatemp_range`: Dynamic temperature range. The final temperature will be in the range of `[temperature - dynatemp_range; temperature + dynatemp_range]` Default: `0.0`, which is disabled.
`dynatemp_exponent`: Dynamic temperature exponent. Default: `1.0`
`top_k`: Limit the next token selection to the K most probable tokens. Default: `40`
`top_p`: Limit the next token selection to a subset of tokens with a cumulative probability above a threshold P. Default: `0.95`
`min_p`: The minimum probability for a token to be considered, relative to the probability of the most likely token. Default: `0.05`
`n_predict`: Set the maximum number of tokens to predict when generating text. **Note:** May exceed the set limit slightly if the last token is a partial multibyte character. When 0, no tokens will be generated but the prompt is evaluated into the cache. Default: `-1`, where `-1` is infinity.
`n_keep`: Specify the number of tokens from the prompt to retain when the context size is exceeded and tokens need to be discarded. The number excludes the BOS token.
By default, this value is set to `0`, meaning no tokens are kept. Use `-1` to retain all tokens from the prompt.
`stream`: It allows receiving each predicted token in real-time instead of waiting for the completion to finish. To enable this, set to `true`.
`stop`: Specify a JSON array of stopping strings.
These words will not be included in the completion, so make sure to add them to the prompt for the next iteration. Default: `[]`
`tfs_z`: Enable tail free sampling with parameter z. Default: `1.0`, which is disabled.
`typical_p`: Enable locally typical sampling with parameter p. Default: `1.0`, which is disabled.
`repeat_penalty`: Control the repetition of token sequences in the generated text. Default: `1.1`
`repeat_last_n`: Last n tokens to consider for penalizing repetition. Default: `64`, where `0` is disabled and `-1` is ctx-size.
`penalize_nl`: Penalize newline tokens when applying the repeat penalty. Default: `true`
`presence_penalty`: Repeat alpha presence penalty. Default: `0.0`, which is disabled.
`frequency_penalty`: Repeat alpha frequency penalty. Default: `0.0`, which is disabled.
`penalty_prompt`: This will replace the `prompt` for the purpose of the penalty evaluation. Can be either `null`, a string or an array of numbers representing tokens. Default: `null`, which is to use the original `prompt`.
`mirostat`: Enable Mirostat sampling, controlling perplexity during text generation. Default: `0`, where `0` is disabled, `1` is Mirostat, and `2` is Mirostat 2.0.
`mirostat_tau`: Set the Mirostat target entropy, parameter tau. Default: `5.0`
`mirostat_eta`: Set the Mirostat learning rate, parameter eta. Default: `0.1`
`grammar`: Set grammar for grammar-based sampling. Default: no grammar
`json_schema`: Set a JSON schema for grammar-based sampling (e.g. `{"items": {"type": "string"}, "minItems": 10, "maxItems": 100}` of a list of strings, or `{}` for any JSON). See [tests](../../tests/test-json-schema-to-grammar.cpp) for supported features. Default: no JSON schema.
`seed`: Set the random number generator (RNG) seed. Default: `-1`, which is a random seed.
`ignore_eos`: Ignore end of stream token and continue generating. Default: `false`
`logit_bias`: Modify the likelihood of a token appearing in the generated text completion. For example, use `"logit_bias": [[15043,1.0]]` to increase the likelihood of the token 'Hello', or `"logit_bias": [[15043,-1.0]]` to decrease its likelihood. Setting the value to false, `"logit_bias": [[15043,false]]` ensures that the token `Hello` is never produced. The tokens can also be represented as strings, e.g. `[["Hello, World!",-0.5]]` will reduce the likelihood of all the individual tokens that represent the string `Hello, World!`, just like the `presence_penalty` does. Default: `[]`
`n_probs`: If greater than 0, the response also contains the probabilities of top N tokens for each generated token given the sampling settings. Note that for temperature < 0 the tokens are sampled greedily but token probabilities are still being calculated via a simple softmax of the logits without considering any other sampler settings. Default: `0`
`min_keep`: If greater than 0, force samplers to return N possible tokens at minimum. Default: `0`
`image_data`: An array of objects to hold base64-encoded image `data` and its `id`s to be reference in `prompt`. You can determine the place of the image in the prompt as in the following: `USER:[img-12]Describe the image in detail.\nASSISTANT:`. In this case, `[img-12]` will be replaced by the embeddings of the image with id `12` in the following `image_data` array: `{..., "image_data": [{"data": "<BASE64_STRING>", "id": 12}]}`. Use `image_data` only with multimodal models, e.g., LLaVA.
`id_slot`: Assign the completion task to an specific slot. If is -1 the task will be assigned to a Idle slot. Default: `-1`
`cache_prompt`: Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. Because (depending on the backend) the logits are **not** guaranteed to be bit-for-bit identical for different batch sizes (prompt processing vs. token generation) enabling this option can cause nondeterministic results. Default: `true`
`system_prompt`: Change the system prompt (initial prompt of all slots), this is useful for chat applications. [See more](#change-system-prompt-on-runtime)
`samplers`: The order the samplers should be applied in. An array of strings representing sampler type names. If a sampler is not set, it will not be used. If a sampler is specified more than once, it will be applied multiple times. Default: `["top_k", "tfs_z", "typical_p", "top_p", "min_p", "temperature"]` - these are all the available values.
`banned_strings`: Specify a JSON array of strings that are prohibited in the generated text. If a banned string is generated, the model rewinds and resamples. Format: `["string1", "string2"]`. Default: `[]`
`banned_regex`: Specify a JSON array of ECMAScript-compatible regular expression patterns that are prohibited in the generated text. If a match is found, the model rewinds and resamples. Format: `["pattern1", "pattern2"]`. Default: `[]`
`banned_regex_case_insensitive`: Specify a JSON array of case-insensitive ECMAScript-compatible regular expression patterns that are prohibited in the generated text. Same behavior as `banned_regex` but matches are case-insensitive. Format: `["pattern1", "pattern2"]`. Default: `[]`
`saturate_predict`: If `true`, ensure that the number of tokens sent in the response equals `n_predict` even if tokens were discarded due to bans. When `false`, `n_predict` counts all generated tokens including those discarded during rewinds. Default: `false`
`banbuffer_size`: Set the token buffer size for ban detection. Larger values detect banned patterns spanning more tokens but delay streaming more. When `0`, automatically sets to the longest banned string/regex length plus 1. Default: `0`
`rewind_count_max`: Set the maximum number of regeneration attempts when banned content is encountered. When `-1`, automatically sets to `max(20, 2 * (number of banned_strings + banned_regex + banned_regex_case_insensitive))`. When `0`, allows infinite retries. Default: `-1`
`banned_n`: Control how many tokens to ban when a banned string is detected at a specific position. For a string tokenizing to `["I", " can", " do"]`, `1` bans only "I", `2` bans "I" and " can", etc. When `-1`, bans all tokens in the match. **Note:** Using `-1` with regex patterns may cause excessive unintended bans. Default: `1`
Response format
-
Note: When using streaming mode (
stream), onlycontentandstopwill be returned until end of completion. -
completion_probabilities: An array of token probabilities for each completion. The array's length isn_predict. Each item in the array has the following structure:
{
"content": "<the token selected by the model>",
"probs": [
{
"prob": float,
"tok_str": "<most likely token>"
},
{
"prob": float,
"tok_str": "<second most likely token>"
},
...
]
},
Notice that each probs is an array of length n_probs.
content: Completion result as a string (excludingstopping_wordif any). In case of streaming mode, will contain the next token as a string.stop: Boolean for use withstreamto check whether the generation has stopped (Note: This is not related to stopping words arraystopfrom input options)generation_settings: The provided options above excludingpromptbut includingn_ctx,model. These options may differ from the original ones in some way (e.g. bad values filtered out, strings converted to tokens, etc.).model: The path to the model loaded with-mprompt: The providedpromptstopped_eos: Indicating whether the completion has stopped because it encountered the EOS tokenstopped_limit: Indicating whether the completion stopped becausen_predicttokens were generated before stop words or EOS was encounteredstopped_word: Indicating whether the completion stopped due to encountering a stopping word fromstopJSON array providedstopping_word: The stopping word encountered which stopped the generation (or "" if not stopped due to a stopping word)timings: Hash of timing information about the completion such as the number of tokenspredicted_per_secondtokens_cached: Number of tokens from the prompt which could be re-used from previous completion (n_past)tokens_evaluated: Number of tokens evaluated in total from the prompttruncated: Boolean indicating if the context size was exceeded during generation, i.e. the number of tokens provided in the prompt (tokens_evaluated) plus tokens generated (tokens predicted) exceeded the context size (n_ctx)
POST /tokenize: Tokenize a given text
*Options:*
`content`: Set the text to tokenize.
`add_special`: Boolean indicating if special tokens, i.e. `BOS`, should be inserted. Default: `false`
POST /detokenize: Convert tokens to text
*Options:*
`tokens`: Set the tokens to detokenize.
POST /embedding: Generate embedding of a given text
The same as the embedding example does.
*Options:*
`content`: Set the text to process.
`image_data`: An array of objects to hold base64-encoded image `data` and its `id`s to be reference in `content`. You can determine the place of the image in the content as in the following: `Image: [img-21].\nCaption: This is a picture of a house`. In this case, `[img-21]` will be replaced by the embeddings of the image with id `21` in the following `image_data` array: `{..., "image_data": [{"data": "<BASE64_STRING>", "id": 21}]}`. Use `image_data` only with multimodal models, e.g., LLaVA.
POST /infill: For code infilling.
Takes a prefix and a suffix and returns the predicted completion as stream.
*Options:*
`input_prefix`: Set the prefix of the code to infill.
`input_suffix`: Set the suffix of the code to infill.
It also accepts all the options of `/completion` except `stream` and `prompt`.
- GET
/props: Return current server settings.
Response format
{
"assistant_name": "",
"user_name": "",
"default_generation_settings": { ... },
"total_slots": 1,
"chat_template": ""
}
assistant_name- the required assistant name to generate the prompt in case you have specified a system prompt for all slots.user_name- the required anti-prompt to generate the prompt in case you have specified a system prompt for all slots.default_generation_settings- the default generation settings for the/completionendpoint, which has the same fields as thegeneration_settingsresponse object from the/completionendpoint.total_slots- the total number of slots for process requests (defined by--paralleloption)chat_template- the model's original Jinja2 prompt template
POST /v1/chat/completions: OpenAI-compatible Chat Completions API
Given a ChatML-formatted json description in messages, it returns the predicted completion. Both synchronous and streaming mode are supported, so scripted and interactive applications work fine. While no strong claims of compatibility with OpenAI API spec is being made, in our experience it suffices to support many apps. Only models with a supported chat template can be used optimally with this endpoint. By default, the ChatML template will be used.
If model supports multimodal, you can input the media file via image_url content part. We support both base64 and remote URL as input. See OAI documentation for more.
Options:
See OpenAI Chat Completions API documentation. llama.cpp /completion-specific features such as mirostat are also supported.
The response_format parameter supports both plain JSON output (e.g. {"type": "json_object"}) and schema-constrained JSON (e.g. {"type": "json_object", "schema": {"type": "string", "minLength": 10, "maxLength": 100}} or {"type": "json_schema", "schema": {"properties": { "name": { "title": "Name", "type": "string" }, "date": { "title": "Date", "type": "string" }, "participants": { "items": {"type: "string" }, "title": "Participants", "type": "string" } } } }), similar to other OpenAI-inspired API providers.
chat_template_kwargs: Allows sending additional parameters to the json templating system. For example: {"enable_thinking": false}
reasoning_format: The reasoning format to be parsed. If set to none, it will output the raw generated text.
thinking_forced_open: Force a reasoning model to always output the reasoning. Only works on certain models.
parse_tool_calls: Whether to parse the generated tool call.
Examples:
You can use either Python openai library with appropriate checkpoints:
import openai
client = openai.OpenAI(
base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
api_key = "sk-no-key-required"
)
completion = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
{"role": "user", "content": "Write a limerick about python exceptions"}
]
)
print(completion.choices[0].message)
... or raw HTTP requests:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "system",
"content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
},
{
"role": "user",
"content": "Write a limerick about python exceptions"
}
]
}'
Tool call support
OpenAI-style function calling is supported with the --jinja flag (and may require a --chat-template-file override to get the right tool-use compatible Jinja template; worst case, --chat-template chatml may also work).
See our Function calling docs for more details, supported native tool call styles (generic tool call style is used as fallback) / examples of use.
POST /v1/responses: OpenAI-compatible Responses API
Options:
See OpenAI Responses API documentation.
Examples:
You can use either Python openai library with appropriate checkpoints:
import openai
client = openai.OpenAI(
base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
api_key = "sk-no-key-required"
)
response = client.responses.create(
model="gpt-4.1",
instructions="You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.",
input="Write a limerick about python exceptions"
)
print(response.output_text)
... or raw HTTP requests:
curl http://localhost:8080/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"model": "gpt-4.1",
"instructions": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.",
"input": "Write a limerick about python exceptions"
}'
This endpoint works by converting Responses requests into Chat Completions requests.
POST /v1/embeddings: OpenAI-compatible embeddings API
*Options:*
See [OpenAI Embeddings API documentation](https://platform.openai.com/docs/api-reference/embeddings).
*Examples:*
-
input as string
curl http://localhost:8080/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer no-key" \ -d '{ "input": "hello", "model":"GPT-4", "encoding_format": "float" }' -
inputas string arraycurl http://localhost:8080/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer no-key" \ -d '{ "input": ["hello", "world"], "model":"GPT-4", "encoding_format": "float" }'
GET /slots: Returns the current slots processing state. Can be disabled with --slots-endpoint-disable.
Response format
[
{
"dynatemp_exponent": 1.0,
"dynatemp_range": 0.0,
"frequency_penalty": 0.0,
"grammar": "",
"id": 0,
"ignore_eos": false,
"logit_bias": [],
"min_p": 0.05000000074505806,
"mirostat": 0,
"mirostat_eta": 0.10000000149011612,
"mirostat_tau": 5.0,
"model": "llama-2-7b-32k-instruct.Q2_K.gguf",
"n_ctx": 2048,
"n_keep": 0,
"n_predict": 100000,
"n_probs": 0,
"next_token": {
"has_next_token": true,
"n_remain": -1,
"n_decoded": 0,
"stopped_eos": false,
"stopped_limit": false,
"stopped_word": false,
"stopping_word": ""
},
"penalize_nl": true,
"penalty_prompt_tokens": [],
"presence_penalty": 0.0,
"prompt": "Say hello to llama.cpp",
"repeat_last_n": 64,
"repeat_penalty": 1.100000023841858,
"samplers": [
"top_k",
"tfs_z",
"typical_p",
"top_p",
"min_p",
"temperature"
],
"seed": 42,
"state": 1,
"stop": [
"\n"
],
"stream": false,
"task_id": 0,
"temperature": 0.0,
"tfs_z": 1.0,
"top_k": 40,
"top_p": 0.949999988079071,
"typical_p": 1.0,
"use_penalty_prompt_tokens": false
}
]
GET /metrics: Prometheus compatible metrics exporter endpoint if --metrics is enabled:
Available metrics:
llamacpp:prompt_tokens_total: Number of prompt tokens processed.llamacpp:tokens_predicted_total: Number of generation tokens processed.llamacpp:prompt_tokens_seconds: Average prompt throughput in tokens/s.llamacpp:predicted_tokens_seconds: Average generation throughput in tokens/s.llamacpp:kv_cache_usage_ratio: KV-cache usage.1means 100 percent usage.llamacpp:kv_cache_tokens: KV-cache tokens.llamacpp:requests_processing: Number of requests processing.llamacpp:requests_deferred: Number of requests deferred.
POST /slots/{id_slot}?action=save: Save the prompt cache of the specified slot to a file.
*Options:*
`filename`: Name of the file to save the slot's prompt cache. The file will be saved in the directory specified by the `--slot-save-path` server parameter.
Response format
{
"id_slot": 0,
"filename": "slot_save_file.bin",
"n_saved": 1745,
"n_written": 14309796,
"timings": {
"save_ms": 49.865
}
}
POST /slots/{id_slot}?action=restore: Restore the prompt cache of the specified slot from a file.
*Options:*
`filename`: Name of the file to restore the slot's prompt cache from. The file should be located in the directory specified by the `--slot-save-path` server parameter.
Response format
{
"id_slot": 0,
"filename": "slot_save_file.bin",
"n_restored": 1745,
"n_read": 14309796,
"timings": {
"restore_ms": 42.937
}
}
POST /slots/{id_slot}?action=erase: Erase the prompt cache of the specified slot.
Response format
{
"id_slot": 0,
"n_erased": 1745
}
GET /lora-adapters: Get list of all LoRA adapters
If an adapter is disabled, the scale will be set to 0.
Response format
[
{
"id": 0,
"path": "my_adapter_1.gguf",
"scale": 0.0
},
{
"id": 1,
"path": "my_adapter_2.gguf",
"scale": 0.0
}
]
POST /lora-adapters: Set list of LoRA adapters
To disable an adapter, either remove it from the list below, or set scale to 0.
Request format
To know the id of the adapter, use GET /lora-adapters
[
{"id": 0, "scale": 0.2},
{"id": 1, "scale": 0.8}
]
More examples
Composite speculative decoding
Use repeated --spec-type SPEC[:k=v,...] entries for explicit stage chains. The currently supported two-stage shape is self-spec first, then mtp or draft fallback.
Example with ngram-mod plus MTP fallback:
./build/bin/llama-server \
--model /models/target-mtp.gguf \
--spec-type ngram-mod:n_max=64,n_min=2,ngram_size_n=8 \
--spec-type mtp:n_max=1,p_min=0.0
Example with ngram-mod plus draft-model fallback:
./build/bin/llama-server \
--model /models/target.gguf \
--model-draft /models/draft.gguf \
--spec-type ngram-mod:n_max=64,n_min=2,ngram_size_n=8 \
--spec-type draft:n_max=4,p_min=0.0
Notes:
- Use
--spec-typefor both single-stage and two-stage startup configuration. - Explicit stage chains currently support at most two stages.
Change system prompt on runtime
To use the server example to serve multiple chat-type clients while keeping the same system prompt, you can utilize the option system_prompt. This only needs to be used once.
prompt: Specify a context that you want all connecting clients to respect.
anti_prompt: Specify the word you want to use to instruct the model to stop. This must be sent to each client through the /props endpoint.
assistant_name: The bot's name is necessary for each customer to generate the prompt. This must be sent to each client through the /props endpoint.
{
"system_prompt": {
"prompt": "Transcript of a never ending dialog, where the User interacts with an Assistant.\nThe Assistant is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\nUser: Recommend a nice restaurant in the area.\nAssistant: I recommend the restaurant \"The Golden Duck\". It is a 5 star restaurant with a great view of the city. The food is delicious and the service is excellent. The prices are reasonable and the portions are generous. The restaurant is located at 123 Main Street, New York, NY 10001. The phone number is (212) 555-1234. The hours are Monday through Friday from 11:00 am to 10:00 pm. The restaurant is closed on Saturdays and Sundays.\nUser: Who is Richard Feynman?\nAssistant: Richard Feynman was an American physicist who is best known for his work in quantum mechanics and particle physics. He was awarded the Nobel Prize in Physics in 1965 for his contributions to the development of quantum electrodynamics. He was a popular lecturer and author, and he wrote several books, including \"Surely You're Joking, Mr. Feynman!\" and \"What Do You Care What Other People Think?\".\nUser:",
"anti_prompt": "User:",
"assistant_name": "Assistant:"
}
}
NOTE: You can do this automatically when starting the server by simply creating a .json file with these options and using the CLI option -spf FNAME or --system-prompt-file FNAME.
Interactive mode
Check the sample in chat.mjs. Run with NodeJS version 16 or later:
node chat.mjs
Another sample in chat.sh. Requires bash, curl and jq. Run with bash:
bash chat.sh
OAI-like API
The HTTP llama-server supports an OAI-like API: https://github.com/openai/openai-openapi
API errors
llama-server returns errors in the same format as OAI: https://github.com/openai/openai-openapi
Example of an error:
{
"error": {
"code": 401,
"message": "Invalid API Key",
"type": "authentication_error"
}
}
Apart from error types supported by OAI, we also have custom types that are specific to functionalities of llama.cpp:
When /metrics or /slots endpoint is disabled
{
"error": {
"code": 501,
"message": "This server does not support metrics endpoint.",
"type": "not_supported_error"
}
}
*When the server receives invalid grammar via /completions endpoint
{
"error": {
"code": 400,
"message": "Failed to parse grammar",
"type": "invalid_request_error"
}
}
Extending or building alternative Web Front End
You can extend the front end by running the server binary with --path set to ./your-directory and importing /completion.js to get access to the llamaComplete() method.
Read the documentation in /completion.js to see convenient ways to access llama.
A simple example is below:
<html>
<body>
<pre>
<script type="module">
import { llama } from '/completion.js'
const prompt = `### Instruction:
Write dad jokes, each one paragraph.
You can use html formatting if needed.
### Response:`
for await (const chunk of llama(prompt)) {
document.write(chunk.data.content)
}
</script>
</pre>
</body>
</html>