* hotswap: keep load-time-derived and transformed tensors coherent after reloads
- Re-derive the MLA combined attn_kv_b (computed_wkv_b) in place when a layer's
attn_k_b/attn_v_b are hot-swapped: the mla>1 prompt-processing path consumes
the derived tensor, so swaps of the source tensors previously had no effect
(KLD stayed exactly 0 in per-tensor benchmarks, e.g. GLM-5.2 attn_k_b/v_b).
- Refuse (loudly) hot-swaps that cannot be correct: views into -mqkv/-muge
merged tensors, khad-folded MLA weights, in-place-scaled ffn_gate_inp_s,
BitNet fused scales, OpenPangu parameter-sink sources, and same-dtype swaps
of mmap-backed tensors. A refused reload produces no 'reloaded tensor' line,
so benchmark drivers quarantine the round instead of recording wrong data.
- Propagate reloaded data to same-name duplicate instances (tied lm head copy
of token_embd, per-layer rope_freqs/rope_factors copies, expert-bias dups),
warning when a duplicate cannot be refreshed.
- Warn that derived state stays stale where a refresh is not possible:
pre-transposed wk_b_pp under -sm graph/attn, requantized MTP head
(output_extra.weight), and k_b/v_b derived from a reloaded attn_kv_b.
- Warn at registration time when -rtr is enabled (restores cannot reproduce
the run-time-repacked state; F16 -> BF16_R16 is lossy).
- server: only attempt the /health hot-swap reload when no slot is processing,
and clear the KV cache + cached prompts after a successful reload (they were
computed with the previous weights).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate llm_compute_wkv_b
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* perplexity: add signal-driven hot-swap mode for persistent KLD/PPL benchmarking
llama-perplexity can now stay resident and be driven through control/status files (reload/compute/exit): it reloads only the tensors that changed on disk and recomputes PPL/KLD without ever reloading the full model. File-based signalling works on Windows, macOS and Linux. The reload returning-to-original path now refreshes tensor data from disk instead of
reattaching stale weights.
* Not Cygwin specific
* host-swap tensor loop
the host-swap functionality is only triggered when the certain env. variables are declared
* target_include_directories tweak
* hot-swap tensor support
two intrusions:
1.) at the model loading to collect the snapshot
2.) the modification of the `/health` HTTP endpoint to be able to trigger the hot-swap via sending the `llama-server` the HTTP-request.
*both a braced by the specific env. variables
* hot-swap tensor support; graph invalidation
ggml_backend_cuda_invalidate_graphs export
* hot-swap tensor support
graph invalidation implementation; extended debug output (commented out)
* llama_reload_changed_tensors export
* tensor hot-swap on-demand reload
cpu-only/hybrid/gpu-only with split mode layer/graph full support implementation
* docs
* reuse the gguf parsing from llama.cpp
gguf_init_from_file, gguf_find_tensor, ggml_get_tensor
* remove the manual scheduling for hybrid inference
* update docs
* tensor shape validation
* update docs
* update docs
accidentally wiped the previous changes; so recovered them
* revert the GGML_CUDA_MAX_DEVICES to 16
* update llama_reload_changed_tensor
update llama_reload_changed_tensor, revert CMakeLists.txt
* update llama_reload_changed_tensor
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* llama_reload_changed_tensor
update llama_reload_changed_tensor definition
* refactory
move the tensor-reloading implementation to llama-reload.cpp, llama-reload-info.h; some bugfixes and code reduction
* revert
added back the missing newline
* update docs
* reload_info constructor
* bugfix: cpu-only
TODO: improve the working environment by compiling for multiple hardware configurations; possibly make a test pipeline
* cpu-only bugfix
set the fix again after unsuccessful sync with main
* windows os compilation fix
#include <string>
* fix windows os build
error C2039: 'string': is not a member of 'std'
* remove dead file
* implement perplexity in server
* Revert "implement perplexity in server"
