mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-08-12 22:31:11 +04:00
master
20
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
4ae84dea27 |
server: add more tool isolation support (ssh remote + podman rootless) (#26774)
* server: add an ssh transport to the tools runtime --tools-runtime ssh:<target> runs the built-in tools on a remote host, where target is whatever ssh already resolves, a user@host or a config alias, so no credentials live in llama.cpp. Only build_argv and upload differ from the docker transport: the remote shell re-parses the command line, so the argv travels through shell_quote_join, and files go over scp with the same quoting on the remote path. Authentication is key-based and the host key must already be trusted, since the tools run without a console and any prompt would hang them. The target is validated before use. The spec can reach us from the x-tool-runtime header, and a leading dash would turn it into an ssh option, which is enough to run a command back on the host. Nothing is created and nothing is reclaimed, so an ssh spec goes straight to the tool call instead of through the container runtime. Note that this is remoting rather than isolation: the tools can do whatever the target account can do, and the isolation is whatever runs them on the far side. * server: support podman in the tools runtime docker and podman expose the same run, exec, cp and inspect verbs with the same argument order, so a single implementation drives both and the engine is carried by the spec prefix: podman:<image> and podman-container:<id> sit next to the docker forms. tools_io_docker becomes tools_io_container and the runtime spawner becomes server_tools_container_runtime, both holding the client binary chosen at parse time. A single parse_container_runtime() resolves every spec, so adding another engine is one string in the table. make_tools_io() now rejects the spawning forms. The spec also reaches it from the x-tool-runtime header, which is client controlled, and only the runtime that owns a container is allowed to create one: a tool call can attach to a running container, nothing more. * ./build/bin/llama-gen-docs * server: simplify the tools runtime and drop the file copy step A server_tools_runtime base with one virtual spec() replaces the container runtime and the bare spec string that ssh needed next to it, so server_tools is back to a single pointer and neither setup nor the handler tests which of the two is set. write_file used to spill its content into a temporary file on the host and copy it in, because run_subprocess had no way to feed a child. It now takes an optional stdin payload and creates the parent directory and the file in a single round trip through a shell in the isolate. That removes the upload virtual and both implementations: no more container cp or scp, no second binary on the host, no sftp subsystem on the target, no predictable temporary in a shared tmp, and none of the content reaching an argv the remote shell re-parses. It also fixes write_file over ssh, which never worked: scp speaks sftp and takes the remote path literally, so quoting it kept the quotes in the file name. Writing the payload before reading the output relies on the child draining stdin as it goes, which holds for cat, its only user today. * ./build/bin/llama-gen-docs * server: harden the tools runtime against argv injection and a stdin stall Validate the container id from x-tool-runtime and --tools-runtime the same way the ssh target already is, so an id shaped like an option (docker-container:--privileged) is rejected before it reaches the engine's exec command line instead of running against a hardened container. Feed the child's stdin after the watchdog is armed, so a transport that stalls mid-write is terminated at the deadline rather than blocking the request forever. Cover both guards and fix the unknown-scheme test, which used ssh: as its example and now names a real runtime. * tests: exercise the tools runtime tests on podman as well as docker Follow-up #26507. The container runtime drives docker and podman through one implementation, so parametrize the availability helper, the container fixture and the attach test on the engine, and cover both engine prefixes in the container id injection test. Each engine skips on its own when it is not installed. The spawn cleanup test stays docker only: it recovers the spawned id from the container hostname, which docker sets to the short id and podman rootless does not guarantee. Podman keeps its coverage through the attach path. * server: release the container handle before respawning Follow-up #26507. create() writes over the handle it is given, so a respawn after the container died on its own leaked the pipes and the process handle of the previous one. * server: trim the tools runtime comments * server: read tool output as raw bytes and harden the runtime on Windows The stdout pipe is read with read() instead of fgets(), so a chunk can hold any byte, including NUL, and still streams as soon as data is available. Past the size cap the pipe keeps draining so the child never blocks on a full pipe. Both pipe fds are forced to binary mode on Windows, where the CRT defaults them to text mode and translates line endings in both directions. Stdin is now always closed after the feed: the child reads a deterministic EOF, and the Windows docker and ssh clients stop outliving their command on a stdin pipe that never closes. The attach form of --tools-runtime has no lifecycle to own, so it becomes a static target validated once at startup. This removes the subprocess that ran on every tool call and serialized calls behind a mutex; a stopped container now surfaces the engine's own error at exec time. The cidfile path is passed as UTF-8, matching the encoding the subprocess layer expects for the CreateProcessW command line, so the spawn form works from a non-ASCII Windows profile. The SIGPIPE note in server.cpp now names the tools runtime children as well as the MCP ones. * clean up comments * less pollute global scope * nits * tests: name the container image after both engines --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
|
dd2c7c4471 |
server: add initial tool isolation support (via docker) (#26507)
* server: add initial tool isolation support (via docker) * add docs * adapt get_info * py: fix type check * cont * separate tools_io_sandbox / tools_io_docker * rename sandbox --> isolate * x-tool-docker --> x-tool-runtime --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Xuan-Son NguyenandGitHub
|
f2b52a87e8 |
server: (tools) add x-tool-cwd header (#26420)
* server: (tools) add x-tool-cwd header * reuse str_to_lower from server-models |
||
|
PascalandGitHub
|
d73c1d6b22 |
server + ui: fix stream routes for model names containing a slash (#26137)
* server + ui: refactor resumable stream routes to query string conv_id The conversation id can embed a model name containing slashes (ggml-org/...) in router mode, which the decoded path splits before the :conv_id param is captured, so stop and resume never matched the session. Move the id to the conv_id query string on the public routes and on the internal router -> child hop, where slashes survive encoding. Handlers are unchanged since query and path params land in the same map. Add a regression test with a slashed model name. * server: move stream route docs to server-stream.h Address review: ngxson wants the main server.cpp registration code kept clean and simple, with route-level explanations living in the header. Move the query string rationale and the lookup ownership note next to the handler declarations in server-stream.h, and shorten the wiring comment to a pointer. * server: cancel a pending request when its stream is stopped during model load The conversation was registered in the conv map only after the blocking autoload wait, so a stop issued while the model loaded found nothing to cancel and the request went on to generate an orphan once the load ended. Register the conversation before the wait and give the entry a ticket: a stop erases the entry, and the parked request checks its ticket after the wait and aborts with 400 instead of starting. A newer request on the same conversation replaces the entry, so only the stopped request is cancelled. Add a regression test that stops during the load window. * server + ui: resume a stream after a page reload during model load A pending request died with the client socket when the page was reloaded while its model was loading, so no session ever existed and the conversation had nothing to recover. A session request that waited for a load now detaches from the client socket and reaches the child regardless, the session buffer receives the generation, and the resume route answers 503 while the owner is loading so the client retries instead of dropping its state. The WebUI persists the pending stream at send time, quietly polls on 503, and attaches once the session exists. Add a regression test that drops the client during the load window. * ui: show the model load progress again after a page refresh The resume wait was invisible, so a conversation refreshed while its model was loading showed nothing until the first byte. On a 503 from the resume probe, mark the conversation as loading again so the assistant row persisted at send time renders the processing info, and target the model frozen in the persisted stream state for the progress, since the row has no model yet and the dropdown may not be restored. * fix CI * fix CI bis |
||
|
|
20455a4ad3 |
server: support MCP stdio (#26062)
* move server_pipe to common * init impl * vendor: update subprocess.h * add server_mcp_stdio * stderr drain * server_mcp_transport * server_mcp_stdio is now framing-only, no json * internal/mcp-stdio: integration + tests + fixes (#26075) * server-mcp: harden transport and wire up the tool integration Builds on the transport/manager architecture (server_mcp_transport + server_pipe) with the hardening and integration the draft did not yet have. Hardening: * Reader and stderr pumps are polled (running-aware) instead of blocking on a read that only ends at EOF. subprocess_terminate() SIGKILLs only the direct child, so a grandchild the MCP server spawned that inherited the pipe would otherwise keep the write end open and hang teardown (both warmup shutdown at startup and process shutdown). The writer is likewise non-blocking + polled. * Windows: resolve the command through PATHEXT so "npx" (npm ships npx.cmd, never npx.exe) spawns, matching POSIX's PATH search; and enumerate the parent environment as UTF-8 (GetEnvironmentStringsW) instead of the active code page. * server_pipe gains an opt-in max_size (default unbounded, so the router's streaming use is unchanged); the MCP reply queue uses it so a server that streams unsolicited notifications between requests cannot grow it without bound. Integration: * --mcp-servers-config / --mcp-servers-json flags; enabling MCP restricts default CORS to localhost, same as --tools. * MCP tools are exposed through /tools (and chat-completions) as <server>_<tool>, skipping names that collide with a built-in or another MCP tool. * Manager lifecycle wired into llama_server(): warmup at start, shutdown() from the signal handler before the HTTP server drains, blocking teardown in clean_up(). * SIGPIPE ignored so a child dying mid-write yields EPIPE rather than killing us. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com> * server-mcp: add MCP test suite with grandchild deadlock regression test 21 tests over the /tools endpoint: tool discovery/invocation, timeouts, crash recovery and respawn cooldown, warmup partial failure, malformed and batched notification+response output, tool-definition shape, and prompt shutdown during a slow call. The last test spawns an MCP server that leaves a grandchild inheriting its stdout/stderr and asserts the server both starts and stops promptly. Verified it fails (5s SIGKILL fallback on a deadlocked reader-join) when the pump is made to ignore the running flag, and passes with the polled reader. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com> * clean up * clean up 2 * even stricter life cycle * nits * nits 2 --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> * fix some edge cases * fix last_error data race * fix response schema + docs * server: fix MCP zombie leak and timeout-induced transport teardown join_pumps() never reaped the child, leaking one zombie per spawn: call subprocess_join() before subprocess_destroy(). A per-call timeout permanently closed from_server and got a healthy transport evicted: add close_on_stop to server_pipe::read() and pass false from send_rpc(), where should_stop is a per-request deadline and a late reply is already skipped on id mismatch. Also drop the unreachable disconnect cancellation in server_mcp_tool::invoke(): support_stream is false, st is always null. (cherry picked from commit e6de1ec043174fd0570b1e60d47f06c7c19d620d) Assisted-by: Claude Opus 4.8 * server: make MCP test fixtures JSON-RPC 2.0 compliant Add the missing notification guard to mcp_malformed_server.py and mcp_burst_server.py (the latter treated id 0 as a notification and replied to unknown ones; its notification table is now unused). Return -32602 instead of -32601 for unknown tools: tools/call is a valid method, the tool name is the invalid parameter. Also fix the test module docstring: tools are named <server>_<tool>. (cherry picked from commit 74a08e8c311dabf3b49d06cc6d754b0097ae7a38) Assisted-by: Claude Opus 4.8 --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Xuan-Son NguyenandGitHub
|
c92e806d1c |
server: allow stream for exec_shell_command (#25526)
* init stream * add stream for shell tool * add test * nits * update docs |
||
|
|
ea1f7bbb5d |
server: refactor server_stream (#25541)
* server: refactoring, remove spipe from server_http_res * wip * remove non-thread-safe rd.stop() call * move server_res_spipe * nits * improve server_stream_create_spipe * server-stream: update dev docs for the improved API --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
|
bbebeec4a8 |
server-stream: follow-up on SSE Replay Buffer (#23226) (#25047)
* server-stream : pimpl * server-stream: prefix free functions with server_stream_ address review from ggerganov: scope the public stream functions under the server_stream_ prefix, matching server_stream_session_manager_start/stop. * server-stream: guard session and manager state with the mutex address review from ggerganov: make done, completed_ts and the GC running flag plain members under their mutex and set the condvar predicates under the lock. keep cancelled atomic for the lock-free should_stop poll. * server-stream: trim comments to the non-obvious address review from ggerganov: drop comments that restate the code, keep the concurrency, lifetime and ordering rationale. de-stale a few comments left by the pimpl: g_stream_sessions is now internal and the /v1/streams listing is gone. * server-stream: update dev docs for the pimpl and prefix reflect server_stream_session_manager_start/stop and the server_stream_ prefix, note the manager is now a file-static singleton hidden in the .cpp * server-stream: move stream traces to debug level keep the bring-up traces for diagnostics but off the default log: skip drain, draining, drain ended, DELETE evict, attach_pipe, and the router stream resume proxy. * server-stream: align router stream resume proxy trace with upstream the child-side bring-up traces are already SRV_TRC on master, move the router stream resume proxy trace to the same level. * server-stream: move stream_read_status enum to the cpp it is only used by the hidden session and consumer types, so it belongs with them behind the pimpl boundary, not on the public header surface. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
|
1a87dcdc45 |
server + ui: SSE Replay Buffer (#23226)
* server: SSE replay buffer, survives client disconnect
Opt in on POST /v1/chat/completions when the client sends
X-Stream-Resume: 1 and a non empty X-Conversation-Id. The conv id is
the session identity end to end, no extra opaque token. The drain
runs detached server side and buffers SSE bytes, the generation
survives HTTP disconnect, F5, or lets users switch from iOS Safari
to another app without losing the actively generated response.
Routes:
GET /v1/stream/<conv_id>?from=N replay
GET /v1/streams[?conversation_id=X] list, drives sidebar spinners
DELETE /v1/stream/<conv_id> Stop, idempotent
Router parent fans out to children for list and delete, probes on GET
to route to the owner, fans out DELETE on POST so "one session per
conv" holds across model swaps.
WebUI: the layout snapshots /v1/streams at mount and on
visibilitychange, the sidebar reflects live inferences across all
convs. The chat page reattaches on mount, append vs fresh is detected
from existing content so continue mid stream keeps its prefix.
update_slots: on llama_memory_seq_rm refusal at a deep position, full
clear of the seq and reprefill from zero instead of GGML_ABORT.
OAI strict path unchanged when the opt in headers are absent.
* server: create stream session only after post_tasks succeeds
* server, ui: drop X-Stream-Resume, X-Conversation-Id alone enables the replay buffer
* server: drop magic 17, derive the X-Conversation-Id header length from sizeof at build time
* refactor: address review feedback from ngxson
* server-context: cleaning
* server-stream: fix use-after-free on rd
Guard stop_producer with a shared alive flag, flipped by on_stream_end
before rd dies. Prevents a late cancel (session eviction by a later
POST on the same conv_id, or a DELETE arriving after the producer
ended) from touching a destroyed rd.
* ui: fix cross-conversation contamination
Scope streaming flags per conv so one finishing does not unflag the
others, guard discoverActiveStream against concurrent runs to avoid
duplicate attaches, and stop racing syncRemoteRunningStreams for the
sidebar set.
* server-http: keep request alive in detached SSE drain
The response next() lambda may reach into *request via &req long
after on_complete reset the request shared_ptr. Capture request in
the detached thread so it outlives the drain.
* ui: address review feedback from coder543
Forward Authorization to /v1/stream and /v1/streams fetches, the resumable routes
must obey --api-key like the rest of the API.
Wrap reader.read() in a try/catch, the underlying connection drop rejects with
TypeError instead of resolving done=true, treat it as a premature end of stream
so the existing resume loop kicks in.
Freeze the model at session start in chatStreamingStates.model and thread it
through cancel and resume, the dropdown selection may have changed since the
POST and the server side identity is fixed at that time.
* format
* ui: remove unused selectedModelName
* server-stream: poll session->is_cancelled() in stream_aware_should_stop
Address review feedback from coder543. The cancel propagation through
rd.stop() relies on the slot eventually processing the cancel task and
posting a result that notifies the recv condvar, remove_waiting_task_ids
does not notify directly. Add a defensive poll on session->is_cancelled()
so the producer-side next() loop exits on its next iteration after
cancel() without waiting for the cancel task to round trip through a slot.
* server-stream, ui: replace GET /v1/streams with POST /v1/streams/lookup
Address review feedback from coder543. Listing live sessions leaks the
conversation_id of every concurrent user, which defeats the random UUID
unguessability. The new route takes {conversation_ids: [...]} in the
body and returns matches only for the ids the caller already owns, so
foreign UUIDs stay private. The router fans out the same POST to every
child and aggregates, the WebUI passes the convs visible in its sidebar.
* ui: read conv ids from IndexedDB in syncRemoteRunningStreams
The conversations store is not hydrated yet at +layout onMount, so the
sidebar spinners stayed off for background convs until the user clicked
on them. Read straight from the DB to dodge the init race.
* server-models: deduplicate stream lookup timeouts behind one constant
* ui: extract visibility kick grace into a stream constant, bump to 1000 ms
* make it safer & more simple
* server-stream: survive client disconnect via stream_pipe::finish_producer
After the RAII rewrite the generation stopped the moment the client
disconnected. httplib bails its content provider on the is_peer_alive
check at the top of write_content_chunked, so returning true from the
provider never keeps it producing: the response resets, rd is destroyed
and its task gets cancelled.
Reinstate the disconnect survival inside the pipe. stream_pipe gains
finish_producer, which pumps the response next() into the ring buffer
until the generation ends, and mark_producer_done for the clean wire
end. server-http only triggers them: mark before sink.done on a clean
close, finish in on_complete when the peer left early. No detach, no
stream logic in server-http beyond the trigger, and the strict OAI path
is untouched when no pipe is attached.
Known limitation: finish_producer pumps synchronously on the http
worker, so a disconnected stream keeps its worker busy until the
generation ends. A follow-up will move the drain off the http worker so
no worker is held.
* server-stream: drain disconnected streams on a manager owned thread
The previous commit pumped the post disconnect drain synchronously in
on_complete, on the http worker, so a disconnected stream kept its
worker busy until the generation ended. Under a wave of reloads or tab
closes that pins workers from the pool.
Move the drain off the http worker. on_complete now hands the response
to stream_session_manager::adopt_orphan, which pumps it to completion on
a manager owned thread and releases the worker at once. One thread per
disconnected stream still generating, stored in a list, joined and
reaped on the next adopt, by the GC, and at shutdown. No detach, the
thread lifecycle is fully owned by the manager. needs_drain gates the
handoff so a cleanly finished stream never spawns a thread, and the
strict OAI path stays untouched when no pipe is attached.
stop_gc now cancels sessions before finalizing them, so an in flight
drain sees is_cancelled and exits instead of blocking the shutdown join
until the generation ends naturally.
* ui: add missing JSDoc
* server-stream: drain on the http worker, drop the manager thread
Address @ngxson review: httplib runs a large dynamic pool and a worker
blocked in next() sits on a condvar instead of burning cpu, so draining
the rest of the generation on that worker is fine and much simpler than
a dedicated thread.
on_complete calls finish_producer directly again. Removes adopt_orphan,
the orphan thread list and its reaping, the stop_gc session cancel that
only existed to unblock those threads, and the now dead drain_shutdown
flag.
* server-stream: split stream_pipe into producer and consumer classes
Address @ngxson review: one class covering both ends was messy. stream_pipe
is now a base holding the session and is_cancelled, with stream_pipe_producer
(write, mark_producer_done, finish_producer, cleanup, finalizes on destruct)
and stream_pipe_consumer (read only, no finalize) deriving from it.
Drops the is_producer_ discriminator and its runtime guards, the type now
encodes the role. res.spipe is retyped to shared_ptr<stream_pipe_producer>
since it is only ever a producer. No behavior change.
* server-stream: rename producer methods to unix pipe semantics
Address @ngxson review: mark_producer_done becomes done(), finish_producer
becomes close(), matching a unix pipe write end. The producer_done_ member
follows as done_. write() is unchanged. No behavior change.
* server, ui: route resumable streams via a conv map, persist resume identity
Address ngxson review: drop the polling probe, proxy_post records a conv_id ->
model map and the stream routes resolve the owning child with one lookup. The
map is the single source of truth, the ::model suffix stays for child session
uniqueness but the router never parses it.
UI: the server keys a session by the POST time identity (conv::model), but reload
probed with the bare conv id and missed model tagged sessions, so F5 stopped the
stream and sidebar spinners stayed off. Persist the model and rebuild the exact
identity on resume, single conv and bulk sidebar both send it.
Add unit coverage for the identity round trip.
* ui: resolve continue target by id to stop cross-conversation flash on switch
* ui: skip stream resume when the abort is intentional
* server: move the conv id to model map into a self contained tracker
Address review from ngxson: server_models held two mutexes side by side, the
global one and a bare conv_model_mu guarding a loose map, which made the locking
hard to follow. Wrap the map and its lock in a small conv_model_tracker struct
that owns its mutex, one mutex per struct. The remember, lookup and forget
methods move inline into the tracker, server_models exposes a single conv_models
member and the routes call models.conv_models.lookup and friends. No behavior
change, the map stays the single source of truth for routing resumable streams
to a child.
* ui: replace stream magic values with enums and shared constants
Address review from allozaur: lift the inline literals around the resumable
stream code into named symbols so the intent is explicit and reusable.
* ui: fold the stream resume and discovery helpers into ChatService
Address review from allozaur: drop the two standalone stream-*.service files.
They were used only by the chat service and store, carried no shared state, and
did not follow the static class pattern the other services use, so a separate
abstraction was not warranted. Move the helpers onto ChatService as static
methods. No behavior change, tests now exercise them through ChatService.
* docs: document the SSE replay buffer in server README-dev
Add the resumable streaming section, list stream_session_manager in the
backend component inventory, and link PR 23226 in the related PRs.
* ui: align attachServerStream call with onCompletionId param in handleStreamResponse
* server-http: rename del_ to del to match get and post
* ui: address review feedback from allozaur
* ui: drop duplicate SSE constants, keep sse.ts canonical
* ui: use svelte:document for the visibilitychange listener
address review from allozaur: replace the manual document.addEventListener
in onMount with a declarative <svelte:document onvisibilitychange>. svelte
handles attach, detach and SSR, so the typeof document guard and the onMount
cleanup go away. onMount keeps only the first load snapshot.
* server: trim redundant stream drain comments
Address review from ngxson
* server: balance and clean up stream comments
remove redundant comments and tighten the verbose ones across the resumable
stream code, keeping the concurrency and lifetime rationale that is not obvious
from the code. also fix two stale comments in server.cpp and server-models.h
that still described the old ::model suffix probe and fan out routing, now
replaced by the conv_id -> model map
Address review from ngxson
* ui: balance and clean up stream comments
dedup repeated rationale (frozen conv::model identity, the lookup privacy note,
the abort patterns) down to one canonical spot, tighten the verbose blocks, and
keep the concurrency and resume-offset reasoning. fix stale comments in
stream-identity.ts and chat.service.ts that still described the old loopback
probe and fan out routing, now the conv_id -> model map.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
|
||
|
Xuan-Son NguyenandGitHub
|
721354fbdf |
server: (router) move model downloading to dedicated process (#24834)
* server: real-time model load progress tracking via /models/sse * update docs * server: move model download to child process * rm unused * fix most problems * clean up * nit fixes * fix test case * do not detact() thread * shorter MODEL_DOWNLOAD_TIMEOUT in test * throttle |
||
|
Xuan-Son NguyenandGitHub
|
2b686a9120 |
server: refactor child --> router communication (#24821)
* server: refactor child --> router communication * fix wakeup case * add docs * improve update_status() * nits |
||
|
Xuan-Son NguyenandGitHub
|
4b4d13ae72 |
server: (router) add model management API (#23976)
* wip * server: (router) add SSE realtime updates API * nits * wip * add download API * add download api * update docs * add delete endpoint * fix std::terminate * fix crash * fix 2 * add tests * nits |
||
|
|
59778f0196 |
ui: Restructure repo to use tools/ui folder and ui / UI / llama-ui / LLAMA_UI naming (#23064)
* webui: Move static build output from `tools/server/public` to `build/ui` directory * refactor: Move to `tools/ui` * refactor: rename CMake variables and preprocessor defines - Rename LLAMA_BUILD_WEBUI -> LLAMA_BUILD_UI (old kept as deprecated) - Rename LLAMA_USE_PREBUILT_WEBUI -> LLAMA_USE_PREBUILT_UI (old kept as deprecated) - Backward compat: old vars auto-forward to new ones with DEPRECATION warning - Rename internal vars: WEBUI_SOURCE -> UI_SOURCE, WEBUI_SOURCE_DIR -> UI_SOURCE_DIR, etc. - Rename HF bucket: LLAMA_WEBUI_HF_BUCKET -> LLAMA_UI_HF_BUCKET - Emit both LLAMA_BUILD_WEBUI and LLAMA_BUILD_UI preprocessor defines - Emit both LLAMA_WEBUI_DEFAULT_ENABLED and LLAMA_UI_DEFAULT_ENABLED * refactor: rename CLI flags (--webui -> --ui) with backward compat - Add --ui/--no-ui (old --webui/--no-webui kept as deprecated aliases) - Add --ui-config (old --webui-config kept as deprecated alias) - Add --ui-config-file (old --webui-config-file kept as deprecated alias) - Add --ui-mcp-proxy/--no-ui-mcp-proxy (old --webui-mcp-proxy kept as deprecated) - Add new env vars: LLAMA_ARG_UI, LLAMA_ARG_UI_CONFIG, LLAMA_ARG_UI_CONFIG_FILE, LLAMA_ARG_UI_MCP_PROXY - C++ struct fields: params.ui, params.ui_config_json, params.ui_mcp_proxy added alongside old fields - Backward compat: old fields synced to new ones in g_params_to_internals * refactor: update C++ server internals with backward compat - Rename json_webui_settings -> json_ui_settings (both kept in server_context_meta) - Rename params.webui usage -> params.ui (both synced, old still works) - JSON API emits both "ui"/"ui_settings" and "webui"/"webui_settings" keys - Server routes use params.ui_mcp_proxy || params.webui_mcp_proxy - Preprocessor guards use #if defined(LLAMA_BUILD_UI) || defined(LLAMA_BUILD_WEBUI) * refactor: rename CI/CD workflows, artifacts, and build script - Rename webui-build.yml -> ui-build.yml; artifact webui-build -> ui-build - Rename webui-publish.yml -> ui-publish.yml; var HF_BUCKET_WEBUI_STATIC_OUTPUT -> HF_BUCKET_UI_STATIC_OUTPUT - Rename server-webui.yml -> server-ui.yml; job webui-build/checks -> ui-build/checks - Update server.yml: job/artifact refs webui-build -> ui-build - Update release.yml: all webui-build/publish refs -> ui-build/publish; HF_TOKEN_WEBUI_STATIC_OUTPUT -> HF_TOKEN_UI_STATIC_OUTPUT - Update server-self-hosted.yml: webui-build -> ui-build - Update build-self-hosted.yml: HF_WEBUI_VERSION -> HF_UI_VERSION - Rename webui-download.cmake -> ui-download.cmake (internal refs updated) - Update labeler.yml: server/webui -> server/ui path label * docs: update CODEOWNERS and server README docs - Update CODEOWNERS: team ggml-org/llama-webui -> ggml-org/llama-ui, path /tools/server/webui/ -> /tools/ui/ - Update server README.md: CLI tables show --ui flags with deprecated --webui aliases - Update server README-dev.md: "WebUI" -> "UI", paths updated to tools/ui/ * fix: Small fixes for UI build * fix: CMake.txt syntax * chore: Formatting * fix: `.editorconfig` for llama-ui * chore: Formatting * refactor: Use `APP_NAME` in Error route * refactor: Cleanup * refactor: Single migration service * make llama-ui a linkable target * fix: UI Build output * fix: Missing change * fix: separate llama-ui npm build output into build/tools/ui/dist subfolder + use cmake npm build instead of downloading ui-build.yml artifacts in CI * refactor: UI workflows cleanup --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
|
4a00bbfed6 |
server: (webui) no more gzip compression (#21073)
* webui: no more gzip
* try changing a small line
* Revert "try changing a small line"
This reverts commit
|
||
|
Xuan-Son NguyenandGitHub
|
20197b6fe3 |
server: add built-in tools backend support (#20898)
* wip: server_tools * refactor * displayName -> display_name * snake_case everywhere * rm redundant field * change arg to --tools all * add readme mention * llama-gen-docs |
||
|
|
fb78ad29bb |
server: (doc) clarify in-scope and out-scope features (#20794)
* server: (doc) clarify in-scope and out-scope features * Apply suggestions from code review Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
Xuan-Son NguyenandGitHub
|
ddcb75dd8a |
server: add auto-sleep after N seconds of idle (#18228)
* implement sleeping at queue level * implement server-context suspend * add test * add docs * optimization: add fast path * make sure to free llama_init * nits * fix use-after-free * allow /models to be accessed during sleeping, fix use-after-free * don't allow accessing /models during sleep, it is not thread-safe * fix data race on accessing props and model_meta * small clean up * trailing whitespace * rm outdated comments |
||
|
Xuan-Son NguyenandGitHub
|
951520ddb0 |
server: delegate result_state creation to server_task (#17835)
* server: delegate result_state creation to server_task * remove unued states * add more docs |
||
|
|
f896d2c34f |
server: improve speed of speculative decoding (#17808)
* server: improve speed of speculative decoding * fix small draft case * add link to the PR * server : fix generation time measurement * server : fix draft acceptance logs (add SRV_CNT, SLT_CNT macros) * server : add comment * add PR to docs --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
Xuan-Son NguyenandGitHub
|
37a4f63244 |
server : add development documentation (#17760)
* first draft * rewrite * update & remove duplicated sections |