* server: add read_image tool (#25875)
Adds a server-tool that allows vision models to analyze server-side images.
This tool is reading a single file for now:
The image data is base64 encoded and passed to the UI, which
decodes it, fills the <img> tag and removes the data URI before
passing the tool result back to the model.
* cleanup read_image tool: move magic strings to constants
* Add dedicated constants file: tools/ui/src/lib/constants/read-image.ts
with PREFIX_IMAGE, PREFIX_SIZE, PREFIX_MIME constants
* Use ATTACHMENT_SAVED_REGEX from agentic.ts in ChatMessageToolCallBlockReadImage.svelte
* Use NEWLINE constant from code.ts instead of hardcoded '\n'
* Use PREFIX_SIZE in regex pattern for size parsing
* Add SERVER_TOOL_READ_IMAGE_PREFIX_* constants in C++ server-tools.cpp
to match the TypeScript PREFIX_* constants for consistency
* server: rename read_image tool to read_media for images and audio
* Rename server_tool_read_image to server_tool_read_media in C++
* Rename enum BuiltInTool.READ_IMAGE to READ_MEDIA
* Rename UI constants, parser, and Svelte component files
* Update display label from 'Read image' to 'Read media'
* ui: consolidate audio data URI handling into shared utility
* Extract getAudioInputFormat to a shared utility (was duplicated inline)
* Store raw base64 in base64Data on the message object
* Use base64Data to construct data URIs for audio rendering
* Update agentic store to build INPUT_AUDIO parts from base64Data
* server: read_media: restrict audio to wav/mp3 and minor fixes
* Server get_mime_from_extension now only advertises audio/wav and
audio/mpeg (the only formats the model's input_audio API accepts)
* Case-insensitive extension matching (fixes .MP3, .Wav, etc.)
* Unknown extensions return an error instead of a multi-MB data URI
that inflates model context with garbage
* Updated tool description to document supported formats
* Frontend AUDIO_MIME_TO_EXTENSION trimmed to match server
* fix a missing import in tools/ui/src/lib/stores/agentic.svelte.ts
* server: read_media: add to --tools help text and README tool list
* ui: fix indentation in ChatMessageToolCallBlockDefault.svelte
* server: read_media tool: fix a cast to use the correct type
* server: read_media: multiple fixes
* server-tools.cpp import cctype, remove UTF-8 char, check mime before reading file
* ui: add MimeTypePrefix.AUDIO and use it in agentic.svelte.ts
* server: make read_media inherit from read_file and add uses_cwd
* ui: fix formating issues
* rm from server
* move it to frontend-only tool
* correct partial commit
* rm unused
* ui: address review from allozaur
Replace the magic strings, regexes and number in the read_media parser
and service with named constants. Path splitting reuses
FILE_PATH_SEPARATOR_REGEX, the size header regex moves to
READ_MEDIA_SIZE_REGEX derived from PREFIX_SIZE, and
FILE_EXTENSION_SEPARATOR lands next to it in constants/code.ts.
---------
Co-authored-by: ckrafft <ckrafft@epyc>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* server: add an ssh transport to the tools runtime
--tools-runtime ssh:<target> runs the built-in tools on a remote host,
where target is whatever ssh already resolves, a user@host or a config
alias, so no credentials live in llama.cpp.
Only build_argv and upload differ from the docker transport: the remote
shell re-parses the command line, so the argv travels through
shell_quote_join, and files go over scp with the same quoting on the
remote path. Authentication is key-based and the host key must already
be trusted, since the tools run without a console and any prompt would
hang them.
The target is validated before use. The spec can reach us from the
x-tool-runtime header, and a leading dash would turn it into an ssh
option, which is enough to run a command back on the host.
Nothing is created and nothing is reclaimed, so an ssh spec goes
straight to the tool call instead of through the container runtime.
Note that this is remoting rather than isolation: the tools can do
whatever the target account can do, and the isolation is whatever runs
them on the far side.
* server: support podman in the tools runtime
docker and podman expose the same run, exec, cp and inspect verbs with the
same argument order, so a single implementation drives both and the engine
is carried by the spec prefix: podman:<image> and podman-container:<id> sit
next to the docker forms.
tools_io_docker becomes tools_io_container and the runtime spawner becomes
server_tools_container_runtime, both holding the client binary chosen at
parse time. A single parse_container_runtime() resolves every spec, so
adding another engine is one string in the table.
make_tools_io() now rejects the spawning forms. The spec also reaches it
from the x-tool-runtime header, which is client controlled, and only the
runtime that owns a container is allowed to create one: a tool call can
attach to a running container, nothing more.
* ./build/bin/llama-gen-docs
* server: simplify the tools runtime and drop the file copy step
A server_tools_runtime base with one virtual spec() replaces the
container runtime and the bare spec string that ssh needed next to it,
so server_tools is back to a single pointer and neither setup nor the
handler tests which of the two is set.
write_file used to spill its content into a temporary file on the host
and copy it in, because run_subprocess had no way to feed a child. It
now takes an optional stdin payload and creates the parent directory
and the file in a single round trip through a shell in the isolate.
That removes the upload virtual and both implementations: no more
container cp or scp, no second binary on the host, no sftp subsystem on
the target, no predictable temporary in a shared tmp, and none of the
content reaching an argv the remote shell re-parses. It also fixes
write_file over ssh, which never worked: scp speaks sftp and takes the
remote path literally, so quoting it kept the quotes in the file name.
Writing the payload before reading the output relies on the child
draining stdin as it goes, which holds for cat, its only user today.
* ./build/bin/llama-gen-docs
* server: harden the tools runtime against argv injection and a stdin stall
Validate the container id from x-tool-runtime and --tools-runtime the
same way the ssh target already is, so an id shaped like an option
(docker-container:--privileged) is rejected before it reaches the
engine's exec command line instead of running against a hardened
container. Feed the child's stdin after the watchdog is armed, so a
transport that stalls mid-write is terminated at the deadline rather
than blocking the request forever.
Cover both guards and fix the unknown-scheme test, which used ssh: as
its example and now names a real runtime.
* tests: exercise the tools runtime tests on podman as well as docker
Follow-up #26507. The container runtime drives docker and podman
through one implementation, so parametrize the availability helper,
the container fixture and the attach test on the engine, and cover
both engine prefixes in the container id injection test. Each engine
skips on its own when it is not installed.
The spawn cleanup test stays docker only: it recovers the spawned id
from the container hostname, which docker sets to the short id and
podman rootless does not guarantee. Podman keeps its coverage through
the attach path.
* server: release the container handle before respawning
Follow-up #26507. create() writes over the handle it is given, so a
respawn after the container died on its own leaked the pipes and the
process handle of the previous one.
* server: trim the tools runtime comments
* server: read tool output as raw bytes and harden the runtime on Windows
The stdout pipe is read with read() instead of fgets(), so a chunk
can hold any byte, including NUL, and still streams as soon as data
is available. Past the size cap the pipe keeps draining so the child
never blocks on a full pipe. Both pipe fds are forced to binary mode
on Windows, where the CRT defaults them to text mode and translates
line endings in both directions. Stdin is now always closed after
the feed: the child reads a deterministic EOF, and the Windows
docker and ssh clients stop outliving their command on a stdin pipe
that never closes.
The attach form of --tools-runtime has no lifecycle to own, so it
becomes a static target validated once at startup. This removes the
subprocess that ran on every tool call and
serialized calls behind a mutex; a stopped container now surfaces
the engine's own error at exec time.
The cidfile path is passed as UTF-8, matching the encoding the
subprocess layer expects for the CreateProcessW command line, so
the spawn form works from a non-ASCII Windows profile.
The SIGPIPE note in server.cpp now names the tools runtime children
as well as the MCP ones.
* clean up comments
* less pollute global scope
* nits
* tests: name the container image after both engines
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: report the isolate working directory from get_info
Without an explicit cwd, get_info fell back to the server process
working directory even when a tools runtime was configured. That named a
host path no tool would ever run in, since an isolate starts in a
directory of its own.
It now asks the isolate for its working directory in that case, and
keeps the process one only when the tools run on the host.
* remove redundant comment
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
The working directory chip showed up as soon as the server exposed any
builtin tool, so a server started with just get_datetime, or a user who
turned every filesystem tool off in the settings, still got a control
that nothing would read.
Tools now declare whether they resolve their paths and run against the
working directory, next to the write permission they already publish in
the /tools listing. The WebUI shows the chip and enables the /cwd
command only when at least one such tool is both served and left
enabled.
* server: don't walk Windows junctions in file_glob_search
std::filesystem reports a junction as a plain directory, so the symlink
guard misses it and a junction pointing back at an ancestor is walked
until the path length gives out
read the reparse tag and treat a symlink and a mount point as links,
leaving any other reparse point walkable so cloud placeholders and dedup
stubs still get searched
look junk directory names up case insensitively on Windows, where NTFS
makes Build the same directory as build
test that a junk directory stays selectable while its contents stay out
of search results
* server: report a directory the walk could not read
a directory that fails to open or to iterate was skipped in silence, so
a caller got a listing that looked complete while a whole subtree was
missing: a path over the platform limit, a volume going away, a name the
filesystem rejects
skip_permission_denied never reaches this path, so an error here is an
incomplete answer rather than a deliberate omission, and it now sets the
truncated flag
* server: simplify the file_glob_search listing plumbing
return a small result struct instead of two out params and a caller path
that only fed an error string, taking list_entries from six parameters
down to three
scope the error code to the directory being read, act on the status code
the entry lookups already returned, and treat an unreadable link state as
a link so the walk never descends on a guess
check the deadline when a directory is popped, not only per entry, so a
tree of empty directories cannot outlive the budget
read the path parameter once, and reject an invalid limit the way an
invalid type is already rejected, instead of silently falling back
normalize the resolved path, so a "." or ".." a caller typed reaches
neither git nor the client, and return the generic path form with '/'
separators on every platform, so the base sent to clients no longer needs
a local fixup
* ui: expire cached picker searches
the cache grew for the lifetime of the component: entries went stale
after the TTL but were never removed, so every distinct query typed in a
session stayed in memory
drop expired entries when a new result is stored
* server: address review from @ngxson
trim comments to one line each, and drop two that restate the code
rename junk_lookup_name to get_effective_name, and move it and the link
check to private static members next to junk_dir_names
merge the Windows and Linux link checks into one is_link, so symlinks are
checked everywhere and junctions only add to it on Windows
* server: convert tool paths as UTF-8 on Windows
a narrow path uses the active code page there, so a file name came back
mangled and a path with an accent could not be opened at all
convert explicitly at every crossing between a std::string, which always
carries UTF-8 here, and fs::path
read the home directory through the wide environment, since the narrow
one returns the profile path in the active code page too
the walker no longer normalizes separators by hand, since paths now come
back in generic form
* server: fold the platform branch inside console_output_to_utf8
match the shape of the other helpers, one definition with the #if inside,
instead of two definitions wrapped in #if and #else
inline the single caller helper and trim the comment
a child process writes in the OEM code page, which is not UTF-8 on a
western Windows install, so accented output reaches the JSON layer as
invalid bytes and gets replaced there, silently losing the characters
run() spawns without a console, so the child never inherits the console
code page and GetOEMCP is the one that applies
decode with MB_ERR_INVALID_CHARS so a wrong code page returns the text
untouched instead of emitting replacement characters, and pass text that
already decodes as UTF-8 through so a child emitting UTF-8 is never
decoded twice
the check drops an incomplete trailing sequence before validating, since
a streamed chunk can end in the middle of a multi-byte character
* server : extend file_glob_search for UI pickers
* ui : add per-conversation working directory with picker
* ui : add path navigation and search scope to cwd picker
Treat path-like queries (starting with / or ~) as directory navigation
instead of glob-matching the whole query: search the parent for the last
segment, and descend into an exactly-typed directory by listing its
children. Show the effective search scope in the footer and auto-search
on open so the current directory and its siblings appear immediately.
Assisted-by: Claude
* db : persist per-call tool cwd on tool result messages
* ui : abbreviate tool paths under home with a tilde
* ui : show the per-call cwd on exec shell rows
* ui : clarify the synthetic cwd message for the model
* ui : reuse the trailing cwd row on a repeated pick
* ui : don't jump when a cwd row is injected mid-chat
* chore: Formatting
* refactor: Cleanup comments
* ui : unify working directory naming and add a synthetic-message flag
* ui : render synthetic cwd rows without a scroll jump
* ui : decouple the working directory picker into utils and sub-components
* ui : add get_info tool call block
* chore: Formatting
* refactor: Cleanup
* refactor: Cleanup
* refactor: Cleanup
* fix: UI
* server : harden file_glob_search listing (kind enum, timeout, symlink guard, absolute base)
* ui : use persisted isSynthetic flag for cwd rows, drop legacy formats
* ui : cache picker search, fail visibly on native resolve
* ui : escape glob metacharacters in picker search glob
* ui : simplify auto-scroll pin
* chore: Format
* fix: Use `SvelteMap`
* refactor: Post-review fixes
* ui: accept Windows roots in the working directory picker
recognize a drive root (C:) and a UNC share (//host/share) as path
navigation, alongside the POSIX root and ~, so a query like D:\repos
lists that directory instead of glob-matching it under the home dir
split below the root, so a bare drive resolves to its root rather than
to a drive-relative prefix
rewrite backslashes into forward slashes only when the query carries a
Windows root, since a backslash is a legal POSIX filename character
paths keep travelling with forward slashes, which is what the server
returns and what Windows accepts
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server: add get_info tool
* fix --rpc in docs
* server: harden get_info probe result handling
Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.
* server: simplify get_info result handling
Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* add common/subproc.h|cpp
* add compile flag LLAMA_SUBPROCESS
* disabled by default on android and ios
* test-jinja: use common subproc
* mtmd: disable video if subproc is not set
* disable subproc on wasm
* make is_created atomic
* migrate server-mcp
* move server_pipe to common
* init impl
* vendor: update subprocess.h
* add server_mcp_stdio
* stderr drain
* server_mcp_transport
* server_mcp_stdio is now framing-only, no json
* internal/mcp-stdio: integration + tests + fixes (#26075)
* server-mcp: harden transport and wire up the tool integration
Builds on the transport/manager architecture (server_mcp_transport + server_pipe)
with the hardening and integration the draft did not yet have.
Hardening:
* Reader and stderr pumps are polled (running-aware) instead of blocking on a read
that only ends at EOF. subprocess_terminate() SIGKILLs only the direct child, so a
grandchild the MCP server spawned that inherited the pipe would otherwise keep the
write end open and hang teardown (both warmup shutdown at startup and process
shutdown). The writer is likewise non-blocking + polled.
* Windows: resolve the command through PATHEXT so "npx" (npm ships npx.cmd, never
npx.exe) spawns, matching POSIX's PATH search; and enumerate the parent environment
as UTF-8 (GetEnvironmentStringsW) instead of the active code page.
* server_pipe gains an opt-in max_size (default unbounded, so the router's streaming
use is unchanged); the MCP reply queue uses it so a server that streams unsolicited
notifications between requests cannot grow it without bound.
Integration:
* --mcp-servers-config / --mcp-servers-json flags; enabling MCP restricts default CORS
to localhost, same as --tools.
* MCP tools are exposed through /tools (and chat-completions) as <server>_<tool>,
skipping names that collide with a built-in or another MCP tool.
* Manager lifecycle wired into llama_server(): warmup at start, shutdown() from the
signal handler before the HTTP server drains, blocking teardown in clean_up().
* SIGPIPE ignored so a child dying mid-write yields EPIPE rather than killing us.
Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
* server-mcp: add MCP test suite with grandchild deadlock regression test
21 tests over the /tools endpoint: tool discovery/invocation, timeouts, crash
recovery and respawn cooldown, warmup partial failure, malformed and batched
notification+response output, tool-definition shape, and prompt shutdown during a
slow call.
The last test spawns an MCP server that leaves a grandchild inheriting its
stdout/stderr and asserts the server both starts and stops promptly. Verified it
fails (5s SIGKILL fallback on a deadlocked reader-join) when the pump is made to
ignore the running flag, and passes with the polled reader.
Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
* clean up
* clean up 2
* even stricter life cycle
* nits
* nits 2
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* fix some edge cases
* fix last_error data race
* fix response schema + docs
* server: fix MCP zombie leak and timeout-induced transport teardown
join_pumps() never reaped the child, leaking one zombie per spawn:
call subprocess_join() before subprocess_destroy().
A per-call timeout permanently closed from_server and got a healthy
transport evicted: add close_on_stop to server_pipe::read() and pass
false from send_rpc(), where should_stop is a per-request deadline
and a late reply is already skipped on id mismatch.
Also drop the unreachable disconnect cancellation in
server_mcp_tool::invoke(): support_stream is false, st is always null.
(cherry picked from commit e6de1ec043174fd0570b1e60d47f06c7c19d620d)
Assisted-by: Claude Opus 4.8
* server: make MCP test fixtures JSON-RPC 2.0 compliant
Add the missing notification guard to mcp_malformed_server.py and
mcp_burst_server.py (the latter treated id 0 as a notification and
replied to unknown ones; its notification table is now unused).
Return -32602 instead of -32601 for unknown tools: tools/call is a
valid method, the tool name is the invalid parameter.
Also fix the test module docstring: tools are named <server>_<tool>.
(cherry picked from commit 74a08e8c311dabf3b49d06cc6d754b0097ae7a38)
Assisted-by: Claude Opus 4.8
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Pascal <admin@serveurperso.com>
read_file with append_loc emits "{n}\u2192 {line}". The space after the
arrow is meant as a separator, but it is indistinguishable from real
indentation. Models strip "{n}\u2192" yet keep the space, so the old_text
passed to edit_file carries a phantom leading space and never matches
(normalize_for_fuzzy_match trims trailing whitespace only, never leading).
Drop the separator space so the arrow abuts content: stripping "{n}\u2192"
now yields the exact line with its real indentation preserved, and the
failure mode cannot occur by construction. Update the description example
to match the new format.
line_start -1 normalized to n+1, so append inserted at lines.begin() + n + 1,
one past end() -> heap-buffer-overflow in vector::_M_range_insert.
Normalize -1 to n (insert at end()), restrict -1 to append mode and reject it
for replace/delete instead of silently clobbering the last line. Parenthesize
the insert offset so empty-file append computes the position as int first,
avoiding a transient begin() - 1 on a null vector data pointer.
* common/peg : refactor until gbnf grammar into an ac automaton
* cont : add a test with multiple strings
* cont : pad state with 0s so rules line up
* cont : clean up comments
* cont : use set everywhere
* cont : inline state num string padding
* cont : add a ref to PR
* cont : fix regression in server-tools.cpp
Previously, unknown tool names passed via --tools were silently ignored.
Now the server validates each tool name at startup and exits with an
error if an unrecognized tool is specified, listing the available tools.
Assisted-by: llama.cpp:local pi
