LlamaNexus

Features

Ships as a single binary (llamanexus) with four subcommands - see Commands for the full reference.

API bridging

  • Presents both /ollama/api/* and /openai/v1/* routes to clients, forwarding to llama-server's single OpenAI-compatible router underneath, translating request/response shapes as needed.
  • Normalizes model identifiers (strips a trailing .gguf/:latest) so a client can request a model by its exact downloaded filename and still match the router's registered id.
  • GET /openai/v1/models reports, per model: real context_length (authoritative, sourced from router.preset.ini), n_ctx_train (the model's trained context length from GGUF metadata, purely informational), and live loaded status.

Distributed inference (RPC)

Manual mode (--rpc <ip:port>,...) or auto-discovery mode (--discovery, UDP heartbeat once per second, automatic llama-server restart whenever a worker appears or drops) - see Distributed inference for the full reference. Worker binaries are built for a portable baseline CPU instruction set so a binary built on one machine doesn't SIGILL on an older worker CPU.

Model downloading (Hugging Face)

  • Downloads via huggingface_hub directly (a Python sidecar) rather than relying on llama-server's own auto-pull, so real byte-level progress streams back to the client in real time.
  • Accepts a single repo:tag or an explicit file list (e.g. every shard of a split GGUF quant) to download as one batch, with combined progress reporting - one router restart at the end instead of one per file.
  • Orphaned .incomplete blob files left behind by a cancelled/failed download (a known huggingface_hub resume bug) are cleaned up automatically.

Hugging Face search

GET /api/hf/search/GET /api/hf/repo-files centralize search/repo-listing here, so HF_TOKEN covers gated/private repos for WebUI and the VS Code extension too, not just this proxy's own downloads - see API reference.

Context-size overrides

Works around llama.cpp allocating its KV cache at model-load time: LlamaNexus writes the new ctx-size into a per-model section of router.preset.ini, restarts llama-server, and waits for the router to come back up before the triggering request proceeds. POST /api/set-ctx-size exposes the same flow directly for external tools.

Model unloading & removal

Detects Open WebUI's "eject" action and Ollama's keep_alive: 0 convention automatically, forwarding to llama-server's router unload endpoint. POST /api/eject-model and POST /api/remove-model expose the same actions directly, for tools like the VS Code extension's model picker - removal deletes the GGUF file(s) from the Hugging Face cache, any ctx-size override, then restarts llama-server.

Speech-to-text & Text-to-speech

Both optional and off by default. --whisper-model/LLAMANEXUS_WHISPER_MODEL starts a managed whisper-server (whisper.cpp, built alongside llama-server in the same image, on the same GPU backend) and exposes POST /openai/v1/audio/transcriptions. --piper-voice/LLAMANEXUS_PIPER_VOICE starts a managed piper.http_server (Piper) and exposes POST /openai/v1/audio/speech. Both are OpenAI-compatible with no client-side request/response adaptation needed.

  • Text-to-speech switches voice live: a request naming a different voice than the one currently loaded downloads it first if needed, restarts piper.http_server with it, then serves the request - the previous voice is restored automatically if the new one fails to load. Speech-to-text doesn't have this yet - whisper-server still loads exactly one model at startup, so changing --whisper-model needs a restart.
  • GET /api/piper/voices/cached/search and GET /api/whisper/models/cached/search let a client build a voice/model picker without hand-typing a filename - see API reference for the full shapes.
  • GPU acceleration: whisper-server is built with the same backend as llama-server on every image variant (CUDA/ROCm/Vulkan). Piper only gets GPU acceleration on the CUDA image (onnxruntime-gpu); ROCm/Vulkan run Piper CPU-only, since onnxruntime's own GPU execution providers don't cover either backend.

Licensing

--license-key/LLAMANEXUS_LICENSE_KEY registers this proxy install against an account's plan, via a stable UUID persisted alongside router.preset.ini. No key configured makes zero license API calls and starts no heartbeat loop. Enforcement is currently disabled - chat requests are never blocked for a missing, invalid, or expired license.

Companion apps

← Previous Overview Next → Commands