Docker setup

LlamaNexus ships only as prebuilt images on Docker Hub - there's no source build to run yourself. It runs in Docker alongside llama-server (GPU-enabled) and Open WebUI.

Image tags

Tags follow <gpu-backend>[-<backend-version>]-<llamanexus-version|latest>:

Pick the tag matching your GPU driver/toolkit, or pin an exact version (e.g. cuda-13.0.0-0.1.21) instead of latest for reproducible upgrades.

Server

services:
  llamanexus:
    image: makershop/llamanexus:cuda-13.0.0-latest
    network_mode: host   # required for --discovery (see Distributed inference)
    command: serve --discovery --llamaport 8080 --port 11434 -- --jinja --context-shift # llama-server specific arguments after double dash
    environment:
      - HF_TOKEN=${HF_TOKEN:-}
      - LLAMANEXUS_CHAT_TIMEOUT_SECONDS=900
    volumes:
      - ./data/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

HF_TOKEN covers both downloads and Hugging Face search/listing (see API reference) - set it once here and it authenticates gated/private repo access for WebUI and VS Code too, since both search through this proxy rather than calling huggingface.co directly.

Speech-to-text & Text-to-speech

Both optional, off by default - add either or both to the command above:

command: serve --whisper-model ggml-base.en.bin --piper-voice en_US-lessac-medium --llamaport 8080 --port 11434

Or their env var equivalents, LLAMANEXUS_WHISPER_MODEL/LLAMANEXUS_PIPER_VOICE. Downloaded models/voices are cached under the same ~/.cache/huggingface volume already mounted above - no separate volume needed. See Features for what each engine supports.

Worker

services:
  llamanexus-worker:
    image: makershop/llamanexus:cuda-13.0.0-latest
    network_mode: host
    command: worker --discovery
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Using a rocm-* or vulkan-latest tag instead? Drop the Nvidia deploy.resources block above and pass through the right device nodes instead - --device=/dev/kfd --device=/dev/dri for ROCm, --device=/dev/dri for Vulkan.

See Distributed inference for --discovery/--rpc details and the network_mode: host requirement.

Volumes

Mount a persistent volume at ~/.cache/huggingface inside the container - it holds the Hugging Face cache (downloaded GGUF files) and router.preset.ini (per-model context-size overrides).