The KAGAMI mark КАГАМИ
kagami.bg/academy · lesson · machine-readable viewVERIFIED 2026-10-01 · UPDATED 2026-10-01
IDENTITY
module
01-01b · Docker Compose production stack for AI
series
Blocks 0–10 · Block 1 — Production Inference · Part 2/2
level
Intermediate
duration
3–4 h
prerequisites
01-01a (vLLM basics)
trust_label
VERIFIED 2026-10-01 (image tags, vLLM metric names, links) · UPDATED 2026-10-01 · NOT end-to-end tested as a whole stack
language
human view: en · bulgarian edition: /academy/blokove/moduli/01-01b_Блок_1_Част_2_Docker_Production.html
next
01-01c_Lab_Production_Stack.html · Lab: production stack
PURPOSE

Run a self-hosted LLM behind a production-grade stack defined in one docker-compose.yml: Nginx (entry, rate limit) → Redis (response cache) → vLLM (inference, OpenAI-compatible) → Prometheus (metrics, alerts) → Grafana (dashboards). Includes an ARM64 variant with Ollama.

KEY CONCEPTS
COMMANDS / PATHS
CHECKLIST
NEXT MODULE

01-01c_Lab_Production_Stack.html · Lab: production stack · then Block 2 (prompt engineering, structured outputs) · offer: Quick experiment (kagami.bg/stalbata/)

SOURCES
TAGS
docker-composevllmnginxredisprometheusgrafanaobservabilityarm64ollama
VERIFIED · 01.10.2026 UPDATED · 01.10.2026

Docker Compose production stack for AI

A model on its own is not a service. It becomes one when it has a front door, a cache, measurement and alerts in front of it. In this lesson we assemble five layers in one docker-compose.yml: Nginx → Redis → vLLM → Prometheus → Grafana, plus a variant for an ARM64 machine with Ollama.

⏱ 3–4 h Intermediate Block 1 · Part 2/2 Docker · observability · security
Docker Compose🔒 local vLLM🔒 local Nginx · Redis🔒 local Prometheus · Grafana🔒 local Hugging Face (model download)🌐 global
🔄
UPDATED · 01.10.2026 — what changed
Image versions were checked against Docker Hub and GitHub: vLLM v0.6.6 → v0.30.0, Redis 7.4 → 8.10, Nginx 1.27 → 1.30 (stable), Prometheus 2.55 → 3.15.0, Grafana 11.3 → 13.2.3. vLLM metrics now use the v1 names (kv_cache_usage_perc instead of gpu_cache_usage_perc; errors are counted via request_success_total{finished_reason="error"}). Bugs in the old example were fixed: the password in redis.conf is not substituted from a variable; --quantization awq fails on a plain model; the script waited for 5 "healthy" services while only 3 had a healthcheck; the Redis and Nginx exporters were missing; the "GPU memory" alert actually measured bandwidth. Dashboard ID 21118 points to an unrelated dashboard — replaced with the official one from vLLM. For ARM64: vLLM now ships -aarch64 images and Ollama has a built-in OpenAI-compatible API, so LiteLLM is optional.

01What you will learn

02Before you start

03Steps

  1. The architecture: five layers for one request

    Every request passes through five layers on its way to the model and back. Why layers and not just the model? Because the model cannot protect itself from overload, does not remember answers it already gave and does not tell you when it is unwell. Each layer takes on one of those jobs.

    LayerPortJob
    Nginx80 / 443Front door: TLS, rate limiting, load balancing, health check.
    Redis6379 (internal)Cache of answers to identical questions, TTL 1 hour.
    vLLM8000Generates: PagedAttention, continuous batching, OpenAI-compatible API.
    Prometheus9090Scrapes metrics every 15 s, keeps 15 days of history, evaluates alerts.
    Grafana3000Real-time dashboards: throughput, latency, GPU, cache, errors.
    project layout
    ai-production/
    ├── docker-compose.yml          # the main file
    ├── .env                        # secrets — NOT in git
    ├── config/
    │   ├── nginx/conf.d/vllm.conf  # upstream + rate limiting
    │   ├── redis/redis.conf        # memory, persistence
    │   └── prometheus/
    │       ├── prometheus.yml      # what gets scraped
    │       └── alerts.yml          # alert rules
    ├── data/
    │   ├── models/                 # Hugging Face cache
    │   ├── prometheus/  grafana/  redis/
    └── scripts/
        ├── start.sh                # start with checks
        └── health_check.sh         # check from cron
  2. docker-compose.yml — the whole stack in one file

    Three rules make the file "production": pinned versions (tomorrow latest may be something else and nothing starts), health checks (Nginx starts only once the model has loaded) and only the front door faces outward — vLLM, Grafana and Prometheus listen on localhost only, while Redis and the exporters live on the internal network only.

    YAML · docker-compose.yml
    # AI production stack · docker compose up -d · versions verified on 2026-10-01
    services:
      vllm:
        image: vllm/vllm-openai:v0.30.0        # pinned version
        restart: unless-stopped
        ipc: host                              # shared memory for PyTorch (per vLLM docs)
        environment:
          HF_TOKEN: ${HF_TOKEN}
        volumes:
          - ./data/models:/root/.cache/huggingface
        ports:
          - "127.0.0.1:8000:8000"              # localhost only — Nginx faces outward
        command: >
          --model ${MODEL_NAME:-meta-llama/Llama-3.1-8B-Instruct}
          --served-model-name ${SERVED_MODEL_NAME:-llama3}
          --max-model-len ${MAX_MODEL_LEN:-16384}
          --max-num-seqs ${MAX_NUM_SEQS:-256}
          --gpu-memory-utilization 0.90
        healthcheck:
          test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
          interval: 30s
          timeout: 10s
          retries: 5
          start_period: 180s                   # the model takes minutes to load
        deploy:
          resources:
            reservations:
              devices:
                - driver: nvidia
                  count: all
                  capabilities: [gpu]
    
      redis:
        image: redis:8.10-alpine
        restart: unless-stopped
        # redis.conf does NOT read ${VARIABLES} — the password is passed from here
        command: ["redis-server", "/usr/local/etc/redis/redis.conf", "--requirepass", "${REDIS_PASSWORD}"]
        environment:
          REDIS_PASSWORD: ${REDIS_PASSWORD}
        volumes:
          - ./config/redis/redis.conf:/usr/local/etc/redis/redis.conf:ro
          - ./data/redis:/data
        healthcheck:
          test: ["CMD-SHELL", "redis-cli -a \"$$REDIS_PASSWORD\" --no-auth-warning ping | grep -q PONG"]
          interval: 10s
          timeout: 5s
          retries: 3
    
      nginx:
        image: nginx:1.30-alpine               # the stable branch
        restart: unless-stopped
        depends_on:
          vllm:  { condition: service_healthy }
          redis: { condition: service_healthy }
        volumes:
          - ./config/nginx/conf.d:/etc/nginx/conf.d:ro
        ports:
          - "80:80"
          # - "443:443"   ← once you add a TLS certificate
        healthcheck:
          test: ["CMD", "nginx", "-t"]
          interval: 30s
    
      prometheus:
        image: prom/prometheus:v3.15.0
        restart: unless-stopped
        command:
          - --config.file=/etc/prometheus/prometheus.yml
          - --storage.tsdb.path=/prometheus
          - --storage.tsdb.retention.time=15d
          - --web.enable-lifecycle             # reload config without a restart
        volumes:
          - ./config/prometheus:/etc/prometheus:ro
          - ./data/prometheus:/prometheus
        ports:
          - "127.0.0.1:9090:9090"
    
      grafana:
        image: grafana/grafana:13.2.3
        restart: unless-stopped
        depends_on: [prometheus]
        environment:
          GF_SECURITY_ADMIN_USER: ${GRAFANA_USER:-admin}
          GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASS}
        volumes:
          - ./data/grafana:/var/lib/grafana
        ports:
          - "127.0.0.1:3000:3000"
    
      # ── exporters: translate Redis, Nginx and GPU into Prometheus' language ──
      redis-exporter:
        image: oliver006/redis_exporter:v1.92.1
        restart: unless-stopped
        environment:
          REDIS_ADDR: redis://redis:6379
          REDIS_PASSWORD: ${REDIS_PASSWORD}
    
      nginx-exporter:
        image: nginx/nginx-prometheus-exporter:1.5.3
        restart: unless-stopped
        command: ["--nginx.scrape-uri=http://nginx:8080/stub_status"]
    
      gpu-exporter:
        image: nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}   # tag — from the NVIDIA NGC catalog
        restart: unless-stopped
        cap_add: [SYS_ADMIN]
        deploy:
          resources:
            reservations:
              devices:
                - driver: nvidia
                  count: all
                  capabilities: [gpu]
    
    networks:
      default:
        name: ai-network
    ✅
    Rule: pin versions, upgrade deliberately
    The versions above were verified on 2026-10-01. When you upgrade, change one at a time and run the checks from the "Check" section. If you want an AWQ or GPTQ model, take a ready quantised checkpoint — vLLM detects the quantisation from its config. The --quantization awq flag on a plain model fails at start-up.
    bash · .env (NEVER committed to git)
    # echo ".env" >> .gitignore   ← mandatory
    HF_TOKEN=<your-hugging-face-token>
    MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
    SERVED_MODEL_NAME=llama3
    MAX_MODEL_LEN=16384
    MAX_NUM_SEQS=256
    GRAFANA_USER=admin
    GRAFANA_PASS=<strong-password>
    REDIS_PASSWORD=<strong-password>
    DCGM_EXPORTER_TAG=<current-tag-from-NGC>
  3. Nginx: the front door

    Nginx has three jobs: protection (limit requests per address), efficiency (persistent connections to vLLM) and scale (an upstream pool where tomorrow you add a second vLLM). Statistics for Prometheus live on a separate internal port 8080 that is not published.

    nginx · config/nginx/conf.d/vllm.conf
    # Rate-limit zones (files in conf.d are in the http context)
    limit_req_zone $binary_remote_addr zone=ai_per_ip:10m rate=10r/s;
    limit_req_zone $server_name        zone=ai_global:10m rate=100r/s;
    
    upstream vllm_backend {
        server vllm:8000 max_fails=3 fail_timeout=30s;
        # server vllm-2:8000;   ← second instance when needed
        keepalive 32;
        keepalive_timeout 65s;
    }
    
    server {
        listen 80;
        server_name _;
        client_max_body_size 10m;
        send_timeout         300s;
        proxy_read_timeout   300s;   # timeout between two reads, not total time
    
        add_header X-Content-Type-Options nosniff;
        add_header X-Frame-Options        DENY;
        # add_header Strict-Transport-Security "max-age=31536000" always;  ← only after TLS
    
        location /v1/ {
            limit_req zone=ai_per_ip burst=20  nodelay;
            limit_req zone=ai_global burst=100 nodelay;
            limit_req_status 429;
            proxy_pass         http://vllm_backend;
            proxy_http_version 1.1;
            proxy_set_header   Connection   "";
            proxy_set_header   Host         $host;
            proxy_set_header   X-Real-IP    $remote_addr;
            proxy_set_header   X-Request-ID $request_id;
            proxy_buffering    off;      # streaming (SSE)
        }
    
        location /health {
            proxy_pass http://vllm_backend/health;
            access_log off;
        }
    
        error_page 429 /429.json;
        location = /429.json { default_type application/json; return 429 '{"error":"Rate limit exceeded","retry_after":1}'; }
        error_page 502 503 504 /5xx.json;
        location = /5xx.json { default_type application/json; return 503 '{"error":"Service temporarily unavailable"}'; }
    }
    
    # Internal port for nginx-exporter only (not published in compose)
    server {
        listen 8080;
        location /stub_status { stub_status; access_log off; }
    }
    ⚠️
    Trap: long generation and 504
    proxy_read_timeout is the time between two successive reads from vLLM. If the answer is generated in full and only then sent, and that takes longer than 300 s, Nginx returns 504. With streaming, every chunk resets the timer — so turn on stream for long answers.
  4. Redis: a cache for repeated questions

    In learning systems (for example RAG for Moodle) the same questions are asked again and again. We cache the answer for one hour: on a hit it comes from memory without touching the GPU. Cache only predictable requests — no streaming and low temperature — otherwise you return a "frozen" answer where variety is wanted.

    💡
    An example of scale (illustrative)
    If in a group of 50 learners 80% of the questions are variations of "Explain RAG" or "How do I install Ollama", only the first request generates and the rest take the cache. The higher the hit rate, the more GPU is left for new questions. The numbers depend on your audience — measure them with /cache/stats, do not take them for granted.
    ini · config/redis/redis.conf
    maxmemory 4gb
    maxmemory-policy allkeys-lru   # when full, evicts the least recently used keys
    save 900 1
    save 300 10
    dbfilename dump.rdb
    dir /data
    protected-mode yes             # the password comes from compose (--requirepass)
    tcp-keepalive 300
    Python · redis_cache_middleware.py (a layer in front of vLLM in your RAG API)
    import os, time, json, hashlib
    import redis, httpx
    from fastapi import FastAPI, Request
    from fastapi.responses import JSONResponse
    
    app = FastAPI()
    r = redis.Redis(host="redis", port=6379,
                    password=os.environ["REDIS_PASSWORD"], decode_responses=True)
    VLLM = os.environ.get("VLLM_URL", "http://vllm:8000")
    
    def cache_key(model, messages, temperature):
        payload = json.dumps({"m": model, "msg": messages, "t": temperature}, sort_keys=True)
        return "llm:" + hashlib.sha256(payload.encode()).hexdigest()[:16]
    
    @app.post("/v1/chat/completions")
    async def cached_chat(request: Request):
        body = await request.json()
        temperature = body.get("temperature", 0.2)
        use_cache = not body.get("stream") and temperature < 0.5
        if use_cache:
            key = cache_key(body.get("model"), body.get("messages", []), temperature)
            hit = r.get(key)
            if hit:
                r.incr("stats:cache_hits")
                return JSONResponse(json.loads(hit), headers={"X-Cache": "HIT"})
        t0 = time.time()
        async with httpx.AsyncClient(timeout=300) as client:
            resp = await client.post(f"{VLLM}/v1/chat/completions", json=body)
        result = resp.json()
        if use_cache and resp.status_code == 200:
            r.setex(key, 3600, json.dumps(result))   # TTL 1 hour
            r.incr("stats:cache_misses")
        return JSONResponse(result, status_code=resp.status_code,
                            headers={"X-Cache": "MISS", "X-Latency-ms": str(round((time.time() - t0) * 1000))})
    
    @app.get("/cache/stats")
    def cache_stats():
        hits = int(r.get("stats:cache_hits") or 0)
        misses = int(r.get("stats:cache_misses") or 0)
        total = hits + misses
        return {"hits": hits, "misses": misses,
                "hit_rate_pct": round(hits / total * 100, 1) if total else 0}
    ⛔
    Never a password in code
    The old version of this lesson had the Redis password hard-coded in the Python file. Passwords come only from the environment (.env → environment) — code goes into git, the password never does.
  5. Prometheus: what we measure and when we alert

    Without measurement you are blind. For an AI service we watch throughput (tokens/s), latency (P50/P99), the queue (waiting requests), the KV cache, errors and GPU temperature. The metric names below come from the current vLLM (v1) documentation.

    YAML · config/prometheus/prometheus.yml
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    rule_files:
      - alerts.yml
    scrape_configs:
      - job_name: vllm
        scrape_interval: 5s
        static_configs: [{ targets: ["vllm:8000"] }]           # built-in /metrics
      - job_name: gpu
        static_configs: [{ targets: ["gpu-exporter:9400"] }]
      - job_name: redis
        static_configs: [{ targets: ["redis-exporter:9121"] }]
      - job_name: nginx
        static_configs: [{ targets: ["nginx-exporter:9113"] }]
      - job_name: prometheus
        static_configs: [{ targets: ["localhost:9090"] }]
    YAML · config/prometheus/alerts.yml
    groups:
      - name: ai_production
        rules:
          - alert: LargeWaitingQueue
            expr: vllm:num_requests_waiting > 50
            for: 1m
            labels: { severity: warning }
            annotations: { summary: "More than 50 waiting requests — consider a second instance" }
          - alert: HighP99Latency
            expr: histogram_quantile(0.99, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m]))) > 10
            for: 2m
            labels: { severity: warning }
            annotations: { summary: "P99 latency above 10 s" }
          - alert: CriticalLatency
            expr: histogram_quantile(0.99, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m]))) > 30
            for: 1m
            labels: { severity: critical }
            annotations: { summary: "P99 above 30 s — the system is congested" }
          - alert: KVCacheFull
            expr: vllm:kv_cache_usage_perc > 0.95                # 1 = 100%
            for: 5m
            labels: { severity: warning }
            annotations: { summary: "KV cache above 95% for 5 minutes" }
          - alert: HighErrorRate
            expr: sum(rate(vllm:request_success_total{finished_reason="error"}[5m])) / sum(rate(vllm:request_success_total[5m])) > 0.05
            for: 2m
            labels: { severity: critical }
            annotations: { summary: "More than 5% of requests end in an error" }
          - alert: GPUTemperatureHigh
            expr: DCGM_FI_DEV_GPU_TEMP > 80
            for: 3m
            labels: { severity: warning }
            annotations: { summary: "GPU above 80 °C — check cooling" }
    ⚠️
    Trap: old metric names
    Many examples online still use vllm:gpu_cache_usage_perc, vllm:num_requests_swapped or vllm:request_failure_total. They do not exist in current vLLM (v1), so the alert simply never fires — silently. Check the names on /metrics of your version.
  6. Grafana: the dashboard

    Do not draw the dashboard from scratch. The vLLM repository has a ready one (examples/observability/prometheus_grafana/grafana.json) — import it into Grafana and add your own panels. The key queries:

    PromQL · panel queries
    # Throughput: generated tokens per second
    rate(vllm:generation_tokens_total[1m])
    # Latency P50 / P99
    histogram_quantile(0.50, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m])))
    histogram_quantile(0.99, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m])))
    # Time to first token P95
    histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
    # Requests: running and waiting
    vllm:num_requests_running
    vllm:num_requests_waiting
    # KV cache (0–1)
    vllm:kv_cache_usage_perc
    # Prefix cache hit rate
    rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m])
    # GPU (DCGM)
    DCGM_FI_DEV_GPU_UTIL
    DCGM_FI_DEV_GPU_TEMP

    A pie chart is built into Grafana now — the old grafana-piechart-panel plugin is not needed.

  7. Safe start-up and a check from cron

    The script does four things before it says "ready": checks the requirements, starts, waits for health checks and sends a real end-to-end request. We wait only for services that have a healthcheck — the others will never be "healthy".

    bash · scripts/start.sh
    #!/bin/bash
    set -euo pipefail
    cd "$(dirname "$0")/.."
    log(){ echo "[$(date +%H:%M:%S)] $1"; }
    err(){ echo "[ERROR] $1" >&2; exit 1; }
    
    [[ -f .env ]] || err ".env is missing — copy it from .env.example"
    command -v docker >/dev/null || err "Docker is not installed"
    docker info >/dev/null 2>&1 || err "Docker daemon is not running"
    nvidia-smi >/dev/null 2>&1 || err "No NVIDIA GPU found"
    FREE=$(df ./data --output=avail -BG | tail -1 | tr -d ' G')
    [[ ${FREE:-0} -gt 50 ]] || log "WARNING: less than 50 GB free in ./data"
    
    mkdir -p data/{models,prometheus,grafana,redis}
    docker compose up -d
    
    log "Waiting for health checks (up to 5 min)..."
    TOTAL=3   # vllm, redis, nginx — only they have a healthcheck
    for i in $(seq 1 60); do
      OK=$(docker compose ps --format '{{.Health}}' | grep -c '^healthy$' || true)
      [[ $OK -ge $TOTAL ]] && { log "All $TOTAL are healthy"; break; }
      [[ $i -eq 60 ]] && err "Timed out — see: docker compose logs vllm"
      sleep 5
    done
    
    log "End-to-end test..."
    curl -sf http://localhost/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{"model":"llama3","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5}' \
      | python3 -c "import json,sys; print(json.load(sys.stdin)['choices'][0]['message']['content'])"
    log "Ready · API: http://localhost/v1 · Grafana: http://localhost:3000"
    bash · scripts/health_check.sh (cron: */5 * * * *)
    #!/bin/bash
    cd "$(dirname "$0")/.."
    ALERT_EMAIL="ops@example.com"     # your address; needs a configured mail command
    ISSUES=()
    up(){ [[ $(curl -so /dev/null -w "%{http_code}" --connect-timeout 5 "$1") == 200 ]]; }
    
    up http://localhost:8000/health || ISSUES+=("vLLM not responding")
    up http://localhost/health      || ISSUES+=("Nginx not responding")
    docker compose exec -T redis sh -c 'redis-cli -a "$REDIS_PASSWORD" --no-auth-warning ping' \
      | grep -q PONG || ISSUES+=("Redis not responding")
    TEMP=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader 2>/dev/null | head -1)
    [[ -n $TEMP && $TEMP -gt 85 ]] && ISSUES+=("GPU: ${TEMP} °C")
    
    if (( ${#ISSUES[@]} )); then
      printf '%s\n' "${ISSUES[@]}" | mail -s "AI stack: alert" "$ALERT_EMAIL"
      logger -t ai-health "ALERT: ${ISSUES[*]}"
    fi
    🛡️
    Why we removed the Prometheus admin API
    The --web.enable-admin-api flag allows deleting data over HTTP. You do not need it in production — fewer open doors, less risk.
  8. A variant for an ARM64 machine (for example with NVIDIA GB10)

    On an ARM64 machine with an NVIDIA GPU (such as GB10-based systems) the simplest path is Ollama 🔒 local: the image is multi-architecture and has a built-in OpenAI-compatible API at /v1. Nginx, Redis, Prometheus and Grafana are multi-architecture and stay the same. If you want vLLM on ARM64 as well, it now publishes images tagged -aarch64 (for example v0.30.0-aarch64).

    YAML · docker-compose.arm64.yml (swaps only the inference layer)
    services:
      ollama:
        image: ollama/ollama:0.35.0            # linux/arm64 and amd64
        restart: unless-stopped
        environment:
          OLLAMA_NUM_PARALLEL: "8"             # parallel requests per model
          OLLAMA_MAX_LOADED_MODELS: "3"
          OLLAMA_KEEP_ALIVE: "60m"
          OLLAMA_FLASH_ATTENTION: "1"
        volumes:
          - ./data/ollama:/root/.ollama
        ports:
          - "127.0.0.1:11434:11434"
        healthcheck:
          test: ["CMD", "ollama", "list"]
          interval: 15s
          timeout: 10s
          retries: 3
        deploy:
          resources:
            reservations:
              devices:
                - driver: nvidia
                  count: all
                  capabilities: [gpu]
      # redis, nginx, prometheus, grafana — as in the main file.
      # In vllm.conf change the upstream to:  server ollama:11434;
      # Clients call the same thing: POST /v1/chat/completions
    ✅
    LiteLLM — optional, not required
    Since Ollama speaks the OpenAI format, your code (LangChain, RAG) works without a translator. LiteLLM 🔒 local is worth it when you want one entry point to several models, a fallback model on failure or cost reporting.
    ⚠️
    Not verified: GPU metrics on GB10
    GB10 uses unified memory shared by CPU and GPU. We have not verified whether DCGM exporter provides every field on this platform — if the GPU panels are empty, use nvidia-smi in the health script and the model server's own metrics.

04Check

Pre-launch checklist

Test

1. Redis runs with maxmemory-policy allkeys-lru. What happens when memory is full on a new write?

2. vllm:num_requests_waiting has been 85 for 3 minutes. Which alert fires first?

3. proxy_read_timeout is 300 s and a non-streamed answer takes 350 s to generate. What happens?

4. Why are vLLM, Grafana and Prometheus bound to localhost while only Nginx faces outward?

05What's next

06Sources

  1. Compose file reference — docker-compose.yml syntax: healthcheck, depends_on, deploy.
  2. Docker: GPU in Compose — reserving NVIDIA GPUs.
  3. vLLM: Docker — the official image, --ipc=host, ARM64 images.
  4. vLLM: metrics — the full list of Prometheus metrics.
  5. vLLM: Prometheus + Grafana example — the ready grafana.json dashboard.
  6. Nginx limit_req — rate limiting, burst and nodelay.
  7. Redis: configuration — maxmemory-policy and persistence.
  8. Prometheus: storage — retention and TSDB.
  9. NVIDIA DCGM Exporter — GPU metrics for Prometheus.
  10. redis_exporter and nginx-prometheus-exporter — Redis and Nginx metrics.
  11. Ollama: OpenAI compatibility — the built-in /v1 API.
  12. LiteLLM — a proxy to many models in the OpenAI format.