Docker Compose production stack for AI
A model on its own is not a service. It becomes one when it has a front door, a cache, measurement and alerts in front of it. In this lesson we assemble five layers in one docker-compose.yml: Nginx → Redis → vLLM → Prometheus → Grafana, plus a variant for an ARM64 machine with Ollama.
kv_cache_usage_perc instead of gpu_cache_usage_perc; errors are counted via request_success_total{finished_reason="error"}). Bugs in the old example were fixed: the password in redis.conf is not substituted from a variable; --quantization awq fails on a plain model; the script waited for 5 "healthy" services while only 3 had a healthcheck; the Redis and Nginx exporters were missing; the "GPU memory" alert actually measured bandwidth. Dashboard ID 21118 points to an unrelated dashboard — replaced with the official one from vLLM. For ARM64: vLLM now ships -aarch64 images and Ollama has a built-in OpenAI-compatible API, so LiteLLM is optional.
01What you will learn
- How a request passes through the five layers and what each one adds.
- How to describe the whole stack in one file: pinned versions, health checks, correct start order, secrets out of git.
- How Nginx protects the model with rate limiting and how Redis saves GPU by caching repeated questions.
- Which vLLM metrics to watch, how to write alerts and how to bring up a ready dashboard in Grafana.
- How to start safely with a script and how to run the same stack on an ARM64 machine with Ollama.
02Before you start
- You have completed Block 1 · Part 1 (vLLM and inference) — you know what PagedAttention and an OpenAI-compatible API are.
- A Linux machine with an NVIDIA GPU, Docker Engine and Docker Compose (the command is
docker compose, no hyphen). - NVIDIA Container Toolkit installed — without it containers cannot see the GPU.
- A Hugging Face account and token if the model is gated (for example Llama).
- About 50 GB of free space for models and data.
03Steps
-
The architecture: five layers for one request
Every request passes through five layers on its way to the model and back. Why layers and not just the model? Because the model cannot protect itself from overload, does not remember answers it already gave and does not tell you when it is unwell. Each layer takes on one of those jobs.
Layer Port Job Nginx 80 / 443 Front door: TLS, rate limiting, load balancing, health check. Redis 6379 (internal) Cache of answers to identical questions, TTL 1 hour. vLLM 8000 Generates: PagedAttention, continuous batching, OpenAI-compatible API. Prometheus 9090 Scrapes metrics every 15 s, keeps 15 days of history, evaluates alerts. Grafana 3000 Real-time dashboards: throughput, latency, GPU, cache, errors. project layoutai-production/ ├── docker-compose.yml # the main file ├── .env # secrets — NOT in git ├── config/ │ ├── nginx/conf.d/vllm.conf # upstream + rate limiting │ ├── redis/redis.conf # memory, persistence │ └── prometheus/ │ ├── prometheus.yml # what gets scraped │ └── alerts.yml # alert rules ├── data/ │ ├── models/ # Hugging Face cache │ ├── prometheus/ grafana/ redis/ └── scripts/ ├── start.sh # start with checks └── health_check.sh # check from cron -
docker-compose.yml — the whole stack in one file
Three rules make the file "production": pinned versions (tomorrow
latestmay be something else and nothing starts), health checks (Nginx starts only once the model has loaded) and only the front door faces outward — vLLM, Grafana and Prometheus listen onlocalhostonly, while Redis and the exporters live on the internal network only.YAML · docker-compose.yml# AI production stack · docker compose up -d · versions verified on 2026-10-01 services: vllm: image: vllm/vllm-openai:v0.30.0 # pinned version restart: unless-stopped ipc: host # shared memory for PyTorch (per vLLM docs) environment: HF_TOKEN: ${HF_TOKEN} volumes: - ./data/models:/root/.cache/huggingface ports: - "127.0.0.1:8000:8000" # localhost only — Nginx faces outward command: > --model ${MODEL_NAME:-meta-llama/Llama-3.1-8B-Instruct} --served-model-name ${SERVED_MODEL_NAME:-llama3} --max-model-len ${MAX_MODEL_LEN:-16384} --max-num-seqs ${MAX_NUM_SEQS:-256} --gpu-memory-utilization 0.90 healthcheck: test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"] interval: 30s timeout: 10s retries: 5 start_period: 180s # the model takes minutes to load deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] redis: image: redis:8.10-alpine restart: unless-stopped # redis.conf does NOT read ${VARIABLES} — the password is passed from here command: ["redis-server", "/usr/local/etc/redis/redis.conf", "--requirepass", "${REDIS_PASSWORD}"] environment: REDIS_PASSWORD: ${REDIS_PASSWORD} volumes: - ./config/redis/redis.conf:/usr/local/etc/redis/redis.conf:ro - ./data/redis:/data healthcheck: test: ["CMD-SHELL", "redis-cli -a \"$$REDIS_PASSWORD\" --no-auth-warning ping | grep -q PONG"] interval: 10s timeout: 5s retries: 3 nginx: image: nginx:1.30-alpine # the stable branch restart: unless-stopped depends_on: vllm: { condition: service_healthy } redis: { condition: service_healthy } volumes: - ./config/nginx/conf.d:/etc/nginx/conf.d:ro ports: - "80:80" # - "443:443" ← once you add a TLS certificate healthcheck: test: ["CMD", "nginx", "-t"] interval: 30s prometheus: image: prom/prometheus:v3.15.0 restart: unless-stopped command: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.path=/prometheus - --storage.tsdb.retention.time=15d - --web.enable-lifecycle # reload config without a restart volumes: - ./config/prometheus:/etc/prometheus:ro - ./data/prometheus:/prometheus ports: - "127.0.0.1:9090:9090" grafana: image: grafana/grafana:13.2.3 restart: unless-stopped depends_on: [prometheus] environment: GF_SECURITY_ADMIN_USER: ${GRAFANA_USER:-admin} GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASS} volumes: - ./data/grafana:/var/lib/grafana ports: - "127.0.0.1:3000:3000" # ── exporters: translate Redis, Nginx and GPU into Prometheus' language ── redis-exporter: image: oliver006/redis_exporter:v1.92.1 restart: unless-stopped environment: REDIS_ADDR: redis://redis:6379 REDIS_PASSWORD: ${REDIS_PASSWORD} nginx-exporter: image: nginx/nginx-prometheus-exporter:1.5.3 restart: unless-stopped command: ["--nginx.scrape-uri=http://nginx:8080/stub_status"] gpu-exporter: image: nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG} # tag — from the NVIDIA NGC catalog restart: unless-stopped cap_add: [SYS_ADMIN] deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] networks: default: name: ai-network✅Rule: pin versions, upgrade deliberatelyThe versions above were verified on 2026-10-01. When you upgrade, change one at a time and run the checks from the "Check" section. If you want an AWQ or GPTQ model, take a ready quantised checkpoint — vLLM detects the quantisation from its config. The--quantization awqflag on a plain model fails at start-up.bash · .env (NEVER committed to git)# echo ".env" >> .gitignore ← mandatory HF_TOKEN=<your-hugging-face-token> MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct SERVED_MODEL_NAME=llama3 MAX_MODEL_LEN=16384 MAX_NUM_SEQS=256 GRAFANA_USER=admin GRAFANA_PASS=<strong-password> REDIS_PASSWORD=<strong-password> DCGM_EXPORTER_TAG=<current-tag-from-NGC> -
Nginx: the front door
Nginx has three jobs: protection (limit requests per address), efficiency (persistent connections to vLLM) and scale (an upstream pool where tomorrow you add a second vLLM). Statistics for Prometheus live on a separate internal port 8080 that is not published.
nginx · config/nginx/conf.d/vllm.conf# Rate-limit zones (files in conf.d are in the http context) limit_req_zone $binary_remote_addr zone=ai_per_ip:10m rate=10r/s; limit_req_zone $server_name zone=ai_global:10m rate=100r/s; upstream vllm_backend { server vllm:8000 max_fails=3 fail_timeout=30s; # server vllm-2:8000; ← second instance when needed keepalive 32; keepalive_timeout 65s; } server { listen 80; server_name _; client_max_body_size 10m; send_timeout 300s; proxy_read_timeout 300s; # timeout between two reads, not total time add_header X-Content-Type-Options nosniff; add_header X-Frame-Options DENY; # add_header Strict-Transport-Security "max-age=31536000" always; ← only after TLS location /v1/ { limit_req zone=ai_per_ip burst=20 nodelay; limit_req zone=ai_global burst=100 nodelay; limit_req_status 429; proxy_pass http://vllm_backend; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Request-ID $request_id; proxy_buffering off; # streaming (SSE) } location /health { proxy_pass http://vllm_backend/health; access_log off; } error_page 429 /429.json; location = /429.json { default_type application/json; return 429 '{"error":"Rate limit exceeded","retry_after":1}'; } error_page 502 503 504 /5xx.json; location = /5xx.json { default_type application/json; return 503 '{"error":"Service temporarily unavailable"}'; } } # Internal port for nginx-exporter only (not published in compose) server { listen 8080; location /stub_status { stub_status; access_log off; } }⚠️Trap: long generation and 504proxy_read_timeoutis the time between two successive reads from vLLM. If the answer is generated in full and only then sent, and that takes longer than 300 s, Nginx returns 504. With streaming, every chunk resets the timer — so turn onstreamfor long answers. -
Redis: a cache for repeated questions
In learning systems (for example RAG for Moodle) the same questions are asked again and again. We cache the answer for one hour: on a hit it comes from memory without touching the GPU. Cache only predictable requests — no streaming and low temperature — otherwise you return a "frozen" answer where variety is wanted.
💡An example of scale (illustrative)If in a group of 50 learners 80% of the questions are variations of "Explain RAG" or "How do I install Ollama", only the first request generates and the rest take the cache. The higher the hit rate, the more GPU is left for new questions. The numbers depend on your audience — measure them with/cache/stats, do not take them for granted.ini · config/redis/redis.confmaxmemory 4gb maxmemory-policy allkeys-lru # when full, evicts the least recently used keys save 900 1 save 300 10 dbfilename dump.rdb dir /data protected-mode yes # the password comes from compose (--requirepass) tcp-keepalive 300Python · redis_cache_middleware.py (a layer in front of vLLM in your RAG API)import os, time, json, hashlib import redis, httpx from fastapi import FastAPI, Request from fastapi.responses import JSONResponse app = FastAPI() r = redis.Redis(host="redis", port=6379, password=os.environ["REDIS_PASSWORD"], decode_responses=True) VLLM = os.environ.get("VLLM_URL", "http://vllm:8000") def cache_key(model, messages, temperature): payload = json.dumps({"m": model, "msg": messages, "t": temperature}, sort_keys=True) return "llm:" + hashlib.sha256(payload.encode()).hexdigest()[:16] @app.post("/v1/chat/completions") async def cached_chat(request: Request): body = await request.json() temperature = body.get("temperature", 0.2) use_cache = not body.get("stream") and temperature < 0.5 if use_cache: key = cache_key(body.get("model"), body.get("messages", []), temperature) hit = r.get(key) if hit: r.incr("stats:cache_hits") return JSONResponse(json.loads(hit), headers={"X-Cache": "HIT"}) t0 = time.time() async with httpx.AsyncClient(timeout=300) as client: resp = await client.post(f"{VLLM}/v1/chat/completions", json=body) result = resp.json() if use_cache and resp.status_code == 200: r.setex(key, 3600, json.dumps(result)) # TTL 1 hour r.incr("stats:cache_misses") return JSONResponse(result, status_code=resp.status_code, headers={"X-Cache": "MISS", "X-Latency-ms": str(round((time.time() - t0) * 1000))}) @app.get("/cache/stats") def cache_stats(): hits = int(r.get("stats:cache_hits") or 0) misses = int(r.get("stats:cache_misses") or 0) total = hits + misses return {"hits": hits, "misses": misses, "hit_rate_pct": round(hits / total * 100, 1) if total else 0}⛔Never a password in codeThe old version of this lesson had the Redis password hard-coded in the Python file. Passwords come only from the environment (.env→environment) — code goes into git, the password never does. -
Prometheus: what we measure and when we alert
Without measurement you are blind. For an AI service we watch throughput (tokens/s), latency (P50/P99), the queue (waiting requests), the KV cache, errors and GPU temperature. The metric names below come from the current vLLM (v1) documentation.
YAML · config/prometheus/prometheus.ymlglobal: scrape_interval: 15s evaluation_interval: 15s rule_files: - alerts.yml scrape_configs: - job_name: vllm scrape_interval: 5s static_configs: [{ targets: ["vllm:8000"] }] # built-in /metrics - job_name: gpu static_configs: [{ targets: ["gpu-exporter:9400"] }] - job_name: redis static_configs: [{ targets: ["redis-exporter:9121"] }] - job_name: nginx static_configs: [{ targets: ["nginx-exporter:9113"] }] - job_name: prometheus static_configs: [{ targets: ["localhost:9090"] }]YAML · config/prometheus/alerts.ymlgroups: - name: ai_production rules: - alert: LargeWaitingQueue expr: vllm:num_requests_waiting > 50 for: 1m labels: { severity: warning } annotations: { summary: "More than 50 waiting requests — consider a second instance" } - alert: HighP99Latency expr: histogram_quantile(0.99, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m]))) > 10 for: 2m labels: { severity: warning } annotations: { summary: "P99 latency above 10 s" } - alert: CriticalLatency expr: histogram_quantile(0.99, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m]))) > 30 for: 1m labels: { severity: critical } annotations: { summary: "P99 above 30 s — the system is congested" } - alert: KVCacheFull expr: vllm:kv_cache_usage_perc > 0.95 # 1 = 100% for: 5m labels: { severity: warning } annotations: { summary: "KV cache above 95% for 5 minutes" } - alert: HighErrorRate expr: sum(rate(vllm:request_success_total{finished_reason="error"}[5m])) / sum(rate(vllm:request_success_total[5m])) > 0.05 for: 2m labels: { severity: critical } annotations: { summary: "More than 5% of requests end in an error" } - alert: GPUTemperatureHigh expr: DCGM_FI_DEV_GPU_TEMP > 80 for: 3m labels: { severity: warning } annotations: { summary: "GPU above 80 °C — check cooling" }⚠️Trap: old metric namesMany examples online still usevllm:gpu_cache_usage_perc,vllm:num_requests_swappedorvllm:request_failure_total. They do not exist in current vLLM (v1), so the alert simply never fires — silently. Check the names on/metricsof your version. -
Grafana: the dashboard
Do not draw the dashboard from scratch. The vLLM repository has a ready one (
examples/observability/prometheus_grafana/grafana.json) — import it into Grafana and add your own panels. The key queries:PromQL · panel queries# Throughput: generated tokens per second rate(vllm:generation_tokens_total[1m]) # Latency P50 / P99 histogram_quantile(0.50, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m]))) histogram_quantile(0.99, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m]))) # Time to first token P95 histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) # Requests: running and waiting vllm:num_requests_running vllm:num_requests_waiting # KV cache (0–1) vllm:kv_cache_usage_perc # Prefix cache hit rate rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) # GPU (DCGM) DCGM_FI_DEV_GPU_UTIL DCGM_FI_DEV_GPU_TEMPA pie chart is built into Grafana now — the old
grafana-piechart-panelplugin is not needed. -
Safe start-up and a check from cron
The script does four things before it says "ready": checks the requirements, starts, waits for health checks and sends a real end-to-end request. We wait only for services that have a healthcheck — the others will never be "healthy".
bash · scripts/start.sh#!/bin/bash set -euo pipefail cd "$(dirname "$0")/.." log(){ echo "[$(date +%H:%M:%S)] $1"; } err(){ echo "[ERROR] $1" >&2; exit 1; } [[ -f .env ]] || err ".env is missing — copy it from .env.example" command -v docker >/dev/null || err "Docker is not installed" docker info >/dev/null 2>&1 || err "Docker daemon is not running" nvidia-smi >/dev/null 2>&1 || err "No NVIDIA GPU found" FREE=$(df ./data --output=avail -BG | tail -1 | tr -d ' G') [[ ${FREE:-0} -gt 50 ]] || log "WARNING: less than 50 GB free in ./data" mkdir -p data/{models,prometheus,grafana,redis} docker compose up -d log "Waiting for health checks (up to 5 min)..." TOTAL=3 # vllm, redis, nginx — only they have a healthcheck for i in $(seq 1 60); do OK=$(docker compose ps --format '{{.Health}}' | grep -c '^healthy$' || true) [[ $OK -ge $TOTAL ]] && { log "All $TOTAL are healthy"; break; } [[ $i -eq 60 ]] && err "Timed out — see: docker compose logs vllm" sleep 5 done log "End-to-end test..." curl -sf http://localhost/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"llama3","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5}' \ | python3 -c "import json,sys; print(json.load(sys.stdin)['choices'][0]['message']['content'])" log "Ready · API: http://localhost/v1 · Grafana: http://localhost:3000"bash · scripts/health_check.sh (cron: */5 * * * *)#!/bin/bash cd "$(dirname "$0")/.." ALERT_EMAIL="ops@example.com" # your address; needs a configured mail command ISSUES=() up(){ [[ $(curl -so /dev/null -w "%{http_code}" --connect-timeout 5 "$1") == 200 ]]; } up http://localhost:8000/health || ISSUES+=("vLLM not responding") up http://localhost/health || ISSUES+=("Nginx not responding") docker compose exec -T redis sh -c 'redis-cli -a "$REDIS_PASSWORD" --no-auth-warning ping' \ | grep -q PONG || ISSUES+=("Redis not responding") TEMP=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader 2>/dev/null | head -1) [[ -n $TEMP && $TEMP -gt 85 ]] && ISSUES+=("GPU: ${TEMP} °C") if (( ${#ISSUES[@]} )); then printf '%s\n' "${ISSUES[@]}" | mail -s "AI stack: alert" "$ALERT_EMAIL" logger -t ai-health "ALERT: ${ISSUES[*]}" fi🛡️Why we removed the Prometheus admin APIThe--web.enable-admin-apiflag allows deleting data over HTTP. You do not need it in production — fewer open doors, less risk. -
A variant for an ARM64 machine (for example with NVIDIA GB10)
On an ARM64 machine with an NVIDIA GPU (such as GB10-based systems) the simplest path is Ollama 🔒 local: the image is multi-architecture and has a built-in OpenAI-compatible API at
/v1. Nginx, Redis, Prometheus and Grafana are multi-architecture and stay the same. If you want vLLM on ARM64 as well, it now publishes images tagged-aarch64(for examplev0.30.0-aarch64).YAML · docker-compose.arm64.yml (swaps only the inference layer)services: ollama: image: ollama/ollama:0.35.0 # linux/arm64 and amd64 restart: unless-stopped environment: OLLAMA_NUM_PARALLEL: "8" # parallel requests per model OLLAMA_MAX_LOADED_MODELS: "3" OLLAMA_KEEP_ALIVE: "60m" OLLAMA_FLASH_ATTENTION: "1" volumes: - ./data/ollama:/root/.ollama ports: - "127.0.0.1:11434:11434" healthcheck: test: ["CMD", "ollama", "list"] interval: 15s timeout: 10s retries: 3 deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] # redis, nginx, prometheus, grafana — as in the main file. # In vllm.conf change the upstream to: server ollama:11434; # Clients call the same thing: POST /v1/chat/completions✅LiteLLM — optional, not requiredSince Ollama speaks the OpenAI format, your code (LangChain, RAG) works without a translator. LiteLLM 🔒 local is worth it when you want one entry point to several models, a fallback model on failure or cost reporting.⚠️Not verified: GPU metrics on GB10GB10 uses unified memory shared by CPU and GPU. We have not verified whether DCGM exporter provides every field on this platform — if the GPU panels are empty, usenvidia-smiin the health script and the model server's own metrics.
04Check
Pre-launch checklist
.envis filled in and listed in.gitignore.docker compose up -druns without errors; vllm, redis and nginx are healthy.curl http://localhost/healthreturns 200.- A chat request through Nginx returns an answer.
- Grafana shows vLLM metrics.
/cache/statsshows hits (if you use the cache layer).- Above the configured limit Nginx returns 429.
- The rules are loaded in Prometheus (Status → Rules).
health_check.shis in crontab.- A benchmark was run: latency and throughput are recorded.
Test
1. Redis runs with maxmemory-policy allkeys-lru. What happens when memory is full on a new write?
2. vllm:num_requests_waiting has been 85 for 3 minutes. Which alert fires first?
3. proxy_read_timeout is 300 s and a non-streamed answer takes 350 s to generate. What happens?
4. Why are vLLM, Grafana and Prometheus bound to localhost while only Nginx faces outward?
05What's next
06Sources
- Compose file reference —
docker-compose.ymlsyntax: healthcheck, depends_on, deploy. - Docker: GPU in Compose — reserving NVIDIA GPUs.
- vLLM: Docker — the official image,
--ipc=host, ARM64 images. - vLLM: metrics — the full list of Prometheus metrics.
- vLLM: Prometheus + Grafana example — the ready
grafana.jsondashboard. - Nginx limit_req — rate limiting, burst and nodelay.
- Redis: configuration — maxmemory-policy and persistence.
- Prometheus: storage — retention and TSDB.
- NVIDIA DCGM Exporter — GPU metrics for Prometheus.
- redis_exporter and nginx-prometheus-exporter — Redis and Nginx metrics.
- Ollama: OpenAI compatibility — the built-in
/v1API. - LiteLLM — a proxy to many models in the OpenAI format.