Why Your Ollama LLMs Are Running Painfully Slow on K3s and how to fix

Why Your Ollama LLMs Are Running Painfully Slow on K3s (And How to Fix GPU Pass-Through)

Imagine this scenario: You finally decided to take control of your data privacy and build an in-house AI infrastructure. You spun up a lightweight Kubernetes cluster using K3s, deployed Ollama, connected it to an Open-WebUI frontend, and asked your local LLM a simple question.

Then... you wait.

The cursor blinks. One word crawls onto the screen every two seconds ($1 \text{ token/s}$). Your CPU fan sounds like a jet engine taking off, system memory spikes to $100\%$, and the user experience is thoroughly ruined.

What went wrong? You have an expensive NVIDIA GPU seated right inside your server hardware! Why is your local AI running like a turtle on a treadmill?

In this deep-dive article, we will unpack the root cause behind Ollama’s sluggish performance on Kubernetes, explore how K3s handles hardware resources, and walk through a step-by-step production guide to enable true NVIDIA GPU Pass-Through using the NVIDIA Container Toolkit and Device Plugin.

The Root Cause: Why Is Ollama Defaulting to CPU?

When you run Ollama locally on bare-metal desktop OS (like macOS or Windows with CUDA drivers installed), Ollama automatically detects available GPUs and offloads neural network layers to VRAM.

However, Kubernetes operates in an isolated, containerized universe.

By default, container runtimes inside Kubernetes (specifically containerd, which comes bundled with K3s) are deliberately sandboxed from the underlying host's physical PCI hardware.

What Happens Behind the Scenes:

1. Hardware Blindness: The containerized Ollama process cannot "see" /dev/nvidia* host device nodes.

2. CPU Fallback: Because no CUDA runtime is exposed inside the pod, Ollama gracefully falls back to CPU matrix multiplication (using AVX/AVX-512 instructions).

3. Resource Exhaustion: CPU inference on a 7B or 13B parameter model requires continuous, high-latency memory transfers between system RAM and the CPU cache. This leads to thread contention, thermal throttling, and sub-par generation speeds compared to 50 tex - 80 text tokens/s on GPU VRAM).

To fix this, we must build a hardware bridge from the Linux kernel driver all the way into our K3s pod namespace.

Architectural Overview: The 3-Layer GPU Bridge

To achieve zero-latency GPU pass-through in K3s, we need three distinct layers working in harmony:

┌─────────────────────────────────────────────────────────┐
│               Ollama Pod (K3s Namespace)                │
│     resources: limits: { nvidia.com/gpu: "1" }          │
└────────────────────────────┬────────────────────────────┘
                             │
┌────────────────────────────▼────────────────────────────┐
│              NVIDIA K8s Device Plugin                   │
│         (Exposes GPU count to K3s Scheduler)            │
└────────────────────────────┬────────────────────────────┘
                             │
┌────────────────────────────▼────────────────────────────┐
│      NVIDIA Container Toolkit & Runtime (Host)          │
│          (Injects CUDA libraries into containerd)       │
└────────────────────────────┬────────────────────────────┘
                             │
┌────────────────────────────▼────────────────────────────┐
│            Linux Kernel & NVIDIA Drivers                │
│              (/dev/nvidia0, /dev/nvidia-uvm)            │
└─────────────────────────────────────────────────────────┘

Let's implement this step-by-step.

Step 1: Prepare the Host Node (NVIDIA Drivers & Container Toolkit)

Before touching Kubernetes or K3s manifests, your Linux host kernel must have functioning NVIDIA drivers and the NVIDIA Container Toolkit installed.

1. Verify NVIDIA Driver Installation

Ensure the official NVIDIA proprietary drivers are loaded on your host machine:

Bash

nvidia-smi

You should see a clean driver status table showing your GPU device name (e.g., RTX 3090 / RTX 4090 / A100), driver version (e.g., 535.x or 550.x), and CUDA version capability.

2. Install NVIDIA Container Toolkit

The Container Toolkit configures containerd to pass host GPU devices into OCI container specs.

On Ubuntu / Debian systems:

Bash

# Add NVIDIA Container Toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb [^ ]* #&[signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] #' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Step 2: Configure K3s Containerd Engine to Support NVIDIA Runtime

By default, K3s uses its own embedded containerd instance located at /var/lib/rancher/k3s/agent/etc/containerd/config.toml.

Crucial Note: Never edit this config.toml directly! K3s overwrites this file every time the service restarts. Instead, we must use the K3s template extension mechanism.

1. Create a Containerd Template File

Create a template configuration file at /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl:

Bash

sudo cp /var/lib/rancher/k3s/agent/etc/containerd/config.toml /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

2. Register the nvidia-container-runtime

Configure the NVIDIA Container Runtime as a default/available runtime handler inside /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl:

Ini, TOML

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/bin/nvidia-container-runtime"

3. Restart K3s Service

Apply the changes by restarting K3s:

Bash

sudo systemctl restart k3s

Step 3: Deploy the NVIDIA Kubernetes Device Plugin

Now that containerd knows how to launch CUDA-enabled containers, we need to advertise physical GPU resources to the Kubernetes Control Plane scheduler.

We use Helm to deploy the official NVIDIA Kubernetes Device Plugin:

Bash

# Add NVIDIA Helm repository
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update

# Install Device Plugin into kube-system namespace
helm install nvidia-device-plugin nvdp/k8s-device-plugin \
  --namespace kube-system \
  --set runtimeClassName=nvidia

Verify GPU Resource Capacity

Check if your K3s node now recognizes the physical GPU as a schedulable capacity unit:

Bash

kubectl get nodes -o jsonpath='{.items[*].status.allocatable}' | jq .

You should see an allocatable resource key similar to this:

JSON

{
  "cpu": "16",
  "memory": "32Gi",
  "nvidia.com/gpu": "1"
}

If [nvidia.com/gpu](https://nvidia.com/gpu) appears in the list, congratulations! Your K3s cluster is officially GPU-aware.

Step 4: Deploying Ollama with GPU Resource Requests

Now comes the moment of truth: deploying Ollama with explicit GPU resource limits so the Kubernetes scheduler mounts the CUDA drivers into the container.

Create a production deployment file named ollama-gpu-deployment.yaml:

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ai-workloads
  labels:
    app: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      runtimeClassName: nvidia  # Enforces NVIDIA container runtime
      containers:
      - name: ollama
        image: ollama/ollama:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 11434
          name: http
        resources:
          requests:
            cpu: "2000m"
            memory: "4Gi"
            nvidia.com/gpu: "1"  # Requests 1 Physical GPU
          limits:
            memory: "16Gi"
            nvidia.com/gpu: "1"  # Caps GPU usage to 1
        volumeMounts:
        - name: ollama-storage
          mountPath: /root/.ollama
      volumes:
      - name: ollama-storage
        persistentVolumeClaim:
          claimName: ollama-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: ollama-service
  namespace: ai-workloads
spec:
  type: ClusterIP
  ports:
  - port: 11434
    targetPort: 11434
    name: http
  selector:
    app: ollama

Apply the manifest:

Bash

kubectl create namespace ai-workloads
kubectl apply -f ollama-gpu-deployment.yaml

Step 5: Verification & Performance Benchmark

Let's verify that Ollama is actively using GPU hardware acceleration instead of CPU rendering.

1. Exec into the Pod & Monitor VRAM Usage

Run nvidia-smi directly from your host or exec into the pod while pulling a model:

Bash

# Exec into running Ollama pod
kubectl exec -ti -n ai-workloads deployment/ollama -- ollama run llama3:8b

On your host machine, open a separate terminal window and run:

Bash

watch -n 1 nvidia-smi

2. Check Key Metrics

Look for the following signals:

• Process Name: You should see ollama or ollama_llama_server registered under the GPU Processes table.

• VRAM Allocation: VRAM usage should jump from $0 \text{ MiB}$ to $4.5\text{--}5.5 \text{ GiB}$ (for a quantized 8B model).

• Generation Speed: Tokens should stream onto your terminal screen instantly at $60\text{--}90 \text{ tokens/s}$!

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14    CUDA Version: 12.4     |
|-----------------------------------+------------------------+--------------------------+
| GPU  Name        Persistence-M    | Bus-Id        Disp.A   | Volatile Uncorr. ECC     |
| Fan  Temp  Perf  Pwr:Usage/Cap    | Memory-Usage           | GPU-Util  Compute M.     |
|===================================+========================+==========================|
|   0  NVIDIA RTX 4090         On   | 00000000:01:00.0  Off  |                  N/A     |
| 30%   45C    P2    120W / 450W   |   5620MiB / 24564MiB   |     88%      Default     |
+-----------------------------------+------------------------+--------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                             GPU Memory |
|        ID   ID                                                              Usage      |
|=======================================================================================|
|    0  N/A  N/A      18294      C   ...bin/ollama_llama_server                  5612MiB |
+---------------------------------------------------------------------------------------+

Troubleshooting Common Edge Cases

Even with a detailed setup, distributed container runtime quirks can occur. Here are the top three issues engineers face when deploying Ollama on K3s:

Issue 1: Insufficient [nvidia.com/gpu](https://nvidia.com/gpu) Error

• Symptom: Pod remains stuck in Pending state. kubectl describe pod shows 0/1 nodes are available: 1 Insufficient [nvidia.com/gpu](https://nvidia.com/gpu).

• Fix: The Device Plugin failed to register the GPU. Verify that nvidia-smi works on the host and restart the nvidia-device-plugin daemonset:

  Bash

  kubectl rollout restart daemonset nvidia-device-plugin-daemonset -n kube-system
  

Issue 2: CUDA Out Of Memory (OOM)

• Symptom: Ollama pod crashes during model loading with CUDA error: out of memory.

• Fix: The model parameter size exceeds your GPU VRAM capacity. Use smaller quantized models (e.g., llama3:8b-instruct-q4_K_M instead of FP16), or enable CPU Offloading parameters in Ollama environment flags (OLLAMA_NUM_PARALLEL=1).

Issue 3: Driver Mismatch After Linux Kernel Updates

• Symptom: GPU pass-through worked yesterday, but broke after a system reboot.

• Fix: Unattended Linux kernel upgrades break proprietary NVIDIA DKMS kernel modules. Disable automatic updates for kernel/NVIDIA packages or pin your driver versions:

  Bash

  sudo apt-mark hold nvidia-driver-550
  

Conclusion & Next Steps

Building a Private AI Cloud on top of K3s gives you the ultimate combination: the lightweight orchestration of K3s paired with the raw computing power of local GPU hardware.

By setting up the NVIDIA Container Toolkit and enforcing runtimeClassName: nvidia on your deployments, you unlock the full throughput of your hardware—slashing LLM response latencies from 30+ seconds down to milliseconds.

Ready to Build Your Complete Private AI Stack?

GPU pass-through for Ollama is just one piece of the puzzle. To run an enterprise-ready local AI ecosystem, you also need:

• Vector Databases (Qdrant / Milvus) for Document Retrieval-Augmented Generation (RAG).

• Automated Cluster Diagnostics using K8sGPT with Telegram/Discord alerts.

• High-Availability Storage (Longhorn) for persistent vector indices.

If you want to master these architectures step-by-step with production-ready manifests and real-world scripts, explore my complete course:

👉 Masterclass: From Zero to Private AI Cloud on K3s HA

Have questions about GPU passthrough, CNI configurations, or local LLM inference? Drop a comment below and let's troubleshoot together!