Running 177B MoE Models on a 12 GB GPU with llama-moe-cache and NVMe Streaming [Part 3]

In Part 1 of this series, we set up Qwen 3.6 35B A3B with FreeToken on an RTX 3060 (12 GB VRAM), clocking a reliable 65 to 70 tokens per second for daily coding loops. In Part 2, we ran 640 public trials and 280 private trials benchmarking terminal coding agents, and found that 35B models solve everyday bugs and features with great reliability.

Then came the inevitable follow-up question: what is the absolute ceiling on single-card consumer hardware? Can we actually run a frontier-scale 100B+ MoE model on a $280 graphics card without renting a cluster or buying four GPUs?

Standard llama.cpp gives you a quick answer: no. A 177-billion-parameter model like Qwen 3.8 Flash Next takes 76.3 GiB of disk space even when aggressively quantized down to 3 bits (UD-IQ3_XXS). If you try to fit 77 GB of weights into a 12 GB card, offloading falls apart. If you try to offload to standard system RAM, CPU memory bandwidth chokes generation down to unworkable speeds.

The solution came from a specialized fork: GenerelSchwerz’s llama-moe-cache branch. By pairing a dynamic GPU expert cache with NVMe demand-paging and 8-bit quantized KV caching, we got Qwen 3.8 Flash Next running locally on our RTX 3060 with a full 64K context window.

Here is what it took to set it up, the memory trap that almost froze my workstation, and the real numbers.

The Model: Qwen 3.8 Flash Next

Qwen 3.8 Flash Next is an unusual architecture:

• 177 billion total parameters: A 125B parameter Mixture-of-Experts backbone combined with a 51B parameter N-gram phrasebook layer.
• 14 billion active parameters: Each token only activates a fraction of the total experts.
• Quantization: We use the Unsloth Dynamic IQ3_XXS quantization (UD-IQ3_XXS), split across three GGUF shards totaling 76.3 GiB (82 GB).

On paper, a 76 GB model on a 12 GB card sounds impossible. But MoE models have a mechanical property that dense models lack: at any given moment, the model is only using a handful of its expert weights.

How llama-moe-cache Fits 77 GB into 12 GB

In standard llama.cpp GPU offloading, you decide how many full layers to place in VRAM and leave the rest in system RAM. For a 177B model, you can only fit 2 or 3 layers into 12 GB of VRAM, forcing 95% of the model to compute on the CPU.

GenerelSchwerz’s moe-cache fork takes a completely different approach:

1. Shared layers in VRAM: The embedding table, attention layers, and expert routing networks stay permanently in GPU VRAM.
2. GPU MoE Expert Cache: The GPU reserves a pool (in our setup, 20 expert slabs) managed with an LRU (Least Recently Used) eviction policy. When the router picks an expert, the engine checks if that slab is already sitting in the GPU cache. If it is a hit, execution runs at native GPU tensor core speed.
3. NVMe Demand Paging: Inactive experts and the massive 51B phrasebook live on disk. When a cache miss occurs, the weights stream directly from the NVMe SSD over PCIe Gen4 into memory.

This turns your NVMe drive into an active storage tier for the model. But getting it to run safely on a machine with 32 GB of system RAM required learning a hard lesson about Linux memory allocation.

The Critical Gotcha: --load-mode mmap vs --load-mode none

When I first launched the server, I copied a launch script that passed --load-mode none.

That was a mistake.

Within twenty seconds of launching, my desktop froze. The mouse stuttered, audio stopped, and the system became completely unresponsive. The disk activity LED stayed solid on.

Here is what happened:

• --load-mode none tells llama.cpp to bypass memory mapping and allocate anonymous memory (malloc) for the model weights.
• On our machine with 32 GB of RAM, llama-server attempted to malloc all 76.3 GiB of GGUF weights into anonymous memory.
• The Linux kernel exhausted all 31 GB of physical RAM in seconds and dumped 40+ GB of pages into swap. The system began thrashing violently.

Checking /proc/<pid>/status after recovering the system showed the damage:

VmSize:  129989116 kB
VmHWM:    27325948 kB
VmRSS:    27031160 kB
RssAnon:  76892416 kB   <-- 76.8 GB of anonymous allocated memory!

The fix is straightforward, but mandatory: you must use --load-mode mmap.

# Correct setting for limited system RAM
--load-mode mmap

When you use mmap, llama.cpp does not allocate anonymous memory for the weights. Instead, it maps the GGUF file from the NVMe SSD into address space. The kernel reads pages on demand and keeps them in the Linux page cache (RssFile).

Checking /proc/<pid>/status with --load-mode mmap:

VmSize:  129989116 kB
VmHWM:    27325948 kB
VmRSS:    27031160 kB
RssAnon:    286312 kB   <-- Only 286 MB of anonymous RAM!
RssFile:  26575968 kB   <-- File-backed clean pages from NVMe

Because RssFile consists of clean file-backed pages, Linux can drop them instantly whenever any application needs RAM, without writing anything to swap. System RAM usage stayed at a calm 4.5 GB, leaving 26 GB available for the rest of the workstation.

Step 1: Building llama-moe-cache with CUDA sm_86

First, install the prerequisites on Arch Linux:

sudo pacman -S base-devel cmake git cuda

Clone the repository and switch to the moe-cache branch:

cd ~/projects
git clone https://github.com/GenerelSchwerz/llama.cpp.git llama-moe-cache
cd llama-moe-cache
git checkout moe-cache

Configure and build with native optimizations and CUDA architecture 86 (matching Ampere cards like the RTX 3060):

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DGGML_NATIVE=ON \
  -DCMAKE_CUDA_COMPILER=/opt/cuda/bin/nvcc

cmake --build build --target llama-server llama-cli -j8

Verify that build/bin/llama-server and build/bin/llama-cli were created cleanly.

Step 2: Downloading the Weights to NVMe

The model weights must live on a fast NVMe SSD. Running this over a SATA SSD or a spinning hard drive will bottleneck PCIe transfers down to 500 MB/s or less, making generation crawl.

Create the model directory on your NVMe partition:

mkdir -p ~/projects/llama-moe-cache/models/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS
cd ~/projects/llama-moe-cache/models/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS

Download the three shards using huggingface-cli:

huggingface-cli download unsloth/Qwen3.8-Flash-Next-GGUF \
  --include "UD-IQ3_XXS/*" \
  --local-dir ~/projects/llama-moe-cache/models/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS \
  --local-dir-use-symlinks False

The three shards take 76.3 GiB total:

• Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf (27.9 GB)
• Qwen3.8-Flash-Next-UD-IQ3_XXS-00002-of-00003.gguf (27.9 GB)
• Qwen3.8-Flash-Next-UD-IQ3_XXS-00003-of-00003.gguf (26.1 GB)

Step 3: Configuring the Server Script

To fit comfortably within 12 GB of VRAM while maximizing expert cache residency, every flag matters:

1. -c 32768: Allocates a 32K token context window. While 64K is supported, right-sizing to 32K saves ~1.2 GB of VRAM, which we can directly reallocate to GPU expert cache slots.
2. -ctk q8_0 -ctv q8_0 -kvo: Quantizes both key and value KV caches to 8-bit. Standard FP16 KV cache would consume excessive VRAM. Quantizing to Q8 cuts KV memory in half.
3. -b 4096 -ub 512: Caps micro-batch size to 512 tokens, reducing transient CUDA workspace overhead.
4. -t 8: Locks CPU computation to 8 threads. The AMD Ryzen 7 5800X has 8 physical cores. Setting -t 8 keeps each thread pinned to a dedicated core for evaluating CPU-side expert tensors.
5. --moe-expert-cache-size 48: Keeps 48 expert slabs cached directly on the GPU in VRAM (up from the conservative 20-slot baseline).
6. --lazy-mode on: Prevents preloading the entire 51B phrasebook into memory on boot.
7. --load-mode mmap: Uses clean file-backed mmap demand-paging, preventing system RAM exhaustion.

Here is the complete startup script (~/projects/llama-moe-cache/start-llama-moe.sh):

#!/usr/bin/env bash
set -euo pipefail

MODEL_DIR="/home/ann/projects/llama-moe-cache/models/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS"
MODEL_FILE="$MODEL_DIR/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf"
BINARY="/home/ann/projects/llama-moe-cache/build/bin/llama-server"
PORT="${LLAMA_PORT:-8080}"

if ss -tulpn 2>/dev/null | grep -q ":${PORT}\b"; then
  echo "llama-server is already running on port ${PORT}."
  exit 0
fi

if [ ! -f "$MODEL_FILE" ]; then
  echo "Error: Model file not found: $MODEL_FILE"
  exit 1
fi

echo "Starting llama-server (moe-cache) on port ${PORT} with Qwen 3.8 Flash Next..."
echo "Configuration: 8 CPU threads (physical cores), 48-slot GPU MoE cache, 32K context."

exec "$BINARY" \
  --model "$MODEL_FILE" \
  --host 127.0.0.1 --port "$PORT" \
  -ngl all -fit off \
  -c 32768 -b 4096 -ub 512 -np 1 \
  -t 8 \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode mmap \
  --lazy-mode on \
  --moe-expert-cache-size 48 \
  --cache-ram 0 \
  --alias qwen38-flash,Qwen3.8-Flash-Next,qwen3.8-flash \
  --jinja

And the companion stop script (~/projects/llama-moe-cache/stop-llama-moe.sh):

#!/usr/bin/env bash
set -euo pipefail

PORT="${LLAMA_PORT:-8080}"

echo "Stopping llama-server on port ${PORT}..."
pkill -f "llama-server.*--port ${PORT}" || true
pkill -f "llama-server.*Qwen3.8-Flash-Next" || true
sleep 1

if ss -tulpn 2>/dev/null | grep -q ":${PORT}\b"; then
  fuser -k "${PORT}/tcp" 2>/dev/null || true
fi

echo "llama-server stopped. VRAM released."

Make both executable and symlink them into ~/.local/bin/:

chmod +x ~/projects/llama-moe-cache/start-llama-moe.sh
chmod +x ~/projects/llama-moe-cache/stop-llama-moe.sh
ln -sf ~/projects/llama-moe-cache/start-llama-moe.sh ~/.local/bin/start-llama-moe
ln -sf ~/projects/llama-moe-cache/stop-llama-moe.sh ~/.local/bin/stop-llama-moe

Step 4: Connecting to Terminal Agents (Pi, Crush, and Oh My Pi)

llama-server exposes a standard OpenAI-compatible API on port 8080.

Configuring Pi (~/.pi/agent/models.json)

Add the llamacpp provider to your Pi configuration:

{
  "providers": {
    "llamacpp": {
      "baseUrl": "http://127.0.0.1:8080/v1",
      "api": "openai-completions",
      "apiKey": "local-development-bypass",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false,
        "maxTokensField": "max_tokens"
      },
      "models": [
        {
          "id": "qwen38-flash",
          "name": "Qwen 3.8 Flash Next 177B MoE",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 65536,
          "maxTokens": 16384
        }
      ]
    }
  }
}

Check that Pi detects the model:

pi --list-models

Output:

provider   model                  context  max-out  thinking  images
llamacpp   qwen38-flash           65.5K    16.4K    yes       no

Configuring Oh My Pi (~/.omp/agent/models.yml)

Add the corresponding block in ~/.omp/agent/models.yml:

providers:
  llamacpp:
    baseUrl: http://127.0.0.1:8080/v1
    api: openai-completions
    apiKey: local-development-bypass
    compat:
      supportsDeveloperRole: false
      supportsReasoningEffort: false
      maxTokensField: max_tokens
    models:
      - id: qwen38-flash
        name: Qwen 3.8 Flash Next 177B MoE
        reasoning: true
        input:
          - text
        contextWindow: 65536
        maxTokens: 16384

Configuring Crush (~/.config/crush/crush.json)

Add the llamacpp-local provider to your Crush configuration:

{
  "options": {
    "request_timeout": 600
  },
  "models": {
    "default": {
      "provider": "llamacpp-local",
      "model": "qwen38-flash"
    },
    "large": {
      "provider": "llamacpp-local",
      "model": "qwen38-flash"
    },
    "small": {
      "provider": "freetoken-local",
      "model": "qwen3.6-35b",
      "reasoning_effort": "low"
    }
  },
  "providers": {
    "llamacpp-local": {
      "name": "llama.cpp Local",
      "base_url": "http://127.0.0.1:8080/v1",
      "type": "openai-compat",
      "api_key": "local-development-bypass",
      "models": [
        {
          "id": "qwen38-flash",
          "name": "Qwen 3.8 Flash Next 177B MoE",
          "context_window": 65536,
          "default_max_tokens": 16384,
          "can_reason": true,
          "supports_attachments": false,
          "cost_per_1m_in": 0,
          "cost_per_1m_out": 0,
          "cost_per_1m_in_cached": 0,
          "cost_per_1m_out_cached": 0
        }
      ]
    }
  }
}

Verify that Crush recognizes the local endpoint:

crush models

Notice the "options": {"request_timeout": 600} setting. By default, Crush enforces a 60-second deadline before the first token arrives (LLM stream received no data for 1m0s). Because cold prompt evaluation on a 177B model across 1,500+ agent tokens can take 70 to 120 seconds before the first token is emitted, setting request_timeout: 600 prevents premature client-side aborts.

The Real Numbers

Once loaded, we ran generation benchmarks against the local endpoint:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen38-flash",
    "messages": [
      {"role": "user", "content": "Write a python one-liner to calculate the factorial of 10."}
    ],
    "temperature": 0.0,
    "max_tokens": 512
  }' | jq .

Here is what the timing and memory measurements showed:

The model cleanly outputs reasoning tokens inside reasoning_content before delivering the final answer in content, matching the native DeepSeek/Qwen thinking format.

We also verified non-interactive execution directly from Pi:

pi -p --provider llamacpp --model qwen38-flash "Print 'HELLO_FROM_QWEN_177B' and nothing else."

In the server logs, Pi’s full 2,603-token agent prompt (including tool schemas and system instructions) evaluated at 10.82 tokens per second (240 seconds total). When the response returned, the follow-up turn reused 99% of the prefix via LCP cache matching, evaluating the new 23 tokens in just 4.27 seconds (5.38 t/s) before printing:

HELLO_FROM_QWEN_177B

The Practical Workflow: Model Swapping

A decode speed of 3 to 5 tokens per second means you probably do not want Qwen 3.8 Flash Next as your interactive auto-complete or rapid iterative test-runner. For typing code or running small unit test fixes, 65 tokens per second on 35B is much more responsive.

Where the 177B model shines is as an architectural thinker:

• Planning a tricky refactor across several files.
• Diagnosing a subtle race condition in distributed code.
• Analyzing complex system boundaries.

Because both servers are packaged cleanly into scripts, switching between them takes under a minute:

# Running daily coding loops (65 t/s, 35B model)
start-ft

# Switch to heavy architectural reasoning (177B model)
stop-ft
start-llama-moe

# Return to fast loops
stop-llama-moe
start-ft

On an NVMe SSD, switching models is just a stop command followed by a 40-second start command.

Wrapping Up

Running a 177-billion-parameter MoE model locally on a single $280 graphics card with 12 GB of VRAM sounded far-fetched until recently. GenerelSchwerz’s llama-moe-cache fork shows how much performance can be extracted from consumer hardware when dynamic caching, NVMe bandwidth, and 8-bit KV quantization work together.

The rules for making it work reliably:

• Use NVMe storage. SATA drives will bottleneck the expert streaming.
• Always use --load-mode mmap to prevent anonymous RAM exhaustion.
• Quantize the KV cache to 8-bit (-ctk q8_0 -ctv q8_0) to leave room in VRAM for the 64K context window and the 20-slot expert cache.
• Pin threads to physical cores (-t 8 on 8-core CPUs) to avoid SMT spin-wait penalties.

This concludes our three-part local coding agent series:

• Part 1: Running Qwen 3.5 MoE Locally on an RTX 3060 with FreeToken, Crush, and Pi
• Part 2: Benchmarking Terminal Coding Agents: 640 Public Trials and 280 Private Trials
• Part 3: Running 177B MoE Models on a 12 GB GPU with llama-moe-cache and NVMe Streaming