Running Qwen 3.6 35B NVFP4 Locally with FreeToken: Fitting a Coding MoE into 12GB VRAM [Part 1]

This is Part 1 of a three-part series on running, benchmarking, and scaling local coding agents. In Part 2, we benchmark this setup across 640 public trials and 280 private trials on a production Go monorepo, including Claude Code, Codex, Antigravity, local Qwen, and two hosted models. In Part 3, we build and test the specialized llama-moe-cache fork to run 177B Qwen3.8-Flash-Next on desktop hardware.

I wanted a capable local reasoning model for coding agents on my workstation, but my GPU is a standard desktop NVIDIA GeForce RTX 3060 with 12GB VRAM. Running modern 30B+ coding models locally usually demands either punishing quantization trade-offs or multiple datacenter cards.

After testing several options, I got Qwen 3.6 35B A3B NVFP4 running reliably with a full 64k token context window using FreeToken (ft), and hooked it up to both Crush and the Pi Coding Agent (pi). Getting there was not plug-and-play.. We ran into Hugging Face download pattern bugs, CUDA JIT compiler lookup failures, an aggressive 8k context ceiling that killed agent loops, and activation OOM spikes during long prefills.

Here is the complete walkthrough of what worked, what broke, and how to reproduce the setup on another machine.

The Hardware and the Model Search

My workstation specs for this run:

• GPU: NVIDIA GeForce RTX 3060 (12GB GDDR6 VRAM)
• System RAM: 32GB DDR4
• OS: Arch Linux (Kernel 6.13, CUDA 12.8 in /opt/cuda)
• Storage: Fast NVMe SSD (/dev/nvme0n1)

A quick note on disk storage: when I first downloaded the weights, the Hugging Face cache lived on a mechanical SATA HDD. Cold starts took several minutes just reading the 21.8 GB safetensors files into memory. Moving the Hugging Face cache directory to a native NVMe SSD (HF_HOME=~/.cache/huggingface) slashed weight loading time down to 8 seconds, with the entire server ready in ~55 seconds. Don’t run offloading from a spinning disk.

Before landing on the right model, we hit two dead ends:

1. Qwen3.8-27B-FP8: This is a dense model, not an MoE. Even in FP8, holding 27B weights requires around 27GB of VRAM just to load. On a 12GB card, it can’t even initialize without thrashing offload into unusable territory.
2. Qwen3.8-Flash-Next: An MoE model, but its expert offload architecture requires roughly 47.7 GiB of pinned host RAM. On a 32GB machine, host memory ran out instantly.

The sweet spot turned out to be nvidia/Qwen3.6-35B-A3B-NVFP4. It is a 35B Mixture-of-Experts model, but only around 3B parameters are active per token (A3B). The weights are quantized to NVIDIA FP4 (NVFP4), totaling 21.8 GB on disk. FreeToken caches the base weights and the hot working set of experts in VRAM, streaming inactive experts from host RAM as needed.

Step 1: Setting Up the Python Environment

FreeToken works best in an isolated environment. I created a dedicated virtual environment with Python 3.13:

# Using virtualfish or standard venv
python -m venv ~/.virtualenvs/freetoken
source ~/.virtualenvs/freetoken/bin/activate

# Install PyTorch with CUDA 12.8 support and FreeToken
pip install --upgrade pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install freetoken

Step 2: The Traps We Hit (and How We Patched Them)

FreeToken is fast, but we ran into several sharp edges during initial model download and kernel compilation.

1. Hugging Face Download Pattern Missing Config Files

When downloading nvidia/Qwen3.6-35B-A3B-NVFP4, FreeToken’s internal download helper used a restrictive pattern that grabbed .safetensors files but missed .json files. Without config.json and tokenizer metadata, the server crashed on startup.

To fix it, we patched ~/.virtualenvs/freetoken/lib/python3.13/site-packages/freetoken/utils/hf.py so snapshot_download includes *.json:

# In freetoken/utils/hf.py around line 211
return snapshot_download(
    model_path,
    allow_patterns=["*.safetensors", "*.json"],
    tqdm_class=DisabledTqdm,
)

2. Triton and NVCC Path Resolution

During model initialization, FreeToken compiles custom Triton kernels for the NVFP4 expert layers. On Arch Linux, the CUDA toolkit lives in /opt/cuda/bin, which was not in the system’s default $PATH. FreeToken failed with missing nvcc errors.

We resolved this with two targeted fixes:

First, create an nvcc wrapper in your local user path:

mkdir -p ~/.local/bin
cat << 'WRAPPER' > ~/.local/bin/nvcc
#!/bin/sh
exec /opt/cuda/bin/nvcc "$@"
WRAPPER
chmod +x ~/.local/bin/nvcc

Second, ensure the virtualenv’s ft launcher automatically injects CUDA_HOME and /opt/cuda/bin:

# Edit ~/.virtualenvs/freetoken/bin/ft right before main import
import os, sys
os.environ.setdefault("CUDA_HOME", "/opt/cuda")
if "/opt/cuda/bin" not in os.environ.get("PATH", ""):
    os.environ["PATH"] = f"/opt/cuda/bin:{os.environ.get('PATH', '')}"

3. PyTorch Memory Allocator Settings

To prevent VRAM memory fragmentation warnings and allocation failures, set the allocator configuration in your shell or profile:

export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"

4. The 8k Token Wall (Context Ceiling in Coding Agents)

This was the biggest functional issue. By default, FreeToken calculates how many KV cache pages to allocate based on remaining VRAM after loading weights. To keep a large MoE expert cache in GPU memory, it defaulted to only 8,204 tokens.

When running a real coding task in Crush or Pi, system instructions, tools, and repo context quickly reach 10,000 to 15,000 tokens. The agent immediately failed with:

Agent processing failed: failed to start agent processing stream: stream error:
prompt is too long: 13283 tokens > 8204 maximum (prompt + generation);
shorten the prompt or increase the KV cache budget.

The fix is to explicitly reserve the KV cache budget with --kv-reserve-tokens 65536.

5. Prefill Memory Spikes and Activation OOM

When you allocate 64k tokens of KV cache on a 12GB card, VRAM is tight. If an agent sends a 15k token prompt and the server tries to process the entire prefill in one forward pass, activation memory spikes and triggers CUDA Out-of-Memory.

We solved this with two parameters:

• --memory-ratio 0.82: Allocates 82% of VRAM to static weights and caches, leaving ~1.7 GB of headroom for dynamic activations.
• --max-prefill-length 4096: Chunks long prompt prefills into 4k token slices, capping activation memory spikes.

6. The 8k Output Token Cap

By default, FreeToken caps maximum generation output at 8,192 tokens. For simple chat completions that is plenty, but agent loops solving multi-file refactors or emitting detailed reasoning chains can easily exhaust an 8k output window in a single turn. When that happens, the model stops mid-sentence, leaving the agent with an incomplete patch or an empty response.

Adding --max-output-tokens 16384 doubles the headroom to 16k tokens.

Step 3: Launching the FreeToken Server

Here is the production command to launch the server:

ft serve \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --moe-backend auto \
  --kv-reserve-tokens 65536 \
  --memory-ratio 0.82 \
  --max-prefill-length 4096 \
  --max-output-tokens 16384 \
  --port 1420

To avoid port collisions or accidentally starting duplicate instances, I wrapped this in a startup script (~/.local/bin/start-ft):

#!/usr/bin/env bash
set -euo pipefail

if ss -tulpn 2>/dev/null | grep -q ":1420\b"; then
  echo "FreeToken server is already running on port 1420."
  exit 0
fi

export HF_HOME="${HF_HOME:-$HOME/.cache/huggingface}"
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"

exec ~/.virtualenvs/freetoken/bin/ft serve \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --moe-backend auto \
  --kv-reserve-tokens 65536 \
  --memory-ratio 0.82 \
  --max-prefill-length 4096 \
  --max-output-tokens 16384 \
  --port 1420

On the very first launch, FreeToken builds the NVFP4 Triton expert banks in serial. This step takes around 5 to 7 minutes on an RTX 3060:

[core|rank=0] INFO expert banks: slow path (serial build)
Loading Qwen3.5 NVFP4 experts: 100%|██████████| 3/3 [06:48<00:00, 136.25s/it]
[core|rank=0] INFO NVFP4 expert backend: triton
[core|rank=0] INFO --moe-cache-auto resolved moe_cache_size=2283 num_pages=32781
[core|rank=0] INFO Allocating 65536 tokens for KV cache
[core|rank=0] INFO Free memory after initialization: 1.72 GiB
[core|rank=0] INFO Application startup complete. Uvicorn running on http://127.0.0.1:1420

Once running, verify the endpoint with a quick curl test:

curl -s http://127.0.0.1:1420/v1/models | jq .

Step 4: Configuring Crush CLI

Crush is an agentic coding CLI. To configure it for our FreeToken server, update ~/.config/crush/crush.json:

{
  "$schema": "https://charm.land/crush.json",
  "models": {
    "default": {
      "provider": "freetoken-local",
      "model": "qwen3.6-35b"
    }
  },
  "providers": {
    "freetoken-local": {
      "name": "FreeToken Local",
      "base_url": "http://127.0.0.1:1420/v1",
      "type": "openai-compat",
      "api_key": "local-development-bypass",
      "models": [
        {
          "id": "qwen3.6-35b",
          "name": "nvidia/Qwen3.6-35B-A3B-NVFP4",
          "context_window": 65536,
          "default_max_tokens": 8192,
          "can_reason": true,
          "supports_attachments": false
        }
      ]
    }
  }
}

Now Crush can run full-context tasks without hitting context length errors:

crush run "Write a bash script to check current memory usage"

Step 5: Configuring Pi Coding Agent

The Pi Coding Agent (pi) is a terminal coding agent that supports custom providers and models via ~/.pi/agent/models.json.

Here is the configuration to register FreeToken:

{
  "providers": {
    "freetoken": {
      "baseUrl": "http://127.0.0.1:1420/v1",
      "api": "openai-completions",
      "apiKey": "local-development-bypass",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false,
        "maxTokensField": "max_tokens"
      },
      "models": [
        {
          "id": "qwen3.6-35b",
          "name": "Qwen 3.6 35B A3B NVFP4",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 65536,
          "maxTokens": 8192
        }
      ]
    }
  }
}

A few important details for Pi:

• apiKey: Pi requires an API key value to show the model as authorized in /model and --list-models, even though our local server does not enforce auth. A dummy string satisfies the check.
• compat.supportsDeveloperRole: false: Ensures Pi sends instructions under the standard system role rather than the OpenAI developer role.
• compat.supportsReasoningEffort: false: Prevents sending unsupported reasoning effort flags that FreeToken ignores or rejects.

Set it as the default startup model in ~/.pi/agent/settings.json:

{
  "theme": "dark",
  "defaultProvider": "freetoken",
  "defaultModel": "qwen3.6-35b"
}

Verify that Pi detects the model:

pi --list-models

You should see:

provider   model        context  max-out  thinking  images
freetoken  qwen3.6-35b  65.5K    8.2K     yes       no

Test it non-interactively:

pi -p "Write a python one-liner to print current date"

Pi will stream its reasoning tokens, execute the request, and finish cleanly. It also executes tools (file reading, writing, and terminal commands) through FreeToken without issues.

Step 6: Configuring Oh My Pi (omp)

We also wired the same local FreeToken endpoint into Oh My Pi (omp). Since FreeToken implements a standard OpenAI-compatible API at http://127.0.0.1:1420/v1, any terminal agent that accepts custom endpoints can use it. In our benchmarks, we tested omp with low reasoning effort and the 16k output token cap.

Realistic Expectations and Caveats

Before relying on this as your daily driver, keep a few trade-offs in mind:

• First boot build time: The initial Triton compilation takes around 6 minutes. Do not interrupt it. Once built, subsequent starts take under 20 seconds.
• Generation speed: Because inactive experts are paged from system RAM, generation speeds sit around 10 to 18 tokens per second on an RTX 3060. That is plenty fast for coding tasks, but slower than running small 7B models entirely resident in VRAM.
• Keep context windows aligned: Make sure --kv-reserve-tokens on the server, context_window in Crush, and contextWindow in Pi all match (65536). If an agent assumes a 128k window while the server is capped at 64k, the server will reject long conversation branches.

Wrapping Up

Running a 35B MoE coding model on a 12GB desktop card used to be out of reach. With FreeToken’s NVFP4 support and offloading, plus the right prefill and KV cache flags on fast NVMe storage, nvidia/Qwen3.6-35B-A3B-NVFP4 runs reliably with a 64k context window.

Crush, Pi, and Oh My Pi all work with this endpoint, giving you a capable, private, and fully local coding assistant without relying on external API credits.

So, how well does this local setup actually code when put up against the frontier cloud models?

In Part 2: Benchmarking Terminal Coding Agents: 640 Public Trials and 280 Private Trials, we run an empirical benchmark across eight public tasks and sixteen agent configurations (640 trials), followed by 280 private trials inside a 15-package production Go monorepo to see where a 35B local model holds up, where it falls short, and what happens when terminal agents try to escape their workspaces.

And in Part 3, we push consumer hardware even further: building and testing the specialized llama-moe-cache fork to stream the 177B Qwen3.8-Flash-Next model on our 12GB card.