Running 177B MoE Models on a 12 GB GPU with llama-moe-cache and NVMe Streaming [Part 3]

I wanted to try Qwen 3.8 Flash Next, a 177B MoE model, on the same RTX 3060 used in Part 1. Its quantized weights occupy 76.3 GiB, and my machine has 12 GB of VRAM and 32 GB of system RAM.

The first launch froze my desktop. Getting it working required changing how the server loaded the weights.

I used the UD-IQ3_XXS quantization and GenerelSchwerz’s llama-moe-cache branch to test GPU expert caching with weights backed by NVMe storage.

[Read More]

Running Qwen 3.6 35B NVFP4 Locally with FreeToken: Fitting a Coding MoE into 12GB VRAM [Part 1]

This is Part 1 of a three-part series on running, benchmarking, and scaling local coding agents. In Part 2, we benchmark this setup across 640 public trials and 280 private trials on a production Go monorepo, including Claude Code, Codex, Antigravity, local Qwen, and two hosted models. In Part 3, we build and test the specialized llama-moe-cache fork to run 177B Qwen3.8-Flash-Next on desktop hardware.

I wanted a capable local reasoning model for coding agents on my workstation, but my GPU is a standard desktop NVIDIA GeForce RTX 3060 with 12GB VRAM. The weights would not all fit in VRAM, so I needed a setup that could offload them.

[Read More]

Arch Linux: Hyprland 0.46.2 Crashes After Upgrade — Here's a Temporary Workaround and a hyprutils-git Conflict Fix

Yesterday I encountered a nasty surprise after upgrading Hyprland to version 0.46.2—it crashed on startup, leaving me unable to log into my Wayland session. If you’re experiencing this problem, here’s a temporary workaround so you can keep working until a fix arrives.


The Issue

Hyprland 0.46.2 appears to crash immediately upon launch, preventing you from getting into your usual tiling window manager environment.


Update Dec 29 2024: hyprutils-git Caused Compatibility Issues

After some further digging, I discovered my Hyprland crash was caused by the AUR hyprutils-git package. I had previously switched to hyprutils-git from the AUR because I was also using hyprpolkitagent-git. Because of this, the entire Hyprland ecosystem ended up depending on the hyprutils-git package, which broke compatibility with the stable release of Hyprland in the last few updates.

[Read More]

Hyprland Quick-Start Guide Part 2: Enhancing Your Setup with Advanced Features

Hyprland is not just a tiling window manager; it’s a gateway to a highly customized and efficient Linux environment. In this second part of our guide, let’s explore some advanced features for your Hyprland setup.

In the previous part, we’ve covered the basics of setting up and configuring Hyprland. We’ve done a lot of work, and it would be a shame to lose it. So, first, let’s start by saving our configurations using GNU Stow in a GitHub repository.

[Read More]

Hyprland Quick-Start Guide: Setting Up a Modern Tiling Window Manager on Linux

Hyprland is a dynamic tiling window manager built for the modern Wayland display server protocol. This guide, we’ll walk you through the process of setting up Hyprland on your Linux system.

Why Wayland?

First and foremost, performance and reduced input lag. Wayland is a modern replacement for X11, and it has much cleaner architecture, better application separation, and improved HiDPI hardware support.

What is a Tiling Window Manager and Why Should I Care?

So, a tiling window manager, as opposed to a floating window manager, organizes your windows in a non-overlapping, grid-like pattern. It will give you a superhuman ability to fly through your windows and workspaces with hotkeys and improve your productivity.

[Read More]