Skip to content
ignisignis

A 125B MoE on a 32 GB card

Qwen3.8-Flash-Next is a 125B-parameter mixture of experts. It does not fit in 32 GB, and on most engines that ends the discussion. ignis stores its experts in a 2.5-bit trellis code, keeps the ones in use in a VRAM cache and the rest in pinned host RAM, and decodes at about 100 tok/s on one lane on an RTX 5090 with a PCIe 3.0 host. On an RTX PRO 6000, where the experts fit, the same model with its MTP head reaches 258 tok/s on one lane of code and 427 tok/s across eight.

How it works

Section titled “How it works”

Each of Flash-Next’s 48 layers holds 512 small MLPs, the experts, and a router picks ten per token. Only about 6B parameters work on a given token, which is why a model this size can decode fast at all. The experts are re-quantized to a trellis code with a K per expert: 2, 2.5, 3 or 4 bits, chosen from calibration error, 2.5 bits per weight on average. At that rate the 120.8B expert parameters take 37.7 GB, pinned in host RAM (ADR 0044).

On a 32 GB card a VRAM expert cache holds the experts in use, replaced least-recently-used, and a token whose experts are not there waits for them to cross PCIe. So decode on this card is bound by the link, and the engine works on that. A router lookahead copies the next layer’s experts ahead of need, under a per-step budget. An opt-in pinned hot set keeps the calibration-hottest experts out of the LRU for the life of the load (ADR 0055). The KV pool is reserved first and the cache takes what is left, so the cache and the pool trade against each other, and the pool follows residency (ADR 0045).

On a card that holds every expert, around 96 GB, the experts are read from the artifact straight into their VRAM slots and never copied again; no host pool exists (ADR 0054). There the decode is limited by weight streaming as the 27B’s is, so Flash-Next’s MTP head pays; on the 5090 it is off by default, because a verify round selects experts for every column and so copies more of them.

Where each piece sits, and why: Models and artifacts.

The numbers

Section titled “The numbers”
Figure Measured on Source
101.5 tok/s, 9.85 ms a token, one lane RTX 5090, PCIe Gen 3 host, after decode fusion Flash-Next decode fusion
99 tok/s at --decode-lanes 1; the 3-lane default costs a lone user about 10 tok/s (99 against 89 in the one measured pair) RTX 5090 Flash-Next on the 5090
2.5 bits a weight on average (K of 2, 2.5, 3 or 4); 37.7 GB of experts pinned calibration-derived allocation ADR 0044
MTP on the 5090: 1.02-1.30x on one lane, 7-15% slower at two and three lanes at the default row budget RTX 5090 MTP is PCIe-bound
258.2 tok/s one lane of code, 427.5 tok/s across eight lanes of code (MTP, 3 drafts), 512 tokens RTX PRO 6000 Server Edition RTX PRO 6000 profile
MTP up to 1.85x speculation off on one lane of code same, every lane decoding same
Lanes keep 15.6-18.3 tok/s while a prompt prefills at decode share 25 RTX 5090 Prefill chunk and decode share

See it, control it

Section titled “See it, control it”

Limits

Section titled “Limits”
  • On a 32 GB card the decode is bound by PCIe, not by the GPU. MTP is off by default there and helps only one user or one agent at a time.
  • The load needs about 46-47 GB of available RAM, plus the n-gram table’s read path (n-gram cache).
  • Below a 12 GiB expert cache the start is refused, because decode slows sharply there.
  • The 258 and 427 tok/s are one run each on one server; the finding states its own conditions.

Read more

Section titled “Read more”