Hugging Face makes Transformers stream MoE experts from disk
- Change Transformers decodes 200 tokens at 13.9 tok/s while Strata runs at 14.8 tok/s, a 1.06x gap, Hugging Face says.
- Detail Transformers offloaded MoE models through Accelerate by copying a whole layer's expert bank to the GPU on every token, with no cache or routing.
- Detail The new experts_offload option is added to AutoModelForCausalLM.from_pretrained in a Transformers pull request.
- Detail Experts mapped to "disk" use the streaming path instead of Accelerate offload, and MoE layers switch to an offloaded implementation.
- Detail The streaming path uses a bounded cache implemented as a shared slot pool tensor plus per-layer tables from expert id to slot.
- Detail Cache misses are read from SSD by 8 I/O threads, using one contiguous read per expert.
- Detail The method can run Qwen3.8-Flash-Next, which has 48 layers of 512 experts (10 active), whose experts are 121 GB in FP8.
- Detail On a 32 GB Mac, experts quantized to 3 bits take 49 GB on disk with 12 GiB cached on device, and dense layers take about 5.4 GB in 8 bits.
- Detail With these optimizations on the same Mac, Transformers decodes 200 tokens at 13.9 tok/s while Strata runs at 14.8 tok/s, a 1.06x gap.
- Affected Builders who want to run MoE models like Qwen3.8-Flash-Next through Transformers with disk-based expert streaming.
Source: huggingface.co ·