SGLang adds opt-in expert paging for models that exceed a GPU’s memory

SGLang added opt-in expert paging for some models that exceed a GPU’s memory.

**Dek:** Version 0.5.23 gives operators another way to serve mixture-of-experts models and introduces optional LoRA serving for selected quantized MoE formats. Its benchmark numbers describe specific configurations.

The SGLang project released version 0.5.23 on November 2 UTC, November 3 in India. The release includes an opt-in Paged Experts feature that the maintainers say can run mixture-of-experts models larger than available GPU memory on a single consumer GPU. It also adds an opt-in LoRA engine for selected quantized MoE models and documents changes to the first-token latency of certain configurations on H200. These are independent release features. A serving deployment may use one without the others. [SGLang 0.5.23 release](https://github.com/sgl-project/sglang/releases/tag/v0.5.23)

What expert paging changes

Mixture-of-experts models route a token through a subset of a model’s expert networks. The model can have many more expert weights in total than it uses for one token. Holding every expert resident in accelerator memory can limit which model sizes a given GPU can serve. Paging gives the serving runtime another option for placing and loading those experts. The relevant trade-off is that weights which are not resident must be brought into the working set when needed, so performance can depend on routing and how often that working set changes.

The maintainers describe an improvement of up to 30 times versus CPU offload for their comparison. The phrase “up to” and the baseline both matter. The announcement does not establish the same gain against a model that fits entirely in GPU memory, nor does it establish that every consumer card supports every model. Before acting on the number, an operator should read the linked changes and identify the test’s model, hardware and workload. [SGLang 0.5.23 release](https://github.com/sgl-project/sglang/releases/tag/v0.5.23)

The release also reports shorter time to first token on particular mixture-of-experts models with 8K-token prompts on H200. That measurement describes a separate improvement and should not be used to estimate the performance of the single-GPU paging setup. First-token latency and decode throughput can move differently, especially when a service handles concurrent requests. [SGLang 0.5.23 release](https://github.com/sgl-project/sglang/releases/tag/v0.5.23)

How to evaluate it for a serving workload

Use a checkpoint and quantization format the project documents, then record startup memory, memory during traffic, time to first token and output throughput. Keep prompt sizes and concurrency the same across comparisons. A cold start, a long conversation and a burst of concurrent users exercise different parts of the serving system. Test the application’s actual response schema and tool calls if it depends on them, since successful token generation by itself does not show that a serving upgrade preserves the client contract.

For small deployments, the feature opens a practical experiment: see whether a model outside the device’s resident-memory budget can meet a specific application’s latency and throughput targets. For larger deployments, the same release’s LoRA and caching changes may be more relevant. The choice should come from the workload and support matrix, not the largest benchmark number in the release notes.