Article may be outdated

This article is 4 days old. Some details may have changed since publication.

Hacker News·5 min read·hard

Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM

W
wondersky
Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM
✦AI Summary

This technical guide explains how to serve the 753-billion parameter GLM-5.3 model across two Nvidia DGX stations using vLLM and tensor parallelism. It details the hardware requirements and configuration needed to manage model weights that exceed the memory capacity of a single GPU.

Why it matters

As AI models grow in size, distributed inference techniques are becoming essential for developers and researchers working with frontier-scale models.

✦Dive DeeperCreate a free account to unlock

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with tensor parallelism splitting the model weights and computation across both GPUs.

The NVIDIA GLM-5.3 NVFP4 model, a quantized version of Z.ai ’s mixture-of-experts (MoE) model, has 753 billion parameters, with 40 billion activated per token. Sparse activation reduces computation, but all model weights still need to remain available because each token can be routed to different experts. At about 464 GB, the checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU. This tutorial uses two DGX Stations to keep the weights in GPU memory, with CPU weight offload disabled.

Before starting, connect the two Stations with the Connect Two DGX Stations playbook .

Continue reading on Headlinne

Create a free account to read the full article.

Read full article →
technologyai
✦

Get smarter about the news

Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.

Create free account

Already have an account? Sign in