GPU selection

Key concepts

• Count all model weights in VRAM for full-GPU residency. For hybrid inference, budget host DRAM and GPU VRAM separately based on the actual placement.
• VRAM bandwidth, PCIe bandwidth, and host DRAM bandwidth are separate. Check runtime support, power, warranty, and price.
• Supported Strata modes store MoE experts in DRAM. Budget the GPU-resident weights, expert cache, GPU portion of KV cache, and workspace. Some modes compute experts on CPU or let the GPU access them.
• Evaluate architecture, quantization, runtime version, context, and concurrency. Test startup, peak DRAM and VRAM, latency, and output quality.
• Check capacity claims: 70B × 2 bytes for BF16 is about 140 GB for weights alone. Add KV cache and runtime buffers. Evetech’s 24 GB figure is not a budget for full, unquantized VRAM residency.

Checkpoint

Your coding model works, but its weights exceed one GPU. Which tradeoff should the system prioritize?

1. Keep the model on one GPU

   Fits interactive workloads when a smaller or quantized model still passes acceptance.

   Tradeoffs · Simpler residency can trade model quality for memory savings.

   Verify · Compare acceptance, first-token latency, and correction time on fixed tasks.

2. Evaluate Strata hybrid inference

   Fits supported MoE models with sufficient host memory and tuning capacity.

   Tradeoffs · Reduces VRAM residency requirements but can shift bottlenecks to host memory, CPU, PCIe, or expert cache.

   Verify · Record model, quantization, version, and placement, then test startup, long contexts, and shared load.

3. Evaluate multiple GPUs

   Fits workloads needing more GPU residency with room to expand.

   Tradeoffs · Adds hardware, power, interconnect traffic, and backend requirements.

   Verify · Verify the runtime partitioning method and benchmark the actual PCIe topology.

Hands-on check

Record DRAM/VRAM placement, peaks on both sides, concurrency, acceptance, time, and quotes.

References

• See the GPU bandwidth cards
• Strata
• Hardware primer · Evetech

Illustration credits

• GPU RESIDENCY · AI-generated concept illustration. Verify actual specifications in product documentation.
• HYBRID MoE · AI-generated concept illustration. Verify actual specifications in product documentation.
• MULTI-GPU · AI-generated concept illustration. Verify actual specifications in product documentation.

← Back to the field lab