GPU clusters that are busy, not waiting
GPUs are the most expensive part of an AI platform, and also the easiest to waste. They sit idle when the network is too slow, when storage cannot feed them data, or when the data centre cannot power and cool them. We design the whole stack around the GPUs: servers, fabric, storage, scheduling, power and cooling.
Why GPU projects disappoint
The data centre is not ready
One 8-GPU server draws about 10 kW. Many enterprise racks are built for 5 to 10 kW in total. The servers arrive and there is nowhere to plug them in.
GPUs wait on the network
Training across servers exchanges data after every step. On ordinary data-centre networking, GPUs spend much of their time waiting.
Storage cannot keep up
Loading data and writing checkpoints on general-purpose storage leaves expensive GPUs idle for minutes at a time.
Nobody can see utilisation
Without GPU-aware scheduling and monitoring, shared clusters run well below capacity while teams queue for access.
How many GPUs do I need?
Pick a model size and what you want to do with it. The estimate shows GPU memory, number of GPUs and servers, and roughly how much power they draw.
Six layers of a GPU platform
| Layer | What it is | What goes wrong if it is weak |
|---|---|---|
| GPU servers | 8-GPU servers (HGX H100, H200, B200, B300 and similar), or PCIe servers (H100 NVL, H200 NVL, L40S, L4) for lighter or inference-only work | Wrong GPU for the job: paying for training-class GPUs to serve small models, or the reverse |
| GPU fabric | A dedicated network between GPUs, usually 400 Gb/s per GPU over InfiniBand or RoCE Ethernet | Multi-server training runs at a fraction of its potential speed |
| Storage | A fast parallel file system for datasets and checkpoints, plus object storage for raw data | GPUs idle while data loads or checkpoints write |
| Scheduling | Slurm for training, Kubernetes with the NVIDIA GPU Operator for serving, or both | Queues, idle GPUs and arguments about who gets access |
| Monitoring | GPU health and utilisation (DCGM), job metrics and cost per team | Failing GPUs and wasted capacity go unnoticed |
| Power and cooling | High-density racks, often 30 kW or more, with rear-door or liquid cooling | The cluster cannot be installed, or runs throttled because it overheats |
Guides and examples
Training cluster
How a multi-server training cluster is laid out, with an animated job that survives a node failure.
Read the guide →Inference platform
Serving models to users: GPU choice, model servers, sharing GPUs and scaling.
Read the guide →GPU network fabric
InfiniBand or Ethernet, rail-optimised design, and how many switches you need.
Read the guide →Storage for AI
Parallel file systems, checkpoint sizes and how fast storage really needs to be.
Read the guide →Power and cooling
kW per server and per rack, air vs liquid cooling, and checking your data centre.
Read the guide →Use case: private LLM platform
An illustrative scenario: 32 GPUs for inference and fine-tuning, sized end to end.
Read the use case →GPU questions, answered
Buy or rent, H100 vs H200 vs B200 vs L40S, lead times, InfiniBand and more.
Read the FAQ →For the full technical reference, see the GPU cluster whitepaper.
Buy, host or rent
| Option | Good for | Trade-off |
|---|---|---|
| Buy and host in your own data centre | Steady, long-term use; data that must stay on your premises | Large upfront cost, lead times, and your data centre must handle the power |
| Buy and place in a colocation facility | Same as above when your own data centre cannot take the power density | Colocation contract and remote hands; still a capital purchase |
| Rent from a GPU cloud | Short projects, experiments, bursts of training | Highest cost per GPU-hour over time; capacity can be limited at busy times |
Many organisations start in a GPU cloud to learn their real usage, then buy for the steady part.
How a GPU engagement runs
- Workload first. Which models, training or inference, how many users, what latency. This decides everything else.
- Sizing. GPUs, servers, fabric, storage and software, with the assumptions written down.
- Data-centre readiness. Power per rack, cooling, floor loading and cabling, checked before anything is ordered.
- Bill of materials. Vendor-neutral, with alternatives where lead times or budget require them.
- Build and burn-in. Installation, fabric validation and stress tests before users get access.
- Handover. Scheduler, monitoring and runbooks in place, with your team trained to run them.
Planning a GPU cluster?
Tell us the models you want to run or train, how many users, and where it will be hosted. We will come back with a first sizing: GPUs, servers, network, storage, and the power and cooling your data centre will need.