GPU Infrastructure for Enterprise AI

Executive summary

GPUs are the most expensive part of an AI platform and the easiest to waste. They sit idle when the network between servers is too slow, when storage cannot feed them, when the data centre cannot power and cool them, or when nobody can see who is using them. A GPU cluster is a system: servers, fabric, storage, scheduler, power and cooling, all designed around the GPUs.

This paper sets out how to size, build and run that system for training and inference, with the formulas and planning figures needed to check a vendor proposal or build a business case.

Why GPU projects disappoint

The pattern is familiar. The servers are ordered quickly because GPUs are scarce, and the rest of the system is designed afterwards. Months later the cluster is installed but slow, half-used or not running at all.

Training and inference sizing, done properly

Memory per parameter

A model’s size in memory is its parameter count times the bytes used per parameter. That depends on precision and on whether you are training or serving.

The KV cache

When a model generates text, it keeps a cache of keys and values for every token in the conversation so it does not recompute them. That cache grows with context length and with the number of users served at once. For a 70B model with 8,000-token contexts, each active user needs about 2.7 GB. Thirty-two users need more memory for the cache than for an 8-bit copy of the model. Most undersized inference platforms got this wrong.

Throughput and latency

Inference has two phases. Reading the prompt (prefill) is compute-heavy. Generating each new token (decode) is limited by memory bandwidth, because the weights are read once per step. Model servers such as vLLM batch many users into each step, so one read of the weights serves dozens of requests. This is why memory bandwidth (3.35 TB/s on H100, 4.8 TB/s on H200) matters as much as compute for serving. Measure time to first token and tokens per second per user, at your real concurrency.

How training spreads across GPUs

Training splits work across GPUs in three ways: data parallel (each GPU sees different data), tensor and pipeline parallel (the model itself is split), and sharded optimiser state (FSDP or DeepSpeed ZeRO). All of them exchange results after every step. Inside a server, NVLink carries this traffic at very high speed; between servers, the compute fabric does. This is why the fabric decides whether 32 GPUs behave like 32 or like 12.

GPU and deployment options compared

GPU choice follows the workload. Most enterprise platforms need two classes: training-class GPUs for fine-tuning and large models, and lower-cost GPUs for serving small and mid-size models.

As planning ranges in 2026, an 8-GPU H100 or H200 class server costs roughly ₹2 to 3 crore, and a PCIe server with four to eight L40S-class GPUs roughly ₹40 lakh to ₹1 crore. Prices and lead times move with supply; treat these as budget ranges, not quotations.

Buy, host or rent

A decision framework

Five questions, answered in order, settle most GPU decisions:

1. Training, inference or both? Inference-only platforms rarely need a high-speed fabric. Training across servers always does.
2. How large is the model, and at what precision? This sets the memory floor per copy of the model, and so the GPU type.
3. How many users, with what context and latency? This sets the KV cache and the number of model copies. A chat assistant needs a fast first token; an overnight document job does not.
4. How steady is demand? Above roughly 50 percent sustained utilisation, owning usually costs less than renting. Below it, rent.
5. Where will it run? Power per rack, cooling, floor loading and data residency decide between your data centre, colocation and a GPU cloud.

Reference architecture

The diagram shows a 32-GPU cluster used for both fine-tuning and serving. Compute, storage and management traffic each have their own network, and the scheduler sees every GPU.

Numbered flows: (1) teams submit jobs to the scheduler under their quota; (2) the scheduler places each job on whole servers, close together on the fabric; (3) during training, GPUs exchange gradients after every step over the compute fabric; (4) datasets are read from, and checkpoints written to, the parallel file system over a separate storage network; (5) raw data and finished models live in object storage and are staged to the fast tier when needed; (6) applications reach models through an inference gateway that signs in each caller, applies quotas and routes requests. GPU and fabric health stream to monitoring, and every server can be reached out of band.

The rail-optimised fabric

Each GPU has its own 400 Gb/s network port, so an 8-GPU server has eight fabric ports. In a rail-optimised design, GPU 1 of every server connects to leaf switch 1, GPU 2 to leaf 2, and so on. Traffic between same-numbered GPUs crosses one switch, which keeps latency low and predictable. Every leaf connects to every spine, with as much bandwidth up as down, so the fabric is non-blocking.

InfiniBand or RoCE Ethernet

Both work. InfiniBand (NDR 400G) is lossless by design, proven at scale for training and simpler to tune, but brings separate skills and mostly one vendor. RoCE runs RDMA over standard 400G Ethernet with priority flow control and congestion notification, is familiar to network teams and offers wider vendor choice, but needs careful configuration and testing to match InfiniBand under load. Choose on your team’s skills and existing network more than on benchmarks. Keep storage and management traffic off the compute fabric either way.

Sizing, power and cost

For a 70B model with 80 layers, 8 KV heads of dimension 128 and a 16-bit cache, one token needs 2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 0.33 MB.

Continued training of an 8B model on 100 billion tokens needs 6 × 8×10⁹ × 10¹¹ = 4.8×10²¹ operations. An H100 delivers about 990 TFLOPS at BF16; at a realistic 40 percent utilisation that is about 400 TFLOPS, so the job needs about 3,300 GPU-hours: roughly four and a half days on 32 GPUs.

With a two-minute write and one interruption a week (10,080 minutes), the interval is √(2 × 2 × 10,080), about 200 minutes: checkpoint every three hours or so. Asynchronous checkpointing, where the framework supports it, hides most of the write time.

Power and cooling per rack

Four H100 servers draw about 4 × 10.2 = 41 kW; with switches, storage and management servers, about 47 kW. In a room built for 8 kW racks that means six mostly empty racks and hot spots. With rear-door heat exchangers at 25 kW per rack it fits in two. Plan with the vendor’s maximum figures: under training load, real draw is close to the maximum.

32 GPUs provide 32 × 43,800 = 1.4 million GPU-hours in five years. At 70 percent utilisation that is about 980,000 used hours, so the cost is roughly ₹150 to 250 per GPU-hour. At 40 percent it rises to roughly ₹265 to 440, which is within or above the range for renting H100-class capacity on commitment. Utilisation, not the purchase price, decides whether owning pays.

Security, sharing and governance

Model weights, fine-tuning datasets and prompts are among the most sensitive data an organisation holds. A GPU cluster shared by several teams needs the same controls as any multi-tenant platform, plus a few of its own.

Sharing GPUs safely

Controls that matter

• One identity. Sign-in through your directory for the scheduler, notebooks and inference gateway, with roles and quotas per team.
• Separate networks. Fabric partitions (InfiniBand partition keys or VLANs on RoCE) between tenants, and management networks reachable only by administrators.
• Data protection. Encryption at rest on the file system and object store; datasets classified before they are used for training; personal data handled under the DPDP Act.
• A model registry. Only approved, versioned models are served, with a record of the data each was trained on.
• Audit. Who ran which job on which data, and who called which model, logged centrally. CERT-In expects incidents reported within six hours and logs kept for 180 days within India.

Running it: scheduling, utilisation and failures

Slurm or Kubernetes

Many clusters run both: Slurm for training, Kubernetes for inference, with nodes moved between them as demand changes. Inference needs steady capacity during working hours; training can use what is left, especially at night, if the scheduler supports priorities and pre-emption.

Measuring utilisation

“Allocated” is not “busy”. Track both: the share of GPUs assigned to jobs, and how hard those GPUs actually work (from DCGM metrics). A well-run training cluster keeps 70 to 85 percent of GPUs allocated, with allocated GPUs genuinely busy. Publish a monthly report of GPU-hours and cost per team; it is the fastest way to find idle reservations.

Burn-in before handover

• Run DCGM diagnostics and stress tests on every GPU for 48 to 72 hours.
• Run NCCL all-reduce tests across all servers and compare bandwidth with the design figure. One bad cable or optic can halve it.
• Test the storage with real checkpoint sizes, and record the baselines for later comparison.

Handling failures

Because every GPU in a training job depends on every other at each step, one failure stops the whole job. At 256 GPUs, plan for an interruption roughly every week; at 32, perhaps once or twice a month. The response should be routine:

• Monitoring flags GPU errors (Xid events, memory errors), NVLink errors and fabric link flaps.
• The scheduler drains the faulty server, and jobs restart automatically from the last checkpoint.
• The spare server joins the pool; the faulty one is repaired under the vendor contract and re-tested before returning.
• Drivers, firmware and fabric software are kept at the same version on every server, upgraded in planned windows.

A practical roadmap

Power and cooling upgrades and hardware lead times usually set the pace. Start the facility check before anything else.

Questions to put to every vendor

Proposals are easier to compare when every bidder answers the same questions in writing:

• Which exact GPU, memory and NVLink configuration, and what is the maximum power per server under full load?
• Is the fabric rail-optimised and non-blocking? Show the switch, cable and optic count.
• What sustained write throughput does the storage reach with our checkpoint size, measured rather than quoted?
• Which burn-in and acceptance tests are included, with what pass criteria for NCCL bandwidth and DCGM diagnostics?
• How quickly is a failed GPU or server replaced, and are spares held in India?
• Which driver, firmware and fabric software versions are supported together, and for how long?

Ten common mistakes

1. Ordering GPUs before checking the facility. Power and cooling take longer to fix than servers take to arrive.
2. Sizing inference from model size alone. The KV cache for real users and contexts can double the memory.
3. Training across servers on ordinary Ethernet. The GPUs spend most of their time waiting.
4. Putting storage traffic on the compute fabric. Training slows in ways that are hard to diagnose.
5. Using training-class GPUs to serve small models. Or the reverse. Match the GPU to the job.
6. Checkpointing too rarely. A failure then costs a day of work on every GPU.
7. Skipping burn-in. A weak cable found in production costs far more than one found in week one.
8. No quotas or per-team reporting. GPUs sit reserved and idle while other teams queue.
9. Time-slicing GPUs between untrusted tenants. Memory is shared; use MIG or whole servers.
10. Buying for peak demand. Own the steady base and rent the peaks.

GPU readiness checklist (36 points)

Use this list to score your current position. Anything you cannot tick with evidence is a gap worth closing.

Glossary