How Many GPUs Does Your AI Actually Need?

How Many H100s Does Your Model Actually Need? A GPU Sizing Guide for Indian Enterprises 

"How many H100s do we need?" is the question almost every AI budget starts with, and the honest first answer is another question: to do what? The number of GPUs that train a model from scratch is not the number that fine-tunes it, which is not the number that serves it to your users. Confusing these is the single most common way an AI infrastructure budget ends up either wildly overspent on idle accelerators or quietly under-built, so the project stalls the day real traffic arrives. 

This guide gives you the sizing logic rather than a number, because anyone who hands you a number without seeing your workload is guessing. It covers the three sizing regimes, the memory rules that turn a model into a GPU requirement, how the GPUs are connected and why that matters, a worked example, and the factors beyond raw memory that decide what you actually buy. By the end, you should be able to reason your way to a defensible range and know how to validate it before you commit capital. 

How Many GPUs Does Your Model Need? 

It depends first on what you are doing with the model, and second on how large the model is. Memory is the binding constraint to start with: a GPU has a fixed amount, and the model, its working data and its overhead all have to fit. Once you know the memory a workload needs, the GPU count is broadly that requirement divided by what each GPU provides, adjusted upward for performance and resilience. Everything else refines that figure. 

So the first fork is the regime: training, fine-tuning or inference. They size very differently, and most sizing mistakes begin by skipping this distinction. 

The Three Regimes: Training, Fine-Tuning and Inference 

Each places a different demand on memory, and therefore on GPU count. 

Full training, building or substantially retraining a model, is the heaviest. Each parameter carries not just its weight but optimiser states and gradients, so a useful planning rule is 16 to 20 bytes of memory per parameter. That is why training a large model needs a cluster, not a card. 

Fine-tuning sits in between, and how far in between depends on the method. A full fine-tune approach to training-scale memory. Parameter-efficient methods such as LoRA, which adjust only a small fraction of the weights, cut that requirement dramatically, often bringing a model that would need a cluster to train down to a handful of GPUs to adapt. For most enterprises, fine-tuning an existing open model rather than training one from scratch is the realistic path, and it changes the GPU bill by an order of magnitude. 

Inference, serving the model, is the lightest per copy. It needs the weights plus working memory: roughly 2 GB of GPU memory per billion parameters at FP16 precision, about a quarter of that at INT4, plus 15 to 20% on top for the context cache, activations and framework overhead. The same model you needed a cluster to train might serve on one or two GPUs. 

The practical lesson is to be precise about which regime you are budgeting for. An enterprise that plans to fine-tune and serve open models has a very different, and usually far smaller, GPU requirement than one that believes it must train from scratch. 

How Much GPU Memory Does Your Model Actually Need? 

Work it out from the parameter count, the regime and the precision. The table applies the rules above to common model sizes as directional planning figures, not guarantees. Treat them as a starting range to validate, not a quote. 

The 70B row makes the point. Even an 80 GB H100 cannot hold a 70B model's weights and optimiser states for full training, so you are into a minimum of around 16 GPUs simply to fit it in memory, before any consideration of how fast you want the training to finish. Serving that same model needs far fewer. The memory rule gets you to the floor; performance targets take you up from there. 

Precision and Quantisation Change the Maths 

The precision you run at moves the memory requirement substantially. Dropping from FP16 to INT8 roughly halves the memory a model needs; INT4 roughly quarters it. For inference, quantisation is often the cheapest lever you have, turning a two-GPU deployment into a single-GPU one with an acceptable accuracy trade-off for many use cases. It is not free, very aggressive quantisation can degrade quality on demanding tasks, but for a great many enterprise workloads, it is the difference between needing one accelerator and needing several. Decide your precision before you count GPUs, not after. 

Which GPU: H100, H200 or B200? 

The right accelerator depends on model size, context length and how much you value training and inference speed. More memory per GPU means fewer GPUs to fit a given model, more room for long contexts, and often a faster, simpler cluster. The table sets out the current options. 

Fewer, larger GPUs can beat more, smaller ones, because a model that fits in fewer accelerators spends less time moving data between them. The cheapest GPU per unit is rarely the cheapest cluster for the job once you account for that overhead. 

How the GPUs are Connected Matters as Much as How Many 

A multi-GPU workload is only as fast as the links between the GPUs. Inside a single server, GPUs communicate over high-speed NVLink; between servers, over a low-latency fabric such as InfiniBand or RoCE Ethernet. When a model is split across GPUs, using data, tensor or pipeline parallelism, those GPUs must exchange data constantly and in step. If the interconnect is slow, they spend their time waiting for each other, and you have paid for accelerators that idle. This is why an eight-GPU server, with all eight on fast NVLink, can outperform the same eight GPUs scattered across slower-linked machines. Sizing is not only "how many GPUs" but "how are they connected", and the second question is the one most spreadsheets ignore. 

A Worked Example 

Suppose you want to serve a 70B open model to a few hundred concurrent users at interactive speed. The memory floor for the model at FP16 is around 140 to 170 GB, so it does not fit on a single 80 GB H100; you need at least two, or a single larger-memory GPU. But the memory floor only fits the model. Serving hundreds of concurrent requests, each with its own growing context cache, and meeting a tight response time, pushes you above that floor for throughput and headroom. You might quantise to INT8 to ease memory, add GPUs for concurrency, and provision a little extra so one failing node does not take the service down.  

The honest output is not "two GPUs" but "a small, well-connected cluster, validated against your real traffic", which is exactly the kind of figure a benchmark settles and a spreadsheet does not. 

What Else Drives the GPU Count? 

Memory sets the floor; performance and resilience set the rest. Four factors push the number up from the minimum. Throughput and concurrency: serving many users at once needs more GPUs than the memory minimum implies. Latency targets: a tight response-time budget requires headroom. Context length: long contexts inflate the working memory per request, sometimes dramatically, and are a frequent cause of under-sizing. And utilisation and resilience: you size for sustained load, and for a node failing without taking the service down. A cluster sized only to fit the model in memory will struggle the moment production traffic arrives. 

Don't Size the GPUs in Isolation 

A GPU count is half a plan. The cluster around it decides whether those GPUs ever reach their potential: storage with the throughput to keep them fed, a low-latency network so they scale together, and power and cooling able to carry the density, which for modern GPU servers runs far higher than a conventional rack. A correctly sized set of GPUs starved by slow storage, throttled by an ordinary network, or capped by a facility that cannot cool them, is wasted capital. Size the GPUs, then size what surrounds them to match. 

The India Angle: Build, Rent, or Both 

Sizing also informs whether you should own the GPUs at all. A steady, well-understood workload at a meaningful scale tends to justify owning the cluster you have sized, particularly where data residency points to on-premises or sovereign infrastructure. A workload that is still finding its shape, or that spikes unpredictably, may be better served by GPU-as-a-Service while you learn its real demand, then sized and built once the pattern is clear. There is no contradiction in renting to discover your requirement and owning to run it.  

For Indian enterprises weighing power costs, availability and residency obligations together, the sizing exercise and the build-versus-rent decision are best taken in the same conversation. 

Common Sizing Mistakes to Avoid 

Three recur. The first is sizing for training, when you will actually fine-tune and serve, which overbuilds by a wide margin. The second is sizing only for the memory floor and ignoring concurrency, latency and context length, which underbuilds and shows up as a sluggish service under load. The third is counting GPUs while ignoring the interconnect, storage and cooling around them, which produces a cluster that looks right on paper and disappoints in practice. Each is avoidable by reasoning through the regime, the performance targets and the surrounding infrastructure before settling on a number. 

From Estimate to a Build That Holds 

The sizing logic is learnable, and this guide gives you enough to challenge any number you are quoted. Turning that logic into a validated cluster, GPUs sized to the models and the traffic, connected by a fabric that lets them scale, fed by storage that keeps them busy, and housed in a facility that can power and cool them, is where experience saves the money a first attempt usually wastes. 

Proactive Data Systems sizes, designs and builds AI infrastructure for Indian enterprises. We are a Cisco Preferred Cloud and AI Partner, Dell Platinum Partner and NetApp Preferred Partner, with 35 years in enterprise IT, more than 1,500 organisations served, and a 24/7 service desk in India. We size the GPUs to your models and your traffic, validate the estimate with a proof of concept, build the network, storage and cooling to match, and deliver it on-premises, hybrid or sovereign. 

Tell us the models you intend to train, fine-tune or serve and your workload profile, and we will size the cluster and validate it. Ask us for a GPU sizing and AI-readiness assessment. 

Sources:  

NVIDIA H200/B200 specs (NVIDIA; Spheron);

LLM GPU memory and count rules (Spheron;  

Lyceum Technology; Lenovo LLM Sizing Guide. 

Disclaimer: The memory rules, GPU counts and worked figures here are directional planning estimates, not guarantees or a quote. Actual requirements depend on your models, regime, precision, context length, concurrency and software stack, and GPU specifications and availability change. Validate sizing with a benchmark on representative workloads, and obtain a formal quotation before purchasing.