GPU Specs Without the Hype: Understanding VRAM, NVLink, and Upgrade Paths

What VRAM Actually Constrains

VRAM—Video Random Access Memory, more accurately called GPU memory or HBM (High Bandwidth Memory) on data-centre cards—is the single most important specification for AI workloads. It is also the most commonly misunderstood. Marketing materials emphasise TFLOPS (compute throughput) and tensor core counts, but for most practical AI work, VRAM is the binding constraint. Understanding what it limits and why is essential to making sensible procurement decisions.

VRAM must hold everything the GPU needs to access during computation. For a training workload, this includes:

• Model parameters: The weights and biases of the neural network. A model with 7 billion parameters in FP16 (half precision) requires approximately 14 GB of VRAM just for the parameters alone.
• Gradients: During backpropagation, the gradient for each parameter is computed and stored. This roughly doubles the parameter memory requirement.
• Optimiser states: Optimisers like Adam maintain two additional state variables per parameter (first and second moment estimates). In FP32, this adds another 28 GB for a 7B parameter model.
• Activations: Intermediate computation results that are retained during the forward pass for use during backpropagation. Activation memory scales with batch size and model depth, and it can dominate total memory usage for large batch sizes.
• Batch data: The input data for the current training step. Larger batch sizes improve training throughput but consume more VRAM.

For inference, the picture is simpler but VRAM remains critical. The model parameters must fit in memory, along with the KV (key-value) cache for transformer models, which grows with sequence length and the number of concurrent requests being served.

Batch Size and VRAM

Batch size is often the first thing that teams reduce when they run out of VRAM. This is a pragmatic response, but it has consequences. Smaller batch sizes mean more training steps to process the same amount of data, which increases total training time. They also change the gradient statistics, which can affect convergence behaviour. Techniques like gradient accumulation can partially compensate, but they add complexity and do not reduce the time spent on computation.

The practical implication is that VRAM determines your effective training throughput. More VRAM allows larger batches, which allows fewer total steps, which reduces time-to-result. This is the fundamental economic calculation that should drive GPU selection.

Combined Memory vs Per-GPU Memory

GPU marketing sometimes describes memory in terms that can be misleading. When NVIDIA describes the RTX A6000 with NVLink, for example, it is important to understand the distinction between per-GPU memory and combined memory.

Each RTX A6000 has 48 GB of GDDR6 VRAM. When two A6000 GPUs are connected via NVLink, the combined memory across the pair is 96 GB. However, this is not the same as having a single GPU with 96 GB. NVLink allows one GPU to access the other GPU's memory directly, but with higher latency and lower bandwidth than accessing its own local memory. For workloads that can be partitioned across two GPUs—using model parallelism or pipeline parallelism—this combined memory is genuinely useful. For workloads that require all memory to be accessible with uniform latency, the per-GPU figure is what matters.

NVLink between A6000 GPUs provides a bidirectional bandwidth of 112.5 GB/s (third-generation NVLink). This is substantially faster than PCIe Gen4 (64 GB/s bidirectional), but it is still slower than the local memory bandwidth of each GPU (768 GB/s for the A6000). The performance characteristics of cross-GPU memory access are therefore quite different from local memory access, and workloads must be designed to account for this asymmetry.

When someone tells you a system has "96 GB of GPU memory," always ask: is that 96 GB on a single device, or 2 x 48 GB connected via NVLink? The answer changes what workloads the system can run efficiently.

When A6000-Class vs A100/H100-Class Matters

The GPU market for AI workloads is broadly segmented into professional workstation GPUs (RTX A6000, now succeeded by the RTX 6000 Ada) and data-centre GPUs (A100, H100, and their successors). The price difference is substantial—an H100 SXM costs several times more than an A6000—so understanding when the premium is justified is critical.

Where A6000-Class GPUs Excel

The A6000 class is well-suited for workloads that need substantial VRAM (48 GB per GPU) but do not require the highest possible compute throughput or multi-GPU interconnect bandwidth. Typical use cases include:

• Fine-tuning pre-trained models on domain-specific datasets
• Inference serving for models that fit within 48 GB
• Exploratory research where iteration speed matters more than absolute training throughput
• Mixed workloads that combine visualisation, simulation, and machine learning

Where A100/H100-Class GPUs Are Necessary

Data-centre GPUs justify their premium when workloads demand characteristics that workstation GPUs cannot provide:

• HBM memory bandwidth: The A100 provides 2 TB/s of memory bandwidth (HBM2e), and the H100 pushes to 3.35 TB/s (HBM3). This is four to five times the bandwidth of the A6000's GDDR6. For workloads that are memory-bandwidth-bound—which includes many transformer architectures—this difference translates directly into higher throughput.
• NVSwitch and high-bandwidth interconnect: In SXM form factor, A100 and H100 GPUs connect via NVSwitch with 600 to 900 GB/s of bisection bandwidth. This enables efficient model parallelism across 4 or 8 GPUs, which is essential for training models too large to fit on a single device.
• FP8 and transformer engine support: The H100 introduced FP8 precision and the transformer engine, which provide up to 3x the training throughput of the A100 for transformer-based models with minimal accuracy loss.
• 80 GB VRAM per device: The A100 80 GB and H100 80 GB variants provide significantly more per-GPU memory than the A6000, which matters for large-model training and high-throughput inference with long context lengths.

Planning GPU Capacity Without Overbuying

The most expensive GPU mistake is not buying the wrong GPU—it is buying more GPU than your workloads require, or buying the right GPU and then failing to utilise it. A structured capacity planning process prevents both outcomes.

Start With Workload Profiling

Before selecting GPU hardware, profile your actual workloads. This means running representative training and inference jobs on available hardware and measuring VRAM utilisation, GPU compute utilisation, memory bandwidth utilisation, and data-loading throughput. Most teams skip this step and select GPUs based on external benchmarks that may not reflect their workload characteristics.

Right-Size to the Binding Constraint

For each workload, identify which GPU resource is the binding constraint. If your models fit within 24 GB of VRAM and GPU utilisation is consistently above 80 percent, you do not need 80 GB cards—you need more cards of the current type. If your models require 60 GB of VRAM, then 48 GB cards are insufficient regardless of their compute performance, and you need the 80 GB class.

Plan in Generations, Not Units

GPU technology advances rapidly. Rather than buying the maximum quantity of current-generation hardware, consider a phased approach: deploy enough GPUs to meet current workload needs with reasonable headroom, and plan refresh cycles aligned with GPU generations. An A100 purchased in 2022 delivers roughly one-third the training throughput of an H100 for transformer workloads. Organisations that overinvested in A100 inventory now face the choice between underperforming hardware and an expensive early refresh.

A practical rule of thumb is to plan GPU deployments for 18 to 24 months of useful life, with a clear understanding of when the next generation becomes available and what performance uplift it provides. This avoids both the risk of overcommitting to current technology and the paralysis of perpetually waiting for the next generation.

GPU procurement is not a one-time decision. It is an ongoing capacity management discipline that requires understanding your workloads, tracking utilisation, and planning upgrades deliberately. The organisations that get this right spend less on hardware and deliver more computational throughput per dollar invested.