AI Infrastructure

Behind every AI breakthrough is an enormous infrastructure story. Training GPT-4 reportedly cost over $100 million in compute alone. Understanding the hardware, cloud platforms, and optimisation techniques that power modern AI is essential for anyone building at scale.

Why GPUs Beat CPUs for AI

CPUs are optimised for sequential tasks — a few powerful cores handling complex logic. GPUs are optimised for parallelism — thousands of simple cores executing the same operation on different data simultaneously.

Neural network training is fundamentally matrix multiplication at scale. A single training step for a large model involves multiplying matrices with billions of elements. A CPU processes these sequentially. A GPU processes thousands of elements in parallel.

An NVIDIA H100 GPU delivers roughly 4,000 TFLOPS of FP8 compute. A high-end server CPU manages perhaps 2 TFLOPS. For the embarrassingly parallel workloads of deep learning, GPUs win by three orders of magnitude.

NVIDIA's Dominance

NVIDIA controls roughly 80-90% of the AI accelerator market. Their moat is not just hardware — it is the CUDA ecosystem. Nearly every ML framework (PyTorch, TensorFlow, JAX) is built on CUDA. Switching to a competitor means rewriting low-level kernels and accepting potential incompatibilities.

The GPU Lineup

| GPU | Memory | FP8 Performance | Use Case | |-----|--------|-----------------|----------| | A100 | 80 GB HBM2e | 624 TFLOPS | Workhorse of current data centres | | H100 | 80 GB HBM3 | 3,958 TFLOPS | Frontier model training | | H200 | 141 GB HBM3e | 3,958 TFLOPS | Memory-bound LLM inference | | B200 | 192 GB HBM3e | 9,000 TFLOPS | Next-generation training and inference |

The H100 to B200 jump represents a 2.3× performance increase — but the real bottleneck is often memory bandwidth, not raw compute. The H200's 141 GB of HBM3e memory specifically targets LLM inference where the KV cache dominates memory usage.

Google TPUs

Google's Tensor Processing Units are custom ASICs designed specifically for matrix operations. Unlike GPUs, which handle graphics and compute, TPUs are purpose-built for neural networks.

• TPU v5p — Google's latest training chip, available in pods of up to 8,960 chips connected via high-speed interconnect
• TPU v5e — Cost-optimised for inference workloads
• Key advantage: tight integration with JAX and the XLA compiler
• Key limitation: only available through Google Cloud — no on-premise option

Google trains all its foundation models (Gemini, PaLM) on TPUs, proving they compete with NVIDIA at the frontier.

AMD and the Competition

AMD's MI300X is the most credible NVIDIA alternative:

• 192 GB HBM3 memory (more than the H100)
• Strong ROCm software stack (improving but still behind CUDA)
• Significant price advantage — roughly 30-40% cheaper than equivalent NVIDIA
• Growing ecosystem of ML frameworks with native ROCm support (PyTorch, JAX)

Microsoft, Meta, and Oracle have all adopted MI300X for inference workloads. The memory advantage is particularly compelling for large language models where the KV cache is the primary bottleneck.

Intel's Gaudi 3 accelerator is also entering the mix, offering competitive training performance with a focus on enterprise price-to-performance ratios. The AI accelerator market is finally becoming a multi-vendor ecosystem.

Cloud Providers Comparison

No company outside the hyperscalers can afford to build its own GPU cluster for frontier AI. The cloud battle is fierce:

| Platform | Key Offering | Strength | |----------|-------------|----------| | AWS SageMaker | End-to-end ML platform | Broadest GPU selection, Inferentia custom chips | | GCP Vertex AI | Managed ML with TPU access | TPU availability, tight BigQuery integration | | Azure ML | Enterprise ML platform | OpenAI partnership, enterprise compliance | | Lambda Labs | GPU cloud for AI | Simplicity, competitive H100 pricing | | CoreWeave | GPU-native cloud | Purpose-built for AI, NVIDIA partnership |

Spot instances can reduce training costs by 60-90%. The trade-off: your training job can be interrupted at any time. Frameworks like PyTorch Lightning and DeepSpeed support checkpointing to resume training seamlessly after preemption.

The GPU Shortage and Geopolitics

AI infrastructure is now a geopolitical issue:

• TSMC (Taiwan) fabricates nearly all advanced AI chips — for NVIDIA, AMD, Apple, and Google. A disruption to TSMC would halt AI progress globally.
• US export controls restrict China's access to advanced chips (A100, H100 banned). China is investing heavily in domestic alternatives (Huawei Ascend, Biren).
• NVIDIA's China-specific chips (H20) offer reduced performance to comply with export rules whilst maintaining market access.
• The CHIPS Act (US) and similar programmes in the EU and Japan aim to diversify semiconductor manufacturing.

Custom Silicon Startups

Several startups are challenging the GPU paradigm entirely:

• Groq — LPU (Language Processing Unit) architecture delivers deterministic, ultra-low-latency inference. No batching required.
• Cerebras — Wafer-scale chip (the size of a dinner plate) eliminates memory bandwidth bottlenecks for training.
• SambaNova — Reconfigurable dataflow architecture optimised for enterprise AI.
• Graphcore (acquired by SoftBank) — IPU architecture with massive on-chip SRAM for graph-structured workloads.

These are not NVIDIA replacements today. But they represent genuine architectural innovation that could reshape the market as AI workloads diversify beyond transformers.

Inference Optimisation

Training gets the headlines, but inference is where the money goes. Techniques that reduce inference cost:

• Dynamic batching — Group incoming requests to maximise GPU utilisation
• KV cache management — vLLM's PagedAttention avoids memory fragmentation
• Speculative decoding — Use a small model to draft tokens, verified in parallel by the large model (2-3× speedup)
• Continuous batching — Process new requests without waiting for all current requests to complete
• Model parallelism — Split large models across multiple GPUs for latency-sensitive serving

The Total Cost of AI Infrastructure

Infrastructure cost extends far beyond GPU rental. A complete picture includes:

• Compute — GPU hours for training and inference (typically 60-70% of total cost)
• Storage — Training data, checkpoints, model artefacts, and logs
• Networking — Data transfer between GPUs, regions, and to end users
• Engineering — ML engineers, platform engineers, and DevOps staff to operate the stack
• Electricity — A single H100 server consumes 10+ kW; data centre power is a growing constraint

At scale, companies like Meta spend over $10 billion annually on AI infrastructure. For startups, cloud costs for a single frontier model training run can exceed $1 million. Understanding and optimising this cost stack is a core competency.

📚 Further Reading

• NVIDIA H100 Tensor Core GPU Architecture — Technical deep-dive into the Hopper architecture
• Google Cloud TPU Documentation — Official guide to TPU architecture and programming
• The GPU Shortage Explained (SemiAnalysis) — Industry analysis of AI chip supply and demand dynamics