Phase 5: GPU & Hardware for AI

Training a neural network involves doing the same mathematical operations — matrix multiplications — billions of times. GPUs can do this 100× faster than CPUs because they have thousands of cores designed for parallel computation. Understanding GPU hardware is essential for serious AI work.

Why GPUs? CPU vs GPU

Key GPU Hardware Concepts

GPU Hardware Landscape (2024)

Tensor Cores — The AI Accelerator

Modern NVIDIA GPUs have specialised Tensor Cores that perform fused multiply-add on 4×4 matrices in a single clock cycle. These are 8–16× faster than standard CUDA cores for the matrix operations used in deep learning.

To use Tensor Cores, use mixed precision training: store weights in FP32 but do forward/backward pass in FP16. PyTorch makes this trivial:

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()

for batch in dataloader:
    optimizer.zero_grad()

    with autocast():  # Runs in FP16 on Tensor Cores
        output = model(batch)
        loss = criterion(output, targets)

    scaler.scale(loss).backward()   # Scale gradients to prevent underflow
    scaler.step(optimizer)
    scaler.update()

Checking GPU Usage

# Check GPU memory and utilisation
nvidia-smi

# Watch live (updates every 1 second)
watch -n 1 nvidia-smi

# In Python
import torch
print(torch.cuda.is_available())       # True
print(torch.cuda.device_count())       # 1
print(torch.cuda.get_device_name(0))  # NVIDIA A100
print(f"Memory: {torch.cuda.memory_allocated()/1e9:.2f}GB used")

Frequently Asked Questions