LLM Systems 02: GPU Programming

This article is the learning note of CMU 11868 LLM Systems course Lecture 2. For more details, please refer to the original slides.

0. Recap

  • Autoregressive Language Model: \(P\left( x_{1..T} \right) = \prod_{t=1}^{T}{P\left( x_{t+1} | x_{1..t} \right)}\) .
  • Scaling of LLMs: 0.1B -> 1000+ B, the need for system optimization.
  • Important Topics in LLM systems
    • Programming model
      • relies on good abstraction
    • Latency/Throughput
      • Data movement
      • Computation vs memory
    • Reliability
    • Security

1. Neural Network Layer and low-level operators

Take a simple feedforward neural network for sentiment classification task as example:

1
Embedding -> Linear -> ReLU -> Linear -> Avg Pooling -> Softmax

The Embedding(lookup table) layer convert the input sequence tokens into embeddings, then feed the embeddings to later layers for computation.

The low-level operators behind these layers include:

  • Matrix multiplication: Linear
  • Element-wise ops (add, scale, ReLU): ReLU
  • Reduce ops (sum, avg): Avg Pooling

Efficient computation requires GPUs.

2. Components of A GPU Server

A typical modern GPU server includes the following components:

  • 2 or more CPUs
  • Several TB of CPU memory
  • Tens of TB of SSD
  • Usually 8 GPUs, interconnected via NVLink
    • For the server in the above figure, the NVLink bandwidth 112.5 GB/s
    • PCIe Gen4 32 GB/s (16 lanes x 2 GB/s per lane)

The hardware connection of CPU/GPU in a modern server is usually as follows:

  • Topology (from diagram):
    • 2 CPUs connected via xGMI (2+1 xGMI). xGMI is the interconnect bus between AMD CPUs.
    • Each CPU connects to 4 GPUs via PCIe Gen5 x16.
    • Total: 8 double-width GPUs.
    • No direct GPU-to-GPU interconnect (e.g., NVLink) shown; GPU communication must go through PCIe and CPU xGMI.
  • Why relevant to Data Parallel training?
    • In data parallel, each GPU computes gradients on its own batch, then GPUs must synchronize gradients (e.g., All-Reduce) to keep model replicas consistent.
    • Moving gradients from one GPU to another requires crossing PCIe (and possibly xGMI across CPUs).

3. GPU Architecture

Some additional materials:

GPU Lineup

Metric NVIDIA B200 (Blackwell) NVIDIA H100 (Hopper) NVIDIA A100 (Ampere)
FP64 37 teraFLOPS 34 teraFLOPS 9.7 teraFLOPS
FP64 Tensor Core 37 teraFLOPS 67 teraFLOPS 19.5 teraFLOPS
FP32 75 teraFLOPS 67 teraFLOPS 19.5 teraFLOPS
FP32 Tensor Core 2.2 petaFLOPS 989 teraFLOPS 312 teraFLOPS
FP16/BF16 Tensor Core 4.5 petaFLOPS 1979 teraFLOPS 624 teraFLOPS
INT8 Tensor Core 9 petaOPs 3958 teraOPs 1248 teraOPs
FP8 Tensor Core 9 petaFLOPS 3958 teraFLOPS -
FP4 Tensor Core 18 petaFLOPS - -
GPU Memory 192GB HBM3e 80GB HBM3 80GB HBM2e
Memory Bandwidth Up to 7.7TB/s 3.2TB/s 2TB/s
SMs 148 132(SXM5)/114(PCIe) 108

Streaming Multiprocessor(SM)

  • SM: basic compute unit of an NVIDIA GPU.
  • 4 partitions per SM:
    • Each partition has 32 FP32 cores → 32 cores can execute one warp (32 threads) at a time.
    • Total: 128 FP32 cores per SM.
  • Registers:
    • 64KB register file per partition (fastest storage).
    • Total: 256KB register file per SM.
  • Cache / Shared Memory:
    • 128KB shared L1 cache per SM.
    • 256KB shared L1 cache per SM for H100.
  • Throughput:
    • 128 FP32 operations per cycle per SM (one per core).

New in H100, FP8 operations

CPU vs. GPU

Metric CPU (AMD EPYC 9754) GPU (A6000)
Number of threads 256 10752
Clock 2.25 GHz 1.8 GHz
Compute 576 GFlops 38.7 TFlops
Power 360 W 300 W

4. Program Execution on GPU

GPU Programming Model

  • CPU – host
    • Run normal program (C++)
  • GPU – device
    • Run cuda kernel code
  • CUDA: one part runs on CPU, one part runs on GPU
  • Needs to move data between system memory and GPU memory

SIMT Execution on GPU

  • SIMT = Single Instruction, Multiple Threads:
    • One instruction is issued to many threads in parallel.
    • Each thread processes different data with its own registers/state.
  • Thread hierarchy:
    • Thread: smallest execution unit.
    • Warp: 32 threads; basic scheduling/issue unit.
    • Thread Block: group of warps; scheduled onto one SM.
    • Grid: group of blocks; launched by one kernel.
  • Kernel execution:
    • A kernel is executed as a grid of blocks of threads.
    • Blocks are distributed across SMs.
    • Warps within a block are scheduled onto SM partitions.
    • Each partition issues one warp per cycle to its 32 FP32 cores.
  • SIMT vs. SIMD:
    • SIMD: one instruction operates on a vector of data.
    • SIMT: one instruction executed by many threads, each with its own registers/PC; supports per-thread branching, but divergence costs performance.

How Kernel Threads are Executed

  • Software-to-hardware mapping:
    • A kernel launches a Grid of Thread Blocks.
    • Thread Blocks are distributed across SMs on the GPU.
    • An SM can host multiple Thread Blocks concurrently.
  • Block → Warp partitioning:
    • SM partitions a Thread Block into warps.
    • Warp = basic unit of GPU creation, management, scheduling, and execution.
  • Warp characteristics:
    • Contains 32 threads (matches 32 FP32 cores per SM partition).
    • All threads in a warp start at the same program address.
    • Each thread has its own program counter (PC) and registers.
    • Execute one common instruction per cycle (SIMT).
      • Hardware implements per-thread PC via:
        • Warp-level PC: current issued instruction.
        • Active mask: which threads are active.
        • SIMT / branch stack: tracks divergent paths, executes them serially, then re-converges.
    • Threads can branch and execute independently:
      • Divergent branches cause serialization (branch divergence), masking idle threads.

Warp Execution on GPU

  • Execution context stays on SM for lifetime of warp (Program counter, Registers, Shared memory)
    • Once a warp is scheduled onto an SM, its context remains resident until the warp finishes.
    • No need to swap context out to memory (unlike CPU).
  • Warp-to-warp context switch is instant
    • All resident warps' contexts are in SM registers/shared memory.
    • Scheduler can switch to another ready warp every cycle.
    • This is how GPUs hide memory latency: when one warp stalls, another runs.
  • Warp scheduler at runtime:
    • Chooses a warp with active threads (not finished, not blocked).
    • Issues the warp's current instruction to its 32 threads.
    • Goal: keep execution units busy and maximize throughput.
  • Number of warps resident on an SM is limited by resources:
    • Registers per thread (register file size).
    • Shared memory per block.
    • Hardware limits (e.g., max 64 warps/SM on H100).
    • More resident warps → better latency hiding; fewer → possible stalls.
  • Key takeaway: GPU hides latency through massive hardware multithreading with zero-cost context switches, not through complex out-of-order execution like CPUs.

Executing one Thread block on one SM

  • Thread Block → Warp → SM partition mapping:
    • A Thread Block contains multiple warps.
    • An SM has 4 partitions, each can issue one warp per cycle.
    • Therefore, up to 4 warps can execute in parallel at one time on each SM (one per partition).
  • Diagram meaning:
    • Warps (Warp1–Warp4) of a Thread Block are distributed across SM partitions.
    • Arrows show warp → partition assignment.
    • Each partition executes its assigned warp on 32 FP32 cores.
  • Key points:
    • A Thread Block may contain more than 4 warps; extra warps wait and are scheduled by the warp scheduler.
    • Multiple Thread Blocks can reside on one SM, but the 4-partition issue limit still applies per cycle.
    • This is the hardware basis of SIMT parallelism: many warps, 4 issued per cycle, hiding latency via fast context switching.
  • Takeaway: One SM can issue up to 4 warps per cycle (one per partition). A Thread Block with ≥4 warps can fully utilize this issue width when scheduled on one SM.

CPU-GPU Data Movement

CUDA Kernel

  • Each kernel is a function (program) that runs on GPU
  • Program itself is serial
  • Can simultaneously run many (10k) threads at the same time
  • Using thread index to compute on right portion of data

Compiling CUDA Code

Summary

  • Neural network
    • is composed of layers, each layer defines input and output vectors/embeddings.
    • Each layer’s computation consists of low-level operators, which is executed on a computing device (GPU, CPU, FPGA, etc).
  • GPU is composed of
    • streaming processing units (SMs)
      • each with four partitions of 32 cores
      • shared L1 cache
    • memory
    • L2 cache: share with all SMs
  • Threads organized in
    • grid of thread blocks
    • each block is divided into warps running in parallel on one SM.
ML System
#CMU 11868 #LLM System
LLM Systems 02: GPU Programming
https://arcsin2.cloud/posts/2032429701/
作者
arcsin2
发布于
2026年9月12日
许可协议