The right GPU architecture for AI can make model training significantly faster than a CPU. This performance comes from a highly parallel hardware design that is fundamentally different from a CPU's. Understanding this architecture helps you interpret GPU specifications, write more efficient code, and diagnose performance bottlenecks.
This article traces a single AI operation from a high-level concept down to its execution on specialized hardware, explaining how the pieces fit together.
Key takeaways
- AI workloads are broken into a grid of thread blocks, which are assigned to the GPU's main processing units, called Streaming Multiprocessors (SMs).
- Inside each SM, threads are grouped into "warps" of 32. Schedulers issue instructions to these warps, quickly switching between them to keep the hardware busy while waiting for data from memory.
- GPU performance is a balance of computation, memory bandwidth, and the ability to keep execution units active. Raw core count is not the only factor.
- Modern GPUs feature specialized units like Tensor Cores for matrix math, which are critical for AI. These work alongside general-purpose CUDA cores.
- Understanding the memory hierarchy—from fast on-chip registers to slower global VRAM—is key to identifying performance limits.
The GPU Architecture Hierarchy: From Grid to Core
The GPU architecture for AI maps a software model onto physical hardware to achieve massive parallelism. This starts with a programming model that organizes a large task into a hierarchy of threads. This hierarchy is then executed on the GPU's physical hardware, which consists of multiple independent processors.
The programming model organizes work into a grid of thread blocks. Each thread block contains a group of threads. Imagine a large image processing task. The grid is the entire image. Each thread block might be responsible for a small patch of the image, and each thread within that block processes a single pixel.
This software model maps directly to the hardware:
- A GPU chip contains multiple Streaming Multiprocessors (SMs). Think of an SM as a powerful, independent processing unit.
- When a task (called a kernel) is launched, the GPU's scheduler assigns thread blocks to available SMs (Source).
- A single SM is responsible for executing all the threads within the block it was assigned (Source).
Inside the SM, threads are executed using a model called Single Instruction, Multiple Thread (SIMT). Threads are bundled into groups of 32, known as warps. Threads in a warp share an instruction stream at the hardware level (Source). While this enables 32 threads to execute one instruction in parallel on different data, the model also handles cases where threads take different control-flow paths. When this "warp divergence" occurs, the hardware executes each path with the inactive threads masked, which can lower overall efficiency (Source).
How is Work Scheduled on an SM? A Step-by-Step Breakdown
A GPU's performance relies on its ability to keep its many execution units constantly fed with work. This is managed by schedulers inside each Streaming Multiprocessor (SM). Schedulers rapidly switch between groups of threads (warps) to hide latency, ensuring that the arithmetic units are always busy instead of waiting for data to arrive from memory.
Here is a step-by-step breakdown of how a typical AI task is scheduled and executed on a GPU.
Kernel Launch: The CPU sends a command to the GPU to run a specific function, called a kernel. This command defines the total number of threads, organized as a grid of thread blocks.
Block Assignment: The GPU’s scheduler assigns thread blocks to available SMs on the chip. Blocks can be assigned in any order, and multiple blocks can be assigned to one SM (Source).
Resource Allocation: An SM accepts thread blocks until its internal resources, such as registers or shared memory, are full. For example, if a kernel uses a large number of registers per thread, fewer blocks can run on an SM simultaneously (Source). This is known as occupancy.
Warp Creation: The SM partitions the threads of its assigned blocks into 32-thread warps. For example, a block with 256 threads becomes 8 warps.
Ready-Warp Scheduling: At each issue opportunity, warp schedulers inside the SM check which warps are ready to execute their next instruction. A warp is "ready" if its required data is available and it isn't waiting for a memory operation to complete. The scheduler selects a ready warp to run (Source).
Instruction Execution: The scheduler issues the instruction for the selected warp. The instruction is sent to the appropriate execution units within the SM, such as CUDA cores for arithmetic or load/store units for memory access.
Latency Hiding: If a warp issues an instruction that requires waiting (like fetching data from slow global memory), it becomes "stalled." The warp scheduler immediately switches to another ready warp and issues its instruction. This rapid switching between ready warps is the GPU's primary technique for hiding memory latency and keeping hardware busy.
Worked Example: Tracing a Vector Addition
Let's trace how a simple operation, C = A + B, is executed, where A, B, and C are large vectors each with 1024 elements. This is a common operation in AI for tasks like applying biases.
First, the programmer defines a kernel to be executed by a grid of threads. A common configuration would be 4 thread blocks, each containing 256 threads. This gives us 4 * 256 = 1024 total threads, one for each element in our vectors.
Kernel Launch: The CPU launches the kernel configured for 4 blocks and 256 threads.
vector_add<<<4, 256>>>(C, A, B);Block Assignment: Let's say the GPU's scheduler assigns Block 0 to SM 1. Other blocks might be assigned to SM 1 or other SMs.
Warp Creation: Inside SM 1, the 256 threads of Block 0 are partitioned into 8 warps (256 / 32 = 8). We'll call them Warp 0 through Warp 7.
Memory Load: A warp scheduler in SM 1 picks a ready warp, say Warp 0 (threads 0-31). Its first instruction is to load an element from vector A into a register.
- Thread 0 needs
A[0]. - Thread 1 needs
A[1]. - ...
- Thread 31 needs
A[31]. Because all 32 threads access adjacent memory locations, the GPU hardware combines these into a single, efficient memory transaction known as a coalesced read. This is a critical optimization for improving GPU memory bandwidth.
- Thread 0 needs
Latency Hiding: The trip to global memory (VRAM) takes hundreds of cycles. While Warp 0 is stalled waiting for its data, the warp scheduler on SM 1 is not idle. It immediately picks another ready warp, perhaps Warp 1, and issues its instruction to load data from
A[32]toA[63]. This continues for other warps, keeping the SM's execution units busy.Arithmetic: Once the data for
A[0]throughA[31]arrives from memory and is loaded into registers for Warp 0, the warp becomes "ready" again. The scheduler can now issue the instruction to load data from vector B. After that data arrives, the scheduler issues theaddinstruction. This instruction is executed in parallel for the 32 active threads using the SM's arithmetic units (CUDA cores).Write-Back: Finally, the scheduler issues an instruction to write the results from each thread's register back to vector C in global memory. Like the read, this is performed as a single coalesced write for maximum efficiency.
What are CUDA Cores, Tensor Cores, and Streaming Multiprocessors?
These terms are central to understanding GPU specifications, but they refer to different levels of the hardware hierarchy. A Streaming Multiprocessor (SM) is the engine, while CUDA Cores and Tensor Cores are specialized tools inside that engine. The SM contains all the necessary components to manage and execute work independently.
Here’s a comparison:
| Feature | Streaming Multiprocessor (SM) | CUDA Core | Tensor Core |
|---|---|---|---|
| Analogy | A factory floor | A single worker with a calculator | A specialized assembly line for one task |
| Function | Schedules work and manages resources | General arithmetic (add, multiply) | Specialized matrix multiply-accumulate |
| Granularity | Executes Thread Blocks and Warps | Executes one thread's instruction | Executes a small matrix operation for a group of threads |
| Role in AI | The 'brain' that orchestrates parallelism | Handles activations, normalization, etc. | Accelerates convolutions and linear layers |
Streaming Multiprocessor (SM)
An SM is the main processing unit on a GPU, designed to execute hundreds of threads at the same time (Source). It contains warp schedulers, execution units (CUDA Cores, Tensor Cores), a register file, and on-chip shared memory. The number of SMs is a key indicator of a GPU's overall processing power.
CUDA Core
A CUDA Core is a general-purpose arithmetic logic unit (ALU) that can execute one floating-point or integer instruction per clock cycle. While marketing often highlights the "core count," this number can be misleading. Performance depends on how effectively the SM's schedulers can keep these cores busy. The organization of cores and schedulers also varies between GPU generations. For example, the Pascal GP100 SM had 64 FP32 cores, while the GP104 SM from the same generation had 128 (Source).
Tensor Core
A Tensor Core is a specialized hardware unit designed to accelerate one specific operation: matrix-multiply-accumulate (D = A * B + C). This operation is the computational heart of most deep learning workloads, like linear layers and convolutions. Tensor Cores process small chunks (or "tiles") of matrices at a time, providing much higher throughput for these tasks than CUDA cores could achieve (Source). However, their performance still relies on the SM efficiently feeding them data from registers and memory. This is a key reason why GPUs are so effective for AI.
Why is the GPU Memory Hierarchy Crucial for AI?
AI models require processing vast amounts of data, making memory access a primary performance bottleneck. A GPU's memory system is a pyramid with small, fast memory at the top and large, slow memory at the bottom. Efficiently managing data movement through this hierarchy is just as important as the number of available cores.
The Memory Pyramid
- Registers: These are the fastest memory available, located directly inside the SM. Each thread has its own private registers. They are used to hold the thread's working variables. The number of available registers on an SM is limited, and using too many can reduce how many thread blocks can run concurrently (Source).
- Shared Memory / L1 Cache: This is a small amount of on-chip memory within each SM. It is much faster than global memory. It can be used as a software-managed scratchpad (shared memory) for threads within the same block to share data, or as a hardware-managed data cache (L1 cache). Using shared memory to reduce trips to global memory is a common optimization technique.
- L2 Cache: This is a larger cache shared by all SMs on the GPU. It helps reduce global memory traffic by automatically storing frequently accessed data.
- Global Memory (VRAM): This is the large pool of memory on the GPU board, typically several gigabytes. It has high bandwidth but also high latency. All threads from any SM can access global memory. Performance in AI workloads heavily depends on accessing this memory efficiently, using patterns like the coalesced reads described earlier.
For AI, the goal is to keep as much data as possible in the faster, on-chip memory levels (registers and shared memory) to feed the compute units. When data must be fetched from global memory, it should be done in large, contiguous chunks to maximize bandwidth and hide the high latency.
What to learn next
- CUDA Programming Basics: An ML Engineer's Guide - Learn CUDA programming basics for machine learning. This guide explains kernels, threads, and memory management with a complete, code exampl
- NVLink vs InfiniBand: GPU Interconnects for AI Explained - Explore the nvlink vs infiniband comparison for AI workloads. Learn their complementary roles, why NVLink scales up GPUs in a server, and In
- Model FLOPs Utilization (MFU): Why Your GPU Is Slow - Learn what model flops utilization (MFU) is and why it's a better metric than GPU utilization for training efficiency. Calculate MFU and fix
- Why GPUs for AI? A Guide to Parallel Computing: Explore the fundamental reasons why the parallel nature of GPUs makes them ideal for deep learning workloads.
- GPU Memory Bandwidth Explained for AI: Dive deeper into how memory bandwidth is measured and why it is often the most critical performance limiter for large models.
- Choosing a GPU for LLM Inference: A Sizing Guide: Apply these architectural concepts to the practical task of selecting the right hardware for deploying large language models.
