Machine learning frameworks like PyTorch and TensorFlow handle most GPU computations automatically. However, to understand performance and write custom operations, grasping the fundamentals of the underlying technology—CUDA—is necessary. This guide covers the CUDA programming basics that help understand how work gets done on the GPU.
CUDA is NVIDIA's platform for parallel computing on its graphics processing units (GPUs) (CUDA C++ Programming Guide). It allows you to use the massive parallel processing power of GPUs for general-purpose tasks, which is why GPUs are so effective for AI.
Key takeaways
- A CUDA kernel is a function executed in parallel by many threads on a GPU, launched from code running on the CPU (the "host").
- The CUDA execution model organizes threads into a hierarchy: threads form blocks, and blocks form a grid. Threads within a block can cooperate using fast shared memory.
- Kernel launches and data transfers between the CPU and GPU have overhead. A GPU is only faster when the time saved by parallel computation outweighs these initial costs.
- Kernel launches are asynchronous. Your CPU code continues executing without waiting, so you must explicitly synchronize with
cudaDeviceSynchronize()before using GPU results on the CPU. - Writing correct CUDA programs requires careful memory management and robust error checking for every API call to catch both immediate and asynchronous failures.
What is a CUDA Kernel? The Core of GPU Programming
A CUDA kernel is a function that runs on an NVIDIA GPU, executed in parallel by many different threads. Kernels are written in CUDA C++, an extension of C++, and are launched from standard CPU code. This division is central to CUDA: your application has host code that runs on the CPU and device code that runs on the GPU (Intro to CUDA C++).
The host (CPU) acts as the orchestrator. It allocates memory on the GPU, copies data to it, launches the kernel on the device, and copies the results back. The device (GPU) is the massively parallel processor that executes the kernel.
You designate a function as a kernel using the __global__ specifier. When the NVIDIA CUDA Compiler (NVCC) processes your code, it separates the host code (compiled with your standard C++ compiler) from the device code (compiled for the GPU architecture) (CUDA C Basics).
| Code Type | Runs On | Role | Example |
|---|---|---|---|
| Host Code | CPU | Manages memory, launches kernels, handles I/O | int main() { ... } |
| Device Code | GPU | Executes in parallel across many threads | __global__ void myKernel() { ... } |
One kernel function is executed by thousands or millions of threads at once. Each thread gets a unique ID that it can use to determine which piece of the data to work on.
How Do CUDA Kernels Actually Execute? A Step-by-Step Breakdown
A kernel launch is a coordinated process between your CPU code, the CUDA runtime, and the GPU hardware. It allows the host to queue up parallel work on the device and then continue with its own tasks. The launch is asynchronous, meaning the CPU call returns immediately without waiting for the GPU to finish (CUDA C Basics).
Here is the step-by-step lifecycle of a typical CUDA program:
- Allocate GPU Memory: The host code uses the CUDA runtime API, such as
cudaMalloc(), to allocate memory on the GPU's VRAM. The CUDA runtime API is layered over the lower-level driver API (Intro to CUDA C++). - Transfer Data: The host copies input data from its main system memory (RAM) to the newly allocated GPU memory using a function like
cudaMemcpy(). - Configure and Launch Kernel: The host defines the kernel's execution configuration, specifying the number of thread blocks and the number of threads per block. It then launches the kernel using the
<<<grid_dim, block_dim>>>syntax. This call queues the kernel for execution and returns immediately. - GPU Schedules and Executes: The GPU's hardware scheduler assigns thread blocks to available Streaming Multiprocessors (SMs), the core processing units of a GPU's architecture (Writing SIMT Kernels).
- Threads Perform Work: Each thread within each block executes the kernel code. It uses its built-in identifiers (
threadIdx,blockIdx) to calculate a unique index and process its assigned data element (Writing SIMT Kernels). - Synchronize (Wait for GPU): Since the kernel launch was asynchronous, the host code must explicitly wait for the GPU to finish before it can safely use the results. A call to
cudaDeviceSynchronize()blocks the CPU thread until all previously issued GPU tasks are complete. - Transfer Results Back: The host calls
cudaMemcpy()again to copy the results from GPU memory back to CPU memory. - Clean Up: The host frees the memory allocated on the GPU using
cudaFree().
Anatomy of a CUDA Program: A Complete Vector-Add Example
A complete, working example can illustrate CUDA programming basics. The following program adds two vectors, a and b, and stores the result in a third vector, c. It includes all the essential steps: memory allocation, data transfer, kernel execution, synchronization, and error checking.
This code can be saved as vector_add.cu and compiled using the NVIDIA CUDA Compiler (NVCC).
#include <iostream>
#include <vector>
// CUDA error-checking wrapper
// This macro is a good practice for robust CUDA programming
#define CUDA_CHECK(err) { \
cudaError_t err_ = (err); \
if (err_ != cudaSuccess) { \
std::cerr << "CUDA error at " << __FILE__ << ":" << __LINE__; \
std::cerr << ": " << cudaGetErrorString(err_) << std::endl; \
exit(EXIT_FAILURE); \
} \
}
// The CUDA Kernel that runs on the GPU
__global__ void vectorAdd(const float *a, const float *b, float *c, int n) {
// Calculate the global thread index
int idx = blockIdx.x * blockDim.x + threadIdx.x;
// Bounds check to avoid writing out of bounds
if (idx < n) {
c[idx] = a[idx] + b[idx];
}
}
int main() {
// 1. Host-side setup
int n = 10000000; // Size of vectors
size_t size = n * sizeof(float);
// Allocate host memory (CPU)
std::vector<float> h_a(n);
std::vector<float> h_b(n);
std::vector<float> h_c(n);
// Initialize host vectors
for (int i = 0; i < n; ++i) {
h_a[i] = static_cast<float>(rand()) / RAND_MAX;
h_b[i] = static_cast<float>(rand()) / RAND_MAX;
}
// Allocate device memory (GPU)
float *d_a, *d_b, *d_c;
CUDA_CHECK(cudaMalloc(&d_a, size));
CUDA_CHECK(cudaMalloc(&d_b, size));
CUDA_CHECK(cudaMalloc(&d_c, size));
// 2. Transfer data from Host to Device
CUDA_CHECK(cudaMemcpy(d_a, h_a.data(), size, cudaMemcpyHostToDevice));
CUDA_CHECK(cudaMemcpy(d_b, h_b.data(), size, cudaMemcpyHostToDevice));
// 3. Launch the Kernel
int threadsPerBlock = 256;
int blocksPerGrid = (n + threadsPerBlock - 1) / threadsPerBlock;
vectorAdd<<<blocksPerGrid, threadsPerBlock>>>(d_a, d_b, d_c, n);
// Check for errors launched by the kernel
CUDA_CHECK(cudaGetLastError());
// 4. Synchronize to wait for the kernel to finish
CUDA_CHECK(cudaDeviceSynchronize());
// 5. Transfer data from Device to Host
CUDA_CHECK(cudaMemcpy(h_c.data(), d_c, size, cudaMemcpyDeviceToHost));
// 6. Verify results on the host (optional)
for (int i = 0; i < 10; ++i) { // Check first 10 elements
if (std::abs(h_c[i] - (h_a[i] + h_b[i])) > 1e-5) {
std::cerr << "Result verification failed at index " << i << "!" << std::endl;
break;
}
}
std::cout << "Vector addition successful!" << std::endl;
// 7. Clean up device memory
CUDA_CHECK(cudaFree(d_a));
CUDA_CHECK(cudaFree(d_b));
CUDA_CHECK(cudaFree(d_c));
return 0;
}Compilation and Execution
To compile this code, you need the CUDA Toolkit installed. Run the following command in your terminal:
nvcc vector_add.cu -o vector_addThen, execute the compiled program:
./vector_addIf successful, it will print: Vector addition successful!
How Do Threads Know Which Data to Process? Indexing Explained
CUDA's effectiveness stems from each thread working on a different piece of data. This is achieved through built-in variables that give each thread a unique identity within the launch grid.
CUDA organizes threads into a hierarchy:
- Grid: The set of all threads for a single kernel launch.
- Block: A group of threads. Threads within a block can communicate efficiently via shared memory and synchronize their execution (Getting Started with CUDA).
- Thread: The smallest unit of execution.
Inside a kernel, you can access these built-in variables:
blockDim.x: The number of threads in a block.blockIdx.x: The ID of the current block within the grid.threadIdx.x: The ID of the current thread within its block.
To get a globally unique index for each thread across the entire grid (for a 1D problem), we use a standard formula:
int global_index = blockIdx.x * blockDim.x + threadIdx.x;
Hardware Limits and Execution Configuration
When launching a kernel, the choice of block and grid dimensions is constrained by the GPU's hardware limits, which vary by architecture. For example, a device might have the following limits:
- Maximum threads per thread block: 1024
- Maximum resident threads per SM: 2048
- Maximum resident blocks per SM: 32
- Shared memory limit per block: 49152 bytes
- Registers per SM: 65536
These resource constraints (threads, blocks, shared memory, registers) determine how many blocks can run concurrently on a Streaming Multiprocessor (SM). Choosing launch parameters requires balancing parallelism with these per-block and per-SM resource limitations (Writing SIMT Kernels).
A Worked Example
Imagine we want to process an array with 16 elements. We decide to launch a kernel with 4 blocks, and each block has 4 threads.
blockDim.xwill be 4.blockIdx.xwill range from 0 to 3.threadIdx.xwill range from 0 to 3.
Let's calculate the global index for a few threads:
- Thread 0 in Block 0:
idx = (blockIdx.x * blockDim.x) + threadIdx.xidx = (0 * 4) + 0 = 0. This thread processesdata[0]. - Thread 3 in Block 0:
idx = (0 * 4) + 3 = 3. This thread processesdata[3]. - Thread 0 in Block 2:
idx = (2 * 4) + 0 = 8. This thread processesdata[8]. - Thread 3 in Block 3:
idx = (3 * 4) + 3 = 15. This thread processesdata[15].
This indexing scheme ensures that every thread computes a unique index, allowing them to work on different elements of the array in parallel. It is also why the bounds check if (idx < n) is critical. If your data size isn't a perfect multiple of your block size, you will launch extra threads that should do nothing.
When is a GPU Slower? A Look at Overhead and Synchronization
A common misconception is that GPUs are always faster than CPUs. For small tasks, the overhead of using a GPU can make it significantly slower. This overhead comes from several sources: memory transfer time, kernel launch latency, and synchronization. The GPU only wins when the parallel computation is large enough to hide these fixed costs.
Let's compare our vector-add example to a simple single-threaded CPU version.
Illustrative Performance Comparison:
| Vector Size (N) | CPU Time (Single-Thread) | GPU Time (Total) | Faster? |
|---|---|---|---|
| 1,000 | 1 microsecond | 50 microseconds | CPU is 50x faster |
| 10,000,000 | 10 milliseconds | 1 millisecond | GPU is 10x faster |
Why is the GPU slower for N=1,000?
- Memory Transfer:
cudaMemcpymust move data over the PCIe bus, which takes time. For a tiny vector, this transfer time can be much longer than the computation itself. Exploring GPU memory bandwidth can help you understand these costs. - Kernel Launch Overhead: There is a fixed latency associated with telling the GPU to start a kernel. This involves the CPU, the driver, and the GPU scheduler.
- Synchronization: When the CPU calls
cudaDeviceSynchronize(), it must halt and wait.
For the large vector (N=10,000,000), the parallel computation is substantial. The GPU executes millions of additions simultaneously, and this massive speedup far outweighs the fixed overhead costs. This is the fundamental trade-off in GPU computing.
CUDA C++ vs. Triton Kernels: Which Should You Learn?
While CUDA C++ offers maximum control, it's not the only way to write kernels. For machine learning engineers, Triton is a popular, higher-level alternative. Triton is a Python-based language and compiler designed for writing efficient custom deep learning kernels (Triton Documentation).
Here’s a comparison to help you choose:
| Feature | CUDA C++ | Triton |
|---|---|---|
| Language | C++ with NVIDIA extensions | Python-based syntax |
| Abstraction | Low-level. Manual memory management, explicit indexing. | High-level. Focus on block-level operations, not individual threads. |
| Productivity | Slower to write and debug. | Faster to write, easier to integrate into Python ML workflows. |
| Performance | Can achieve maximum performance with expert tuning. | Compiler handles many optimizations (e.g., memory coalescing) automatically. |
| Best For | Building core libraries, complex non-DNN algorithms, absolute peak performance. | Rapidly creating custom operators for ML models (e.g., fused attention). |
For most ML engineers looking to write a custom operator for a PyTorch model, starting with Triton is often more productive. For those building foundational libraries or needing to solve complex parallel problems outside the ML domain, CUDA C++ is the essential tool.
What Are the Most Common Mistakes for Beginners?
When starting with CUDA, a few common pitfalls can cause frustrating bugs. Being aware of them can save you hours of debugging.
- Forgetting to Synchronize: A kernel launch returns immediately. If you try to read the results on the CPU without calling
cudaDeviceSynchronize()first, you will likely get incorrect or uninitialized data because the GPU hasn't finished. - Mixing Host and Device Pointers: Dereferencing a GPU memory pointer in CPU code (or vice-versa) will cause a segmentation fault. You must use
cudaMemcpyto move data between the two memory spaces (CUDA C Basics). - Missing Bounds Checks: Launching more threads than data elements is a common and valid strategy. However, failing to add a bounds check (
if (idx < n)) inside your kernel will cause threads to write past the end of your allocated memory, leading to crashes or corrupted data. - Ignoring API Errors: CUDA API functions return an error code. Ignoring these codes hides the root cause of problems like out-of-memory errors or invalid launch configurations. Always wrap CUDA calls in an error-checking macro.
- Assuming Cross-Block Communication: Threads in different blocks cannot be synchronized with
__syncthreads()and cannot easily share data. Block-level cooperation is a powerful tool, but it is limited to threads within the same block.
What to learn next
- GPU Architecture for AI: A Beginner's Guide: Understand the hardware components like Streaming Multiprocessors (SMs) where your kernels actually run.
- GPU Memory Bandwidth Explained for AI: Dive deeper into the costs of data movement between the CPU and GPU, a key performance bottleneck.
- Why GPUs for AI? A Guide to Parallel Computing: Explore the broader context of why parallel architectures like GPUs are fundamental to modern machine learning.
