Modern artificial intelligence models are built on a mathematical operation performed an immense number of times: matrix multiplication. To make this feasible, GPU manufacturers created specialized hardware to accelerate this exact workload. This article provides a complete tensor cores explained guide, from the underlying hardware to the code you write.

Key takeaways

  • Tensor Cores are specialized hardware units inside modern GPUs. They accelerate one specific operation: matrix multiply-accumulate, often written as D = A × B + C.
  • They achieve massive speedups by using mixed precision. This means they perform multiplications on low-precision data (like FP16 or INT8) and add the results to a higher-precision total (like FP32).
  • You don't program Tensor Cores directly. Your code uses them automatically when you use frameworks like PyTorch with the correct data types and matrix sizes on a compatible GPU.
  • Peak Tensor Core performance (TFLOPS) is a theoretical maximum. Real-world speedup depends on factors like GPU memory bandwidth, kernel overhead, and if your matrix dimensions are efficiently supported.
  • While created for AI, any task dominated by dense matrix multiplication can benefit, including some scientific computing jobs.

What are Tensor Cores and how do they fit inside a GPU?

Tensor Cores are specialized execution units within a GPU's Streaming Multiprocessors (SMs) built for one purpose: accelerating matrix math. To understand them, it helps to see the hierarchy. A modern GPU's architecture consists of many SMs. Each SM contains hundreds of general-purpose CUDA Cores and a smaller number of highly specialized Tensor Cores. For example, the original Volta architecture had 8 Tensor Cores per SM (Source).

A CUDA Core is like a flexible worker who can perform simple, single calculations like x + y or a * b. A Tensor Core is like a dedicated assembly line. It's much larger and less flexible, but it can perform an entire multi-step operation, D = A × B + C, on a small matrix all at once. This "fused" operation is far more efficient than asking many CUDA Cores to do the same job with dozens of separate instructions.

Groups of threads, called warps, work together to feed data to a Tensor Core. A single thread doesn't control the whole operation; instead, the entire warp collectively issues a warp-level matrix instruction (Source). This cooperative model allows the GPU to process large blocks of data with extreme parallelism.

How do Tensor Cores speed up matrix multiplication?

Tensor Cores accelerate matrix multiplication through a combination of specialized hardware, massive parallelism, and mixed-precision arithmetic. This process turns a large mathematical problem into millions of tiny, manageable tasks that run simultaneously. The underlying operation is a General Matrix Multiply (GEMM), which computes C = αAB + βC (Source). The entire process, managed by libraries like cuBLAS, happens in a few steps.

  1. Tiling the Problem: A large matrix multiplication, like one in a neural network layer, is too big for a single core. The CUDA library breaks it down into many smaller, independent matrix multiplication "tiles."

  2. Distributing Tiles: The GPU assigns each tile to a group of threads called a thread block. This thread block loads its assigned input tiles from slow global GPU memory into the much faster shared memory of its assigned SM.

  3. Fused Multiply-Accumulate: A warp within the block feeds fragments of these tiles to a Tensor Core. The Tensor Core then performs the entire D = A × B + C operation in a single hardware instruction (Source). A single Volta-generation Tensor Core could perform 64 floating-point fused multiply-accumulate operations per clock cycle (Source).

  4. Accumulating in High Precision: This is where mixed precision comes in. The input matrices (A and B) are often in a low-precision format like FP16. This saves memory and bandwidth. The hardware multiplies them and adds the result to an accumulator matrix (C/D) that is kept in a higher-precision format, like FP32. This preserves numerical accuracy over many thousands of operations (Source).

This process repeats in parallel across all the SMs on the GPU. Each SM processes different tiles of the larger matrix, and the final results are written back to global memory.

A worked example: The Fused Multiply-Accumulate operation

The core of a Tensor Core's power is the fused matrix multiply-accumulate (MMA) operation. Let's visualize how D = A × B + C works with simple, illustrative 2x2 matrices. Real Tensor Cores operate on specific tile sizes (e.g., a 4x4x4 matrix operation on the original Volta architecture) depending on the GPU architecture and data type, but the principle is identical (Source).

Imagine our input matrices A and B, and our initial accumulator matrix C, are:

  • Matrix A: [[1, 2], [3, 4]]
  • Matrix B: [[5, 6], [7, 8]]
  • Accumulator C: [[10, 10], [10, 10]]

A Tensor Core performs the entire operation in one step, but we can break it down logically:

  1. Multiply A × B: The hardware computes the dot product of A's rows with B's columns.

    • Result [0,0] = (1 * 5) + (2 * 7) = 5 + 14 = 19
    • Result [0,1] = (1 * 6) + (2 * 8) = 6 + 16 = 22
    • Result [1,0] = (3 * 5) + (4 * 7) = 15 + 28 = 43
    • Result [1,1] = (3 * 6) + (4 * 8) = 18 + 32 = 50
    • The product matrix is [[19, 22], [43, 50]].
  2. Add Accumulator C: The hardware adds the result of the multiplication to the initial accumulator matrix C.

    • Result [0,0] = 19 + 10 = 29
    • Result [0,1] = 22 + 10 = 32
    • Result [1,0] = 43 + 10 = 53
    • Result [1,1] = 50 + 10 = 60

The final output matrix D is [[29, 32], [53, 60]].

A Tensor Core executes this entire sequence as a single, atomic hardware instruction. This avoids the overhead of loading and storing intermediate results, which is a key reason GPUs are used for AI.

What is mixed precision and which format should you use?

Mixed-precision computing is the technique of using lower-precision numbers for most calculations to increase speed and reduce memory, while using higher-precision numbers for critical parts of the computation to maintain accuracy. Tensor Cores are designed for this. They multiply low-precision inputs and "accumulate" the results into a high-precision total.

Choosing the right format involves a trade-off between speed, memory usage, and numerical precision.

Format Common Use Case Key Trade-off
FP32 (Standard) General computing, initial model training High precision, but slow and memory-intensive; does not use Tensor Cores.
FP16 (Half Precision) AI training and inference Enables Tensor Core acceleration by using lower precision inputs, which requires care to maintain numerical stability.
INT8 (8-bit Integer) AI inference after quantization Offers high performance and low memory usage on supported hardware, but requires a calibration process (quantization) to maintain accuracy.
  • For training: Using FP16 with Automatic Mixed Precision (AMP) libraries can provide significant speedups. These libraries handle the details of loss scaling to prevent numerical errors.
  • For inference: FP16 is a common choice for a good balance of speed and ease of use. For the best performance on supported hardware, INT8 is ideal, but it requires an extra step to quantize the model.

How do you actually use Tensor Cores?

You don't write code for Tensor Cores directly. Instead, you enable them by meeting a set of conditions that allow high-level libraries to dispatch work to them. Frameworks like PyTorch and TensorFlow use underlying libraries like cuDNN (for convolutions) and cuBLAS (for matrix multiplications) to execute operations on the GPU (Source). These libraries automatically select Tensor Core kernels when possible.

Here is a simple checklist to ensure your code can leverage them:

  1. Use a Compatible GPU: Tensor Cores were introduced with the Volta architecture (e.g., V100 GPU) and are present in subsequent generations like Turing, Ampere, and beyond. Ensure your hardware has them.

  2. Use Supported Data Types: Convert your model and data to a supported low-precision format. This is the most critical step. For example, in PyTorch, you can use Automatic Mixed Precision (AMP) or manually convert your data.

    Python
    # Example in PyTorch
    import torch
    
    # Enable Automatic Mixed Precision (AMP)
    scaler = torch.cuda.amp.GradScaler()
    model = MyModel().cuda()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
    
    for epoch in epochs:
        for input, target in data:
            optimizer.zero_grad()
            with torch.cuda.amp.autocast(): # This context manager enables mixed precision
                output = model(input)
                loss = loss_fn(output, target)
            
            scaler.scale(loss).backward()
            scaler.step(optimizer)
            scaler.update()
  3. Ensure Matrix Dimensions are Compatible: For maximum efficiency, matrix dimensions should often be multiples of a certain number, typically 8 or 16 for FP16 data, though this varies by GPU and library version (Source). While modern libraries have improved support for odd shapes, aligning dimensions can still improve performance by avoiding overhead.

  4. Use Up-to-Date Libraries: Keep your GPU driver, CUDA toolkit, and deep learning frameworks updated. Newer versions of cuDNN and cuBLAS often contain more optimized kernels and better heuristics for deciding when to use Tensor Cores.

Just setting the data type to float16 is not a guarantee. The library makes the final decision based on a cost-benefit analysis, considering the GPU, matrix sizes, and available kernels (Source).

When should you NOT use Tensor Cores?

While powerful, Tensor Cores are not a universal solution. They are specialized tools, and attempting to use them in the wrong context can lead to no speedup or even a slowdown.

  • High-Precision Requirements: If your application is sensitive to numerical precision and cannot tolerate the reduced precision of formats like FP16, you should stick to FP32 or FP64. This is common in certain scientific simulations where small errors can accumulate and corrupt the final result.
  • Scalar or Element-wise Workloads: Tensor Cores only accelerate matrix multiplication and related operations. Code that is dominated by element-wise operations (like activation functions), scalar math, or irregular data access patterns will not benefit.
  • Small Matrix Sizes: There is a certain amount of overhead in launching any GPU kernel, including those that use Tensor Cores. If your matrix multiplications are very small, the overhead of calling a specialized Tensor Core kernel may be greater than the benefit of the hardware acceleration.
  • Memory-Bound Problems: If your application's bottleneck is reading and writing data from memory, then making the computation part faster won't help overall performance. The Tensor Cores will simply spend more time waiting for data.
  • Incompatible Hardware or Software: If your GPU does not have Tensor Cores or your software libraries are outdated, you cannot use them.

What to learn next

Sources and further reading