When scaling AI training, the choice of interconnects becomes critical. The nvlink vs infiniband debate is common, but it frames two complementary technologies as competitors. NVLink is a high-bandwidth connection for GPUs inside a server, while InfiniBand is a networking fabric that connects many servers into a cluster. Understanding their distinct roles is key to building efficient AI infrastructure.

Key takeaways

  • Complementary, not competing: NVLink and InfiniBand solve different problems. Large AI clusters use NVLink for ultra-fast communication inside a server and InfiniBand to connect different servers.
  • Scale-Up vs. Scale-Out: NVLink "scales up," packing more GPU power into a single, tightly-coupled domain. InfiniBand "scales out," connecting many independent server nodes into a larger computing fabric.
  • Bandwidth numbers are misleading: Directly comparing a 900 GB/s NVLink figure to a 400 Gb/s InfiniBand figure is an apples-to-oranges comparison. They use different units (Bytes vs. bits), measurement scopes, and directionality.
  • Both are critical for performance: In distributed AI training, communication libraries must efficiently use both NVLink and InfiniBand paths to synchronize data between GPUs, a process that can easily become a bottleneck.

To understand the difference, picture a typical AI cluster. An entire cluster consists of many servers, called nodes, connected together. Each node contains multiple GPUs. NVLink and InfiniBand operate at different levels of this hierarchy. NVLink is the local network for GPUs inside a node, while InfiniBand is the broader network connecting all the nodes.

  • Intra-Node (Inside the Server): A server might have eight GPUs. To exchange data quickly, these GPUs are connected by a mesh of NVLink connections, often through a central NVSwitch. This creates a high-bandwidth "scale-up" domain where any of the eight GPUs can talk to any other at full speed. This path is used for frequent, high-volume data transfers.
  • Inter-Node (Between Servers): To connect this server to another one, each node has a high-speed network adapter. This adapter connects to a larger network fabric made of InfiniBand switches. When a GPU in server A needs to send data to a GPU in server B, the data travels out of the server and across this "scale-out" fabric.

Think of it like an office building. NVLink is the high-speed elevator system connecting floors within one building. InfiniBand is the network of roads and highways connecting all the different buildings in the city. You need both to operate at scale.

How Does Interconnect Performance Limit AI Scaling?

Fast interconnects are essential because training large AI models requires splitting the work across hundreds or thousands of GPUs. This process, called distributed training, introduces communication overhead. GPUs must constantly synchronize with each other, and if this communication is slow, they spend more time waiting for data than performing calculations.

This synchronization often happens through collective communication operations. A collective is a communication pattern involving a group of processors. The most common one in AI training is all-reduce. In an all-reduce operation, every GPU has a piece of data (like a calculated gradient). The goal is for every GPU to end up with the sum of all those pieces. This involves aggregating data from all GPUs and then distributing the final result back to all of them.

If communication is slow, all-reduce becomes a major bottleneck. As you add more GPUs to a training job, you increase the amount of communication required. At some point, the time spent communicating outweighs the benefit of adding more compute power. Scaling stops being linear when communication becomes a substantial fraction of each training step (Source). A faster interconnect raises the ceiling, allowing more GPUs to work together effectively before this bottleneck takes over.

NVLink is a proprietary, high-bandwidth interconnect created by NVIDIA specifically to connect its GPUs (Source). It provides a much faster and more direct data path between GPUs than the standard Peripheral Component Interconnect Express (PCIe) bus that connects most computer components. This direct link is crucial for performance in multi-GPU servers.

How it works

  1. Direct Links: At its simplest, NVLink provides a point-to-point connection between two GPUs.
  2. NVSwitch: To connect more than two GPUs efficiently, NVIDIA developed the NVSwitch. An NVSwitch aggregates NVLink connections into a larger fabric, providing switched, high-bandwidth connectivity between multiple GPUs in a supported topology (Source).
  3. Bandwidth Evolution: NVLink has improved with each GPU architecture generation. For example, the V100 GPU offered 300 GB/s of total bidirectional bandwidth and the A100 offered 600 GB/s, while the H100 provides 900 GB/s of total NVLink bandwidth for multi-GPU I/O and shared-memory accesses (Source, Source). These are aggregate, bidirectional figures.
  4. NVLink Network: Historically, NVLink was an intra-server technology. However, with recent generations, NVLink Network technology uses NVSwitches to connect up to 256 GPUs across multiple nodes (Source). This blurs the line between a single server and the rack, creating a much larger scale-up domain. Importantly, endpoints connected via NVLink Network do not automatically share a common memory address space; software must explicitly establish connections, similar to traditional networking interfaces (Source).

What Is InfiniBand? The Cluster Scale-Out Fabric

InfiniBand is an industry-standard networking technology designed for high-performance computing (HPC) and AI clusters (Source). Unlike NVLink, it's not specific to GPUs. It connects entire servers (nodes) together with very high bandwidth and low latency, using components like Host Channel Adapters (HCAs) in the servers and dedicated InfiniBand switches.

Key features for AI

  1. High Bandwidth and Low Latency: InfiniBand offers speeds like 400 Gb/s (NDR) and higher, with much lower latency than traditional Ethernet.
  2. Remote Direct Memory Access (RDMA): This is InfiniBand's most important feature for AI. RDMA allows a network adapter in one server to write data directly into the memory of another server, bypassing the CPU, caches, and operating system on both ends (Source). This dramatically reduces latency and frees up CPU cycles for other tasks.
  3. GPUDirect RDMA: This NVIDIA technology extends RDMA to GPUs. It allows the InfiniBand HCA to pull data directly from one GPU's memory and push it directly to another GPU's memory in a different node. This avoids a slow, multi-step copy through the server's main memory, which is a critical optimization. Activating this feature requires compatible hardware and correct software configuration from end to end (Source).

Directly comparing bandwidth numbers is the most common point of confusion. A 900 GB/s NVLink figure seems vastly larger than a 400 Gb/s InfiniBand figure. However, they measure different things.

  • Units: NVLink is measured in GigaBytes per second (GB/s), while InfiniBand is in Gigabits per second (Gb/s). 1 Byte = 8 bits.
  • Directionality: NVLink figures are often a bidirectional aggregate, meaning the sum of data sent and received. Network speeds like InfiniBand's are typically unidirectional, meaning the speed in one direction.
  • Scope: The NVLink figure is the total bandwidth available to a single GPU from all its links. The InfiniBand figure is the speed of a single network port. A server may have multiple ports.

Here is a normalized comparison between NVLink on an H100 GPU and a standard NDR InfiniBand port.

Feature NVLink (4th Gen on H100) InfiniBand (NDR)
Primary Role GPU Scale-Up (Intra-Node/Rack) Node Scale-Out (Inter-Node)
Technology Type Proprietary NVIDIA GPU Interconnect Open Industry Standard Network
Typical Scope Connects 8 to 256 GPUs in a high-speed domain Connects multiple servers in a scale-out fabric
Advertised Bandwidth 900 GB/s total bandwidth (for multi-GPU I/O and shared-memory accesses) 400 Gb/s (per port, unidirectional)
Normalized Unidirectional BW 450 GB/s (900 GB/s / 2) 50 GB/s (400 Gb/s / 8 bits per Byte)
Key Enabling Tech NVSwitch RDMA / GPUDirect RDMA
Latency Profile Extremely low within its domain Very low for a network between nodes

As the table shows, even after normalizing, NVLink's per-GPU bandwidth is significantly higher. This is by design. Communication within a node should be much faster than communication between nodes.

An Example: The All-Reduce Operation in Action

Let's trace a simplified all-reduce operation across four GPUs in two different servers to see how NVLink and InfiniBand work together.

Setup:

  • Node A: Contains GPU-0 and GPU-1, connected by NVLink.
  • Node B: Contains GPU-2 and GPU-3, connected by NVLink.
  • Node A and Node B are connected by InfiniBand.

A communication library like NVIDIA's NCCL orchestrates this process (Source).

  1. Intra-Node Reduce (NVLink):

    • GPU-1 sends its data to GPU-0 over the fast NVLink connection. GPU-0 adds it to its own data.
    • Simultaneously, GPU-3 sends its data to GPU-2 over NVLink. GPU-2 adds it to its own data.
    • Path used: NVLink
  2. Inter-Node Exchange (InfiniBand):

    • GPU-0 (now holding the combined data from Node A) sends its partial sum to GPU-2 in Node B.
    • This data travels from GPU-0's memory, across the PCIe bus to the InfiniBand network adapter, over the InfiniBand fabric, to the adapter in Node B, and finally to GPU-2's memory (ideally using GPUDirect RDMA).
    • GPU-2 adds this value to its own partial sum. It now holds the final, globally reduced value.
    • Path used: InfiniBand
  3. Inter-Node & Intra-Node Broadcast:

    • GPU-2 sends the final result back to GPU-0 over the InfiniBand network.
    • GPU-0 then broadcasts the final result to GPU-1 over NVLink.
    • Simultaneously, GPU-2 broadcasts the final result to GPU-3 over NVLink.
    • Paths used: InfiniBand and NVLink

At the end, all four GPUs have the final correct sum. This hierarchical process uses the fastest local links (NVLink) for local aggregation and reserves the relatively slower inter-node links (InfiniBand) for communicating between group leaders.

What to learn next

Sources and further reading