Ethernet, InfiniBand, and RDMA
RDMA means remote direct memory access. It moves data between registered memory regions with less remote-CPU involvement in the data path.
An RDMA data path
- Registered source memory
Application and software establish access
- Compatible adapters + fabric
Move the data with the supported transport
- Registered destination memory
Not unrestricted access to arbitrary memory
It still needs a network
InfiniBand supports RDMA. RoCE brings RDMA to Ethernet. Neither lets an application access arbitrary memory without the required setup and permissions.
A compatible adapter, fabric, and software stack must work together.
Fast links can still stall
When senders compete for a shared path, queues grow. Congestion controls help manage that pressure.
ECN signals congestion. PFC can pause selected traffic classes. Their use depends on the validated design; copying settings from another cluster is not a safe shortcut.
MTU limits packet size at a particular interface or layer. Validate the supported setting along the whole path. There is no universal GPU MTU.
Try it yourself
Small messages work, but large concurrent transfers struggle. What would you inspect before changing settings?
Check your answer
Compare endpoints, link rates, errors, queue behavior, congestion signals, and MTU consistency. Use equivalent tests on healthy and affected paths. RDMA reduces some overhead; it does not remove network bottlenecks.