Several server trends are driving the increased demand for
I/O bandwidth: the arrival of multicore CPUs, use of virtualization
(driving up server utilization), powerful new cluster applications,
increased reliance on networked storage and the rise of blade servers.
With high bandwidth and end-to-end latency in the
range of 1 microsecond, 20Gbps InfiniBand (commonly called DDR, or
double data rate) is helping to address data center I/O challenges and
is rapidly gaining acceptance in high-performance computing centers.
InfiniBand is a standard defined by the InfiniBand Trade Association and supported by many vendors and the Open Fabrics Alliance, an open source software community.
The standard defines several key technologies that help it deliver high performance and reliability with low latency: credit-based flow control, channelized I/O with hardware-based QoS, and a transport optimized for moving massive amounts of traffic with minimal load on the server.
Many network protocols have the ability to retransmit dropped packets, usually at the transport layer, but communications typically is slowed by these protocols to ensure recovery, severely degrading performance.
Most packet loss in an Ethernet network occurs when network equipment is heavily congested and buffer space becomes full: Packets are dropped because the speed of the traffic is too high to stop the transmitting source in time.
InfiniBand uses a credit-based, flow-control mechanism to ensure the integrity of the connection, so packets rarely are dropped. With InfiniBand, packets will not be transmitted until there is verified space in the receiving buffer. The destination issues credits to signal available buffer space, after which the packets are transmitted. This eliminates congestion as a source of packet loss, greatly improving efficiency and overall performance.
InfiniBand also uses strict QoS controls implemented in hardware. When multiple servers share network resources, it's important to prevent a flood of low-priority traffic from blocking time-sensitive traffic, and the problem is compounded when multiple virtual servers are implemented on a group of servers.
InfiniBand's credit-based flow control is provided separately across many channels, providing a simple yet robust QoS mechanism for protecting traffic. Traffic protection is critical for converged wire strategies because it lets a single interconnect replace multiple, parallel networks required for clustering, storage, communications and management traffic. It also is critical for virtualized environments.
Because InfiniBand was designed to connect servers and storage efficiently in close proximity, the InfiniBand transport protocol was optimized for this environment. TCP, on the other hand, is the most ubiquitous transport protocol - implemented on devices ranging from refrigerators to supercomputers - but generality comes at a price: It is complex, the code is large and full of special cases, and it's difficult to offload. InfiniBand transport was defined later, during the era of multigigabit networking and high-performance servers, and is more streamlined, making it suitable for offloading to purpose-built, efficient hardware adapters. Offloading the InfiniBand transport enables very high performance with minimal load on the host CPU, so the CPU can focus on useful application processing.
However, most users of InfiniBand also run TCP/IP for application compatibility. Some traffic between servers requires the performance and offloading of the InfiniBand transport, while other traffic requires the protocol and application compatibility provided by TCP/IP. Many InfiniBand-powered data centers use both.
As the technology behind InfiniBand becomes more familiar, the benefits in performance, cost savings and scaling are being applied to a wider variety of applications, both in technical clusters and in the data center. InfiniBand is an example of a technology that is a perfect fit for the server and storage interconnect challenges for which it was conceived.
Tuchler is senior director, product management at Mellanox Technologies. He can be reached at dan@mellanox.com.
InfiniBand is a standard defined by the InfiniBand Trade Association and supported by many vendors and the Open Fabrics Alliance, an open source software community.
The standard defines several key technologies that help it deliver high performance and reliability with low latency: credit-based flow control, channelized I/O with hardware-based QoS, and a transport optimized for moving massive amounts of traffic with minimal load on the server.
Many network protocols have the ability to retransmit dropped packets, usually at the transport layer, but communications typically is slowed by these protocols to ensure recovery, severely degrading performance.
Most packet loss in an Ethernet network occurs when network equipment is heavily congested and buffer space becomes full: Packets are dropped because the speed of the traffic is too high to stop the transmitting source in time.
InfiniBand uses a credit-based, flow-control mechanism to ensure the integrity of the connection, so packets rarely are dropped. With InfiniBand, packets will not be transmitted until there is verified space in the receiving buffer. The destination issues credits to signal available buffer space, after which the packets are transmitted. This eliminates congestion as a source of packet loss, greatly improving efficiency and overall performance.
InfiniBand also uses strict QoS controls implemented in hardware. When multiple servers share network resources, it's important to prevent a flood of low-priority traffic from blocking time-sensitive traffic, and the problem is compounded when multiple virtual servers are implemented on a group of servers.
InfiniBand's credit-based flow control is provided separately across many channels, providing a simple yet robust QoS mechanism for protecting traffic. Traffic protection is critical for converged wire strategies because it lets a single interconnect replace multiple, parallel networks required for clustering, storage, communications and management traffic. It also is critical for virtualized environments.
Because InfiniBand was designed to connect servers and storage efficiently in close proximity, the InfiniBand transport protocol was optimized for this environment. TCP, on the other hand, is the most ubiquitous transport protocol - implemented on devices ranging from refrigerators to supercomputers - but generality comes at a price: It is complex, the code is large and full of special cases, and it's difficult to offload. InfiniBand transport was defined later, during the era of multigigabit networking and high-performance servers, and is more streamlined, making it suitable for offloading to purpose-built, efficient hardware adapters. Offloading the InfiniBand transport enables very high performance with minimal load on the host CPU, so the CPU can focus on useful application processing.
However, most users of InfiniBand also run TCP/IP for application compatibility. Some traffic between servers requires the performance and offloading of the InfiniBand transport, while other traffic requires the protocol and application compatibility provided by TCP/IP. Many InfiniBand-powered data centers use both.
As the technology behind InfiniBand becomes more familiar, the benefits in performance, cost savings and scaling are being applied to a wider variety of applications, both in technical clusters and in the data center. InfiniBand is an example of a technology that is a perfect fit for the server and storage interconnect challenges for which it was conceived.
Tuchler is senior director, product management at Mellanox Technologies. He can be reached at dan@mellanox.com.
No comments:
Post a Comment