InfiniBand was developed to improve upon the limitations of older Ethernet technologies, and it is currently the fastest networking option available for an HPC cluster. The advantages of InfiniBand are high throughput (40Gb/s) and low latency, which is especially important if you are planning to rely on internet connectivity for your cluster.
InfiniBand acts as a independent messaging service between the various nodes in the HPC cluster. Ethernet networks require communication between nodes to go through the operating system and thus utilize more CPU resources. InfiniBand, on the other hand, bypasses the operating system and allows for direct communication between nodes.
InfiniBand is currently the best networking technology for HPC clusters, but it is more expensive than Ethernet. Applications which require maximum possible performance - such as oil and gas exploration modeling and financial computation, for example - should build their clusters with InfiniBand networking technology.