InfiniBand Low Level Performance

Paderborn Center for Parallel Computing: Jens Simon (simon(at)upb.de)

  • Interconnect: InfiniBand 4x SDR (10 Gbit/s) und DDR (20 Gbit/s)

author                         

OFED                             

benchmark program

ib_*

parameters receiver

ib_rdma_lat

 

ib_read_bw  -n 10000 -s 1024

 

ib_read_bw  -n 10000 -s 1000000

 

ib_read_bw -b -n 10000 -s 1024

 

ib_read_bw -b -n 10000 -s 1000000

parameters sender

… <receiver IP>

Machines NIC  / Software RDMA latency [µs] unidir. bandwidth [Mbyte/s] bidir. bandwidth [Mbyte/s] remarks
  Switch   1byte pack. 1kB pack. 1MB pack. 1kB pack. 1MB pack.
Supermicro H8QMi-2
Quad Socket 2.8 GHz AMD Opteron 8220 Dual-Core,
32x2 Gbyte FB-DDR2-667 (operated 555)		 Mellanox PCI-e 8x ConnectX 4x DDR /
Voltair I9024D-M-24p 4x DDR		 OFED
Redhat LS 5		            
1,19 719 1410 1080 2340 DDR Switch: 1 hop
       
04.03.2008
Supermicro H8QMi-2
Quad Socket 2.8 GHz AMD Opteron 8220 Dual-Core,
32x2 Gbyte FB-DDR2-667 (operated 555)		 Mellanox PCI-e 8x ConnectX 4x DDR /
Voltair I9024D-M-24p 4x DDR		 OFED
SuSE Linux		            
1,19 719 1221 1091 2300 DDR Switch: 1 hop
       
04.03.2008

 

 

author:

Mellanox

benchmark program

perf_main

parameters sender:

 --send -rwrite -mlat -s1 -n100000

 --send -rwrite -mlat -s1024 -n100000

 --send -rwrite -mbw -s1024 -n1000000 -b0 -c1

 --send -rwrite -mbw -s1048576 -n10000 -b0 -c1

 --send -rwrite -mbw -s1024 -n1000000 -b1 -c1

 --send -rwrite -mbw -s1048576 -n10000 -b1 -c1

parameters receiver:

 -a<IP>

Machines NIC  / Software RDMA latency [µs] unidir. bandwidth [Mbyte/s] bidir. bandwidth [Mbyte/s] remarks
  Switch   1byte pack. 1kB pack. 1kB pack. 1MB pack. 1kB pack. 1MB pack.
Fujitsu-Siemens TX300-S2
Dual 3.0 GHz Xeon 5160 Dual-Core, Intel 5000V
4x2 Gbyte FB-DDR2-555		 SilverStorm PCI-e 8x HCA 9000 4x DDR /
Voltair I9024D-M-24p 4x DDR		 IB Gold 1.8.3
Scientific Linux, 2.6.9-42		              
2,99 5,55 890 1294 1501 2144 DDR Switch: 1 hop
       
03.04.2007
Fujitsu-Siemens TX300-S2
Dual 3.0 GHz Xeon 5160 Dual-Core, Intel 5000V
4x2 Gbyte FB-DDR2-555		 SilverStorm PCI-e 8x HCA 9000 4x DDR /
SilverStorm IO5000 4x SDR		 IB Gold 1.8.3
Scientific Linux, 2.6.9-42		              
3,08 6,25 739 919 1481 1654 SDR Switch: 1 hop
       
15.01.2007
Fujitsu-Siemens TX300-S2
Dual 3.0 GHz Xeon 5160 Dual-Core, Intel 5000V
4x2 Gbyte FB-DDR2-555		 SilverStorm PCI-e 8x HCA 9000 4x DDR /
SilverStorm IO5000 4x SDR		 IB Gold 1.8.3-RC20
RedHat AS4 U4, 2.6.9-42		 3,27 7,37 567 685 818 1200 PCIe-4 lanes; back-to-back
3,63 8,22 560 686 871 1201 PCIe-4 lanes; SDR Switch: 1 hop
       
15.01.2007
Supermicro X7DA8
Dual 2.66 GHz Xeon 5345 Quad-Core, FSB1333, Intel 5000X
4x2 Gbyte FB-DDR2-666		 SilverStorm PCI-e 8x HCA 9000 4x DDR /
Voltair 9024D-M-24p 4x DDR		 IB Gold 1.8.3-RC20
RedHat AS4 U4, 2.6.9-42		              
2,99 5,33 925 1475 1603 2426 DDR Switch: 1 hop
       
15.01.2007
Supermicro X7DA8
Dual 2.66 GHz Xeon 5345 Quad-Core, FSB1333, Intel 5000X
4x2 Gbyte FB-DDR2-666		 SilverStorm PCI-e 8x HCA 9000 4x DDR /
Voltair 9024D-M-24p 4x SDR		 IB Gold 1.8.3-RC20
RedHat AS4 U4, 2.6.9-42		              
      933   1677 SDR Switch: 1 hop
       
15.01.2007
Supermicro X7DA8
Dual 2.66 GHz Xeon 5345 Quad-Core, FSB1333, Intel 5000X
4x2 Gbyte FB-DDR2-666		 SilverStorm PCI-e 8x HCA 9000 4x DDR /
Mellanox MTS2400-24p 4x SDR		 IB Gold 1.8.3-RC20
RedHat AS4 U4, 2.6.9-42		 2,98 5,19 926 1471 1592 2411 back-to-back:
3,13 6,1 750 933 1549 1670 SDR Switch: 1 hop
       
15.01.2007
Supermicro X7DA8
Dual 2.4 GHz Xeon 5345 Quad-Core, FSB1066, Intel 5000X
4x2 Gbyte FB-DDR2-666		 SilverStorm PCI-e 8x HCA 9000 4x DDR /
Silverstorm InfinIO 5000 4x SDR		 IB Gold 1.8.2-RC3
RedHat AS4 U4, 2.6.9-42		 2,99 5,73 822 1252 1379 1654 SDR back-to-back:
3,39 6,57 694 879 1357 1532 SDR Switch: 1 hop
       
02.01.2007
Supermicro SMX6DAE-G2
Dual 3.2 GHz Xeon EM64T, FSB800, E7525
4x1 Gbyte DDR2		 InfiniCon PCI-e 8x InfiniServ 9000
Firmware 4.6.1 /
Mellanox
MTEK43132-C08-S		 Mellanox vapi-linux 4.0.1
hmst 3.3.6
RedHat AS3 U3, 2.4.21-27.EL		             back-to-back: PCIe 8x; NIC: Standard firmware parameters
3,79 7,8 523 924 1337 1828 Switch: 1 hop;  PCIe 8x; NIC: Standard firmware parameters
       
Supermicro SMX6DAE-G2
Single 3.2 GHz Xeon EM64T, FSB800, E7525
8 x 512Mbyte DDR2-RAM
RedHat AS3 U2, 2.4.21-15.EL		 Mellanox PCI-e 8x MT25208 III Ex
Firmware 4.5.3./
Mellanox MTS2400-24p		 Mellanox thca 3.2-build-024
hmst 3.2.3		             back-to-back: PCIe 8x; NIC: Standard firmware parameters
3,65 6,55 669 918 1347 1783 Switch: 1 hop;  PCIe 8x; NIC: Standard firmware parameters
3,68 7,95 643 715 926 1311 Switch: 1 hop;  PCIe 4x; NIC: Standard firmware parameters
       
INTEL
Dual 3.4 GHz Xeon EM64T, FSB800
2 x 256 Mbyte DDR2-RAM
RedHat AS3 U2, 2.4.21-15.EL		 Mellanox PCI-e 8x MT25208 III Ex
Firmware 4.5.0./
Mellanox MTS2400-24p		 Mellanox thca 3.1.1-build-01
hmst 3.2.3		 3,44 6,39 632 926 1284 1789 back-to-back: PCIe 8x; NIC: Standard firmware parameters
3,6 6,63 632 919 1280 1784 Switch: 1 hop;  PCIe 8x; NIC: Standard firmware parameters
       
INTEL
Dual 3.4 GHz Xeon EM64T
Lindenhurst, FSB800
0.5 Gbyte DDR2-RAM
RedHat AS3 U2, 2.4.21-15.EL		 MT25208 III Ex
Firmware 4.0.1 /
Mellanox MTS2400-24p		 Mellanox thca 3.1.1-build-01
hmst 3.2.3		 3,44 6,39 668 926     back-to-back: PCIe 8x; NIC: Standard firmware parameters
3,67 6,63 668 918     Switch: 1 hop;  PCIe 8x; NIC: Standard firmware parameters
             
             
             
INTEL
Dual 3.4 GHz Xeon EM64T, FSB800
2 x 256 Mbyte DDR2-RAM
RedHat AS3 U2, 2.4.21-15.EL		 nfiniCon PCI-x InfiniServ HCA 7104
Firmware 3.0.1./
Mellanox MTS2400-24p		 Mellanox thca 3.1.1-build-01
hmst 3.2.3		 4,5 8,6 498 833 713 876 back-to-back: PCI-X 133MHz; NIC: Standard firmware parameters
4,73 8,83 498 833 714 885 Switch: 1 hop;  PCI-X 133MHz; NIC: Standard firmware parameters
             
             
             
HP rx2600
Dual 1.3 GHz Itanium2
Chip set ZX1
4 Gbyte DDR-RAM
Redhat AS 3 U2,  2.4.21-15.EL		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.2 /
InfiniCon InfinIO 3032		 Mellanox thca 3.2-build-024
hmst 3.1		 4,52 8,96 553 722 623 781 back-to-back 133MHz PCI-X: HCA TPT map=12, system page size=2^14, PCI-X outstanding reads=4
4,65 9,06 552 722 624 779 Switch 1 hop; 133MHz PCI-X: HCA TPT map=12, system page size=2^14, PCI-X outstanding reads=4
4,88 9,3 552 722 623 782 Switch 3 hops; 133MHz PCI-X: HCA TPT map=12, system page size=2^14, PCI-X outstanding reads=4
             
           
HP rx2600
Dual 1.3 GHz Itanium2
Chip set ZX1
4 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.37smp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0 /
InfiniCon InfinIO 3032		 Mellanox thca 3.1.1-build-01
hmst 3.1		 4,71 9,18 557 723     back-2-back; 133MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
4,84 9,3 558 723 623 779 Switch 1 hop; 133MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
5,07 9,53 558 723 629 782 Switch 3 hop; 133MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
           
HP rx2600
Dual 1.3 GHz Itanium2
Chip set ZX1
4 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.37smp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.0 /
Mellanox MTS9600-36p		 Mellanox thca 3.1-build-017
hmst 3.1		 4,98 9,45 559 723     Switch 1 hop; 133MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
5,55 10,04 557 723     Switch 3 hop; 133MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
6,11 10,6 558 723     Switch 5 hop; 133MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
           
IHP rx2600
Dual 1.3 GHz Itanium2
Chip set ZX1
4 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.37smp		 MTPB23108-CE128 rev. A1
Firmware 3.0 /
MTS9600-36p		 thca-ia64
3.0.1-build-001
mst 1.4.8		 5,37 10,03 502 723     Switch 1 hop: NIC slot1: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
5,92 10,59 502 723     Switch 3 hop: NIC slot1: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
6,46 11,12 502 723     Switch 5 hop: NIC slot1: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
             
IHP rx2600
Dual 1.3 GHz Itanium2
Chip set ZX1
4 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.37smp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.0 /
Mellanox
MTEK43132-C08-S		 thca-ia64
3.0.1-build-001
mst 1.4.8		 5,38 11,12 502 723     Switch 1 hop; 133MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
           
           
           
Intel Tiger4
Quad 1.5 GHz Itanium2
Chip set IntelE8870
16 Gbyte DDR-RAM
Redhat 2.1AS, 2.4.18-e.37smp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.0 /
Mellanox
MTEK43132-C08-S		 thca-ia64
3.0.1-build-001
mst 1.4.8		 5,55 11,15 495 761     Switch 1 hop; 133MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
           
           
           
HP rx2600
Dual 1.3 GHz Itanium2
Chip set ZX1
8 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.12smp		 MTPB23108-CE128 rev. A1
Firmware 2.0 /
MTS9600-36p		 thca-ia64
0.2.0-build-001
mst 1.4.8		 5,4 9,86 450 723     Switch 1 hop: NIC slot1: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
             
             
             
HP zx6000
Dual 1 GHz Itanium2
Chip set ZX1
8 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.12smp		 MTPB23108-CE128 rev. A1
Firmware 2.0 /
MTS9600-36p		 thca 1.0
mst 1.4.8		 Failed allocating WQEs buffers.         driver thca1.0 not supported  for Linux kernel 2.4.18-e.12smp
             
             
             
             
HP zx6000
Dual 1 GHz Itanium2
Chip set ZX1
12 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.37smp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.2 /
Mellanox
Mellanox MTS9600-36p		 Mellanox thca 3.2-build-024
hmst 3.2.3		 5,43 11,77 380 454 413 461 Switch 1 hop; 100MHz PCI-X: HCA TPT map=12, system page size=2^14, PCI-X outstanding reads=4
             
   
   
   
HP zx6000
Dual 1 GHz Itanium2
Chip set ZX1
12 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.37smp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0 /
Mellanox
MTEK43132-C08-S		 Mellanox thca 3.1-build-017
hmst 3.1		 5,45 12,85 380 454     Switch 1 hop; 100MHz PCI-X: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
   
   
   
HP zx6000
Dual 1 GHz Itanium2
Chip set ZX1
8 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.12smp		 MTPB23108-CE128 rev. A1
Firmware 1.18 /
MTS9600-36p		 thca-ia64
0.2.0-build-001
mst 1.4.8		 6,98 13,29 375 453     Switch: 5 hops; NIC: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
6,48 12,79 375 454     Switch: 3 hops; NIC: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
5,87 12,18 375 454     Switch: 1 hop; NIC: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
             
HP zx6000
Dual 1 GHz Itanium2
Chip set ZX1
8 Gbyte DDR-RAM
Redhat 2.1AW, 2.4.18-e.12smp		 MTPB23108-CE128 rev. A1
Firmware 1.18 /
MTS9600-36p		 thca-ia64
0.2.0-build-001
mst 1.4.8		 6,98 13,29 375 453     Switch: 5 hops; NIC: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
6,48 12,79 375 454     Switch: 3 hops; NIC: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
5,87 12,18 375 454     Switch: 1 hop; NIC: HCA TPT map=7, system page size=2^14, PCI-X outstanding reads=4
             
             
Iwill DK8S2-IB
Single 1.4 GHz Opteron 240
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-9.Elsmp		 onboard Mellanox
MT23108
Firmware 3.2.0  /
Mellanox
MTS 2400-24		 Mellanox thca 3.2-build-024
hmst 3.2.3		 4,09 7,89 530 631 691 833 back-to-back; NIC: standard parameters
4,33 8,15 534 628 693 837 Switch: 1 hop; NIC: standard parameters
             
           
           
Iwill DK8S2-IB
Single 1.4 GHz Opteron 240
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-9.Elsmp		 onboard Mellanox
MT23108
Firmware 3.1.0  /
Mellanox
MTS 2400-24		 Mellanox thca 3.2-build-024
hmst 3.2.3		 4,1 7,94 532 631 692 833 back-to-back; NIC: standard parameters
4,11 7,94 528 631 697 770 back-to-back; HCA TPT map=7, PCI-X outstanding reads=4
4,33 8,15 529 628 692 837 Switch: 1 hop; NIC: standard parameters
           
           
Fujitsu-Siemens v810
Dual 2.2 GHz Opteron
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-9.Elsmp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0  /
Mellanox
MTS 2400-24		 Mellanox thca 3.1.1-build-01
hmst 3.1		 4,07 7,94 552 821     back-to-back; NIC: HCA TPT map=7, PCI-X outstanding reads=4
4,3 8,19 552 831     Switch: 1 hop; NIC: HCA TPT map=7, PCI-X outstanding reads=4
             
         
           
Fujitsu-Siemens v810
Dual 2.2 GHz Opteron
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-9.Elsmp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.20  /
Mellanox
MTS 2400-24		 Mellanox thca 3.2-build-024
hmst 3.2.3		 3,93 7,85 581 823 709 841 back-to-back; NIC: HCA TPT map=7, PCI-X outstanding reads=4
4,18 8,09 517 623 691 831 Switch: 1 hop; NIC: standard parameters
4,17 8,09 584 832 714 845 Switch: 1 hop; NIC: HCA TPT map=7, PCI-X outstanding reads=4
           
           
Fujitsu-Siemens v810
Dual 2.2 GHz Opteron
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-9.Elsmp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0  /
Mellanox
MTS 2400-24		 Mellanox thca 3.2-build-024
hmst 3.2.3		             back-to-back; NIC: HCA TPT map=7, PCI-X outstanding reads=4
4,12 8,06 579 831 717 846 Switch: 1 hop; NIC: HCA TPT map=7, PCI-X outstanding reads=4
             
         
           
Fujitsu-Siemens v810
Dual 2.2 GHz Opteron
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-9.Elsmp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0  /
Mellanox
MTEK 43132-C08-S		 Mellanox thca 3.1.1-build-01
hmst 3.1		 4,07 7,94 552 821     back-to-back; NIC: HCA TPT map=7, PCI-X outstanding reads=4
4,36 9,29 554 822 690 847 Switch: 1 hop; NIC: HCA TPT map=7, PCI-X outstanding reads=4
             
         
           
Fujitsu-Siemens v810
Dual 2.2 GHz Opteron
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-9.Elsmp		 InfiniCon InfiniServ HCA
 /
InfiniCon InfinIO 3032		 Mellanox thca 3.1-build-017
hmst 3.1		 4,47 8,34 494 796     back-to-back
             
             
             
             
Fujitsu-Siemens v810
Dual 2.2 GHz Opteron
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-9.Elsmp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0  /
InfiniCon InfinIO 3032		 thca 3.1-build-017
hmst 3.1		 4,27 8,14 553 823     back-to-back; NIC: HCA TPT map=?, PCI-X outstanding reads=4
             
             
             
             
Newisys 2100
Dual 1.4 GHz Opteron 240
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-4.Elsmp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0  /
Mellanox
MTS 2400-24		 Mellanox thca 3.1.1-build-01
hmst 3.1		 4,43 8,77 563 722     back-to-back; NIC: HCA TPT map=7, PCI-X outstanding reads=4
4,63 8,98 564 722     Switch: 1 hop; NIC: HCA TPT map=7, PCI-X outstanding reads=4
             
         
           
Newisys 2100
Dual 1.4 GHz Opteron
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-4.Elsmp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0  /
InfiniCon InfinIO 3032		 thca 3.1-build-017
hmst 3.1		 4,41 8,72 544 731     back-to-back; NIC: HCA TPT map=7, PCI-X outstanding reads=4
4,55 8,82 543 731     Switc h: 1 hop; NIC: HCA TPT map=7, PCI-X outstanding reads=4
4,79 9,09 543 731     Switch: 3 hops; NIC: HCA TPT map=7, PCI-X outstanding reads=4
             
             
Newisys 2100
Dual 1.4 GHz Opteron 240
Chip set AMD 8131
2 x 1 Gbyte DDR-RAM
RedHat AS 3  2.4.21-4.Elsmp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 3.1.0  /
Mellanox
MTEK 43132-C08-S		 Mellanox thca 3.1.1-build-01
hmst 3.1		 4,43 8,77 563 722     back-to-back; NIC: HCA TPT map=7, PCI-X outstanding reads=4
4,7 10,08 564 722     Switch: 1 hop; NIC: HCA TPT map=7, PCI-X outstanding reads=4
             
         
           
HP xw8000
Dual 2.4 GHz Xeon
E7505, FSB533
2 Gbyte DDR-RAM
RedHat 7.3, 2.4.18-3smp		 Mellanox
MTPB23108-CE128 rev. A1
Firmware 2.0 /
Mellanox
MTEK43132-C08-S		 thca 1.01
mst 1.4.8		 5,36 10,76 440 775     Switch: 1 hop;  NIC: HCA TPT map=7, PCI-X outstanding reads=4 
5,06 9,4 440 775     back-to-back NIC: HCA TPT map=7, PCI-X outstanding reads=4
           

PC˛

http://www.uni-paderborn.de/pc2  

Mellanox Technologies:

http://www.mellanox.com

All measurements have been done in our Lab by our own.

My Staffweb