Peeking at your Exadata infiniband traffic

As a DBA you are probably very curious on what is going on, on your system. So when you have a shiny Exadata you probably had a look at the infiniband fabric that is connecting the compute nodes and storage nodes together. When you want to see what kind traffic is going from the compute nodes to the storage nodes, or on the RAC interconnects you can use tcpdump to do so (if it is not install you can do a ‘yum install tcpdump’):

1 2 3 4 5 6 7

[root@dm01db02 ~]# tcpdump -i bond0 -s 0 -w /tmp/tcpdump.pcap tcpdump: WARNING: arptype 32 not supported by libpcap - falling back to cooked socket tcpdump: listening on bond0, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes 2073 packets captured 2073 packets received by filter 0 packets dropped by kernel [root@dm01db02 ~]#

This will give you a dump file (/tmp/tcpdump.pcap) which you can analyze with your favorite network analyzer (probably Wireshark). If you are new to this you can download and install Wireshark here: http://www.wireshark.org/download.html

Using tcpdump you can sniff all the IPOIB traffic (ip over infiniband), but can you take a peak at the other traffic that is going on the Infiniband wire? Yes there is a way, you can use Mellanox’s ibdump. This tool is not installed by default on your compute nodes so need to download it and install it on the node of your choice (as a reminder: don’t install anything on your cellservers!):

1 2 3 4 5 6 7 8 9 10 11 12

[root@dm01db02 ~]# wget http://www.mellanox.com/downloads/tools/ibdump-1.0.5-4-rpms.tgz --2012-02-11 15:13:27--  http://www.mellanox.com/downloads/tools/ibdump-1.0.5-4-rpms.tgz Resolving www.mellanox.com... 98.129.157.233 Connecting to www.mellanox.com|98.129.157.233|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 486054 (475K) [application/x-gzip] Saving to: `ibdump-1.0.5-4-rpms.tgz'   100%[==========================================================================================================================================>] 486,054      290K/s   in 1.6s   2012-02-11 15:13:29 (290 KB/s) - `ibdump-1.0.5-4-rpms.tgz' saved [486054/486054] [root@dm01db02 ~]

Extract the tarball:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

[root@dm01db02 ~]# tar -xvf ibdump-1.0.5-4-rpms.tgz ibdump-1.0.5-4-rpms/ ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.i386-rhel5.4.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.ppc64-rhel5.4.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.i386-rhel5.5.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.ppc64-rhel5.5.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.i386-rhel5.6.rpm ibdump-1.0.5-4-rpms/ibdump_release_notes.txt ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.x86_64-rhel5.4.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.x86_64-rhel5.5.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.ppc64-rhel5.6.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.x86_64-rhel5.6.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.i686-rhel6.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.ppc64-rhel6.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.x86_64-rhel6.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.i586-sles10sp3.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.ppc64-sles10sp3.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.x86_64-sles10sp3.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.i586-sles11.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.ppc64-sles11.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.i586-sles11sp1.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.ppc64-sles11sp1.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.x86_64-sles11sp1.rpm ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.x86_64-sles11.rpm [root@dm01db02 ~]#

Next step, install it. It will be placed into your /usr/bin folder:

1 2 3 4

[root@dm01db02 ~]# rpm -i ./ibdump-1.0.5-4-rpms/ibdump-1.0.5-4.x86_64-rhel`lsb_release -r|awk '{print $2}'`.rpm [root@dm01db02 ~]# ls -la /usr/bin/ibdump -rwxr-xr-x 1 root root 41336 Dec 19  2010 /usr/bin/ibdump [root@dm01db02 ~]#

Now you are ready to play with ibdump, running it without parameters will make ibdump sniffing interface mlx4_0 (which is ib0) and writes the frames into a file called sniffer.pcap in your working directory. Some parameters can be added such as the dump file location:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

[root@dm01db02 ~]# ibdump -o /tmp/ibdump.pcap  ------------------------------------------------  IB device                      : "mlx4_0"  IB port                        : 1  Dump file                      : /tmp/ibdump.pcap  Sniffer WQEs (max burst size)  : 4096  ------------------------------------------------   Initiating resources ... searching for IB devices in host Port active_mtu=2048 MR was registered with addr=0x1bc58590, lkey=0x8001c34e, rkey=0x8001c34e, flags=0x1 QP was created, QP number=0x60005b   Ready to capture (Press ^c to stop): Captured:     11711 packets, 10978982 bytes   Interrupted (signal 2) - exiting ...   [root@dm01db02 ~]#

There are some drawback to ibdump though:

• ibdump may encounter packet drops upon a burst of more than 4096 (or 2^max-burst) packets.
• Packets loss is not reported by ibdump.
• Outbound retransmitted and multicast packets may not be collected correctly.
• ibdump may stop capturing packets when run on the same port of the Subnet Manager (E.G.: opensm). It is advised not to run the SM and ibdump on the same port.

Be aware of the issues above, besides that: Have fun peeking around at your Exadata infiniband fabric!