BBCP: the multistreaming transfer utility

For moving larger files, the multistreaming transfer utility BBCP is recommended. The BBCP utility is capable of breaking up your transfer into multiple simultaneously transferring streams, thereby transferring data much faster than single-streaming utilities such as scp and sftp.

Before you can use the BBCP utility, it must be installed on both the local and remote systems. It is currently available on each OLCF system and should be a part of each user’s default environment. Several pre-compiled binaries as well as the source can be downloaded from the Stanford Linear Accelerator Center (SLAC) BBCP page.

Installation from Source Tips

• Download the BBCP from the Stanford Linear Accelerator Center (SLAC) BBCP page.
• Decompress the downloaded file into a directory on the local system.

$ gunzip bbcp.tar.Z
$ tar -xvf bbcp.tar

• From within the decompressed BBCP directory, run make. This should build the BBCP executable into the created bin directory. The build has been tested on Linux-based systems and should build with few or no modifications. If you system’s uname command does not return Linux, you may need to modify the Makefile.

$ cd bbcp
$ uname
Linux
$ gmake

Common variable modifications

• In MakeSname, the test command is hard coded to /usr/bin/test. If this is not the location of test on your system, you can change the following line to the correct path (which test should return the path to test on your system):

if /usr/bin/test -${1} $2; then

• If the uname command is not in /bin on your system, change the uname variable in the MakeSname file. You will also need to change the following line in the file Makefile:

@cd src;$(MAKE) make`/bin/uname` OSVER=`../MakeSname`

• If the libz.a library is not located at /usr/local/lib/libz.a on your system, change the libzMakefile file.
• The file Makefile contains compiler and compiler flag options for the BBCP build. You can change the compilers and flags by modifying variables in this file. For example, to change the compilers used on a Linux system, modify the variables LNXCC and LNXcc.

Usage

To transfer the local file /local/path/largefile.tar to the remote system remotesystem as/remote/path/largefile.tar, use the following:

bbcp -P 2 -V -w 8m -s 16 /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

where

-P 2 produces progress messages every 2 seconds.
-V produces verbose output, including detailed transfer-speed statistics.
-w 8m sets the size of the disk input/output (I/O) buffers.
-s 16 sets the number of parallel network streams to 16.

BBCP assumes the remote system’s non-interactive environment contains the path to the BBCP utility. This can be determined with the following command:

ssh remotesystem which bbcp

If this is not the case, the -T BBCP option can be used to specify how to start BBCP on the remote system. For example, you could use the following:

bbcp -P 2 -V -w 8m -s 16 -T 'ssh -x -a -oFallBackToRsh=no %I -l %U %H /remote/path/to/bbcp' /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

Often, during large transfers the connection between the transferring systems is lost. The -a option gives BBCP the ability to pick up where it left off. For example, you could use the following:

bbcp -k -a /remotesystem/homedir/.bbcp/ -P 2 -V -w 8m -s 16 /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

To transfer an entire directory tree, use the following:

bbcp -r -P 2 -V -w 8m -s 16 /local/path/* remotesystem:/remote/path

We strongly recommend that you use the Data Transfer Nodes when transferring files to and from the OLCF. If you are, however, connecting directly to systems such as the Cray XK, it is necessary to specify a particular node as the destination host because the host name (i.e. titan.ccs.ornl.gov) actually points to a server load-balancing device that returns node addresses in a round-robin fashion. For example, you could use the following:

bbcp -r -P 2 -V -w 8m -s 16 /local/path/* titan-login3.ccs.ornl.gov:/remote/path

You may encounter an error similar to the following:

bbcp: Accept timed out on port 5031
bbcp: Unable to allocate more than 0 of 8 data streams.
Killed by signal 15.

If this happens, add the -z option to your bbcp command. This tells bbcp to use the “reverse connection protocol” and can be helpful when a transfer is being blocked by a firewall.

Further Reading

More information on BBCP can be found by typing “bbcp -h” on OLCF systems as well as on the Stanford Linear Accelerator Center (SLAC) BBCP page.