Using the Bulk Transfer Server at NCSA¶

NCSA has setup a dedicated bulk transfer server, lsst-xfer.ncsa.illinois.edu, for handling large data moves in and out of LSST’s storage at NCSA. The following bulk data transfer clients are configured on lsst-xfer.ncsa.illinois.edu:

Globus Online
iRODS
bbcp

Globus Online¶

If you don’t already have a Globus Online account, you’ll need to create one at https://www.globusonline.org/SignIn#step=SignUp.

Authentication is handled by a certificate created locally on the lsst-xfer system using the local unix account. You will need to email lsst-admin at ncsa.illinois.edu and request them to manually set a password for your unix account on lsst-xfer.ncsa.illinois.edu. Once the password is temporarily set, you can change the password by logging into lsst-xfer.ncsa.illinois.edu with your SSH public key and using the passwd command.

The globus connect endpoint has been created as a public endpoint named: lsst#lsst-xfer.

If you need to use Globus Online from a local computer that is not already configured with the Globus Online service, you can quickly and easily add your computer as a local Globus Connect Endpoint. Go to https://www.globusonline.org/xfer/ManageEndpoints and select add Globus Connect for instructions and software downloads.

iRODS Client¶

The iRODS client can be used to transfer data from iRODS servers such as those at IN2P3. Documentation for using the iRODS client can be found at https://www.irods.org/index.php/icommands.

To connect to the IN2P3 storage, set the following in ~/.irodsEnv:

irodsHost 'ccirods.in2p3.fr'
irodsPort 5579
irodsUserName 'lsstread'
irodsHome '/lsst-fr/data'
irodsZone 'lsst-fr'

bbcp Client¶

BBCP is an alternative to Gridftp when transferring large amounts of data, capable of breaking up your transfer into multiple simultaneous transferring streams, thereby transferring data much faster than single-streaming utilities such as SCP and SFTP. See: