Using the Bulk Transfer Server at NCSA¶
NCSA has setup a dedicated bulk transfer server,
lsst-xfer.ncsa.illinois.edu, for handling large data moves in and out of LSST’s storage at NCSA.
The following bulk data transfer clients are configured on
If you don’t already have a Globus Online account, you’ll need to create one at https://www.globusonline.org/SignIn#step=SignUp.
Authentication is handled by a certificate created locally on the
lsst-xfer system using your NCSA account.
The globus connect endpoint has been created as a public endpoint named:
If you need to use Globus Online from a local computer that is not already configured with the Globus Online service, you can quickly and easily add your computer as a local Globus Connect Endpoint. Go to https://www.globusonline.org/xfer/ManageEndpoints and select add Globus Connect for instructions and software downloads.
The iRODS client can be used to transfer data from iRODS servers such as those at IN2P3. Documentation for using the iRODS client can be found at https://www.irods.org/index.php/icommands.
To connect to the IN2P3 storage, set the following in
irodsHost 'ccirods.in2p3.fr' irodsPort 5579 irodsUserName 'lsstread' irodsHome '/lsst-fr/data' irodsZone 'lsst-fr'
BBCP is an alternative to Gridftp when transferring large amounts of data, capable of breaking up your transfer into multiple simultaneous transferring streams, thereby transferring data much faster than single-streaming utilities such as SCP and SFTP.
lsst-xfer.ncsa.illinois.edu is setup to use TCP ports 65001-65535 for BBCP transfers with the following external networks:
- IN2P3: 184.108.40.206/16
- LSST Commissioning Cluster: 220.127.116.11/24
- NCSA: 18.104.22.168/16
- NOAO Tucson: 22.214.171.124/16
- SLAC: 126.96.36.199/16
- UC Davis: 188.8.131.52/16
- UW: 184.108.40.206/16 or 220.127.116.11/19
Additional subnets can be requested by filing a JIRA ticket in the IHS project.
Since SSH access into
lsst-xfer.ncsa.illinois.edu is limited to from lsst-login nodes, transfers may need to be initiated on
lsst-xfer rather than remote hosts.