Using the lsst-login Servers

The following login nodes are run by NCSA for access to select Rubin Observatory development resources at NCSA:

  • lsst-login01.ncsa.illinois.edu
  • lsst-login02.ncsa.illinois.edu
  • lsst-login03.ncsa.illinois.edu

To get an account, see the Onboarding Checklist.

This page is designed to assist developers in use of the lsst-login servers:

  1. Overview
  2. Connecting and Authenticating
  3. Development Work
  4. Select Appropriate Developer Tools
  5. Load the LSST Environment
  6. Validation/Test Data Sets
  7. Configure Git LFS
  8. Configure Remote Display with xpra

Overview

The lsst-login servers are primarily intended as bastions used to access other resources at NCSA. Additional capabilities include:

  • light Development Work with short-running processes that require modest resources (e.g., build docs, short compilations against LSST software stack)
  • view files (e.g., FITS files)

Users are encouraged to submit batch jobs to perform work that requires more significant resources. Please see Using the Rubin Batch Systems for more information.

The lsst-login nodes have access to the LDF file systems.

For system status and issues:

Connecting and Authenticating

You can log into Rubin Observatory development servers at NCSA with your NCSA account as follows:

  • NCSA username and password OR valid Kerberos ticket from workstation/laptop, AND
  • NCSA Duo authentication

You can reset your NCSA password at the following URL:

Information on setting up NCSA Duo is available at the following URL:

If you are using OpenSSH on your local machine and you wish to use Kerberos from your local machine (instead of entering your password on the login node), you could add something like this to your local ~/.ssh/config file:

GSSAPIAuthentication yes
PreferredAuthentications gssapi-with-mic,keyboard-interactive,password

The Kerberos domain for the lsst-login servers is NCSA.EDU, so something like this may work to generate a Kerberos ticket on your local machine:

kinit username@NCSA.EDU

# you may get an error like this: 'kinit: Cannot find KDC for realm "NCSA.EDU" while getting initial credentials';
# if that's the case, the Kerberos config on the local machine may need to be updated with 'dns_lookup_kdc = true'

You may wish to use an lsst-login node as a “jump host” (a gateway to an interior node). If using OpenSSH on your local machine you can do this as follows:

Host lsst-someinternalhost.ncsa.illinois.edu
   User ncsausername
   ProxyJump lsst-login01.ncsa.illinois.edu

When using an lsst-login node as a “jump host” you may also wish to configure port forwarding through the lsst-login node to the internal cluster node. To do that you would include something like this in your OpenSSH config file:

Host lsst-someinternalhost.ncsa.illinois.edu
   User ncsausername
   ProxyJump lsst-login01.ncsa.illinois.edu
   DynamicForward yourportnumber

You may also wish to reuse a single connection to/through an lsst-login node via an OpenSSH ControlMaster socket. This allows you to authenticate to the login node one time and reuse that initial connection to make additional connections without authenticating again. See for example OpenSSH Cookbook - Multiplexing.

A relatively complete ~/.ssh/config “recipe” for streamlining your SSH connections (assuming OpenSSH, e.g., on Linux or macOS) through the lsst-login nodes might look like this:

# Set common config for the lsst-login nodes
Host lsst-login*
   # if your account on your local workstation/laptop does not match your LSST username, indicate the latter should be used;
   # substitute your own NCSA username
   User ncsausername
   # allow use of a Kerberos ticket on your local machine for auth to LSST machines
   GSSAPIAuthentication yes
   # prefer Kerberos ticket auth, amongst other possibilities (order/include others as desired)
   PreferredAuthentications gssapi-with-mic,keyboard-interactive,password
   # forward your local Kerberos ticket to the login node if you need to continue to another LSST server after the login
   GSSAPIDelegateCredentials yes
   # configure OpenSSH Control Master "multiplexing" (to allow reuse of an initial connection)
   ControlMaster auto
   ControlPath ~/.ssh/cm_socket_%r@%h:%p
   ControlPersist 5m

# Define aliases onto full hostnames for each login node
Host lsst-login01
   HostName lsst-login01.ncsa.illinois.edu
Host lsst-login02
   HostName lsst-login02.ncsa.illinois.edu
Host lsst-login03
   HostName lsst-login03.ncsa.illinois.edu

# Define an alias and config for an internal node, which can only be reached through a login node
Host lsst-cnd-sub01
   HostName lsst-condordac-sub01.ncsa.illinois.edu
   # you may need to specify your NCSA username again
   User ncsausername
   # when connecting to this internal host, tunnel/jump through a login node (using an alias you defined above)
   ProxyJump lsst-login01
   # if you want to use your local Kerberos ticket to authenticate on the interior node, configure that:
   GSSAPIAuthentication yes
   PreferredAuthentications gssapi-with-mic
   # if the internal node is a batch submit node where you might want a Kerberos ticket (e.g., to
   # submit jobs to HTCondor), you can choose to forward your credentials:
   GSSAPIDelegateCredentials yes
   # if you need to configure port forwarding to the internal node, you can do that here;
   # substitute your actual port number
   DynamicForward yourportnumber

With such config in ~/.ssh/config on your local machine, your SSH connections can be significantly streamlined. Your experience may look like this:

  1. Your first connection attempt involves typing your password once, on your local machine, along with a Duo push for the login node. There’s no need to type your password on the login node or the internal node due to GSSAPI authentication. And your local Kerberos ticket is forwarded into your session on the internal node:
localuser@localmachine ~ % kinit ncsauser@NCSA.EDU
ncsauser@NCSA.EDU's password:
localuser@localmachine ~ % ssh lsst-cnd-sub01
Duo two-factor login for ncsauser

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-####

Passcode or option (1-1): 1
Last login: Fri Aug 14 15:06:35 2020 from 141.142.181.18
lsst-condordac-sub01.ncsa.illinois.edu (141.142.181.231)
  OS: CentOS 7.8.2003   HW: Dell   CPU: 24x 2.60GHz   RAM: 252 GB
  Site: ncsa  DC: npcf  Cluster: condor_dac  Role: condor_submit
[ncsauser@lsst-condordac-sub01 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_11111_OrKJ2p97xr
Default principal: ncsauser@NCSA.EDU

Valid starting       Expires              Service principal
08/14/2020 15:06:12  08/15/2020 01:05:59  krbtgt/NCSA.EDU@NCSA.EDU
[ncsauser@lsst-condordac-sub01 ~]$
  1. In a 2nd terminal window, you can connect again without any need to authenticate whatsoever (thanks to your ControlMaster config):
localuser@localmachine ~ % ssh lsst-cnd-sub01
Last login: Fri Aug 14 15:07:34 2020 from 141.142.181.18
lsst-condordac-sub01.ncsa.illinois.edu (141.142.181.231)
  OS: CentOS 7.8.2003   HW: Dell   CPU: 24x 2.60GHz   RAM: 252 GB
  Site: ncsa  DC: npcf  Cluster: condor_dac  Role: condor_submit
[ncsauser@lsst-condordac-sub01 ~]$
  1. Your control master master connection will persist in the background after your initial client connection terminates, according to the value of ControlPersist. To terminate your control master connection immediately, do the following on your local machine:
localuser@localmachine ~ % ssh -O exit lsst-login03
Exit request sent.
localuser@localmachine ~ %

NOTE: This will break all connections in any terminals that depend on this master connection, e.g.:

[ncsauser@lsst-condordac-sub01 ~]$ client_loop: send disconnect: Broken pipe
localuser@localmachine ~ %

Tip

More tips on working Kerberos tickets and OpenSSH ControlMaster

  • Your Kerberos ticket on your local machine will need to be renewed occasionally, which you can do with kinit -R.
  • Renewing the ticket on your local machine will not generally renew any tickets you have forwarded to remote machines. (NOTE: OpenSSH has a GSSAPIRenewalForcesRekey option that will cascade your ticket renewal out wherever you have forwarded them, however it is not implemented on all platforms, e.g. macOS.)
  • The example above shows you can request a ticket with a maximum lifetime (25 hours) and maximum renewable life time (7 days), again, kinit -l 25h -r 7d ....
  • If your local ticket expires before you renew it you will have to kinit (and authenticate with your password) to create a new ticket.

Development Work

lsst-login nodes can be used for (light) development work. Users are encouraged to utilize batch compute nodes when more significant resources are required. Please see Using the Rubin Batch Systems for more information.

Select Appropriate Developer Tools

Note

Although the material presented below remains valid, the shared stack from May 2020 onwards (/software/lsstsw/stack_20200504) provides the complete toolchain required for Science Pipelines development. It is no longer necessary to load a software collection to work with the shared stack.

The lsst-login systems are configured with the latest CentOS 7.x as its operating system. This release of CentOS provides an old set of development tools, centered around version 4.8.5 of the GNU Compiler Collection (GCC). Updated toolchains are made available through the “Software Collection” system. The following Software Collections are currently available:

Name Description
devtoolset-6 Updated compiler toolchain providing GCC 6.3.1.
devtoolset-7 Updated compiler toolchain providing GCC 7.3.1.
devtoolset-8 Updated compiler toolchain providing GCC 8.3.1.

To enable a particular Software Collection use the scl command. For example:

scl enable devtoolset-8 bash
gcc --version
gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Warning

Code compiled by different versions of GCC may not be compatible: it is generally better to stick to a particular toolchain for a given project. In particular, if you are using a shared stack you must use the matching toolchain.

You may wish to automatically enable a particular software collection every time you log in to lsst-login01 and other Rubin Observatory development systems at NCSA. Take care if you do this: it’s easy to accidentally to either start recursively spawning shells and run out of resources or lock yourself out of machines which don’t have the particular collection you’re interested in installed. If you are using Bash — the default shell on lsst-dev servers — try placing the following at the end of ~/.bash_profile and customising the list of desired_scls.

# User-specified space-delimited list of SCLs to enable.
desired_scls="devtoolset-8"

# Only do anything if /usr/bin/scl is executable.
if [ -x /usr/bin/scl ]; then

    # Select the union of the user's desired SCLs with those which are both
    # available and not currently enabled.
    avail_scls=$(scl --list)
    for scl in $desired_scls; do
        if [[ $avail_scls =~ $scl && ! $X_SCLS =~ $scl ]]; then
            scls[${#scls[@]}]=$scl
        fi
    done

    # Use `tty -s` to output messages only if connected to a terminal;
    # avoids causing problems for non-interactive sessions.
    if [ ${#scls[@]} != 0 ]; then
        tty -s && echo "Enabling ${scls[@]}."
        exec scl enable ${scls[@]} bash
    else
        tty -s && echo "No software collections to enable."
    fi
fi

Load the LSST Environment

We provide a ready-to-use “shared” version of the LSST software stack to enable developers to get up and running quickly with no installation step. The shared stack includes a fully-fledged Miniconda-based Python environment, a selection of additional development tools, and a selection of builds of the lsst_distrib meta-package. It is located on GPFS-based network storage; as such, it is cross-mounted across a variety of Rubin Observatory development systems at the Data Facility including those configured as part of the HTCondor pool and Verification Cluster. The currently stack is regularly updated to include the latest weekly release, which is tagged as current.

The following stacks are currently being updated:

Path Toolchain Description
/software/lsstsw/stack_20200515 Internal (Conda)

Provides weekly w_2020_19 and later of lsst_distrib and w_2020_20 and later of lsst_sims. Based on scipipe_conda_env 46b24e8 with the following additional packages installed:

  • bokeh
  • cx_Oracle
  • dask-jobqueue
  • datashaderpyct
  • fastparquet
  • holoviews
  • hvplot
  • ipdb
  • jupyter
  • numba
  • panel
  • pep8
  • psycopg2
  • pyflakes
  • pyviz_comms

Note

When using a shared stack, you must use the corresponding developer toolchain. If this is listed in the table above as “Internal (Conda)” then no further action on your part is required; otherwise, see above for details of how to Select Appropriate Developer Tools.

In addition, the following symbolic links point to particular versions of the stack:

Path Description
/software/lsstsw/stack The latest version of the stack.

Add a shared stack to your environment and set up the latest build of the LSST applications by running, for example:

source /software/lsstsw/stack/loadLSST.bash
setup lsst_apps

(substitute loadLSST.csh, loadLSST.ksh or loadLSST.zsh, depending on your preferred shell).

Tip

Initializing the stack will prepend the string (lsst-scipipe) to your prompt. If you wish, you can disable this by running

conda config --set changeps1 false

Although the latest weeklies of LSST software are regularly installed into the shared stacks, the rest of their contents is held fixed (to avoid API or ABI incompatibilities with old stack builds). We therefore periodically retire old stacks and replace them with new ones. The following retired stacks are currently available:

Path Toolchain Description
/software/lsstsw/stack_20171023 devtoolset-6 Provides a selection of weekly and release builds dating between October 2017 and October 2018.
/software/lsstsw/stack_20181012 devtoolset-6 Provides weeklies w_2018_41 through w_2019_12; release candidates v17_0_rc1, v17_0_rc2, and v17_0_1_rc1; and releases v_17_0 and v_17_0_1. Based on the pre-RFC-584 Conda environment.
/software/lsstsw/stack_20190330 devtoolset-6 Provides weekly w_2019_12 through w_2019_38 and daily d_2019_09_30. Based on the post-RFC-584 Conda environment.
/software/lsstsw/stack_20191001 devtoolset-8 Provides weeklies w_2019_38 through w_2019_42.
/software/lsstsw/stack_20191101 devtoolset-8 Provides weekly w_2019_43 through w_2020_09 of lsst_distrib, and w_2019_43 through w_2020_07 of lsst_sims. Based on scipipe_conda_env 4d7b902 (RFC-641).
/software/lsstsw/stack_20200220 devtoolset-8 Provides weekly w_2020_07 through w_2020_17 of lsst_distrib, and weekly w_2020_10 through w_2020_16 of lsst_sims. Based on scipipe_conda_env 984c9f7 (RFC-664).
/software/lsstsw/stack_20200504 Internal (Conda) Provides weeklies w_2020_18 and w_2020_19 of lsst_distrib. Based on scipipe_conda_env 2deae7a (RFC-679).

Administrators may wish to note that the shared stack is automatically updated using the script ~lsstsw/shared-stack/shared_stack.py, which is executed nightly by Cron.

Validation/Test Data Sets

There are two cron jobs that will update a set of validation data repositories and test data repositories. These updates will trigger overnight on the lsst-dev system. In most cases, this will be a fairly straightforward git pull, but if corruption is detected, the repository will be cloned afresh. The verification data are currently being used primarily by validate_drp to measure various metrics on the reduced data. The test data serve a variety of purposes, but generally are included via a setupOptional in a package table file.

Test data location is: /project/shared/data/test_data

Included test data repositories are:

testdata_jointcal
testdata_cfht
testdata_subaru
testdata_decam
testdata_lsst
ap_verify_testdata
ap_pipe_testdata
ci_hsc
afwdata

Validation data location is: /project/shared/data/validation_data

Included validation data repositories are:

validation_data_hsc
validation_data_decam
validation_data_cfht

These are maintained by the lsstsw user (this is the same user that curates the shared stack on the lsst-dev system). Please ask in the #dm-infrastructure Slack channel in case of problems.

Configure Git LFS

Note

Although the material presented below remains valid, the shared stack from May 2020 onwards (/software/lsstsw/stack_20200504) provides Git LFS as part of the environment: it is no longer necessary to explicitly run setup, as described below (although it is still necessary to follow DM’s Git LFS guide. The setup step is only necessary for older shared stacks — those marked with toolchain: devtoolset-8 (or -6) in the table above.

For newer shared stacks (toolchain: Internal (Conda)), they are not relevant.

After you have initialized a shared stack, you can enable Git LFS using EUPS:

setup git_lfs

The first time you use Git LFS you’ll need to configure it by following these steps from DM’s Git LFS guide:

  1. Basic configuration
  2. Configuring Git LFS

Configure Remote Display with xpra

xpra can be thought of as “screen for X” and offers advantages over VNC. It can be very handy and efficient for remote display to your machine from Rubin Observatory development compute nodes (e.g., debugging with ds9) because it is much faster than a regular X connection when you don’t have a lot of bandwidth (e.g., working remotely), and it saves state between connections. Here’s how to use it:

On lsst-login01:

xpra start :10
export DISPLAY=:10

You may have to choose a different display number (>10) if :10 is already in use.

On your local machine, do:

xpra attach ssh:lsst-login01.ncsa.illinois.edu:10

## IF YOU EXPERIENCE AUTHENTICATION ISSUES, TRY THE FOLLOWING INSTEAD TO SPECIFY AUTH METHODS OF SSH
xpra attach --ssh="ssh -vvv -o='PreferredAuthentications=gssapi-with-mic,keyboard-interactive,password'" ssh:lsst-login01.ncsa.illinois.edu:10

You may leave that running, or put it in the background and later use:

xpra detach

Then you can open windows on lsst-login01 (with DISPLAY=:10) and they will appear on your machine. If you now kill the xpra attach on your machine, you’ll lose those windows. When you reattach, they’ll reappear.

Note

xpra requires the use of Python 2.

If you are using a Python 3 LSST Stack, you’ll encounter a error like the following:

[...]
File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pyyaml/3.11.lsst2/lib/python/yaml/__init__.py", line 284
  class YAMLObject(metaclass=YAMLObjectMetaclass):
                            ^
SyntaxError: invalid syntax

The solution in this case is to start xpra in a separate shell where you haven’t yet setup the Python 3 LSST Stack.

Note

If you run into issues getting xpra to authenticate when you attempt to attach, you may find that including explicit authentication options helps:

xpra attach -ssh="ssh -o='PreferredAuthentications=gssapi-with-mic,keyboard-interactive,password'" ssh:lsst-login01.ncsa.illinois.edu:100

Note

It is possible to use xpra through a tunneled connection to an “interior” node that also has xpra, e.g., when using a login nodes as a “jump host” to reach a submit node, as described above, you may wish to use xpra on the submit node.

First, make your tunneled connection to the destination host (as detailed above).

Then attach xpra to the submit host by also telling xpra to jump/tunnel through the login node:

xpra attach ssh:lsst-condorprod-sub01.ncsa.illinois.edu:10 --ssh="ssh -J lsst-login01.ncsa.illinois.edu"