The LSST Software Build Tool

Introduction

The lsst-build tool recursively clones, builds, and installs the LSST Stack. While the motivation for its creation was to serve as a backend for the continuous integration system, anyone may download and run it to do stack-wide integration builds (including building with alternative versions of external packages, etc.). The core tool lives in the lsst_build repository.

Howevever the lsstsw tool provides a more convenient way to set up a personal build environment. It is available from the lsstsw repository:

git clone https://github.com/lsst/lsstsw.git

The lsstsw package is primarily used to manage the CI system, but it is self contained and can be used anywhere by anyone.

After the clone, bootstrap the environment as follows:

cd lsstsw
./bin/deploy

This will:

  • Download and install Miniconda in ./miniconda;
  • Download and install Git in ./lfs;
  • Download and install EUPS in ./eups;
  • Make an empty stack directory in ./stack, with a default manifest.remap;
  • Run git clone lsst_build master
  • Run git clone versiondb master

Once the bootstrap has completed, set up the environment as directed. For example:

export LSSTSW=<where you've set it up>
export EUPS_PATH=$LSSTSW/stack
. $LSSTSW/bin/setup.sh

setenv LSSTSW <where you've set it up>
setenv EUPS_PATH $LSSTSW/stack
source $LSSTSW/bin/setup.csh

Note

It is only necessary to run deploy once, although it is clever enough to turn into a no-op when it detects it has successfully run already.

Recursively Cloning & (Re)building a Package

Assuming you’ve done all of the above, simply run:

rebuild lsst_apps

lsst_apps is an empty package that depends on all of the LSST packages. You can use the rebuild command to build other packages by name as well.

Once you have built a package you may want to clone the new EUPS tag to current, so you can setup the package without specifying a particular tag. For example if lsstsw just built a package using EUPS tag b6132 you clone that to current using:

eups tags --clone b6132 current

The rebuild command is is a wrapper around the lower-level lsst-build tool (described below). It will:

  • Search for and clone the package from our Git repositories (as configured in etc/repos.yaml) into $LSSTSW/build;
  • Recursively clone all of its dependencies (also into $LSSTSW/build);
  • Recursively build all its dependencies bottom-up, installing the built packages into $LSSTSW/stack, using an eups tag of the form bNNN (where N is a digit), e.g. b6132;
  • Build the package as well, and install it into $LSSTSW/stack.

Note

Rebuilding afwdata may take awhile, since it must download several GB of data. If you already have copy that is checked out from the same URL as in the etc/repos.yaml file, you can use it by making a symlink to $LSSTSW/build/afwdata. In fact it is very useful to store afwdata outside the lsstsw directory and symlink to it, as it allows you to delete your copy of lsstsw at any time to start fresh, without downloading afwdata again. To do this, run bin/deploy on your fresh copy of lsstsw, then make the symlink before running rebuild. If you don’t want a copy of afwdata at all then you can add it to the etc/exclusions.txt file.

Customizations are possible by editing the etc/settings.cfg.sh file, or by running lsst-build manually. See the documentation in bin/rebuild to see how.

To rebuild the entire stack, pick one of the top-level packages (e.g., lsst_distrib or lsst_apps).

Building Branches

You can build specific branch(es) by running:

rebuild -r branch1 -r branch2 -r ... <packagename>

Before building, the code above will attempt to checkout branch1 (both in the package and its dependencies), and fall back to branch2 if it doesn’t exist, and then fall back to master (or another default branch configured in etc/repos.yaml).

Other command-line options for rebuild

-p will clone the required packages and then stop, without building anything.

-u will bring over a current copy of etc/repos.yaml before starting the build. This can be handy if repositories have moved or been added and is used by our continuous integration system.

-t <tag> is deprecated. Use this instead: eups tags --clone=oldtag newtag.

Low(er)-level tool: lsst-build

Here is an example of how to run lsst-build:

lsst-build prepare \
  --exclusion-map=exclusions.txt \
  --version-git-repo=versiondb \
  ./build lsst_distrib
lsst-build build ./build

Note

For full details of the lsst-build setup procedure, see the README file included in the package.

The lsst-build prepare command will begin by cloning the lsst_distrib product into ./build/lsst_distrib, it will read its dependencies from the table file, and then recursively repeat the process with each one of them until all leafs of the dependency graph are reached. If you just want to clone all packages needed to build a certain package from Git, this is the tool. More than one top-level package can be prepared at the same time (e.g., run it with ... lsst_distrib git anaconda).

In addition to the mass clone, running lsst-build prepare will also create a “build manifest” file in build/manifest.txt. This is a topologically sorted list of all cloned products and the versions that were computed for them. The versions are of the form <tag>[+<N>] (if an annotated tag exists on a commit), or <branch>-g<sha>[+<N>] if there’s no tag. The way the code tracks which +N number to use is through the versiondb database (which is just a specially formatted git repository; again see the README for details).

The second command then takes the cloned repositories and the information in manifest.txt and builds the products, installs them into the stack pointed to by EUPS_PATH, and tags them with a “build ID” (a unique ID computed for each manifest.txt, and listed in the manifest.txt itself as BUILD=bNNN). Therefore, running the two commands will build and install a complete, functioning stack for you. The log of build output for each package is in _build.log in its directory (e.g., in ./build/afw), as well as in the directory where it’s installed (if the build is successful).

Importantly, lsst-build prepare can take one or more --ref <branch_or_tag> arguments. So, you can say:

lsst-build prepare \
  ... \
  --ref tickets/1234 --ref next --ref master \
  build lsst_distrib

and, upon cloning each repository, it will attempt to checkout tickets/1234, falling back to next if it doesn’t exist, and finally to master. This is how we test whether the changes on a branch break the stack.

Implementing that was the easy part. The hard part was making these tools efficient, while being robust (and there is still room for improvement). As an example, on subsequent times you run lsst-build prepare (possibly with different arguments), it will avoid cloning the repositories it already has (and the hard-hard part was making this robust so it works even in presence of forced pushes, dirty directories, removed or changed tags, changed remote URLs, and all sorts of evil nastiness that we shouldn’t have but almost certainly will). Also, lsst-build prepare is guaranteed to produce the same version for the same source code + dependencies. That enables lsst-build build to check if the product with that version already exists in the stack, before building it. Therefore, lsst-build build will only build the packages that need to be built (either because they or their depencencies have changed), and can skip the already built ones.

Warning

The timings cited below are old and likely unrepresentative of a modern (2016) stack.

Using lsst-build, it is possible to rebuild the complete stack (everything up to lsst_distrib) in ~25 minutes in lsst-dev:/ssd. If something above afw has changed, the build time drops to ~10-ish minutes.

Warning

The material below is old and may be outdated; refer to the Continuous Integration Services documentation for the current story.

This machinery is now also installed into ~lsstsw, and Buildbot will use it from there. Buildbot will ultimately manage both the lsst-dev stack and the distribution server. The old tools (e.g., submitRelease, ...) are gone. The old stack (the one in /lsst/DC3/....) will be gone as well. The new (automated) workflow is as follows:

  1. The new lsst-dev stack is in ~lsstsw/stack. Set your EUPS_PATH to point to it.
  2. lsst-build right now periodically runs from cron and builds the master branch any time it changes. The results end up in ~lsstsw/stack. each build is EUPS-tagged with a unique build number (e.g., b1, b2, b3, ...). The latest build gets EUPS tagged as current. There’s no more need to run submitRelease, since everything is available.
  3. When we want to release the stack, someone with ~lsstsw access will log into ~lsstsw and runs the standard eups distrib create, possibly EUPS-tagging it as something more memorable than bNNN (e.g., Winter2014). If it’s useful, we could also automatically release the bNNN builds. Right now there is a set of product with b1 EUPS tag there. These are a build of master as of yesterday, which I Git-tagged as 8.0.0.0. Consider this a release candidate for Winter‘14, and take a look. I’ll proceed to build an EUPS distribution as well soon.