The LSST Software Build Tool¶
Introduction¶
The lsst-build tool recursively clones, builds, and installs the LSST Stack. While the motivation for its creation was to serve as a backend for the continuous integration system, anyone may download and run it to do stack-wide integration builds (including building with alternative versions of external packages, etc.). The core tool lives in the lsst_build repository.
However the lsstsw tool provides a more convenient way to set up a personal build environment. It is available from the lsstsw repository:
git clone https://github.com/lsst/lsstsw.git
The lsstsw package is primarily used to manage the CI system, but it is self contained and can be used anywhere by anyone.
After the clone, bootstrap the environment as follows:
cd lsstsw
./bin/deploy
This will:
- Download and install Miniconda in
./miniconda
; - Download and install Git in
./lfs
; - Download and install EUPS in
./eups
; - Make an empty stack directory in
./stack
, with a defaultmanifest.remap
; - Download and install a copy of etc/repos.yaml from lsst/repos;
- Run git clone lsst_build master
- Run git clone versiondb master
Once the bootstrap has completed, set up the environment as directed. For example:
export LSSTSW=<where you've set it up>
export EUPS_PATH=$LSSTSW/stack
. $LSSTSW/bin/setup.sh
setenv LSSTSW <where you've set it up>
setenv EUPS_PATH $LSSTSW/stack
source $LSSTSW/bin/setup.csh
Note
It is only necessary to run deploy once, although it is clever enough to turn into a no-op when it detects it has successfully run already.
Recursively Cloning & (Re)building a Package¶
Assuming you’ve done all of the above, simply run:
rebuild lsst_apps
lsst_apps
is an empty package that depends on all of the LSST packages.
You can use the rebuild command to build other packages by name as
well.
Once you have built a package you may want to clone the new EUPS tag to
current
, so you can setup the package without specifying a particular tag.
For example if lsstsw just built a package using EUPS tag b6132
you clone that to current using:
eups tags --clone b6132 current
The rebuild command is is a wrapper around the lower-level lsst-build tool (described below). It will:
- Search for and clone the package from our Git repositories (as configured in
etc/repos.yaml
) into$LSSTSW/build
; - Recursively clone all of its dependencies (also into
$LSSTSW/build
); - Recursively build all its dependencies bottom-up, installing the built
packages into
$LSSTSW/stack
, using an eups tag of the formbNNN
(whereN
is a digit), e.g.b6132
; - Build the package as well, and install it into
$LSSTSW/stack
.
Note
Rebuilding afwdata
may take awhile, since it must download several GB
of data. If you already have copy that is checked out from the same URL as
in the etc/repos.yaml
file, you can use it by making a symlink to
$LSSTSW/build/afwdata
. In fact it is very useful to store
afwdata
outside the lsstsw
directory and symlink to it, as it
allows you to delete your copy of lsstsw
at any time to start fresh,
without downloading afwdata
again. To do this, run bin/deploy
on your fresh copy of lsstsw
, then make the symlink before running
rebuild
. If you don’t want a copy of afwdata
at all
then you can add it to the etc/exclusions.txt
file.
Customizations are possible by editing the etc/settings.cfg.sh
file,
or by running lsst-build manually. See the documentation in
bin/rebuild to see how.
To rebuild the entire stack, pick one of the top-level packages (e.g.,
lsst_distrib
or lsst_apps
).
Building Branches¶
You can build specific branch(es) by running:
rebuild -r branch1 -r branch2 -r ... <packagename>
Before building, the code above will attempt to checkout branch1
(both in the
package and its dependencies), and fall back to branch2
if it doesn’t exist,
and then fall back to master (or another default branch configured in
etc/repos.yaml
).
Other command-line options for rebuild¶
-p
will clone the required packages and then stop, without building
anything.
-u
will bring over a current copy of etc/repos.yaml
before
starting the build. This can be handy if repositories have moved or been
added and is used by our continuous integration system.
-t <tag>
is deprecated. Use this instead: eups tags
--clone=oldtag newtag.
Low(er)-level tool: lsst-build¶
Here is an example of how to run lsst-build:
lsst-build prepare \
--exclusion-map=exclusions.txt \
--version-git-repo=versiondb \
./build lsst_distrib
lsst-build build ./build
Note
For full details of the lsst-build setup procedure, see the
README
file included in the package.
The lsst-build prepare command will begin by cloning the
lsst_distrib
product into ./build/lsst_distrib
, it will read its
dependencies from the table file, and then recursively repeat the process with
each one of them until all leafs of the dependency graph are reached. If you
just want to clone all packages needed to build a certain package from Git,
this is the tool. More than one top-level package can be prepared at the same
time (e.g., run it with ... lsst_distrib git anaconda).
In addition to the mass clone, running lsst-build prepare will also
create a “build manifest” file in build/manifest.txt
. This is a
topologically sorted list of all cloned products and the versions that were
computed for them. The versions are of the form <tag>[+<N>]
(if an
annotated tag exists on a commit), or <branch>-g<sha>[+<N>]
if there’s no
tag. The way the code tracks which +N
number to use is through the
versiondb
database (which is just a specially formatted git repository;
again see the README
for details).
The second command then takes the cloned repositories and the information in
manifest.txt
and builds the products, installs them into the stack
pointed to by EUPS_PATH
, and tags them with a “build ID” (a unique
ID computed for each manifest.txt
, and listed in the
manifest.txt
itself as BUILD=bNNN
). Therefore, running the two
commands will build and install a complete, functioning stack for you. The log
of build output for each package is in _build.log
in its directory
(e.g., in ./build/afw
), as well as in the directory where it’s
installed (if the build is successful).
Importantly, lsst-build prepare can take one or more --ref
<branch_or_tag>
arguments. So, you can say:
lsst-build prepare \
... \
--ref tickets/1234 --ref next --ref master \
build lsst_distrib
and, upon cloning each repository, it will attempt to checkout
tickets/1234
, falling back to next
if it doesn’t exist, and finally to
master
. This is how we test whether the changes on a branch break the
stack.
Implementing that was the easy part. The hard part was making these tools efficient, while being robust (and there is still room for improvement). As an example, on subsequent times you run lsst-build prepare (possibly with different arguments), it will avoid cloning the repositories it already has (and the hard-hard part was making this robust so it works even in presence of forced pushes, dirty directories, removed or changed tags, changed remote URLs, and all sorts of evil nastiness that we shouldn’t have but almost certainly will). Also, lsst-build prepare is guaranteed to produce the same version for the same source code + dependencies. That enables lsst-build build to check if the product with that version already exists in the stack, before building it. Therefore, lsst-build build will only build the packages that need to be built (either because they or their depencencies have changed), and can skip the already built ones.
Warning
The timings cited below are old and likely unrepresentative of a modern (2016) stack.
Using lsst-build, it is possible to rebuild the complete stack
(everything up to lsst_distrib
) in ~25 minutes in lsst-dev:/ssd
. If
something above afw
has changed, the build time drops to ~10-ish minutes.
Warning
The material below is old and may be outdated; refer to the Jenkins documentation for the current story.
This machinery is now also installed into ~lsstsw
, and CI will
use it from there. CI will ultimately manage both the lsst-dev
stack
and the distribution server. The old tools (e.g., submitRelease,
...) are gone. The old stack (the one in /lsst/DC3/....
) will be gone as well.
The new (automated) workflow is as follows:
- The new
lsst-dev
stack is in~lsstsw/stack
. Set yourEUPS_PATH
to point to it. - lsst-build right now periodically runs from cron and
builds the
master
branch any time it changes. The results end up in~lsstsw/stack
. each build is EUPS-tagged with a unique build number (e.g.,b1
,b2
,b3
, ...). The latest build gets EUPS tagged ascurrent
. There’s no more need to runsubmitRelease
, since everything is available. - When we want to release the stack, someone with
~lsstsw
access will log into~lsstsw
and runs the standard eups distrib create, possibly EUPS-tagging it as something more memorable thanbNNN
(e.g.,Winter2014
). If it’s useful, we could also automatically release thebNNN
builds. Right now there is a set of product withb1
EUPS tag there. These are a build of master as of yesterday, which I Git-tagged as8.0.0.0
. Consider this a release candidate for Winter‘14, and take a look. I’ll proceed to build an EUPS distribution as well soon.
Updating your EUPS version¶
In order to install a new version of EUPS, first check your current installed version with the ‘-V’ flag:
% eups -V
Next, define the relevant environment variable as so: export EUPS_VERSION=x.y.z (where x.y.z is the version you would like to install). Then install the new EUPS version by changing to your ${LSSTSW} directory and doing:
% ./bin/deploy
there. This will install that version of EUPS and set your default version to it, henceforth.
Older versions will still be available under ${LSSTSW}/eups, and you can switch back to those by simply setting again export EUPS_VERSION=a.b.c, opening a new terminal window, and then executing source $LSSTSW/bin/setup.sh in that window to make a.b.c the default version. Opening any new terminal window from here will keep using this version, also.
The simplest place to find all available versions of EUPS is by looking at this page on github.