Buildbot Documentation¶

Web Views¶

Waterfall View¶

The waterfall view is a time-oriented display of multiple Builders’ progress (aka: the columns) through each Builder’s specific build steps.

Specific Builder View¶

The view of a specific Builder provides status on past, present and future build requests and user input fields allowing the user to:

• Force Build - Initiate a build using the information supplied by the user in fields: “Your full email:” and, optionally, “Branches (blank separated)” followed by the ‘Force Build’ button. The ‘everyman’ account may use this entry.
• Rebuild - Reruns the last build using the identical setup including the same user input. The everyman account may use this entry.
• Stop This Build - Terminates the currently running build. This is restricted to Buildbot admins.
• Cancel Pending Build - remove a prospective build from the queue. This is restricted to Buildbot admins.

1. Access the Buildbot Web interface¶

Browse to: http://lsst-buildx.ncsa.illinois.edu:8010/waterfall . At this point, without being logged in, you may view the current and historic build status for the Builders provided. The Builders are the columns indicating a state: idle, building, offline.

Experiment with the other view options provided on the banner line; mostly you’ll use either the One-Line-Per-Build view or the Waterfall view, the user input page for the DM_stack builder, and the status page for a specific Build which is identified by its Build number.

The Waterfall view does not auto-refresh. To add a 60 second refresh, append: ”?reload=60” to its web address. E.g. http://lsst-buildx.ncsa.illinois.edu:8010/waterfall?reload=60 . Note that the view of the Build log (identified as stdio) does auto-refresh without intervention.

2. Setup A Build¶

Select the DM_stack hot link to transfer to the builder status page where you enter your build request. This page also links to the current and past builds for this Builder.

LSST DM Developer Guide > Triggering a Buildbot Build > .. image:: /_static/buildbot/To_Force.jpg

A. Enter your Email Address¶

Buildbot will send you direct email regarding your build’s completion status. This field is required and must be formatted: “name <user@email.com>”.

If you resisted using the required email field format, the following is displayed.

B. Enter Optional Git-Branch Use¶

You may also provide an ordered list of branches to be preferentially used during git-package extraction. The list does not need to include the ‘master’ branch since that is automatically appended to the end of the list as the branch of last resort. The default is to only use the ‘master’ branch. The list is blank separated.

In the example below, notice that 2 branches are provided. This build and integration run will determine if the provided branches are ready for merging into the master stack based on clean compilation, build, and unit and end-to-end testing.

3. Trigger the Build¶

Select the “Force Build” button to trigger your build. If the Builder is not in the midst of another build, then your job is started and the view updates to show your job as current (see below). If there is a build in-progress, your build will be entered into the Pending Build Requests list; it will be processed in first-in, first out order.

4. Viewing a Build’s Status¶

If you select a Build Number hot-link, you will see a very detailed status report on that particular build. In the sample below, notice the “Steps and Logfiles” with a hotlink to stdio – the log of this build’s run. Notice the “Forced Build Properties” section which shows the use of multiple branches being test-integrated into the stack. And finally, notice the Rebuild button to force an identically setup run.

5. Viewing the DM_stack’s Log¶

Alternate access to the full build’s stdio log is also available from the Waterfall view - as shown below.

6 Login and Cancelling a Pending or In-progress Build¶

A. Login to Buildbot User Interface¶

Buildbot requires authentication only when the user wants to terminate a pending or in-progress run. Authentication uses the htpasswd protocol. Developers are requested to enter the username and encrypted htpasswd into lsst-dev:~lsstsw/BUILDBOT_POWERS_HERE; The SQuaRE Team will load your data into the appropriate location; to ensure it’s done in a timely manner, mention it in the SQuaRE chatroom.

Once your account is setup, enter your username and password into the fields at the upper right hand corner and select ‘Login’. The system automatically logs you out once a day.

You are now logged in and ready to let Buildbot know which Build you want terminated.

Determine the Build Number¶

You can recreate the DM stack environment created by a specific build as long as you can identify the Buildbot build number. If you triggered the build using the ForceBuild form, you will receive an email with the build number identified in the text. Look for ‘@ NNN:’ in the first line of the emailed table; in the example below, that would be “@ 340”.

Determine the Eups-Tag¶

You have three options for determining the eups-tag:

• If you have access to the lsst-dev system, you can use the lookup table: lsst-dev:~lsstsw/build/eupsTag_buildbotNum to find the eups-tag corresponding to the buildbot build number assigned.

  • The format is: <eupsTag>:<build#>:<completion status>:<blank separated list of branches>.

  • Note that the <build#> changes with each new build but the

    <eupsTag> is repeated until a new source change triggers a new eups-tag.

• If you have the email sent on completion of a build, you can easily find the Eups-tag in the lead table of that email. See example above for “Eups Tag b182”.

• If you don’t have the email but still need to deduce the eups tag used for a specific build, then you need to search the build’s stdio log for the specific eups-tag assigned to the build. Look in the stdio log for string: “# BUILD b<number>”; your tag is then “b<number>”.

  • For Build #: 340, the eups-tag is: “b182”.
  • Note 1: The confusing search string, which might conceptually define either a Buildbot Build or an Eups Build, is an unfortunate output diagnostic word choice.

Setup the DM stack access¶

The following sets up the LSST-cluster’s DM stack such that it is ready for the developer’s use:

export EUPS_PATH=/lsst/home/lsstsw/stack
source /lsst/home/lsstsw/eups/bin/setups.sh
setup anaconda
setup git
setup lsst

Setup the Specific Build’s Stack¶

First you need to determine just how successful the stack build was: did it successfully complete all 3 phases: a) build the stack without failure, b) generate the doxygen document for the stack, c) run the end-to-end tester. Once the Build and Unittest phase completes, the stack has been updated for all packages in the release. At this point, even if either of the later two phases fails, the full stack is available for normal, if suspect, use. Buildbot processing terminates at the first fatal exit.

• If the initial (scons and unittest) build phase failed, proceed to section: “Setup a Failed Build’s Environment”.
• If the stack build failed during creation of the doxygen documentation or during the end-to-end testing, then proceed to section: “Setup a Successful Build’s Environment”. However, remember that the buildbot processing terminated early for a reason which was either due to failed doxygen documentation creation or a failed end-to-end test. Those will need to be debugged.

Eups-based commands used during eups-package setup include:

eups list -t  <tag>   # display all packages with the named eups-tag.
setup -t <tag> <package>  # eups-setup <package> and its dependencies.

Setup the Environment based on a Successful SCONS & Unittest Build¶

If the named build refers to the successful completion of the SCONS build and unittest phase then eups-setup the “lsst_distrib” package for the specific build’s eups-tag. Since the “lsst_distrib” package depends on every package in the DM Release stack, all relevant packages will be setup. You’ll need to determine the eups-tag - see section: “Determine the EUPS-Tag”.

Syntax:

setup -t <tag> <package>

For the current example, use:

setup -t b76   lsst_distrib

Setup the Environment based on a Failed SCONS or Unittest Build

Setting up a failed build’s environment is more complicated. In this case you will use the information provided in the build’s failure email to acquire the failed package’s manifest details. Note the pointers to the failed build’s artifacts listed in the sample failure email below.

The “Failed Package Info” directory: /lsst/home/lsstsw/build/FailedLogs/340/ , contains a <package> subdirectory for the package which failed.

Since the failure was during the scons or unittest build, the dependent packages of the failed packages need to be setup. Look-up the name of the package which failed - either in the stdio log or an ‘ls’ of the Failed Package Info directory listed in the email:

export FailedPackageDir=/lsst/home/lsstsw/build/FailedLogs/340/
export Manifest=_build.tags
setup -m $FailedPackageDir/<package>/$Manifest

Now you’re ready to git-extract the failed package into your local space, set it up as a LOCAL instantiation. For the current example, you’d:

cd <favorite scratch space>
git clone git@github.com:lsst/afw.git
cd afw
setup -t b182 -r .   # or (see Note below): setup -T b182 -r .
scons -Q opt=3

Note: If your debugging leads you to modify other packages whilst using the same underlying stack, you move into eups management magic. As DM’s eups guru said regarding the details of eups management: With ‘-t’, you override any dependencies; with ‘-T’ you only use the tag if no version is specified. You probably want something more like:

setup -t rhl -T <tag> package

(get the <tag> package, but override with my own ones).

Finding Concordance between eups-tag and buildbot-number¶

A table, maintained on the lsst cluster, contains the mapping between assigned eups-tags and the buildbot builds which used them. The table is found in ~lsstsw/build/eupsTag_buildbotNum . Each row of the table contains the following colon (:) separated fields:

1. eups-tag - the tag assigned by the CI software to the stack comprising the products being built on a specific buildbot slave.
2. buildbot build number - the monotonically-increasing build run id for a specific buildbot slave.

#. status of build - one of (“success”, “failure”); refers to the build status for the suite of products specified for the build. branches used during the build - blank separated list of branches, in eups-VRO order, which were used to select the specific package branch being built.

A sample table follows::

b202:432:failure:master b202:433:success:master b202:434:success:master

The first line indicates that buildbot run: 432, failed while build-and-unit testing a package. The eups-tag: b202, was generated as soon as the transient package manifest for the full stack build was generated. As the build progresses, the successfully built&tested packages are added to the permanent manifest for the stack. For a more detailed description of the stack tagging process see the README in DM package: devenv/lsst_build.

The second line indicates that the subsequent buildbot build succeeded when using the same packages and branches as the last build. The archived stack manifest for eups-tag b202 would now list every package used during the build.

The third line demonstrates what happens when a subsequent build occurs which again uses the same packages and branches as the original build. The CI software deduces that no new packages were to be added to what has already been done, so it doesn’t need to process further.

A successful build is one in which all packages for the buildbot run’s product have been built and unit tested successfully. Currently that includes all of sims, qserv and lsst_distrib products and their dependencies.

Shared lock¶

Error:

eups list: Unable to take shared lock on /lsst/home/lsstsw/stack: an exclusive lock is held by [user=lsstsw,pid=19919]" (see DM stack Build #: 166)

Cause:

If the buildbot slave is forcefully terminated whilst it is processing an eups-locked action on an eups-stack.

Solution:

The eups-locks remain set until cleared by the owner of the eups stack. In this case, that’s user ‘lsstsw’:

ssh lsstsw@lsst-dev
# The eups DM stack is automatically setup when 'lsstsw' logs in.
eups admin listLocks
eups admin clearLocks
exit

Glossary¶

Buildmaster

the Buildbot supervisory process: polls the git-repositories for change; triggers Builders on schedule; on git-change or on another Builder’s request; serves the Buildbot web pages; and emails status reports on Builder completion.

Builder

a process initiated by the Buildbot master to effect a desired processing goal. The Builder may initiate sequential sub-processes on the Buildmaster host and/or on a captive Buildslave host, The most common Builders direct the building of a code stack according to pre-defined sequence of operations, known as build steps.

Buildslave

provides computational resources on a remote host for a Builder process.

build step

is a self-contained sub-process which is invoked either on the Buildmaster or a Buildslave as dictated by the Builder invoking the step.

trigger

an event which alerts the Buildmaster to initiate a pre-determined Builder. Types of triggers are:

• on-change or on-git-change: a trigger event which occurs when the source code repository is changed;
• on-schedule a trigger event which occurs at a defined point in time - hourly, daily, weekly, etc.
• on-builder-request a trigger event which initiates a secondary builder.

DRP

LSST-speak for a stack which can execute a Data Release Production given suitable input data and computing resources.