Testing the LSST DM Stack with the Jenkins stack-os-matrix Job¶

stack-os-matrix is the continuous integration (CI) job that DM developers must run to ensure tests pass before merging changes to any LSST EUPS Stack package. This page explains how to run the stack-os-matrix job and find build and test logs.

For more information about the DM’s Jenkins continuous integration service, see Jenkins ci.lsst.codes Overview.

Running a stack-os-matrix job¶

Open the stack-os-matrix dashboard in a browser directly or by clicking on the stack-os-matrix job from the main dashboard at https://ci.lsst.codes.
Click on the Run button near the top of the page.
Configure the run in the pop-up dialog:
- Set the list of Git refs (branch or tag names) to check out from Stack Git repositories.
  
  For example, if you enter tickets/DM-2 tickets/DM-1, the build system will attempt to check out the tickets/DM-2 branch in Stack packages. If a package does not have a tickets/DM-2 branch, it will attempt to check out the tickets/DM-1 branch.
  
  If a package has neither branch, it falls back to checking out the main branch or the branch configured in repos.yaml in the case of forked third-party packages. Because of those third-party packages, you never want to specify main explicitly in this field. To check that a main-only build passes, leave the refs box entirely blank.
- Set the list of EUPS packages to build. Use the default (lsst_distrib lsst_ci) to build and test your changes with a full Stack. To improve build times you can instead specify the name of the package you are actively developing. Before you merge a ticket branch to main, you must run a stack-os-matrix Job with at least the default packages so that the full Stack is built and tested with your changes.
  
  There are also several packages that can be appended to the default to do more thorough testing at the cost of much longer build times:
  - ci_cpp exercises the Calibration Products Pipeline in ways that are too computationally expensive for unit tests in cp_pipe.
  - ci_hsc exercises most of the Data Release Production pipelines, including single-frame processing and coaddition, on HSC engineering data. This package is also run as part of the nightly build.
  - ci_imsim runs the same on simulated Rubin Observatory data. Like ci_hsc, it’s run nightly in addition to as part of stack-os-matrix.
- Set the conda environment to run on. The default is the latest rubin-env package. You should only need to change this when testing a new environment, or to temporarily work around bugs introduced in the latest version.
Click the Run button in the dialog to start the build.

Monitoring the run status¶

Your new job is added to the top of the table on the stack-os-matrix dashboard. Click on the row to see a detail page for the run.

The pipeline diagram at the top of the detail page shows the status of the build on each supported platform. Click on the icon for the platform you’re interested in.

The progress for each build step is listed on the run detail page. The most important build step is ./buildbot-scripts/jenkins_wrapper.sh where the lsstsw rebuild is run. Click on that step to see build and test progress for each package.

Getting stack-os-matrix notifications in Slack¶

Jenkins sends status notifications to the #dmj-stack-os-matrix channel on Slack when your job starts and finishes. See Slack job notifications for more information.

Viewing build and test results¶

Click on the Tests tab from the run detail pages to see the status of individual tests.

For more complex build and test failures, it may be most efficient to inspect the full build logs and *.failed files that show unit test failures. You can find these *.failed files, along with build logs, under the Artifacts tab from the run detail pages. You can even download all the logs and *.failed files for a build by scrolling to the bottom and clicking on the Download All button.