Adding a New Package to the Build¶
New packages intended for distribution to end users should generally be added
as a dependency of a “top-level product:” these are the roots of the LSST
package hierarchy. They include
Before adding a new dependency to any of these products, it must be approved through the RFC process. Consensus must be reached regarding both the name and the suitability of the new package. Before adopting the RFC, implementation tickets should be created to cover package creation.
After approval, code written internally by Data Management should be packaged following the template in the lsst/templates repository. DM packaging of third party code should proceed as described in Distributing Third-Party Packages with EUPS.
New packages must be added to the LSST organization on GitHub and access must be granted to appropriate teams. For DM-written code, these include “Data Management” and “Overlords.” For third-party code, use the “DM Externals” and “Overlords” (but not “Data Management”) teams.
The new package must be added to the etc/repos.yaml file in the lsst/repos repository along with its corresponding GitHub URL. This file is governed by a “self-merge” policy: upon opening a pull request, it will be checked by the stack-os-matrix Jenkins job, and developers may merge without further review on success. Refer to RFC-75 for background.
The new package then needs to be added to the
ups/*.table file (and
ups/*.cfg file) of one or more other packages in the
stack where it is used.
Configuring GitHub Repositories¶
All LSST DM repositories on GitHub must be configured to protect the
master branch and to ensure that the merge button for pull requests can not be pushed without the branch being up to date with
There are a number of settings required to ensure this and they are described below with URLs referring to the
afw with the relevant package name to get to the correct page on GitHub.
- On the main settings page for the repository, https://github.com/lsst/afw/settings, disable squash and rebase merging:
master branch to enable protections.
afw this is located at https://github.com/lsst/afw/settings/branches/master and can also be found from the “Branches” sidebar item on the settings screen.
Select the “Protect this branch” option and this will make available additional checks.
You should also enable status checks and require that branches be up to date before merging.
With this enabled people will be able to use the GitHub merge button on Pull Requests and know that the standard process is being adhered to.
In the image above no automated status checks are being performed. Whilst not required, we recommend that some basic checks are enabled on repositories using Travis. This will not replace normal testing with Jenkins job but for packages that have been updated to use flake8 it is useful to add a simple Travis script like the following:
sudo: false language: python matrix: include: - python: '3.6' install: - pip install flake8 script: flake8
For packages containing C++ that have been tidied up using
clang tools, you may consider adding a Travis check that runs the tidy tool and does a
diff with the repository.
Once the Travis script has been written, Travis can be enabled on the “Integrations & services” settings screen, https://github.com/lsst/afw/settings/installations, and Travis can then be added to the branch protection settings. Your branch protections screen should then look something like this:
It takes a few minutes from enabling Travis to GitHub making the Travis option available on the branch protection screen.
Handling Git LFS-backed repos¶
New Git LFS-backed repos (or existing repos being converted to LFS) require additional configuration.
The repos.yaml entry must declare that the repository is LFS backed:
afwdata: url: https://github.com/lsst/afwdata.git lfs: true
See the comment block at the top of repos.yaml for additional details.
At present, the EUPS distrib packaging mechanism does not support LFS-backed repos. These products must not be added to any top-level meta-package or as a mandatory (non-
optional) recursive dependency of a top-level package.
We recommend that you attach the modification PR to a DM Jira issue on the