Adding a New Package to the Build

New packages intended for distribution to end users should generally be added as a dependency of a “top level product”: these are the roots of the LSST package hierarchy. They include lsst_apps, lsst_distrib, qserv_distrib and lsst_sims.

Before adding a new dependency to any of these products, it must be approved through the usual Discussion and Decision Making Process (RFC, RFD): consensus must be reached regarding both the name and the suitability of the new package. Before adopting the RFC, implementation tickets should be created to cover package creation.

After approval, code written internally by Data Management should be packaged following the template in the lsst/templates repository. DM packaging of third party code should proceed as described in Distributing Third-Party Packages with EUPS.

New packages must be added to the LSST organization on GitHub and access must be granted to appropriate teams. For DM written code, these include “Data Management” and “Overlords”; for third party code, “DM Externals” and “Overlords” (but not Data Management).

The new package must be added to the etc/repos.yaml file in the lsstsw package along with its corresponding GitHub URL. This file is governed by a “self-merge” policy: upon opening a pull request, it will be checked by the Travis CI system, and developers may merge without further review on success. Refer to RFC-75 for background.

The new package then needs to be added to the ups/*.table file (and possibly the ups/*.cfg file) of one or more other packages in the stack where it is used.

Handling Git LFS backed repos

New Git LFS (see Using Git Large File Storage (LFS) for Data Repositories) backed repos (or existing repos being converted to lfs) require additional configuration.

  • The repos.yaml entry must declare that the repository is LFS backed.

    afwdata:
  url: https://github.com/lsst/afwdata.git
  lfs: true

    See the comment block at the top of repos.yaml for additional details.

  • At present, the EUPS distrib packaging mechanism does not support lfs backed repos. These products must not be added to any top level meta-package or as a mandatory (non-optional) recursive dependency of a top level package.

  • Optional dependencies must be added to manifest.remap to prevent the creation of broken EUPS distrib packages. Please note that the “self-merge” policy of RFC-75 does not apply to manifest.remap.

    Unlike changes merged into repos.yaml, modifications to manifest.remap do not take immediate affect

    recommend procedure is to attach the modification PR to a DM Jira issue on the Continuous Integration component.