Distributing Third-Party Packages with EUPS¶

Getting Approval¶

Creating a new third-party package that will be a dependency of the LSST code typically requires an RFC. If the code is to be distributed via eups, as this page describes, the license for the third-party code should be verified and cited in the text of that RFC. The license must be compatible with the license under which we distribute our code, currently GPL3. See this page for a list of compatible licenses.

Creating the Package¶

Repositories containing third-party packages exist in the LSST GitHub organization. (Unfortunately, it is currently difficult to distinguish between an LSST package and a third-party package: the table file in the lsst_thirdparty package and the documentation on third party software may help.) In order to distribute a new third-party package, someone with administrator privileges will have to create a new repository of this form for you. Create a development branch on that repository and set it up to distribute the package as described below. You will be able to test the package distribution off of your development branch before you merge to master.

The repository, once created, needs to contain the following directories:

upstream/

This directory should contain a gzipped tarball of the source code for the third-party package. Literally, that is all it should contain. The code should not be altered from whatever is distributed by the package’s author. Any changes that need to be made to the source code should be done with patches in the patches/ directory.

ups/

This directory should contain the packages EUPS table file as well as an optional file eupspkg.cfg.sh which will contain any customized commands for installing the third-party package.

patches/

This directory is optional. It contains any patches to the third-party package (which EUPS will apply using the patch command) that are required to make the package work with the stack.

We discuss the contents of ups/ and patches/ in more detail below.

setupRequired(base)
setupRequired(sconsUtils)
setupOptional(doxygen)

envPrepend(LD_LIBRARY_PATH, ${PRODUCT_DIR}/lib)
envPrepend(DYLD_LIBRARY_PATH, ${PRODUCT_DIR}/lib)
envPrepend(LSST_LIBRARY_PATH, ${PRODUCT_DIR}/lib)
envPrepend(PYTHONPATH, ${PRODUCT_DIR}/python)

export SCONSFLAGS=$SCONSFLAGS" USE_UNKNOWN_VARS=true TMV_DIR="$TMV_DIR" \
       PREFIX="$PREFIX" PYPREFIX="$PREFIX"/lib/python                   \
       EXTRA_LIB_PATH="$TMV_DIR"/lib EXTRA_INCLUDE_PATH="$TMV_DIR"/include"

# EupsPkg config file. Sourced by 'eupspkg'

prep()
{
    # Select the apropriate Anaconda distribution
    OS=$(uname -s -m)
    case "$OS" in
        "Linux x86_64")       FN=Anaconda-2.1.0-Linux-x86_64.sh ;;
        "Linux "*)        FN=Anaconda-2.1.0-Linux-x86.sh ;;
        "Darwin x86_64")  FN=Anaconda-2.1.0-MacOSX-x86_64.sh ;;
        *)          die "unsupported OS or architecture ($OS). try installing Anaconda manually."
    esac

    # Prefer system curl; user-installed ones sometimes behave oddly
    if [[ -x /usr/bin/curl ]]; then
        CURL=${CURL:-/usr/bin/curl}
    else
        CURL=${CURL:-curl}
    fi

    "$CURL" -s -L -o installer.sh http://repo.continuum.io/archive/$FN
}

build() { :; }

install()
{
    clean_old_install

    bash installer.sh -b -p "$PREFIX"

    if [[ $(uname -s) = Darwin* ]]; then
        #run install_name_tool on all of the libpythonX.X.dylib dynamic
        #libraries in anaconda
        for entry in $PREFIX/lib/libpython*.dylib
        do
            install_name_tool -id $entry $entry
        done
    fi

    install_ups
}

git diff -u originalFile correctedFile > someFileName.patch

Testing the Package¶

Before finalizing the distribution, it is useful to be able to test that the distribution as set up does, in fact, build. This can be accomplished using the lsstsw build tool: see its detailed documentation for the full story. Broadly, the steps are:

• Clone and set up the lsstsw package in its own directory using the instructions on the lsstsw documentation page pointed to above.

• In the lsstsw package, use the command:

  ./bin/rebuild -r yourBranch yourPackage
  

  to build the development branch of your package.

• Ideally, you should try this process on at least two different machines (one running OSX and one running a Linux distribution) to make sure that you did not accidentally benefit from the system environment of your test machine when building.

Distributing the Package¶

Once the package builds and passes review (or vice-versa), you need to tell eups that it is available for distribution to the wide world. To do this, add an annotated tag to your package repository using:

git tag -a versionNumber -m "Some comment."

The initial versionNumber should match the external package’s version number. If the package does not supply an appropriate version number, one can be generated from an upstream git SHA1 or equivalent version control revision number: use the format 0.N.SHA1, where N is 1 for the first release of the package, 2 for the second, etc. Note that the version number should never start with a letter, as EUPS regards that as semantically significant.

If changes are required to the packaging (in the ups or patches directories) but not the external package source (in the upstream directory), the string .lsst1 (and .lsst2 etc. thereafter) should be appended to the external package’s version number. Merge your changes to master, then push your changes to the remote repository. Push your tags to the remote repository using:

git push --tags

Now you must log onto lsst-dev as the user lsstsw (this will require special permissions): see the documentation on using this machine. Once logged in as lsstsw, the steps are:

• Build your package with the command:

  rebuild yourPackage
  

  This will cause lsst-dev to build your package and all of its dependencies. This build will be assigned a build number formatted as bNNN``

• Once the build is complete, release it to the world using:

  publish -b bNNN yourPackage
  

  This will make your package installable using:

  eups distrib install yourPackage versionNumber
  

  If you wish to add a distribution server tag to your package, you can do so by changing the publish command to:

  publish -b bNNN -t yourTag yourPackage
  

  Warning

  Do not use the tag ‘current’ as that will overwrite all other packages marked as current and break the stack. Let the people in charge of official releases handle marking things as ‘current.’ it is not usually necessary to distribution-server-tag a particular third party package.

• Generally, if you’re publishing a third party package, it should be because it is a dependency in the build of some (or all) top-level package(s). When the top-level package(s) are next published (and optionally tagged), your new package will be incorporated. If you need something sooner, you can do this publishing yourself using the steps above with the top-level package. In this case, a distribution-server-tag (something like qserv-dev) is usually desirable. That makes the top-level product (or any of its dependency components, including your third-party package) installable using:

  eups distrib install -t yourTag packageName
  

Updating the Package¶

To update the version of your external package after a new upstream release, start with a copy of the LSST stack (installed using the lsstsw tool). Then:

• Create a ticket for the package update (and/or an RFC, if it may cause more trouble), and note the ticket number NNNN.

• cd build/yourPackage

• git checkout -b tickets/DM-NNNN (where NNNN is the ticket number above)

• git clean -id

• Download a copy of the tarball from wherever the external package is distributed. Don’t unzip or untar it.

• git rm the copy of the tarball that is currently in upstream/. Copy the new version of the external tarball into upstream/ and git add it.

• git commit

• git push

• See if the stack will build with your branch in Jenkins. For the branch name, specify the branch you created above (i.e. tickets/DM-NNNN), leaving the rest of the fields as they are.

• While Jenkins is building, you can test whether the new package solves whatever issue caused you to need the upgrade. From within build/yourPackage:

  • eupspkg -er -v 1 prep
  • eupspkg -er -v 1 config
  • eupspkg -er -v 1 build
  • eupspkg -er -v 1 install
  • eupspkg -er -v 1 decl
  • eups list yourPackage should now show a new version named tickets/DM-NNNN-gBLAHBLAH where gBLAHBLAH is the git has revision of the package.
  • setup lsst_apps -t YOURTAG
  • setup yourPackage tickets/DM-NNNN-gBLAHBLAH
  • Run your tests.

• Merge to master after tests pass.