LSST DM Developer Guide

This is an internal guide for LSST DM staff during project construction, and for Rubin Observatory Data Production staff in operations. It’s also openly available so that others can understand how we’re building the LSST’s data management subsystem. In some cases, other Rubin groups (for example Telescope & Site Software) have chosen to follow various sections as it applies to them.

This guide includes a mix of normative requirements and helpful, descriptive, pages. When it’s particularly important that you closely follow a standard, we include an annotation box at the top of the page.

Any member of DM can contribute to this guide. It’s published from the https://github.com/lsst-dm/dm_dev_guide GitHub repo. Check out the README to get started.

---

Jump to: Team · Communications · Project documentation · Work management

Development guides: Overview · C++ · Python · Pybind11 · JavaScript · ReStructuredText · DM Stack · Git · Editors · Legal · User documentation style

Services: Overview · Jenkins

---

Team

Basic information about the LSST Data Management Subsystem and our culture. Learn more about the Data Management Subsystem in LDM-294.

• Onboarding Checklist

• Team Culture and Conduct Standards

• Focus Friday

• Empowerment of DM team members

Team-specific pages:

• Data Release Production

Communications

How to use DM’s communication channels.

• Slack Culture

• Configuring your GitHub username in your Slack profile

• Request for comments (RFC)

• Request for discussion (RFD)

• Policy on DM Confluence Use

• Attending and Presenting at Conferences

• Providing Support to the Community

• DM Calendars

Project documentation

Controlled documentation and publications.

• Writing change-controlled documentation

• Publishing Papers and Proceedings

Technical notes.

• Technotes for Stand-Alone Technical Documentation

• Test Documentation

Work management

How DM coordinates work and gets things done.

• DM Development Workflow with Git, GitHub, JIRA and Jenkins

• Project Planning for Software Development

• JIRA Agile Usage

• JIRA Work Management Recipes

Development guides

Overview of Code Style Guides & Policies

• Introduction to DM’s Code Style Guides

• Software Unit Test Policy

C++

• DM C++ Style Guide

• Documenting C++ Code

• Using clang-format for LSST Development

• Unit-Testing Private C++ Functions

• Using C++ Templates

• Using Boost

• Using Eigen

• C++ performance profiling

• Creating a compilation database

Python

• DM Python Style Guide

• Formatting Python Code Automatically

• Python Unit Testing

• Documenting Python APIs with docstrings

• Using Astropy

• Python performance profiling

• Python Command Line Interfaces

Pybind11

• DM Pybind11 Style Guide

• Python wrappers for C++ with pybind11

JavaScript

• Documenting JavaScript APIs with JSDoc

ReStructuredText

• ReStructuredText Style Guide

DM Stack

General policies and procedures.

• Supported Platforms

• Transferring Code Between Packages

• Deprecating Interfaces

Development.

• Building a package with the installed Science Pipelines stack

• Developing packages on the LSST Science Platform

• Logging

• Debugging Tasks with lsstDebug

Documentation.

• Overview of the Stack documentation system

• Documentation in packages:

  • Layout of the doc/ directory

  • Topic types for package documentation:

    • Package homepage topic type

    • Module homepage topic type

    • Task topic type

    • Config topic type

    • Command-line script topic type

    • Argparse-based script topic type

    • Generic guide topic type

• Documentation in the main repository:

  • How to add a package to pipelines.lsst.io

• Building docs:

  • Building single-package documentation locally

  • Building the pipelines.lsst.io site locally

  • Building pipelines.lsst.io with Jenkins (sqre/infra/documenteer)

Testing.

• Testing the LSST DM Stack with the Jenkins stack-os-matrix Job

• Coverage Analysis

Packaging.

• EUPS Tutorial

• The LSST Software Build Tool

• Adding a New Package to the Build

• Moving to GitHub Actions from Travis

• Managing license and copyright in Stack packages

• Distributing Third-Party Packages with EUPS

• Renaming a Stack package

• Conda and rubin-env

Git

• Git Configuration Standards & Recommendations

• Using Git LFS (Large File Storage) for data repositories

Editors

Crowd-sourced recommendations for configuring editors for LSST development (listed alphabetically)

• Emacs Configuration for LSST Development

• SublimeText Configuration for LSST Development

• Vim Configuration for LSST Development

• Visual Studio Code Configuration for LSST Development

Legal

• Licensing LSST DM source code and content

• Copyrights for LSST DM work and the COPYRIGHT file

User documentation style

• User documentation style guide

• LSST-specific style guide

• Writing and structuring documentation for users

Service guides

IT overview

• LSST Information Technology Services Committee (ITSC)

Jenkins CI

• Jenkins rubin-ci.slac.stanford.edu Overview

Interim Data Facility

• Interim Data Facility Overview

US Data Facility

• SLAC Onboarding Procedure

• S3DF: SLAC Shared Science Data Facility Hosts USDF

• SLAC Kubernetes Overview

• Data Access: Storage Locations and Butler

• Batch Resources

• Stack Access

• Common Dataset Organization and Policy

• Data Access: Object Store