DM Python Style Guide¶

All DM Python code MUST work with Python 3¶

All the Python code written by LSST Data Management must be runnable using Python 3. Python 2 will cease to be supported before LSST is operational (PEP 373). The current baseline version is Python 3.5.

DM Python library code with an external user base MUST support Python 2.7 and 3.x¶

During construction we are expecting external users to be experimenting with some of the DM code and LSST DM library code is being used in their applications. This user community is currently transitioning from Python 2.7 to Python 3.x, and we should ensure that code works on both Python versions until our dependencies drop support for Python 2.7. In particular, the Science Pipelines code (commonly referred to as lsst_distrib) must support 2.7 and 3.x.

Standalone applications, code providing services, and internal programs and modules (such as Qserv, SQuaSH and dax) do not have to support Python 2. If code is currently supporting both 2.7 and 3.x, dropping support for Python 2.7 requires an RFC. New code that has never supported Python 2.7 and which will not be externally usable library code or a dependency of a package that supports 2.7 does not require an RFC to request that 2.7 is not supported.

Exceptions to PEP 8¶

The following table summarizes all PEP 8 guidelines are not followed by the DM Python Style Guide. These exceptions are phrased as error codes that may be ignored by the flake8 linter (see Code MAY be validated with flake8).

E133

Closing bracket is missing indentation. This pycodestyle error (via flake8) is not part of PEP 8.

E226

Missing whitespace around arithmetic operator. See Binary operators SHOULD be surrounded by a single space except for [*, /, **, //, %].

E228

Missing whitespace around bitwise or shift operator. See Binary operators SHOULD be surrounded by a single space except for [*, /, **, //, %].

E251

Unexpected spaces around keyword / parameter equals. See Keyword assignment operators SHOULD be surrounded by a space when statements appear on multiple lines.

N802

Function name should be lowercase. See 6. Naming Conventions.

N803

Argument name should be lowercase. See 6. Naming Conventions.

Maximum line length

See Line Length MUST be less than or equal to 110 columns.

Code MAY be validated with flake8¶

The flake8 tool may be used to validate Python source code against the portion of PEP 8 adopted by Data Management. In addition, flake8 statically checks Python for code errors. The separate pep8-naming plugin validates names according to the DM Python coding style.

pip install flake8
pip install pep8-naming

flake8 --ignore=E133,E226,E228,N802,N803 --max-line-length=110 .

[flake8]
max-line-length = 110
ignore = E133, E226, E228, N802, N803

Lines that intentionally deviate from DM’s PEP 8 MUST include a noqa comment¶

Lines of code may intentionally deviate from our application of PEP 8 (see above) because of limitations in flake8. In such cases, authors must append a # noqa comment to the line that includes the specific error code being ignored. See the flake8 documentation for details . This prevents the line from triggering false flake8 warnings to other developers, while also linting unexpected errors.

For example, to import a module without using it (to build a namespace, as in a __init__.py):

from .module import AClass  # noqa: F401

autopep8 MAY be used to fix PEP 8 compliance¶

Many PEP 8 issues in existing code can be fixed with autopep8:

autopep8 . --in-place --recursive \
    --ignore E133,E226,E228,E251,N802,N803 --max-line-length 110

The . specifies the current directory. Together with --recursive, the full tree of Python files will be processed by autopep8. Alternatively, a single file can be specified in place of ..

autopep8ʼs changes must always be validated before committing.

Style changes must be encapsulated in a distinct commit (see Commits should represent discrete logical changes to the code in Workflow document).

Line Length MUST be less than or equal to 110 columns¶

Limit all lines to a maximum of 110 characters. This conforms to the DM C++ Style Guide (see 4-6).

This differs from the PEP 8 recommendation of 79 characters.

Python’s implied continuation inside parens, brackets and braces SHOULD be used for wrapped lines¶

The preferred way of wrapping long lines is by using Python’s implied line continuation inside parentheses, brackets and braces.

If necessary, you can add an extra pair of parentheses around an expression, but sometimes using a backslash looks better. In this example, continuation is naturally implied within the __init__ method argument lists, while both \ and parentheses-based continuations are used in the if statements.

class Rectangle(Blob):
    """Documentation for Rectangle.
    """
    def __init__(self, width, height,
                 color='black', emphasis=None, highlight=0):

        # Discouraged: continuation with '\'
        if width == 0 and height == 0 and \
               color == 'red' and emphasis == 'strong' or \
               highlight > 100:
            raise ValueError("sorry, you lose")

        # Preferred: continuation with parentheses
        if width == 0 and height == 0 and (color == 'red' or
                                           emphasis is None):
            raise ValueError("I don't think so")

        Blob.__init__(self, width, height,
                      color, emphasis, highlight)

Be aware that the continued line must be distinguished from the following lines through indentation. For example, this will generate an E129 error:

if (width == 0 and
    height == 0):
    pass

Instead, the continued line should be indented:

if (width == 0 and
        height == 0):
    pass

Blank lines SHOULD NOT be added before or after a docstring¶

Do not use a blank line on either side of a docstring.

Consistency with the DM C++ Coding Guide namespaces SHOULD be followed¶

Consistency with the LSST C++ Coding Standards namespaces exists.

Good:

• from lsst.foo.bar import myFunction is analogous to using lsst::foo::bar::myFunction
• import lsst.foo.bar as fooBar is analogous to namespace fooBar = lsst::foo::bar

Disallowed in both Coding Standards (except in __init__.py library initialization context):

• from lsst.foo.bar import * is analogous to using namespace lsst::foo::bar

The minimum number of parentheses needed for correctness and readability SHOULD be used¶

Yes:

a = b(self.config.nSigmaToGrow*sigma + 0.5)

Less readable:

a = b((self.config.nSigmaToGrow*sigma) + 0.5)

Binary operators SHOULD be surrounded by a single space except for [*, /, **, //, %]¶

Always surround these binary operators with a single space on either side; this helps the user see where one token ends and another begins:

• assignment (=),
• augmented assignment (+=, -=, etc.),
• comparisons (==, <, >, !=, <>, <=, >=, in, not in, is, is not),
• Booleans (and, or, not).

Use spaces around these arithmetic operators:

• addition (+),
• subtraction (-)

Never surround these binary arithmetic operators with whitespace:

• multiplication (*),
• division (/),
• exponentiation (**),
• floor division (//),
• modulus (%). Note that a single space must always surround % when used for string formatting.

For example:

i = i + 1
submitted += 1
x = x*2 - 1
hypot2 = x*x + y*y
c = (a + b)*(a - b)
print('Hello %s' % 'world!')

This deviates from PEP 8, which allows whitespace around these arithmetic operators if they appear alone. Error codes: E226 and E228.

Keyword assignment operators SHOULD be surrounded by a space when statements appear on multiple lines¶

However, if keyword assignments occur on a single line, where should be no additional spaces.

Thus this:

# whitespace around multi-line assignment
funcA(
    karg1 = value1,
    karg2 = value2,
    karg3 = value3,
)

# no whitespace around single-line assigment
funcB(x, y, z, karg1=value1, karg2=value2, karg3=value3)

Not this:

funcA(
    karg1=value1,
    karg2=value2,
    karg3=value3,
)

aFunction(x, y, z, karg1 = value1, karg2 = value2, karg3 = value3)

Opposes PEP 8. Error code: E251.

Comments MUST always remain up-to-date with code changes¶

Comments that contradict the code are worse than no comments. Always make a priority of keeping the comments up-to-date when the code changes!

Sentences in comments SHOULD NOT be separated by double spaces¶

Following PEP 8, comments should be complete sentences.

However, sentences should not be separated by two spaces; a single space is sufficient.

This differs from PEP 8.

Block comments SHOULD reference the code following them and SHOULD be indented to the same level¶

Block comments generally apply to some (or all) code that follows them, and are indented to the same level as that code. Each line of a block comment starts with a # and a single space (unless it is indented text inside the comment).

Paragraphs inside a block comment are separated by a line containing a single #.

Docstrings SHOULD be written for all public modules, functions, classes, and methods¶

Write docstrings for all public modules, functions, classes, and methods. See Documenting Python APIs.

Docstrings are not necessary for non-public methods, but you should have a comment that describes what the method does. This comment should appear after the def line.

User defined names SHOULD NOT shadow python built-in functions¶

Names which shadow a python built-in function may cause confusion for readers of the code. Creating a more specific identifier is suggested to avoid collisions. In the case of filter, filterName may be appropriate; for filter objects, something like filterObj might be appropriate.

Class Attribute Names SHOULD be camelCase with leading lowercase¶

Opposes PEP 8. Error codes: N802 and N803.

Module methods (free functions) SHOULD be camelCase with leading lowercase¶

Opposes PEP 8. Error code: N802.

Modules which contain class definitions SHOULD be named after the class name¶

Modules which contain class definitions should be named after the class name (one module per class).

When a Python module wraps a C/C++ extension module, the C/C++ module SHOULD be named <module>Lib¶

When an extension module written in C or C++ has an accompanying Python module that provides a higher level (e.g. more object oriented) interface, the C/C++ module should append Lib to the module’s name (e.g. socketLib).

Names l (lowercase: el), O (uppercase: oh), I (uppercase: eye) MUST be avoided¶

Never use these characters as single character variable names:

• l (lowercase letter el),
• O (uppercase letter oh), or
• I (uppercase letter eye).

In some fonts, these characters are indistinguishable from the numerals one and zero. When tempted to use l, use L instead.

A Python source file name SHOULD be camelCase-with-leading-lowercase and ending in ‘.py’¶

A module containing a single class should be a camelCase-with-leading-lowercase transliteration of the class’s name.

The name of a test case should be descriptive without the need for a trailing numeral to distinguish one test case from another.

ASCII Encoding MUST be used for new code¶

Always use ASCII for new Python code.

• Do not include a coding comment (as described in PEP 263) for ASCII files.
• Existing code using Latin-1 encoding (a.k.a. ISO-8859-1) is acceptable so long as it has a proper coding comment. All other code must be converted to ASCII or Latin-1 except for 3rd party packages used “as is.”

Standard code order SHOULD be followed¶

Within a module, follow the order:

1. Shebang line, #! /usr/bin/env python (only for executable scripts)
2. Module-level comments (such as the license statement)
3. Module-level docstring
4. Imports
5. __all__ statement, if any
6. Private module variables (names start with underscore)
7. Private module functions and classes (names start with underscore)
8. Public module variables
9. Public functions and classes
10. Optional test suites

super SHOULD NOT be used unless the author really understands the implications (e.g. in a well-understood multiple inheritance hierarchy).¶

Python provides super() so that each parent class’ method is only called once.

To use super(), all parent classes in the chain (also called the Method Resolution Order) need to use super() otherwise the chain gets interrupted. Other subtleties have been noted in an article by James Knight:

• Never call super() with anything but the exact arguments you received, unless you really know what you’re doing.
• When you use it on methods whose acceptable arguments can be altered on a subclass via addition of more optional arguments, always accept *args, **kw, and call super() like super(MyClass, self).currentmethod(alltheargsideclared, *args, **kwargs). If you don’t do this, forbid addition of optional arguments in subclasses.
• Never use positional arguments in __init__ or __new__. Always use keyword args, and always call them as keywords, and always pass all keywords on to super().

For guidance on successfully using super(), see Raymond Hettinger’s article Super Considered Super!

is and is not SHOULD only be used for determining if two variables point to same object¶

Use is or is not only for the case that you need to know that two variables point to the exact same object.

To test equality in value, use == or != instead.

is and is not SHOULD be used when comparing to None¶

There are two reasons:

1. is None works with NumPy arrays, whereas == None does not;
2. is None is idiomatic.

This is also consistent with PEP 8, which states:

Comparisons to singletons like None should always be done with is or is not, never the equality operators.

For sequences, (str, list, tuple), use the fact that empty sequences are False.

Yes:

if not seq:
    pass

if seq:
    pass

No:

if len(seq):
    pass

if not len(seq):
    pass

Supporting Python 2.7 and 3.x simultaneously¶

A mutable object MUST NOT be used as a keyword argument default¶

Never use a mutable object as default value for a keyword argument in a function or method.

When used a mutable is used as a default keyword argument, the default can change from one call to another leading to unexpected behavior. This issue can be avoided by only using immutable types as default.

For example, rather than provide a default empty list:

def proclist(alist=[]):
    pass

this function should create a new list in its internal scope:

def proclist(alist=None):
    if alist is None:
        alist = []

Context managers (with) SHOULD be used for resource allocation¶

Use the with statement to simplify resource allocation.

For example to be sure a file will be closed when you are done with it:

with open('/etc/passwd', 'r') as f:
    for line in f:
        pass

Avoid dict.keys() when iterating over keys or checking membership¶

For iterating over keys, iterate over the dictionary itself, e.g.:

for x in mydict:
    pass

To test for inclusion use in:

if key in myDict:
    pass

This is preferred over keys(). Use keys() when storing the keys for later access:

k = list(mydict.keys())

where list ensures that a view or iterator is not being retained.

The subprocess module SHOULD be used to spawn processes¶

Use the subprocess module to spawn processes.

lambda SHOULD NOT be used¶

Avoid the use of lambda. You can almost always write clearer code by using a named function or using the functools module to wrap a function.

The set type SHOULD be used for unordered collections¶

Use the set type for unordered collections of objects.

The argparse module SHOULD be used for command-line scripts¶

Use the argparse module for command-line scripts.

Command line tasks for pipelines should use lsst.pipe.base.ArgumentParser instead.

Iterators and generators SHOULD be used to iterate over large data sets efficiently¶

Use iterators, generators (classes that act like iterators) and generator expressions (expressions that act like iterators) to iterate over large data sets efficiently.