Python Unit Testing¶
This page provides technical guidance to developers writing unit tests for DM’s Python code base.
See Software Unit Test Policy for an overview of LSST Stack testing.
LSST tests should be written using the
unittest framework, with default test discovery, and should support being run using the pytest test runner as well as from the command line.
If you want to jump straight to a full example of the standard LSST Python testing boilerplate without reading the background, read the section on memory testing later in this document.
This document will not attempt to explain full details of how to use
unittest but instead shows common scenarios encountered in the LSST codebase.
unittest example is shown below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
import unittest import math class DemoTestCase1(unittest.TestCase): """Demo test case 1.""" def testDemo(self): self.assertGreater(10, 5) with self.assertRaises(TypeError): 1 + "2" class DemoTestCase2(unittest.TestCase): """Demo test case 2.""" def testDemo1(self): self.assertNotEqual("string1", "string2") def testDemo2(self): self.assertAlmostEqual(3.14, math.pi, places=2) if __name__ == "__main__": unittest.main()
The important things to note in this example are:
- Python tests explicitly should not contain a shebang (
#!/usr/bin/env python) and should not be executable (so cannot be run directly with
./test_Example.py). This avoids problems encountered running tests on macOS and helps ensure consistency in the way that tests are executed.
- Test file names must begin with
test_to allow pytest to automatically detect them without requiring an explicit test list, which can be hard to maintain and can lead to missed tests.
- If the test is being executed using python from the command line the
unittest.maincall performs the test discovery and executes the tests, setting exit status to non-zero if any of the tests fail.
- Test classes are executed in the order in which they appear in the test file.
In this case the tests in
DemoTestCase1will be executed before those in
- Test classes must, ultimately, inherit from
unittest.TestCasein order to be discovered, and it is recommended that
lsst.utils.tests.TestCasebe used as the base class when
afwobjects are involved. The tests themselves must be methods of the test class with names that begin with
test. All other methods and classes will be ignored by the test system but can be used by tests.
- Specific test asserts, such as
assertIn, should be used wherever possible. It is always better to use a specific assert because the error message will contain more useful detail and the intent is more obvious to someone reading the code. Only use
assertFalseif you are checking a boolean value, or a complex statement that is unsupported by other asserts.
- When testing that an exception is raised always use
assertRaisesas a context manager, as shown in line 10 of the above example.
- If a test method completes, the test passes; if it throws an uncaught exception the test has failed.
We write test files to allow them to be run by pytest or to allow them to be run directly from the command line using python.
Whilst pytest provides more flexibility and enhanced reporting when running tests (such as specifying that only certain tests run), it is sometimes expedient to run them using python since there is a faster start time and all the output is visible.
Additionally, although pytest allows
unittest.TestCase.subTest to be used they are treated as normal tests and will fail immediately whereas running with python the sub tests will all be executed before triggering stopping the test.
pytest and its plugins are standard stack EUPS packages and do not have to be installed separately.
All LSST products that are built using scons will execute Python tests using pytest so all tests should be written using it. pytest provides a much richer execution and reporting environment for tests and can be used to run multiple test files together.
The pytest scheme for discovering tests inside Python modules is much more flexible than that provided by
unittest, but LSST test files should not take advantage of that flexibility as it can lead to inconsistency in test reports that depend on the specific test runner, and it is required that an individual test file can be executed by running it directly with python.
In particular, care must be taken not to have free functions that use a
test prefix or non-
TestCase test classes that are named with a
Test prefix in the test files.
When pytest is run by scons full warnings are reported, including
Previously these warnings were hidden in the test output but now they are more obvious, allowing you to fix any problems early.
The tests/SConscript file¶
The behavior of pytest when invoked by scons is controlled by the
At minimum this file should contain the following to enable testing with automated test discovery:
from lsst.sconsUtils import scripts scripts.BasicSConscript.tests(pyList=)
pyList is used to specify which Python test files to run.
Here the empty list is interpreted as “allow pytest to automatically discover tests”
pytest will scan the directory tree itself to find tests and will run them all together using the number of subprocesses matching the
-j argument given to scons.
For this mode to work, all test files must be named
pyList=None (the historical default) is used, the scons
tests target will be used to locate test files using a glob for
*.py in the
This list will then be passed explicitly to the pytest command, bypassing its automatic test discovery.
Automatic test discovery is preferred as this ensures that there is no difference between running the tests with scons and running them with pytest without arguments, and it enables the possibility of adjusting pytest test discovery to add additional testing of all Python files in the package.
If there is pybind wrapper code in
tests/ that must be compiled for the python tests to run (for example, a test C++ library that must be loaded by the python tests), there must be a
BasicSConscript.pybind11() entry before the
BasicSConscript.tests() entry in the
pybind11 come first ensures the necessary code will be compiled before any tests are loaded and run.
Running tests standalone¶
pySingles is an optional argument to the
tests method that can be used for the rare cases where a test must be run standalone and not with other test files in a shared process.
The tests are still run using pytest but executed one at a time without using any multi-process execution.
Use of this should be extremely rare.
base package one test file is used to confirm that the LSST import code is working; this can only be tested if we know that it hasn’t previously been imported as part of another test.
The other reason, so far, to run a test standalone is for test classes that dynamically generate large amounts of test data during the set up phase.
Until it is possible to pin test classes to a particular process with
pytest-xdist, tests such as these interact badly when test methods within the class are allocated to different subprocesses since each subprocess will generate the test files.
This can use significantly more disk and CPU when the test runs, and can even cause Jenkins to fail.
It is important to ensure that any files listed in
pySingles should be named such that they will not be discovered by pytest.
The convention is to name these files
test*.py without the underscore.
Where does the output go?¶
When scons runs any tests, the output from those tests is written to the
tests/.tests directory, and a file is created for each test that is executed.
For the usual case where pytest is running on multiple test files at once, there is a single file created,
pytest-*.out, in that directory, along with an XML file containing the test output in JUnit format.
If a test command fails, that output is renamed to have a
.failed extension and is reported by scons.
For convenience the output from the main pytest run (as opposed to the rare standalone usages) is also written to standard output so it is visible in the log or in the shell along with other scons output.
This section describes some common problems that are encountered when using pytest.
Testing global state¶
to run all files in the
tests directory named
test_*.py. To ensure that the order of test execution does not matter it is useful to sometimes run the tests in reverse order by listing the test files manually:
$ pytest `ls -r tests/test_*.py`
pytest plugins are usually all enabled by default.
In particular, if you install the, otherwise excellent,
pytest-random-order plugin to randomize your tests, this will most likely break your builds as it interacts badly with
pytest-xdist used by scons when
-j is used.
You can install it temporarily for investigative purposes so long as it is uninstalled afterwards.
Test Skipping and Expected Failures¶
When writing tests it is important that tests are skipped using the proper
unittest skipping framework rather than returning from the test early.
unittest supports skipping of individual tests and entire classes using decorators or skip exceptions.
LSST code sometimes raises skip exceptions in
setUpClass class methods.
It is also possible to indicate that a particular test is expected to fail, being reported as an error if the test unexpectedly passes.
Expected failures can be used to write test code that triggers a reported bug before the fix to the bug has been implemented and without causing the continuous integration system to die.
One of the primary advantages of using a modern test runner such as pytest is that it is very easy to generate machine-readable pass/fail/skip/xfail statistics to see how the system is evolving over time, and it is also easy to enable code coverage.
Jenkins now provides test result information.
Enabling additional Pytest options: flake8¶
As described in Code MAY be validated with flake8, Python modules can be configured using the
This configuration is supported by pytest and can be used to enable additional testing or tuning on a per-package basis.
pytest uses the
[tool:pytest] block in the configuration file.
To enable automatic flake8 testing as part of the normal test execution the following can be added to the
[tool:pytest] addopts = --flake8 flake8-ignore = E133 E226 E228 N802 N803 N806 N812 N813 N815 N816 W503
addopts parameter adds additional command-line options to the pytest command when it is run either from the command-line or from scons.
A wrinkle with the configuration of the
pytest-flake8 plugin is that it inherits the
exclude settings from the
[flake8] section of
setup.cfg but you are required to explicitly list the codes to ignore when running within pytest by using the
One advantage of this approach is that you can ignore error codes from specific files such that the unit tests will pass, but running flake8 from the command line will remind you there is an outstanding issue.
This feature should be used sparingly, but can be useful when you wish to enable code linting for the bulk of the project but have some issues preventing full compliance.
For example, at the time of writing this is an extract from the
setup.cfg file for the
[flake8] max-line-length = 110 max-doc-length = 79 ignore = E133, E226, E228, N802, N803, N806, N812, N813, N815, N816, W503 exclude = __init__.py, tests/testLib.py [tool:pytest] addopts = --flake8 flake8-ignore = E133 E226 E228 N802 N803 N806 N812 N813 N815 N816 W503 # These will not be needed when we use numpydoc baseMeasurement.py E266 forcedMeasurement.py E266
Here two files trigger an error because Doxygen syntax sometimes requires non-compliant comment code.
With this configuration each Python file tested by pytest will have flake8 run on it. If scons has not been configured to use pytest in automatic test discovery mode, you will discover that flake8 is only being run on the test files themselves rather than all the Python files in the package.
LSST Utility Test Support Classes¶
lsst.utils.tests provides several helpful functions and classes for writing Python tests that developers should make use of.
- Asserts that floating point scalars and/or arrays are equal within the specified tolerance.
The default tolerance is significantly tighter than the tolerance used by
numpy.testing.assert_almost_equal; if you are replacing either of those methods you may have to specify
atolto prevent failing asserts.
- Asserts that floating point scalars and/or arrays are identically equal.
- Asserts that floating point scalars and/or arrays are not equal.
Additionally, lsst.geom, lsst.afw.geom, and lsst.afw.image provide additional asserts that get loaded into
lsst.utils.tests.TestCase when the associated module is loaded.
These include methods for Geom (SpherePoints, Angles, Pairs, Boxes), and Images, such as:
- Assert that two sphere points (
SpherePoint) are nearly equal (provided by
- Assert that two angles (
Angle) are nearly equal, ignoring wrap differences by default (provided by
- Assert that two planar pairs (e.g.
Extent2D) are nearly equal (provided by
- Assert that two boxes (
Box2I) are nearly equal (provided by
skyToPixelfor two WCS over a rectangular grid of pixel positions (provided by
- Assert that two images are nearly equal, including non-finite values (provided by
- Assert that two masks are equal (provided by
- Assert that two masked images are nearly equal, including non-finite values (provided by
In some cases the test to be executed is a shell script or a compiled binary executable.
In order for the test running environment to be aware of these tests, a Python test file must be present that can be run by pytest.
If none of the tests require special arguments and all the files with the executable bit set are to be run, this can be achieved by copying the file
$UTILS_DIR/tests/test_executables.py to the relevant
The file is reproduced here:
1 2 3 4 5 6 7 8 9 10 11 12
import unittest import lsst.utils.tests class UtilsBinaryTester(lsst.utils.tests.ExecutablesTestCase): pass EXECUTABLES = None UtilsBinaryTester.create_executable_tests(__file__, EXECUTABLES) if __name__ == "__main__": unittest.main()
EXECUTABLES variable can be a tuple containing the names of the executables to be run (relative to the directory containing the test file).
None indicates that the test script should discover the executables in the same directory as that containing the test file.
The call to
create_executable_tests initiates executable discovery and creates a test for each executable that is found.
In some cases an explicit test has to be written either because some precondition has to be met before the test will stand a chance of running or because some arguments have to be passed to the executable.
To support this the
assertExecutable method is available:
def testBinary(self): self.assertExecutable("binary1", args=None, root_dir=os.path.dirname(__file__))
binary1 is the name of the executable relative to the root directory specified in the
root_dir optional argument.
Arguments can be provided to the
args keyword parameter in the form of a sequence of arguments in a list or tuple.
The LSST codebase is currently in transition such that
sconsUtils will run executables
itself as well as running Python test scripts that run executables.
Do not worry about this duplication of test running.
When the codebase has migrated to consistently use the testing scheme described in this section
sconsUtils will be modified to disable the duplicate testing.
File descriptor leak testing¶
This example shows the basic structure of an LSST Python unit test module,
MemoryTestCase (the highlighted lines indicate the leak testing modifications):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
import unittest import lsst.utils.tests class DemoTestCase(lsst.utils.tests.TestCase): """Demo test case.""" def testDemo(self): self.assertNotIn("i", "team") class MemoryTester(lsst.utils.tests.MemoryTestCase): pass def setup_module(module): lsst.utils.tests.init() if __name__ == "__main__": lsst.utils.tests.init() unittest.main()
which ends up running the single specified test plus the two running as part of the leak test:
$ pytest -v test_runner_example.py ============================= test session starts ============================== platform darwin -- Python 3.6.2, pytest-3.2.1, py-1.4.31, pluggy-0.3.1 -- ~/lsstsw/miniconda/bin/python cachedir: .cache rootdir: .../python/examples, inifile: collected 3 items test_runner_example.py::DemoTestCase::testDemo PASSED test_runner_example.py::MemoryTester::testFileDescriptorLeaks <- .../lsstsw/stack/DarwinX86/utils/12.0.rc1+f79d1f7db4/python/lsst/utils/tests.py PASSED =========================== 2 passed in 0.28 seconds ===========================
MemoryTestCase must always be the
final test suite.
For the memory test to function properly the
lsst.utils.tests.init function must be invoked before any of the tests in the class are executed.
Since LSST test scripts are required to run properly from the command-line and when called from within pytest, the
init function has to be in the file twice: once in the setup_module function that is called by pytest whenever a test module is loaded (pytest will not use the
__main__ code path), and also just before the call to
unittest.main call to handle being called with python.
Decorators for iteration¶
lsst.utils.tests.classParameters is a class decorator for generating classes with different combinations of class variables.
This is useful for when the
setUp method generates the object being tested:
placing the decorator on the class allows generating that object with different values.
The decorator takes multiple lists of named parameters (which must have the same length) and iterates over the combinations.
@classParameters(foo=[1, 2], bar=[3, 4]) class MyTestCase(unittest.TestCase): ...
will generate two classes, as if you wrote:
class MyTestCase_1_3(unittest.TestCase): foo = 1 bar = 3 ... class MyTestCase_2_4(unittest.TestCase): foo = 2 bar = 4 ...
Note that the values are embedded in the class name, which allows identification of the particular class in the event of a test failure.
lsst.utils.tests.methodParameters is a method decorator for running a test method with different value combinations.
This is useful for when you want an individual test to iterate over multiple values.
classParameters, the decorator takes multiple lists of named parameters (which must have the same length) and iterates over the combinations.
class MyTestCase(unittest.TestCase): @methodParameters(foo=[1, 2], bar=[3, 4]) def testSomething(self, foo, bar): ...
will run tests:
testSomething(foo=1, bar=3) testSomething(foo=2, bar=4)
Note that the method being decorated must be within a subclass of
unittest.TestCase, since it relies on the existence of the
subTest method for identifying the individual iterations.
This use of
subTest also means that all iterations will be executed, not stopping at the first failure.
pytest.mark.parametrize, which is more complex but offers similar functionality;
however, this module is not yet approved for use.
It is now commonplace for Unicode to be used in Python code and the LSST test cases should reflect this situation. In particular file paths, externally supplied strings and strings originating from third party software packages may well include code points outside of US-ASCII. LSST tests should ensure that these cases are handled by explicitly including strings that include code points outside of this range. For example,
- file paths should be generated that include spaces as well as international characters,
- accented characters should be included for name strings, and
- unit strings should include the µm if appropriate.