This document explains how to migrate from old-style LSST Python tests that use a
suite -based approach and custom test runner from the
If you are interesting in writing new tests or learning the current policies for Python testing please refer to the LSST developer guide.
The standard for Python unit testing in LSST is to use the
Older versions of the coding standard did not use the builtin
unittest test discovery mechanism but instead the convention was to specify a list of explicit test classes by specifying a
This document explains how to migrate the tests from this legacy approach to a scheme that supports the
py.test test runner.
A fully worked example describing the new scheme can be found later in this document.
If your test file has the following boilerplate in it, you should read this document to learn how to convert it to be
py.test - compatible:
import lsst.utils.tests <Test classes here subclasses of lsst.utils.tests.TestCase or unittest.TestCase> def suite(): """Returns a suite containing all the test cases in this module.""" lsst.utils.tests.init() suites =  suites += unittest.makeSuite(FootprintSetTestCase) suites += unittest.makeSuite(PeaksInFootprintsTestCase) suites += unittest.makeSuite(lsst.utils.tests.MemoryTestCase) return unittest.TestSuite(suites) def run(shouldExit=False): """Run the tests""" lsst.utils.tests.run(suite(), shouldExit) if __name__ == "__main__": run(True)
Tests are set up like this to allow tests to be disabled by commenting out a single line, and also to allow the memory test to be included in every test file.
The memory test case is used to check for memory leaks in the C++ code and the
lsst.utils.tests.init() call is there to initialize the memory tester.
run() method ensures that tests can exit with bad exit status if they fail, making it possible for
sconsUtils to determine whether a particular test file passed or failed.
sconsUtils testing works as follows: the
tests directory is scanned looking for executable binaries and Python scripts; binaries are executed directly and Python scripts are executed using the
python binary in the path; the exit status from this call determines whether the test passes or fails; the output from the tests is redirected to a file in a
.tests subdirectory using the test name as the name of the output file and if the test failed
.failed is appended to the filename; if
scons finds any
.failed files the
test target itself fails and
scons aborts the build.
unittest provides a reasonable environment for writing unit tests it does not make it easy for the software running the tests to obtain pass/fail/skip statistics when they are run, or report the duration of individual tests.
To do this a test runner environment is required such as nose or pytest.
Following the lead of Astropy LSST is migrating away from a simple exit status approach to test running and switching to use pytest.
Pytest provides a very flexible testing environment and is much simpler to use than
At this time we are not proposing that
unittest be dropped to make use of these testing simplifications and new tests should not be written relying solely on pytest.
We are switching to using pytest as the test runner.
Pytest can use the standard
unittest test discovery mechanism which results in the
suite boilerplate described above being bypassed when the test file is executed via
The LSST test files must therefore be vetted to ensure that the tests function with automated test discovery and result in all tests being executed.
In many cases the tests will run properly with
py.test with no modifications, although the memory test case will not be executed.
unittest test discovery is based on the presence of classes inherited from
unittest.TestCase, including subclasses of
lsst.utils.tests.TestCase, containing methods that begin with the string
The simplest way to determine what changes are required is to run the test once using
python and once using
$ python tests/statistics.py ......................... ---------------------------------------------------------------------- Ran 26 tests in 0.149s OK $ py.test tests/statistics.py ============================= test session starts ============================== platform darwin -- Python 2.7.11, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 rootdir: /Users/timj/work/lsstsw/src/afw/tests, inifile: collected 24 items tests/statistics.py ........................ ========================== 24 passed in 0.64 seconds ===========================
Here it is clear that 24 tests ran instead of 26.
This is expected because the, currently 2, resource leak test cases will not be “discovered” because they are hidden in the
If the test counts differ by more than one, or if they match, then more work may be required. Possible issues are:
- A routine exists in the file with a
testprefix but is not itself a test.
suite()does not list all the test classes that are in the file.
- A class is named with a
Testprefix (but does not inherit from
pytestmay issue warnings if no tests can be found or if an
__init__method is present (usually these are not meant to be treated as tests but
pytestdoes not know that; rename the class to fix this).
- Tests in a base class are being run inadvertently.
The issue of base classes needs some explanation.
In some test files, in order to avoid code duplication a base class is used defining the tests and then subclasses are used which override certain parameters.
Only the subclasses will be listed in the
If the base class inherits from
unittest.TestCase the tests in the base class will be executed even though it is likely that conditions are such that the tests will fail.
One solution to this problem is for the base class not to inherit from
unittest.TestCase and to have the test subclasses themselves inherit from both the base class and the test base class.
It is also important that tests are skipped explicitly using the
unittest skipping feature (such as raising
unittest.SkipTest in the
setup() phase, or using the
unittest.skipUnless() decorator) rather than the test not being run without comment (which can be interpreted as a pass).
Skipping statistics are very important and large numbers of skipping tests can be indicative of a wider issue with the test suite.
One final comment is that the tests executed by pytest will not be in the same namespace as when they are run from the command line with Python.
If tests rely on knowing their own namespace they should use
__name__ rather than
Every LSST test file includes the
lsst.utils.tests.MemoryTestCase test for leaked resources in the C++ code.
This is the final test run from within each file and it relies on the
lsst.utils.tests.init() method being called before any of the tests start.
In the current system this reset occurs when
suite() is called before being passed to the test runner.
Pytest test discovery works by finding all the tests to be invoked first, and then running them so pytest must be configured to reset the memory leak counter before test classes are executed.
This can be done by adding the following to the top of the test file:
def setup_module(module): lsst.utils.tests.init()
Making the memory test itself available to pytest can be achieved by adding it explicitly at the end of the test file as the final test class:
class MyMemoryTestCase(lsst.utils.tests.MemoryTestCase): pass
This will then be run once the other tests in that file have been run.
As an additional protection, when the tests complete the leak counter is reset to allow new test files to start from a blank slate.
It is recommended to be explicit and call
lsst.utils.tests.init() in the
To enable a switch to pytest for all LSST testing, executable test binaries that have been built by the
sconsUtils test target must be tested from a Python wrapper rather than being executed by
lsst.utils.tests package provides a simple means of doing this by providing a base test case class that can automatically discover binary executables and create a test for each one.
To enable this feature copy
$UTILS_DIR/tests/testExecutables.py to the
tests directory of your application.
In many cases this will just work, but it is also possible to restrict the testing to an explicit list of binaries.
1 2 3 4 5 6 7 8 9 10 11 12
import unittest import lsst.utils.tests class UtilsBinaryTester(lsst.utils.tests.ExecutablesTestCase): pass EXECUTABLES = ("binary1", "binary2") UtilsBinaryTester.create_executable_tests(__file__, EXECUTABLES) if __name__ == "__main__": unittest.main()
by explicitly listing the test executables in a tuple. The tuple of executables can contain any executable that can be run from the shell which will return zero exit status if it works and non-zero if it fails. The output from the test executable is captured.
In some cases, explicit tests should be written for each executable, such as when a test should be skipped if some precondition is not met or if some command-line arguments need to be given.
In that case use can be made of the
assertExecutable() method available in the
def testBinary(self): self.assertExecutable("binary1", args=None, root_dir=os.path.dirname(__file__))
The optional second argument is required to allow the test to run regardless of the working directory from which the test is invoked.
In this case
binary1 would be located relative to the testing Python wrapper.
args optional keyword parameter can be used to supply a list of arguments to the executable.
Cleaning up persistent state¶
Pytest is a test runner that is designed to be able to run tests from multiple files simultaneously.
This means that rather than each test file running in a separate process, pytest may run all of them sequentially within a single process.
This means that any persistent state defined in one test file must be reset so that it does not contaminate subsequent tests.
Currently, large test suites, such as those in
meas_astrom can give different answers depending on the order of the test files given to
The pytest test runner integrated into
sconsUtils will be designed explicitly to not guarantee the order in which test files will be executed.
When testing after migration to pytest please ensure that the tests run in a single process:
$ py.test tests/*.py
and that the test file order does not matter:
$ py.test `ls -r tests/*.py`
In many cases this will trigger unexpected failures in tests that work standalone or even in conjunction with some, but not all, of the other tests. These problems can be due to tests setting global state, in particular environment variables, and not resetting it, or tests not correctly releasing resources (for example running out of file handles). In extreme cases this could be indicative of memory corruption issues in the C++ code.
Remove the suites¶
Once the tests have been modified to support standard test discovery the suite handling boilerplate can be removed and replaced with:
if __name__ == "__main__": lsst.utils.tests.init() unittest.main()
lsst.utils.tests.init() is only needed for tests that include the
It will then be possible to run the tests using
python directly, but the recommendation is that tests should be executed by
py.test if at all possible, to emulate the CI environment.
Putting it all together¶
With all the changes described above your test files should be simpler. An example test file should look something like this:
import unittest import lsst.utils.tests class DemoTestCase(lsst.utils.tests.TestCase): """Demo test case.""" def testDemo(self): self.assertNotIn("i", "team") class MemoryTester(lsst.utils.tests.MemoryTestCase): pass def setup_module(module): lsst.utils.tests.init() if __name__ == "__main__": lsst.utils.tests.init() unittest.main()
The Python packages are loaded at the top and then all the tests are defined.
At the end there is code to initialize the memory leak tester in
py.test and when running from the command line.
If you are not using any LSST C++ code then the test will simply have the
unittest.main() call at the end.