Memory Test¶

Every LSST test file includes the lsst.utils.tests.MemoryTestCase test for leaked resources in the C++ code. This is the final test run from within each file and it relies on the lsst.utils.tests.init() method being called before any of the tests start. In the current system this reset occurs when suite() is called before being passed to the test runner. Pytest test discovery works by finding all the tests to be invoked first, and then running them so pytest must be configured to reset the memory leak counter before test classes are executed. This can be done by adding the following to the top of the test file:

def setup_module(module):
    lsst.utils.tests.init()

Making the memory test itself available to pytest can be achieved by adding it explicitly at the end of the test file as the final test class:

class MyMemoryTestCase(lsst.utils.tests.MemoryTestCase):
    pass

This will then be run once the other tests in that file have been run. As an additional protection, when the tests complete the leak counter is reset to allow new test files to start from a blank slate. It is recommended to be explicit and call lsst.utils.tests.init() in the setup_module() function.

Testing Binaries¶

To enable a switch to pytest for all LSST testing, executable test binaries that have been built by the sconsUtils test target must be tested from a Python wrapper rather than being executed by sconsUtils directly. The lsst.utils.tests package provides a simple means of doing this by providing a base test case class that can automatically discover binary executables and create a test for each one. To enable this feature copy $UTILS_DIR/tests/testExecutables.py to the tests directory of your application. In many cases this will just work, but it is also possible to restrict the testing to an explicit list of binaries. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import unittest
import lsst.utils.tests


class UtilsBinaryTester(lsst.utils.tests.ExecutablesTestCase):
    pass

EXECUTABLES = ("binary1", "binary2")
UtilsBinaryTester.create_executable_tests(__file__, EXECUTABLES)

if __name__ == "__main__":
    unittest.main()

by explicitly listing the test executables in a tuple. The tuple of executables can contain any executable that can be run from the shell which will return zero exit status if it works and non-zero if it fails. The output from the test executable is captured.

In some cases, explicit tests should be written for each executable, such as when a test should be skipped if some precondition is not met or if some command-line arguments need to be given. In that case use can be made of the assertExecutable() method available in the ExecutablesTestCase class:

def testBinary(self):
    self.assertExecutable("binary1", args=None,
                          root_dir=os.path.dirname(__file__))

The optional second argument is required to allow the test to run regardless of the working directory from which the test is invoked. In this case binary1 would be located relative to the testing Python wrapper. The args optional keyword parameter can be used to supply a list of arguments to the executable.

Cleaning up persistent state¶

Pytest is a test runner that is designed to be able to run tests from multiple files simultaneously. This means that rather than each test file running in a separate process, pytest may run all of them sequentially within a single process. This means that any persistent state defined in one test file must be reset so that it does not contaminate subsequent tests. Currently, large test suites, such as those in afw and meas_astrom can give different answers depending on the order of the test files given to py.test. The pytest test runner integrated into sconsUtils will be designed explicitly to not guarantee the order in which test files will be executed. When testing after migration to pytest please ensure that the tests run in a single process:

$ py.test tests/*.py

and that the test file order does not matter:

$ py.test `ls -r tests/*.py`

In many cases this will trigger unexpected failures in tests that work standalone or even in conjunction with some, but not all, of the other tests. These problems can be due to tests setting global state, in particular environment variables, and not resetting it, or tests not correctly releasing resources (for example running out of file handles). In extreme cases this could be indicative of memory corruption issues in the C++ code.

Remove the suites¶

Once the tests have been modified to support standard test discovery the suite handling boilerplate can be removed and replaced with:

if __name__ == "__main__":
    lsst.utils.tests.init()
    unittest.main()

Where the lsst.utils.tests.init() is only needed for tests that include the MemoryTestCase. It will then be possible to run the tests using python directly, but the recommendation is that tests should be executed by py.test if at all possible, to emulate the CI environment.

Putting it all together¶

With all the changes described above your test files should be simpler. An example test file should look something like this:

import unittest
import lsst.utils.tests


class DemoTestCase(lsst.utils.tests.TestCase):
    """Demo test case."""

    def testDemo(self):
        self.assertNotIn("i", "team")


class MemoryTester(lsst.utils.tests.MemoryTestCase):
    pass


def setup_module(module):
    lsst.utils.tests.init()


if __name__ == "__main__":
    lsst.utils.tests.init()
    unittest.main()

The Python packages are loaded at the top and then all the tests are defined. At the end there is code to initialize the memory leak tester in py.test and when running from the command line. If you are not using any LSST C++ code then the test will simply have the unittest.main() call at the end.