Generating dynamic Python tests using metaclasses

One common testing anti-pattern is a single testcase that loops over multiple yet independent inputs:

from django.test import TestCase

class MyTests(TestCase):
    def test_all(self):
        for x in (
            'foo',
            'bar',
            'baz',
            # etc.
        ):
            self.assertEqual(test(x), None)

Whilst this code style typically occurs in an honourable attempt to avoid a DRY violation, these tests:

1. Can only report the first failure
2. Prevent individual inputs from being tested independently
3. Are typically slower than their neighbours, or a performance hotspot generally
4. Do not allow for parallel computation, a feature recently added to Django 1.9

(Note that whilst foo, bar, etc. are defined statically above for simplicity, the values could be determined dynamically by, for example, iterating over the filesystem.)

If you have such tests, consider splitting them out using a metaclass like so:

class MyTestsMeta(type):
    def __new__(cls, name, bases, attrs):
        for x in (
            'foo',
            'bar',
            'baz',
        ):
            attrs['test_%s' % x] = cls.gen(x)

        return super(MyTestsMeta, cls).__new__(cls, name, bases, attrs)

    @classmethod
    def gen(cls, x):
        # Return a testcase that tests ``x``.
        def fn(self):
            self.assertEqual(test(x), None)
        return fn

class MyTests(TestCase):
    __metaclass__ = MyTestsMeta

This has the effect of replacing the single testcase with individual test_foo, test_bar & test_baz testcases. Each test can then be run separately:

$ ./manage.py test myproject.myapp.tests.MyTests.test_baz
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.039s

OK
Destroying test database for alias 'default'...

... or we can test them all in parallel:

$ ./manage.py test myproject.myapp.tests.MyTests --parallel
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 3 tests in 0.065s

OK
Destroying test database for alias 'default'...

You must ensure that the tests have unique names to avoid cases masking each other. In the above example we could simply use the input string itself, but if you have no obvious candidate you could try using Python's enumerate method to generate a unique, if somewhat opaque, suffix:

for idx, x in enumerate((
    'foo',
    'bar',
    'baz',
)):
    attrs['test_%d' % idx] = cls.gen(x)

One alternative approach to a metaclass is to generate the test methods and use setattr, to bind them to the TestCase class. However, using a metaclass:

1. Is cleaner and/or more "Pythonic"
2. Avoids a number of subtle pitfalls with parameter binding
3. Prevents your TestCase class from being polluted with loop variables, etc.
4. Can be composed or abstracted into reusable testing components

Note that you can still use setUp and all the other unittest.TestCase and django.test.TestCase methods as before:

from .utils import MyBaseTestCase # custom superclass

class MyTestsMeta(type):
    # <snip>

    @classmethod
    def gen(cls, x):
        def fn(self):
            self.assertRedirects(...)
        return fn

class MyTests(MyBaseTestCase):
    __metaclass__ = MyTestsMeta

    def setUp(self):
        super(MyTests, self).setUp()
        # Code here is run before every test

    def test_other(self):
        # Test some other functionality here

UPDATE: Stu Cox asked: Do you even need the metaclass, or could you do this with __new__ straight on the TestCase? Curiously, unittest does not initialise classes in the typical way if you do not explicitly define at least one test_ method.

You can subscribe to new posts via email or RSS.