You are here: Home ‣ Dive Into Python 3 ‣

Difficulty level: ♦♦♢♢♢

Unit Testing

❝ Certitude is not the test of certainty. We have been cocksure of many things that were not so. ❞
— Oliver Wendell Holmes, Jr.

(Not) Diving In

In this chapter, you’re going to write and debug a set of utility functions to convert to and from Roman numerals. You saw the mechanics of constructing and validating Roman numerals in “Case study: roman numerals”. Now step back and consider what it would take to expand that into a two-way utility.

The rules for Roman numerals lead to a number of interesting observations:

There is only one correct way to represent a particular number as a Roman numeral.
The converse is also true: if a string of characters is a valid Roman numeral, it represents only one number (that is, it can only be interpreted one way).
There is a limited range of numbers that can be expressed as Roman numerals, specifically 1 through 3999. (The Romans did have several ways of expressing larger numbers, for instance by having a bar over a numeral to represent that its normal value should be multiplied by 1000, but you’re not going to deal with that. For the purposes of this chapter, let’s stipulate that Roman numerals go from 1 to 3999.)
There is no way to represent 0 in Roman numerals.
There is no way to represent negative numbers in Roman numerals.
There is no way to represent fractions or non-integer numbers in Roman numerals.

Let’s start mapping out what a roman.py module should do. It will have two main functions, to_roman() and from_roman(). The to_roman() function should take an integer from 1 to 3999 and return the Roman numeral representation as a string…

Stop right there. Now let’s do something a little unexpected: write a test case that checks whether the to_roman() function does what you want it to. You read that right: you’re going to write code that tests code that you haven’t written yet.

This is called unit testing. The set of two conversion functions — to_roman(), and later from_roman() — can be written and tested as a unit, separate from any larger program that imports them. Python has a framework for unit testing, the appropriately-named unittest module.

Unit testing is an important part of an overall testing-centric development strategy. If you write unit tests, it is important to write them early (preferably before writing the code that they test), and to keep them updated as code and requirements change. Unit testing is not a replacement for higher-level functional or system testing, but it is important in all phases of development:

Before writing code, it forces you to detail your requirements in a useful fashion.
While writing code, it keeps you from over-coding. When all the test cases pass, the function is complete.
When refactoring code, it assures you that the new version behaves the same way as the old version.
When maintaining code, it helps you cover your ass when someone comes screaming that your latest change broke their old code. (“But sir, all the unit tests passed when I checked it in...”)
When writing code in a team, it increases confidence that the code you’re about to commit isn’t going to break someone else’s code, because you can run their unit tests first. (I’ve seen this sort of thing in code sprints. A team breaks up the assignment, everybody takes the specs for their task, writes unit tests for it, then shares their unit tests with the rest of the team. That way, nobody goes off too far into developing code that doesn’t play well with others.)

⁂

A Single Question

A test case answers a single question about the code it is testing. A test case should be able to...

...run completely by itself, without any human input. Unit testing is about automation.
...determine by itself whether the function it is testing has passed or failed, without a human interpreting the results.
...run in isolation, separate from any other test cases (even if they test the same functions). Each test case is an island.

Given that, let’s build a test case for the first requirement:

The to_roman() function should return the Roman numeral representation for all integers 1 to 3999.

It is not immediately obvious how this code does… well, anything. It defines a class which has no __init__() method. The class does have another method, but it is never called. The entire script has a __main__ block, but it doesn’t reference the class or its method. But it does do something, I promise.

[download romantest1.py]

import roman1
import unittest

class KnownValues(unittest.TestCase):               ①
    known_values = ( (1, 'I'),
                     (2, 'II'),
                     (3, 'III'),
                     (4, 'IV'),
                     (5, 'V'),
                     (6, 'VI'),
                     (7, 'VII'),
                     (8, 'VIII'),
                     (9, 'IX'),
                     (10, 'X'),
                     (50, 'L'),
                     (100, 'C'),
                     (500, 'D'),
                     (1000, 'M'),
                     (31, 'XXXI'),
                     (148, 'CXLVIII'),
                     (294, 'CCXCIV'),
                     (312, 'CCCXII'),
                     (421, 'CDXXI'),
                     (528, 'DXXVIII'),
                     (621, 'DCXXI'),
                     (782, 'DCCLXXXII'),
                     (870, 'DCCCLXX'),
                     (941, 'CMXLI'),
                     (1043, 'MXLIII'),
                     (1110, 'MCX'),
                     (1226, 'MCCXXVI'),
                     (1301, 'MCCCI'),
                     (1485, 'MCDLXXXV'),
                     (1509, 'MDIX'),
                     (1607, 'MDCVII'),
                     (1754, 'MDCCLIV'),
                     (1832, 'MDCCCXXXII'),
                     (1993, 'MCMXCIII'),
                     (2074, 'MMLXXIV'),
                     (2152, 'MMCLII'),
                     (2212, 'MMCCXII'),
                     (2343, 'MMCCCXLIII'),
                     (2499, 'MMCDXCIX'),
                     (2574, 'MMDLXXIV'),
                     (2646, 'MMDCXLVI'),
                     (2723, 'MMDCCXXIII'),
                     (2892, 'MMDCCCXCII'),
                     (2975, 'MMCMLXXV'),
                     (3051, 'MMMLI'),
                     (3185, 'MMMCLXXXV'),
                     (3250, 'MMMCCL'),
                     (3313, 'MMMCCCXIII'),
                     (3408, 'MMMCDVIII'),
                     (3501, 'MMMDI'),
                     (3610, 'MMMDCX'),
                     (3743, 'MMMDCCXLIII'),
                     (3844, 'MMMDCCCXLIV'),
                     (3888, 'MMMDCCCLXXXVIII'),
                     (3940, 'MMMCMXL'),
                     (3999, 'MMMCMXCIX'))           ②

    def test_to_roman_known_values(self):           ③
        '''to_roman should give known result with known input'''
        for integer, numeral in self.known_values:
            result = roman1.to_roman(integer)       ④
            self.assertEqual(numeral, result)       ⑤

if __name__ == '__main__':
    unittest.main()

To write a test case, first subclass the TestCase class of the unittest module. This class provides many useful methods which you can use in your test case to test specific conditions.
This is a list of integer/numeral pairs that I verified manually. It includes the lowest ten numbers, the highest number, every number that translates to a single-character Roman numeral, and a random sampling of other valid numbers. The point of a unit test is not to test every possible input, but to test a representative sample.
Every individual test is its own method, which must take no parameters and return no value. If the method exits normally without raising an exception, the test is considered passed; if the method raises an exception, the test is considered failed.
Here you call the actual to_roman() function. (Well, the function hasn’t be written yet, but once it is, this is the line that will call it.) Notice that you have now defined the API for the to_roman() function: it must take an integer (the number to convert) and return a string (the Roman numeral representation). If the API is different than that, this test is considered failed. Also notice that you are not trapping any exceptions when you call to_roman(). This is intentional. to_roman() shouldn’t raise an exception when you call it with valid input, and these input values are all valid. If to_roman() raises an exception, this test is considered failed.
Assuming the to_roman() function was defined correctly, called correctly, completed successfully, and returned a value, the last step is to check whether it returned the right value. This is a common question, and the TestCase class provides a method, assertEqual, to check whether two values are equal. If the result returned from to_roman() (result) does not match the known value you were expecting (numeral), assertEqual will raise an exception and the test will fail. If the two values are equal, assertEqual will do nothing. If every value returned from to_roman() matches the known value you expect, assertEqual never raises an exception, so testToRomanKnownValues eventually exits normally, which means to_roman() has passed this test.

Once you have a test case, you can start coding the to_roman() function. First, you should stub it out as an empty function and make sure the tests fail. If the tests succeed before you’ve written any code, you’re doing it wrong — your tests aren’t testing your code at all! Write a test that fails, then code until it passes.

# roman1.py

function to_roman(n):
    '''convert integer to Roman numeral'''
    pass                                   ①

At this stage, you want to define the API of the to_roman() function, but you don’t want to code it yet. (Your test needs to fail first.) To stub it out, use the Python reserved word pass, which does precisely nothing.

Execute romantest1.py on the command line to run the test. If you call it with the -v command-line option, it will give more verbose output so you can see exactly what’s going on as each test case runs. With any luck, your output should look like this:

you@localhost:~$ python3 romantest1.py -v
to_roman should give known result with known input ... FAIL            ①

======================================================================
FAIL: to_roman should give known result with known input
----------------------------------------------------------------------
Traceback (most recent call last):
  File "romantest1.py", line 73, in test_to_roman_known_values
    self.assertEqual(numeral, result)
AssertionError: 'I' != None                                            ②

----------------------------------------------------------------------
Ran 1 test in 0.016s                                                   ③

FAILED (failures=1)                                                    ④

Running the script runs unittest.main(), which runs each test case. Each test case is a method within each class in romantest.py that inherits from unittest.TestCase. For each test case, the unittest module will print out the docstring of the method and whether that test passed or failed. As expected, this test case fails.
For each failed test case, unittest displays the trace information showing exactly what happened. In this case, the call to assertEqual() raised an AssertionError because it was expecting to_roman(1) to return 'I', but it didn’t. (Since there was no explicit return statement, the function returned None, the Python null value.)
After the detail of each test, unittest displays a summary of how many tests were performed and how long it took.
Overall, the unit test failed because at least one test case did not pass. When a test case doesn’t pass, unittest distinguishes between failures and errors. A failure is a call to an assertXYZ method, like assertEqual or assertRaises, that fails because the asserted condition is not true or the expected exception was not raised. An error is any other sort of exception raised in the code you’re testing or the unit test case itself.

Now, finally, you can write the to_roman() function.

[download roman1.py]

roman_numeral_map = (('M',  1000),
                     ('CM', 900),
                     ('D',  500),
                     ('CD', 400),
                     ('C',  100),
                     ('XC', 90),
                     ('L',  50),
                     ('XL', 40),
                     ('X',  10),
                     ('IX', 9),
                     ('V',  5),
                     ('IV', 4),
                     ('I',  1))                 ①

def to_roman(n):
    '''convert integer to Roman numeral'''
    result = ''
    for numeral, integer in roman_numeral_map:
        while n >= integer:                     ②
            result += numeral
            n -= integer
    return result

roman_numeral_map is a tuple of tuples which defines three things: the character representations of the most basic Roman numerals; the order of the Roman numerals (in descending value order, from M all the way down to I); the value of each Roman numeral. Each inner tuple is a pair of (numeral, value). It’s not just single-character Roman numerals; it also defines two-character pairs like CM (“one hundred less than one thousand”). This makes the to_roman() function code simpler.
Here’s where the rich data structure of roman_numeral_map pays off, because you don’t need any special logic to handle the subtraction rule. To convert to Roman numerals, simply iterate through roman_numeral_map looking for the largest integer value less than or equal to the input. Once found, add the Roman numeral representation to the end of the output, subtract the corresponding integer value from the input, lather, rinse, repeat.

If you’re still not clear how the to_roman() function works, add a print() call to the end of the while loop:


while n >= integer:
    result += numeral
    n -= integer
    print('subtracting {0} from input, adding {1} to output'.format(integer, numeral))

With the debug print() statements, the output looks like this:

>>> import roman1
>>> roman1.to_roman(1424)
subtracting 1000 from input, adding M to output
subtracting 400 from input, adding CD to output
subtracting 10 from input, adding X to output
subtracting 10 from input, adding X to output
subtracting 4 from input, adding IV to output
'MCDXXIV'

So the to_roman() function appears to work, at least in this manual spot check. But will it pass the test case you wrote?

you@localhost:~$ python3 romantest1.py -v
to_roman should give known result with known input ... ok

----------------------------------------------------------------------
Ran 1 test in 0.016s

OK

Hooray! The to_roman() function passes the “known values” test case. It’s not comprehensive, but it does put the function through its paces with a variety of inputs, including inputs that produce every single-character Roman numeral, the largest possible input (3999), and the input that produces the longest possible Roman numeral (3888). At this point, you can be reasonably confident that the function works for any good input value you could throw at it.

“Good” input? Hmm. What about bad input?

⁂

“Halt And Catch Fire”

It is not enough to test that functions succeed when given good input; you must also test that they fail when given bad input. And not just any sort of failure; they must fail in the way you expect.

>>> import roman1
>>> roman1.to_roman(4000)
'MMMM'
>>> roman1.to_roman(5000)
'MMMMM'
>>> roman1.to_roman(9000)  ①
'MMMMMMMMM'

That’s definitely not what you wanted — that’s not even a valid Roman numeral! In fact, each of these numbers is outside the range of acceptable input, but the function returns a bogus value anyway. Silently returning bad values is baaaaaaad; if a program is going to fail, it is far better that it fail quickly and noisily. “Halt and catch fire,” as the saying goes. The Pythonic way to halt and catch fire is to raise an exception.

The question to ask yourself is, “How can I express this as a testable requirement?” How’s this for starters:

The to_roman() function should raise an OutOfRangeError when given an integer greater than 3999.

What would that test look like?

[download romantest2.py]

class ToRomanBadInput(unittest.TestCase):                                 ①
    def test_too_large(self):                                             ②
        '''to_roman should fail with large input'''
        self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, 4000)  ③

Like the previous test case, you create a class that inherits from unittest.TestCase. You can have more than one test per class (as you’ll see later in this chapter), but I chose to create a new class here because this test is something different than the last one. We’ll keep all the good input tests together in one class, and all the bad input tests together in another.
Like the previous test case, the test itself is a method of the class, with a name starting with test.
The unittest.TestCase class provides the assertRaises method, which takes the following arguments: the exception you’re expecting, the function you’re testing, and the arguments you’re passing to that function. (If the function you’re testing takes more than one argument, pass them all to assertRaises, in order, and it will pass them right along to the function you’re testing.)

Pay close attention to this last line of code. Instead of calling to_roman() directly and manually checking that it raises a particular exception (by wrapping it in a try...except block), the assertRaises method has encapsulated all of that for us. All you do is tell it what exception you’re expecting (roman2.OutOfRangeError), the function (to_roman()), and the function’s arguments (4000). The assertRaises method takes care of calling to_roman() and checking that it raises roman2.OutOfRangeError.

Also note that you’re passing the to_roman() function itself as an argument; you’re not calling it, and you’re not passing the name of it as a string. Have I mentioned recently how handy it is that everything in Python is an object?

So what happens when you run the test suite with this new test?

you@localhost:~$ python3 romantest2.py -v
to_roman should give known result with known input ... ok
to_roman should fail with large input ... ERROR                         ①

======================================================================
ERROR: to_roman should fail with large input                          
----------------------------------------------------------------------
Traceback (most recent call last):
  File "romantest2.py", line 78, in test_too_large
    self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, 4000)
AttributeError: 'module' object has no attribute 'OutOfRangeError'      ②

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (errors=1)

You should have expected this to fail (since you haven’t written any code to pass it yet), but... it didn’t actually “fail,” it had an “error” instead. This is a subtle but important distinction. A unit test actually has three return values: pass, fail, and error. Pass, of course, means that the test passed — the code did what you expected. “Fail” is what the previous test case did (until you wrote code to make it pass) — it executed the code but the result was not what you expected. “Error” means that the code didn’t even execute properly.
Why didn’t the code execute properly? The traceback tells all. The module you’re testing doesn’t have an exception called OutOfRangeError. Remember, you passed this exception to the assertRaises() method, because it’s the exception you want the function to raise given an out-of-range input. But the exception doesn’t exist, so the call to the assertRaises() method failed. It never got a chance to test the to_roman() function; it didn’t get that far.

To solve this problem, you need to define the OutOfRangeError exception in roman2.py.

class OutOfRangeError(ValueError):  ①
    pass                            ②

Exceptions are classes. An “out of range” error is a kind of value error — the argument value is out of its acceptable range. So this exception inherits from the built-in ValueError exception. This is not strictly necessary (it could just inherit from the base Exception class), but it feels right.
Exceptions don’t actually do anything, but you need at least one line of code to make a class. Calling pass does precisely nothing, but it’s a line of Python code, so that makes it a class.

Now run the test suite again.

you@localhost:~$ python3 romantest2.py -v
to_roman should give known result with known input ... ok
to_roman should fail with large input ... FAIL                          ①

======================================================================
FAIL: to_roman should fail with large input
----------------------------------------------------------------------
Traceback (most recent call last):
  File "romantest2.py", line 78, in test_too_large
    self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, 4000)
AssertionError: OutOfRangeError not raised by to_roman                 ②

----------------------------------------------------------------------
Ran 2 tests in 0.016s

FAILED (failures=1)

The new test is still not passing, but it’s not returning an error either. Instead, the test is failing. That’s progress! It means the call to the assertRaises() method succeeded this time, and the unit test framework actually tested the to_roman() function.
Of course, the to_roman() function isn’t raising the OutOfRangeError exception you just defined, because you haven’t told it to do that yet. That’s excellent news! It means this is a valid test case — it fails before you write the code to make it pass.

Now you can write the code to make this test pass.

[download roman2.py]

def to_roman(n):
    '''convert integer to Roman numeral'''
    if n > 3999:
        raise OutOfRangeError('number out of range (must be less than 3999)')  ①

    result = ''
    for numeral, integer in roman_numeral_map:
        while n >= integer:
            result += numeral
            n -= integer
    return result

This is straightforward: if the given input (n) is greater than 3999, raise an OutOfRangeError exception. The unit test does not check the human-readable string that accompanies the exception, although you could write another test that did check it (but watch out for internationalization issues for strings that vary by the user’s language or environment).

Does this make the test pass? Let’s find out.

you@localhost:~$ python3 romantest2.py -v
to_roman should give known result with known input ... ok
to_roman should fail with large input ... ok                            ①

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

Hooray! Both tests pass. Because you worked iteratively, bouncing back and forth between testing and coding, you can be sure that the two lines of code you just wrote were the cause of that one test going from “fail” to “pass.” That kind of confidence doesn’t come cheap, but it will pay for itself over the lifetime of your code.

⁂

More Halting, More Fire

Along with testing numbers that are too large, you need to test numbers that are too small. As we noted in our functional requirements, Roman numerals cannot express 0 or negative numbers.

>>> import roman2
>>> roman2.to_roman(0)
''
>>> roman2.to_roman(-1)
''

Well that’s not good. Let’s add tests for each of these conditions.

[download romantest3.py]

class ToRomanBadInput(unittest.TestCase):
    def test_too_large(self):
        '''to_roman should fail with large input'''
        self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, 4000)  ①

    def test_zero(self):
        '''to_roman should fail with 0 input'''
        self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, 0)     ②

    def test_negative(self):
        '''to_roman should fail with negative input'''
        self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, -1)    ③

The test_too_large() method has not changed since the previous step. I’m including it here to show where the new code fits.
Here’s a new test: the test_zero() method. Like the test_too_large() method, it tells the assertRaises() method defined in unittest.TestCase to call our to_roman() function with a parameter of 0, and check that it raises the appropriate exception, OutOfRangeError.
The test_negative() method is almost identical, except it passes -1 to the to_roman() function. If either of these new tests does not raise an OutOfRangeError (either because the function returns an actual value, or because it raises some other exception), the test is considered failed.

Now check that the tests fail:

you@localhost:~$ python3 romantest3.py -v
to_roman should give known result with known input ... ok
to_roman should fail with negative input ... FAIL
to_roman should fail with large input ... ok
to_roman should fail with 0 input ... FAIL

======================================================================
FAIL: to_roman should fail with negative input
----------------------------------------------------------------------
Traceback (most recent call last):
  File "romantest3.py", line 86, in test_negative
    self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, -1)
AssertionError: OutOfRangeError not raised by to_roman

======================================================================
FAIL: to_roman should fail with 0 input
----------------------------------------------------------------------
Traceback (most recent call last):
  File "romantest3.py", line 82, in test_zero
    self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, 0)
AssertionError: OutOfRangeError not raised by to_roman

----------------------------------------------------------------------
Ran 4 tests in 0.000s

FAILED (failures=2)

Excellent. Both tests failed, as expected. Now let’s switch over to the code and see what we can do to make them pass.

[download roman3.py]

def to_roman(n):
    '''convert integer to Roman numeral'''
    if not (0 < n < 4000):                                              ①
        raise OutOfRangeError('number out of range (must be 0..3999)')  ②

    result = ''
    for numeral, integer in roman_numeral_map:
        while n >= integer:
            result += numeral
            n -= integer
    return result

This is a nice Pythonic shortcut: multiple comparisons at once. This is equivalent to if not ((0 < n) and (n < 4000)), but it’s much easier to read. This one line of code should catch inputs that are too large, negative, or zero.
If you change your conditions, make sure to update your human-readable error strings to match. The unittest framework won’t care, but it’ll make it difficult to do manual debugging if your code is throwing incorrectly-described exceptions.

I could show you a whole series of unrelated examples to show that the multiple-comparisons-at-once shortcut works, but instead I’ll just run the unit tests and prove it.

you@localhost:~$ python3 romantest3.py -v
to_roman should give known result with known input ... ok
to_roman should fail with negative input ... ok
to_roman should fail with large input ... ok
to_roman should fail with 0 input ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.016s

OK

⁂

And One More Thing…

There was one more functional requirement for converting numbers to Roman numerals: dealing with non-integers.

>>> import roman3
>>> roman3.to_roman(0.5)  ①
''
>>> roman3.to_roman(1.5)  ②
'I'

Oh, that’s bad.
Oh, that’s even worse. Both of these cases should raise an exception. Instead, they give bogus results.

Testing for non-integers is not difficult. First, define a NonIntegerError exception.

# roman4.py
class OutOfRangeError(ValueError): pass
class NotIntegerError(ValueError): pass

Next, write a test case that checks for the NonIntegerError exception.

class ToRomanBadInput(unittest.TestCase):
    .
    .
    .
    def test_non_integer(self):
        '''to_roman should fail with non-integer input'''
        self.assertRaises(roman4.NotIntegerError, roman4.to_roman, 0.5)

Now check that the test fails properly.

you@localhost:~$ python3 romantest4.py -v
to_roman should give known result with known input ... ok
to_roman should fail with negative input ... ok
to_roman should fail with non-integer input ... FAIL
to_roman should fail with large input ... ok
to_roman should fail with 0 input ... ok

======================================================================
FAIL: to_roman should fail with non-integer input
----------------------------------------------------------------------
Traceback (most recent call last):
  File "romantest4.py", line 90, in test_non_integer
    self.assertRaises(roman4.NotIntegerError, roman4.to_roman, 0.5)
AssertionError: NotIntegerError not raised by to_roman

----------------------------------------------------------------------
Ran 5 tests in 0.000s

FAILED (failures=1)

Write the code that makes the test pass.

def to_roman(n):
    '''convert integer to Roman numeral'''
    if not (0 < n < 4000):
        raise OutOfRangeError('number out of range (must be 0..3999)')
    if not isinstance(n, int):                                          ①
        raise NotIntegerError('non-integers can not be converted')      ②

    result = ''
    for numeral, integer in roman_numeral_map:
        while n >= integer:
            result += numeral
            n -= integer
    return result

The built-in isinstance() function tests whether a variable is a particular type (or, technically, any descendant type).
If the argument n is not an int, raise our newly minted NotIntegerError exception.

Finally, check that the code does indeed make the test pass.

you@localhost:~$ python3 romantest4.py -v
to_roman should give known result with known input ... ok
to_roman should fail with negative input ... ok
to_roman should fail with non-integer input ... ok
to_roman should fail with large input ... ok
to_roman should fail with 0 input ... ok

----------------------------------------------------------------------
Ran 5 tests in 0.000s

OK

Now stop coding.

☜ ☞