Fizz Buzz Testing

By | 2013-08-09
0 Comment

Unit testing is the name we give to the idea of writing code that tests our application code automatically. Essentially, its job is to exercise each piece of your application in isolation, and ensure that any data dependent bugs you might introduce accidentally, never make it into production releases. Personally, I also find it makes programming more enjoyable; it takes your large application and lets you play with just a little piece of it – programming becomes more like a little bite sized puzzle. That being said, it’s not always easy to write test code, especially in large intricate libraries you will find you have to start faking parts of the system to feed to your test code. I’m not going to go into that today.

I used to do my unit testing like this:

#ifdef UNITTEST
int main()
{
    try {
        /* tests */
    } catch( exception &e ) {
        log() << e.what() << endl;
        return 255;
    }
}
#endif

This is certainly workable; and has the advantage of being very simple. However, I recently got pointed at cppunit by a posting on stackoverflow, and I like the structure it imposes on tests. After a bit of reading and messing I got enough boilerplate code that I find I’d rather use cppunit for unit tests, over my simple method. It’s got the advantage that I can create a single unit test executable covering all modules, or test individual modules as I wish. It’s got some nice output formats and allows easy separation and isolation of individual tests.

Now, I could just give you my boilerplate template, but I thought it would be nice to actually show you it in use on something real. So, let’s attack the problem that 199 out of 200 applicants for developer positions can’t do (incidentally that doesn’t mean if you can do this you’re in the top 0.5%) and unit test the result with cppunit. That problem is FizzBuzz.

Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.

First let me say this: a lot of good programmers, who can do this test, for a bit of fun coded up elaborate and clever solutions. For fun, that’s fine, but clever solutions make for poor, unreadable, code. I’ll not bother with anything clever then, I’ll just do a work horse solution.

void FizzBuzz() {
    for(int i = 0; i <= 100; i++) {
        bool f = (i % 3) == 0;
        bool b = (i % 5) == 0;
        if(f)
            cout << "Fizz";
        if(b)
            cout << "Buzz";
        if(!b && !f)
            cout << i;
        cout << endl;
    }
}

This solution is rubbish. It’s not rubbish because it doesn’t work, or because it’s not a clever solution, it’s rubbish because to test it we have to run it then manually eyeball 100 lines of output – worse, what if it was 1000 lines of output? That means it’s difficult to test.

That difficulty represents a smell from our code. Well written code will be testable. Let’s first split it into something that’s testable by separating our maths from our loop.

enum class eFizzBuzz {
    Null,
    Fizz,
    Buzz,
    FizzBuzz,
};

ostream &operator<<(ostream &s, eFizzBuzz fb)
{
    switch(fb) {
        case eFizzBuzz::Fizz:
            return s << "Fizz";
        case eFizzBuzz::Buzz:
            return s << "Buzz";
        case eFizzBuzz::FizzBuzz:
            return s << "FizzBuzz";
        case eFizzBuzz::Null:
            return s << "Null";
    }
}

eFizzBuzz FizzBuzz(int n)
{
    bool f = (i % 3) == 0;
    bool b = (i % 5) == 0;
    if(f && b)
        return eFizzBuzz::FizzBuzz;
    if(f)
        return eFizzBuzz::Fizz;
    if(b)
        return eFizzBuzz::Buzz;
    return eFizzBuzz::Null;
}

void FizzBuzz() {
    for(int i = 0; i <= 100; i++) {
        auto fb = FizzBuzz(i);
        if(fb == eFizzBuzz::Null) {
            cout << i << endl;
        } else {
            cout << fb << endl;
        }
    }
}

A lot more code; and you might say “how is more code better, if it does the same job?”. The answer is that it does the same job only superficially. Writing testable code has forced us to write cleaner, more general code. We’ve separated our algorithm from our application, and we’ve also generalised our output.

This is now in a form that we can write a unit test. Let’s do so. Make sure you have the cppunit library available (apt-get install libcppunit-dev on Debian).

First the test case, note there is absolutely no requirement that this go in the same module as the code it’s testing. I like to do that and keep my test code wrapped in #ifdef UNITTEST, but you can do as you wish.

#include <cppunit/extensions/HelperMacros.h>

class FizzBuzzTest : public CppUnit::TestFixture
{
  public:
    void setUp(void) {}
    void tearDown(void) {}

    void test_fizzbuzz() {
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Fizz, FizzBuzz(3));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Buzz, FizzBuzz(5));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::FizzBuzz, FizzBuzz(15));
    }

    // -----------------------

    // HelperMacros to manufacture `static CppUnit::Test *suite()`
    CPPUNIT_TEST_SUITE(FizzBuzzTest);
    CPPUNIT_TEST(test_fizzbuzz);
    CPPUNIT_TEST_SUITE_END();
};
// Add manufactured FizzBuzzTest::suite() to the global test
// register
CPPUNIT_TEST_SUITE_REGISTRATION(FizzBuzzTest);

cppunit’s helper macros, while slightly unnerving looking, are straight-forward to use. Everything in the top half is just a standard class derived from CppUnit::TestFixture, setUp() and tearDown() are called before and after each test – these supply a clean environment for each test to run in; we’re not using them here. test_fizzbuzz() is arbitrarily named, and is added to the list of tests (cppunit calls it the test suite) in this class with the CPPUNIT_TEST(test_fizzbuzz) line. The test suite represented by this class is then scheduled for test with CPPUNIT_TEST_SUITE_REGISTRATION(FizzBuzzTest).

Now we just need the code to run the tests.

#include <cppunit/ui/text/TestRunner.h>
#include <cppunit/TextOutputter.h>
#include <iostream>
#include <stdexcept>
int main()
{
    // --- Boilerplate cppunit code
    // Set up runner to run all test in registry
    CppUnit::TextUi::TestRunner runner;
    runner.addTest( CppUnit::TestFactoryRegistry::getRegistry().makeTest() );
    // Redirect output to clog,
    runner.setOutputter(new CppUnit::TextOutputter(&runner.result(), std::clog));
    // Run all and give success indiciation
    return runner.run() ? 0 : 1;
}

Thanks to the use of the test registry there is absolutely no code in this block specific to our tests – you can use this without change in every single project you have. It runs every test suite registered, outputs to std::clog (stderr) and returns 0 for test success, and 1 for test failure – this is typical for command line programs.

The meat of our unit test is in test_fizzbuzz(), and we are free to do anything we wish. The most useful thing we can do though is to use the cppunit assertions to perform some tests.

void test_fizzbuzz() {
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Fizz, FizzBuzz(3));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Buzz, FizzBuzz(5));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::FizzBuzz, FizzBuzz(15));
}

CPPUNIT_ASSERT_EQUAL() takes two arguments, expected, and result. cppunit expects to be able to output both of these to a stream should the test fail, so our operator<<() implementation for eFizzBuzz types is a necessity here.

Here’s my compile and build:

clang++ -ggdb3 -O2 -Wall -Wextra -std=c++11 -Wfatal-errors   -c -o fizzbuzz.o fizzbuzz.cc
clang++ -pthread -rdynamic -lcppunit   -o fizzbuzz fizzbuzz.o
.


OK (1 tests)

cppunit output is terse when there are no failures. It’s expected that there will be a great many tests for a project, and it’s far better to have only the failures show any verbose output. The “.” indicates a test ran, and the summary tells us everything was okay with 1 tests run.

Our tests as they stand are not very good. We’ve made the classic mistake of only testing for the match cases, not for the non-match cases.

void test_fizzbuzz() {
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(0));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(1));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(2));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Fizz, FizzBuzz(3));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(4));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Buzz, FizzBuzz(5));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(6));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(7));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(8));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(9));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(10));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(11));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(12));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(13));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(14));
    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::FizzBuzz, FizzBuzz(15));
}

This might look odd, as programmers we’re used to turning sequences like this into a loop; but remember what we’re testing is pretty fundamental – whether we’ve used C++’s modulo operator correctly. It would be a pretty bad test then if we used the very thing we were testing in our test – that’s like saying “if(testresult == testresult)”. Our only option then is to hard code tests without using “%” and a loop counter. This is the equivalent of us eyeballing every line of output from the first example.

Let’s run our test again…

clang++ -ggdb3 -O2 -Wall -Wextra -std=c++11 -Wfatal-errors   -c -o fizzbuzz.o fizzbuzz.cc
clang++ -pthread -rdynamic -lcppunit   -o fizzbuzz fizzbuzz.o
rm fizzbuzz.o
.F


!!!FAILURES!!!
Test Results:
Run:  1   Failures: 1   Errors: 0


1) test: FizzBuzzTest::test_fizzbuzz (F) line: 97 fizzbuzz.cc
equality assertion failed
- Expected: Null
- Actual  : FizzBuzz

Very useful, we’ve already caught something. A test failure when “null” was expected but the result was “fizzbuzz” (we’re seeing the advantage of our operator<<() in this output). You can’t see my line numbers of course, but this is the faulty line:

    CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(0));

Spotted the fault? It’s actually a fault in our test. Zero is divisible by 3 and by 5 and by everything in fact. The expected output is therefore “FizzBuzz” not “Null”.

We’ve also only tested from 0 to 15 so far; we should do more:

void test_fizzbuzz() {
    for( int i = 0; i < 1000; i += 15 ) {
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::FizzBuzz, FizzBuzz(i + 0));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 1));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 2));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Fizz, FizzBuzz(i + 3));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 4));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Buzz, FizzBuzz(i + 5));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 6));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 7));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 8));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 9));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 10));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 11));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 12));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 13));
        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 14));
    }
}

Rerun:

!!!FAILURES!!!
Test Results:
Run:  1   Failures: 1   Errors: 0


1) test: FizzBuzzTest::test_fizzbuzz (F) line: 104 fizzbuzz.cc
equality assertion failed
- Expected: Null
- Actual  : Fizz

Line 104:

        CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 6));

Another test case fault; 6 is divisible by 3. Similarly 9 and 12. Similarly 10 is divisible by 5. Our test case has listed them all as “Null”.

    void test_fizzbuzz() {
        for( int i = 0; i < 100*15; i += 15 ) {
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::FizzBuzz, FizzBuzz(i + 0));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 1));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 2));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Fizz, FizzBuzz(i + 3));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 4));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Buzz, FizzBuzz(i + 5));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Fizz, FizzBuzz(i + 6));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 7));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 8));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Fizz, FizzBuzz(i + 9));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Buzz, FizzBuzz(i + 10));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 11));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Fizz, FizzBuzz(i + 12));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 13));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(i + 14));
        }
    }

This will give us zero errors finally. We can be pretty confident in our maths now. Or can we? What about the edge cases? We’ve done zero, but what about int_max? What about negative numbers? Remember when you’re testing, you’re trying to find faults, you should test for every circumstance – overflow, underflow, off-by-one errors, sign errors. I’ll leave those for you (or you can check out my cpp11 repository and look at the fizzbuzz example). The limit test is interesting though because it shows the use of C++’s numeric_limits to check the boundary cases.

    int digits = numeric_limits<int>::digits;
    int max = numeric_limits<int>::max();
    int min = numeric_limits<int>::min();
    switch( digits ) {
        // max = (2^n-1) ; min = -(2^n)
        // All happen to be identically, Null
        case 15:
        case 23:
        case 31:
        case 63:
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(min));
            CPPUNIT_ASSERT_EQUAL(eFizzBuzz::Null, FizzBuzz(max));
            break;
        default:
            CPPUNIT_ASSERT_EQUAL(0, digits);
            throw logic_error("unsupported architecture");
    }

Are we done then? No. We’ve used a facility in our main application without testing it.

void FizzBuzz() {
    for(int i = 0; i <= 100; i++) {
        auto fb = FizzBuzz(i);
        if(fb == eFizzBuzz::Null) {
            cout << i << endl;
        } else {
            cout << fb << endl;
        }
    }
}

Our operator<< implementation. Testing it is easy though because we’ve generalised it to work on all streams, we can output to a string instead of the terminal and have cppunit check those strings.

    void test_fizzbuzz_operator() {
        ostringstream null, fizz, buzz, fizzbuzz;
        null << eFizzBuzz::Null;
        CPPUNIT_ASSERT_EQUAL(string("Null"), null.str());
        fizz << eFizzBuzz::Fizz;
        CPPUNIT_ASSERT_EQUAL(string("Fizz"), fizz.str());
        buzz << eFizzBuzz::Buzz;
        CPPUNIT_ASSERT_EQUAL(string("Buzz"), buzz.str());
        fizzbuzz << eFizzBuzz::FizzBuzz;
        CPPUNIT_ASSERT_EQUAL(string("Fizzbuzz"), fizzbuzz.str());
    }

    // ...

    CPPUNIT_TEST_SUITE(FizzBuzzTest);
    CPPUNIT_TEST(test_fizzbuzz_operator);
    CPPUNIT_TEST(test_fizzbuzz);
    CPPUNIT_TEST_SUITE_END();

Output:

!!!FAILURES!!!
Test Results:
Run:  2   Failures: 1   Errors: 0


1) test: FizzBuzzTest::test_fizzbuzz_operator (F) line: 107 fizzbuzz.cc
equality assertion failed
- Expected: Fizzbuzz
- Actual  : FizzBuzz

Hopefully the fault in line 107 is obvious:

    CPPUNIT_ASSERT_EQUAL(string("Fizzbuzz"), fizzbuzz.str());

I’ve left the “buzz” lowercase.

With that fixed we’re in a position to be confident about our FizzBuzz implementation. By using a unit test system we’ve also forced ourselves to not only write a working FizzBuzz, but also a well designed one. You should notice that, strangely, we haven’t actually tested the actual FizzBuzz() function.

As written, this isn’t testable. I’m not going to present that because it’s just another, much larger, string comparison; after a change to make FizzBuzz() able to output to a string; and we’ve already seen how to do that, so it becomes a typing exercise.

ostream &FizzBuzz(ostream &s, int n)
{
    for(int i = 0; i <= n; i++) {
        auto fb = FizzBuzz(i);
        if(fb == eFizzBuzz::Null) {
            s << i << endl;
        } else {
            s << fb << endl;
        }
    }
    return s;
}

Leave a Reply