Test introduction

In this chapter, I’d like to present my view on testing, especially on web application testing. Through it will probably bring nothing new for most of the readers, I feel my view is very different from one that prevail in Django cmmunity and it’s fundamental for library design, usage and further development.

When speaking about tests, I mean only tests written by developers themselves. QA has it’s own tools and procedures that are out of scope of this document (and library).

Developer tests and xUnit

For past few years, unit-testing and test-driven development became nice buzzwords and together with agile and extreme programming created new style of programming. To take full advantage of testing, however, your precious test-suite must be handled properly, otherwise you will fail into futile world of slow, long test suites and constant need for suite refactor, which questions value of tests.

First of all, test suite must be deterministic. Under all supported configurations, tests suite must pass green; when some tests could not pass because of configured environment (like using some soft of application backend that do not support some features), tests must mark themselves as skipped.

This lead us to holy grail of agile methods, unit tests. Unit tests are fast and deterministic, because they do NOT:

  • Talk to database
  • Communicate over network
  • Talk to 3rd party software

Instead, they use some sort of “dummy stub” (called mock) to replace real thing with dummy, predictable object that directly returns expected results and allows us to test proper response of our system. If you’re doing this, inherit your cases from UnitTestCase and gain huge speedup provided.

For most Django-based apps, however, this is not practical; replacing database with dummy stubs would mean a lot of reimplementation (might be worh it sometimes, however). Still; one should bear in mind what unit test is and try to write them, because they lead to decoupled design and one might consider to create proper classes and functions (and test them separately) instead of binding it all to Model() instance methods.

Before digging into some of those challenges, let us take a brief look on tests we’re talking about.

Test phases (and database)

Developer tests have, generally, four phases:

  1. Prepare environment
  2. Execute system interaction
  3. Verify expected results [1]
  4. Bring environment back to expected state

For true unittests, environment preparation (represented by setUp method in xUnit de-facto standard) consist mostly of mocking system components with those dummy thingies and replacing them back on tearDown.

Databases are, however, strange beasts. Most of tests using them need generators for unique, autogenerated fields (mostly id’s) and tests are relying on them to be in expected states. Thus, suite must reinitialize database before doing such a test and that is a very costly operation that slow down your suite from running few hundred tests per second to few ones per second.

There is, however, compromise solution: transactions. Rolling back a transaction is not as costly as flushing whole database and thus can be used for testing with database object, where mocking is not a good time/profit solution. If it’s your case, inherit from DatabaseTestCase and go. Beware, however, that you can’t use it in multithreaded tests or when you have to commit during tests; then, you’re stuck with DestructiveDatabaseTestCase that requests full cleanup. Otherwise, another tests will be strangely failing and interacting tests are really thing you don’t want to have.

###FIXME: What follows is merely an extended feature intro, rewrite to follow in consistent way

HTTP Tests

While Django’s TestClient (available for DatabaseTestCase and above) is cool, it’s not usable for all cases (like, when you want to test your HTTP Basic/Digest view protection). If you want to test it, use HttpTestCase (which is sadly also DestructiveDatabaseTestCase) and framework will fire up multithreaded Django live server for you.

If this is not enough (and might not be, Django server is still kinda incomplete), you can have your Django served with CherryPy’s production-ready, multi-threaded server. Just set CHERRYPY_TEST_SERVER=True in your settings and enjoy server you can repeatably connect to.

For HTTP requests, use included function urlopen (wrapper for eponymous function from urllib2), which can handles server-side traceback.

Web tests

Web tests are futile attempt to automatize acceptance tests, and also to test javascript between various browsers. Selenium is a great tool for that and we’re providing support for it. Grab SeleniumTestCase, export Selenium tests in PyUnit format and enjoy self.selenium.

Developer’s workflow

Common scenario when fixing a bug:

  1. Write a failing regression tests
  2. Debug to find wher test actually is
  3. Write a failing unittest
  4. Fix broken code
  5. Run tests again to confirm we fixed the right thing
  6. Run the whole suite to be sure we haven’t broken anything

Single tests can be run with ./manage.py test package.package.module:TestCase.singleTest. Using plugins, you can pipeline test runner to run only unittest and corresponding tests on developer machine and let your CI server to run full (and probably longer) suite to check your back.

Few words against Doctests

This library nor my thoughts dont cover doctests. It’s for simple reason: they are lousy for testing. Doctest is excellent tool to verify your documentation and acceptable for making acceptance tests (thus write “system user stories”).

However, for usual developers tests (and mainly unit tests), they are very bad idea. Few reasons:

  1. You must flush database between them as there is no teardown to clear inconsistencies when test fail
  2. Attempt for recovery when condition fails: One stares at hundred lines of traceback and must look for first condition that caused all the fails
  3. Fixture support is lacking and must be done manually
  4. They’re hard to write: no support from editor, lot of >>>’s and ...’s

...but wait, isn’t their so easy to write, because you just cut&paste your console output? If this is your case, then I’d say your development model is broken, and that is probably because your testing suite is broken. You are manually setting up your environment and friends and still feeling more productive then when using your suite, your suite is to blame (perhaps because you can’t select only this one tes you are writing now? Well, you can do it with us). Fix it, because you’re still doing all four phases, just wasting time doing it by hand instead of having it automatized.

Footnotes

[1]To help defect localization, there should be only one condition tested. Rule of thumb is “one assert per test”