Engineering Blog

Write your tests in another language!


For acceptance testing, you should consider using a scripting language. I recommend using Python with Contexts, but I suppose I’m biased.

Unit testing with MSpec

Here at Huddle, most of our code-base is written in C#. For unit testing, we use Machine.Specifications (or MSpec for short). MSpec works some C# magic to help you write tests that read like sentences. Here’s an example of a unit test for our awesome new zip-and-download feature.

class When_bulk_downloading_a_non_existent_document
  const int ExistentDocumentId = 249587;
  const int NonexistentDocumentId = 4592;
  static Exception _exception;

  Establish that_the_cmd_contains_a_non_existent_doc_id = () =>
    _command = new BulkDownloadCommand {
      FolderId = _folder.ID,
      DocumentIdsToProcess = new[] { ExistentDocumentId, NonexistentDocumentId },
      Id = _bulkDownloadId


    _handler = new BulkDownloadCommandHandler(
      new NullLogger()

    Because we_handle_the_command = () => _exception = Catch.Exception(() => _handler.Handle(_command));

    It should_throw_an_object_not_found_exception = () => _exception.ShouldBeOfType<ObjectNotFoundException<Document>>();
    It should_not_write_an_async_command = () => _asyncCommandRepo.Verify(r => r.Add(Moq.It.IsAny<AsynchronousCommand>()), Times.Never());
    It should_not_put_the_command_in_redis = () => _redisClient.Verify(c => c.Set(Moq.It.IsAny<string>(), Moq.It.IsAny<BulkDownloadCommand>()), Times.Never());

(I’ve snipped some of the less interesting code.) The whole class is a single test case. Establish, Because and It are MSpec’s keywords: Establish sets up the system under test, Because performs the action you’re testing, and the Its are the assertions.

The syntax doesn’t look much like C#, but you end up being able to read the names of the methods like a sentence. You don’t have to work hard to figure out what the test is all about. If you’re trying to write a test that isn’t readily described by this three-point sentence format, you may be testing the wrong thing.

Why not use C# for API tests too?

Building a good suite of acceptance tests is almost like building a whole new application. Surrounding the tests is a constellation of other classes and functions: networking, test data builders, configuration, and so on. As with any application, it’s important to choose your tools carefully.

C#’s safe, expressive type system makes it great for server-side logic. But I’ve found that these same qualities can actually become a hindrance when you’re writing a REST API client such as a test suite. This is because of the fundamental difference between client-side and server-side programming: the client does not control the shape of the data that it’s dealing with.

For example, .NET’s JavaScriptSerialiser converts .NET objects to and from JSON. This means your tests will have to use the same DTOs as the implementation code, so they’re not black-box tests any more. The alternative is duplicating the code. Of course, if you’re testing in a different language, accidentally reusing code like that is not an option - you’re forced to take an outside-in approach.

If you decide not to deserialise your JSON into objects, and just use dictionaries or dynamics, I’ve found that C#’s verbosity gets in the way. When you don’t need a statically typed language, the type signatures all over the code can be annoying. Here’s how you create a list of dictionaries (a very common JSON structure) in C#:

List<Dictionary<string, string>> company_members = new List<Dictionary<string, string>>
    new Dictionary<string, string>
        {"FirstName", "Benjamin"},
        {"LastName", "Hodgson"},
        {"Role", "Graduate"},
        {"EmailAddress", ""}
    new Dictionary<string, string>
        {"FirstName", "Jon"},
        {"LastName", "Finerty"},
        {"Role", "Developer"},
        {"EmailAddress", ""}

What a lot of curly braces! The visual noise in the code balloons when the structure gets more complicated. I’ve seen tests which resort to hand-writing JSON with string-interpolation to avoid having to write code as verbose as this! Here’s an equivalent structure in Python, which is much more concise:

company_members = [
    {'FirstName': 'Benjamin',
     'LastName': 'Hodgson',
     'Role': 'Graduate',
     'EmailAddress': ''},
    {'FirstName': 'Jon',
     'LastName': 'Finerty',
     'Role': 'Developer',
     'EmailAddress': ''}

So, we decided to start writing API tests using a scripting language. Dynamic languages come into their own in client-side logic. We considered JavaScript, but we ended up choosing Python, partly because the developers wanted to try something new, but largely because of the side-project I’d been working on for a couple of weeks.

Introducing Contexts

Meanwhile, I was becoming frustrated that the Python ecosystem doesn’t offer a tool like MSpec. I wanted to be able to do ‘behaviour-driven development’, but in a lighter-weight manner than tools like Lettuce require. Lettuce and its ilk (such as Cucumber and Fitnesse) are designed for high-level acceptance testing with a formalised process, but I wanted something that would be appropriate at all levels of the Testing Pyramid.

One of the great things about working at Huddle is Tuesday Time, a whole day of every week during which we get to work on anything we want. (Yes, we shamelessly cribbed this idea off Google.) So I set about building a test framework for Python, which I’m calling Contexts.

Contexts, like MSpec, subscribes to a ‘context-specification’ style of testing (hence the name!), wherein a test is spread out over a whole class, with descriptively-named methods for each step in the test. Contexts aims to provide a domain-specific language for testing, while still looking largely like a normal Python class, by treating methods specially based on their names. Here’s an example from Contexts’s own self-test suite.

import contexts

class WhenCatchingAnException:
  def context(self):
    self.thrown = ValueError("test exception")

    def throwing_function(a, b, c, d=[]):
      self.call_args = (a,b,c,d)
      raise self.thrown
    self.throwing_function = throwing_function

  def because_we_call_catch(self):
    self.caught = contexts.catch(self.throwing_function, 3, c='yes', b=None)

  def it_should_call_the_function_with_the_supplied_arguments(self):
    assert self.call_args == (3, None, 'yes', [])

  def it_should_catch_and_return_the_exception(self):
    assert self.caught is self.thrown

Hopefully you can see the parallels between Contexts and MSpec’s testing styles. It’s not just a clone, though - I’ve tried to avoid some of the places I think MSpec has made mistakes. For example, Contexts is a lot more flexible about the keywords it accepts in your method names. If you prefer ‘Given-When-Then’, go ahead! Contexts also has some modern features like parametrised tests (which MSpec doesn’t support) and reporting on the content of assertions made using Python’s assert statement.

How to get hold of Contexts

Contexts works with Python 3.3, and you can install it via the Cheese Shop using Pip. It’s open source, and the code is hosted on Github. I’m actively developing it - cool new features get released every few weeks - and you can see my to-do list on Trello. I’m also open to feature requests, bug reports and pull requests through the Github repo.

Stay tuned for another post in the next few weeks, about how we’re getting on with Contexts at Huddle and lessons I’ve learned from developing it in my Tuesday Time.

comments powered by Disqus