Quality of tests

Writing good automated tests is about more than just code coverage - over and above exercising the code path of interest without it blowing up, we usually want the tests to assert something meaningful about its behaviour. Normally, if we have structured our code to be testable and picked the right boundary to test against, this will be in the form of supplying some known input and making sure we get the right output, or the right side effect happens.

A test is like a science experiment - we want solve for one variable, the logic we want to validate, and hold all the others constant. This may mean varying one input at a time, or controlling the behaviour of our subject’s collaborators using test doubles so that it becomes known and not part of the test.

Anatomy of a test

Every automated test has three phases:

• Arrange
  • Set up the world to a known starting state
• Act
  • Perform the action under test
• Assert
  • Check that the system has been transitioned to the desired end state

Test Data

We want our tests to be deterministic, so in general we want the inputs to a test to be controlled, so that we can trigger different scenarios and check the correct outcomes. This is especially true of Unit Tests, which test the different cases of a piece of business logic.

Sometimes though, especially in Integration or End-to-End tests, we just want valid data to test with. Using realistic, anonymized data to test different scenarios increases the functional coverage of tests and reduces the likelihood for surprises when your system encounters real-world data for the first time.

Test data can be written by hand into tests, but can also be:

• Generated - testing tools or scripts can be used to create synthetic, yet realistic, test data based on the known properties of your data model. This can help to make sure edge cases are exercised, and is especially useful when a large volume of data is needed.
• Migrated from real data - for an existing application, the most realistic dataset is from the production system. Production data can contain the Personally Identifiable Information (PII) of real people, so this approach is risky. The data has to sanitised and anonymised to prevent accidental leak or exposure, and may still be identifiable.

Generating data should be sufficient for most use cases. Analysis of production data can be used to confirm the test data contains all the needed variance.

In any case, we usually want a mechanism to clear out testing data between automated test runs, to prevent test pollution. This can be done by clearing or recreating a database if accessible, or via an application’s API.

One exception is the manual testing data that the product team may build up in a User Acceptance Testing environment while accepting User Stories. This environment is expected to be stable, and so the team should be warned if any destructive changes to its data are necessary.

Testing antipatterns

• Lack of assertions - it may seem obvious, but to actually test something, a test must have at least one assertion connected to the code under test. More subtle errors like asserting the behaviour of code not involved in the test (such as mocks) can lead to meaningless tests.

• Overuse of mocking - replacing all or part of the behaviour you want to test with a mock that you control means that code remains untested. If your testing boundary is the public API of some object or service, then don’t mock out other parts of that entity, especially private ones.

• Testing private methods - Tempting to want to test directly because you know they’re there, but private areas of code are an object’s own business - what matters to consumers is how they interact through their public API, and they should be tested using that, even if it’s less convenient. Private code is liable to be refactored for maintainability, and rightly so if it still fulfils its public contract. See here for more details.

• Not seeing the test fail - If you’ve never seen your test fail, it may never fail. It’s crucial to see the test fail for the reasons you expect (and not some other error) and then make sure that the code under test makes it pass. This can be done by “red-greening” the test - by breaking the production code logic in some small way and then fixing it again. Test Driven Development bakes this step into the workflow.

• Test Pollution - for reliability, tests should be independent, able to run on their own or in any order. If they use some shared state between them that is not cleared out between tests, this will lead to unexpected behaviour and unpredictable failures. Independent tests also enable optimisation of the test suite run time through parallelisation.

• Flakey tests - tests that pass or fail unpredictably on subsequent runs are known as flakey. Common culprits include test pollution and concurrency or timing issues. Although it can seem random, test flakiness always has some root cause that can be tracked down and eliminated.