Problem
Unit testing is, IMHO, the essential bare minimum automated testing that should be employed on any software development project beyond the very simplest CRUD application. So once you have unit tests in place what is the next step?
My preference is for there to be Business Driven Development (BDD) style user acceptance tests. Ideally these would be specified by, where available, the users themselves with help from the developer. One problem with this however is that the user will probably be thinking in terms of a user interface - be it a web page, an SMS service, a web service or anything in between. If you test through a user interface then a lot more work is going to be involved and the tests will be a lot more fragile.
Solution
The best option is to test through an API. One that will be used by the UI programmer to interact with the system as a whole. That said, systems rarely stand alone and often interact with other systems and again this makes the testing hard. Even introducing a database or any other state retaining part of the system can
make the testing harder as it will need to be reset to a known state between each test. For these dependencies then it is better to mock them out of the tests – ensuring that they will not fail no matter what environment you are running on.
The following diagram illustrates the priority I would assign to each kind of test. The priority indicates not only how early in the project the tests should be developed but also how many tests I would expect to see in each case and how often you would execute them.
With respect to the number of tests at each priority there is no hard and fast mathematical rule. However I am inclined to think of it as an exponential decrease in number, for instance 1000 tests at priority 1, 100 at priority 2, 10 at priority 3 and 1 at 4. Actual numbers will depend on how critical the system is and how much effort you are willing to put in to avoid manual testing.
1 API-Mock Testing
This is your bread and butter testing. Where unit tests leave off so these take over – you are aiming for really good coverage here. These should be run all the time – either in a continuous testing system or at least before check-in.
2 API-Actual Testing
First stage of integration testing. You may be less reliant on the actual results of tests against the dependencies and more concerned that the structure of the communication protocol is correct. How often these are run will count on how difficult it is to get your dependencies reset to a baseline. Database dependencies should be easier in this respect and so the tests can be run before each check-in. Dependencies on hard to reset (like a payment gateway) or expensive to replicate for testing (like a telephone exchange) systems may be relegated to only being automatically ran just before user/manual testing.
3 UI-Mock Testing
First stage of UI testing. You may be just ensuring that the screens flow correctly. There should be no need to test your business services behaviour as such since this is tested by the type 1 tests. These should be run by your build system. Generally you wouldn’t want it to break the build if they fail as it may be a small issue or perhaps you are working up to a fully functional system. Failing tests can either be a warning to manual testers of what to avoid, a cue to the test writers that the test needs updating or a cue to the developers that they got something wrong. It should never indicate that business logic is wrong as that should be covered by type 1 API-Mock testing.
4 UI-Actual Testing
Full UI integration testing. Your best bet here is to simply smoke test that everything is configured correctly – something that can be quite challenging if you are using DI or IoC. At times I have used a single screen for this that simply displays pass/fail indicators for every integration point. True – this isn’t really a UI test but it is useful in confirming configuration and connectivity and can be used in the live environment. These tests should be run as part of your deployment testing to each environment.
Conclusion
Once you have good coverage with unit tests or a test driven approach to development then it is time to level up to Business/Domain Driven Development where your tests are closer to what the customer understands and needs. Although customers often think in terms of user interfaces your tests will be less fragile if you test against a coded application programming interface that will then be used by the user interface to drive the domain logic. Also mock out your dependencies to ensure quick and reliable test execution. Write fewer tests against the UI and dependencies which will only be executed as part of the build process or in deployment.
Sources
This is not entirely original thought. Unfortunately I can’t remember where I first read about this approach. If I find it or remember I’ll link here.