Article
August 24, 2022 · 4 min read timeA while back I was holding a presentation on the testing practices of an API serving a complex mobile application. I got a question: "How many percent of a project's code should be test code?". This gave me pause because the question itself exposes a fundamental disconnect between the code and the real world, and made me start thinking how to make people notice and avoid that disconnect.
When you start measuring quality in arbitrary numbers, you inevitably stop measuring it against reality. Quantitative measures can be useful, but should always be recognized for the crutches they are. They certainly shouldn't be the starting point of the reasoning behind your work. The actual answer to the question starts not from the code but in the real world: what you are trying to do with your code, and in what circumstances it's built.
Running software is not the goal, it’s a byproduct
My employer values pragmatism very highly. Success is measured foremost in problem solving, not financial turnover. If it's possible and sensible, we want to solve the customer's problem without a single line of code. Getting various software components up and running is not the end goal, it's a byproduct of what software development is actually about: solving real world problems.
The heart of agile development has never been about the speed of decisions or development. It's been about forcing everyone to zoom out and think about reality more, instead of staring at things that were carved in stone a year or two back – in other words, using common sense and asking the right questions.
On a social level, business experts, project leaders, architects, designers, and developers all work on the same painting from their own corner, therefore, it's critical that they all see the same big picture, even when working on the details. Everyone needs to have a certain empathy towards every other role involved in the work. And whatever you're doing, in any of these roles, you can always ask the question: "but does this make sense?".
If you can't figure out the answer, you should say it aloud. If nobody wants to hear it, that often indicates problems in communication. If it's still hard to figure out the answer, the idea behind the whole painting might not be fleshed out enough, or it could be too big and in need of streamlining or breaking down into smaller parts. These things are often out of your hands, in which case you can only do the best you can and avoid fighting windmills. You know, the pragmatic choice.
Technology must adapt to the problem, not the other way around.
On a technical level, one of the biggest pitfalls developers end up in that undermines pragmatism is falling in love with your specific development toolset – be it a programming language, protocol, or development pattern – and forcing it on your surroundings. With a hammer in hand, everything looks like a nail indeed. Technological choices shouldn't be locked in before learning the details of a project. It's those choices that must adapt to the problem, not the other way around.
You might be able to develop the project very efficiently and elegantly with Rust, but if your app's lifecycle is longer than your presence, how easily can the customer find experts on the open market to work on the project after you're gone? You might love Serverless (I know I do). But do the expected usage patterns of this project happen to favour a more traditional container-based approach? Often questions like these aren't a reason to not use your preference (after all it's a preference for a reason), but you should still always ask them. Even the best craftspeople also use a screwdriver every now and then.
Learn to treasure your pragmatism. It is your own spidey sense, applicable to every problem you face. Develop it, and trust it.
All this in mind, coming back to the original question: how much test code should I have?
First of all, measuring code by amount has little correlation with its value. You should test every functionality in a way that makes sense, with a number of test cases that make sense. This could be literally zero, or it could end up dwarfing the amount of business code.
Like ancient philosophers, I ended up answering the question with many more questions in the hope that the answers will eventually come from within the questioner:
What are the real world effects of your app (f.ex. on people’s health or finances)?
How complex is your app?
How many and how complex integrations does your app have with other apps?
How often do you expect changes to be made to your code?
How long is your app's lifecycle?
How many people touch the codebase?
How diverse are the motivations of these people touching the codebase?
How large is your app’s attack surface for malicious parties?
How much legislative regulation is there in your app's business domain?
Written by
Joonas Laitio often finds himself building foundations and creating possibilities for others, be it as an engineer, referee, or bassist. In the software world, he works developing full-stack with an emphasis on efficient backend solutions and serverless computing.
