Automated Tests: Get Dividends With Every Test Run
What Is Automated Testing
Traditional software development involves manual testing which works well on a small scale. That being said, once your team grows in size, engineers write more code, manually testing each new and existing feature is not effective anymore. Bugs get into production making customers unhappy. This is where automated testing comes in. Automated tests are a codified version of manual tests. The tests are written by engineers. Once they’re written they can run over and over again while your engineers are working on the text feature.
Advantages of Automated Testing
Write Robust and Modular Code
This is perhaps one of the most important advantages of automated testing. What makes testing easier is structuring your code in a modular way. Modularizing your architecture allows you to swap out or refactor each module in isolation without affecting the rest of the system.
In addition to your code being modular, your code will also be more robust. When running your tests with some randomized data or by using mutation testing, your code gets executed under different conditions. Seeing the code you wrote fail under some conditions you didn’t think about is eye-opening. It allows you to account for more edge cases and make your code more robust.
Refactor
Your product changes, your customer change, and your code need to change as a result of it. Refactoring makes changing your code easier. Automated tests allow you to make sure your code works as expected once the refactor is complete. This way you can refactor without any fear of breaking your already working system.
Minimize Bugs
Automated tests catch bugs. With the constantly growing amount of code, we humans are not able to keep all code paths in our heads which leads to unexpected bugs. Automated tests help exercise all code paths involved so you can focus on the task at hand as opposed to constantly retesting the entire system and hypothesizing where it can break.
Tests are Your Documentation
Tests document the existing behavior of your system. When structured correctly, and especially with the use of BDD syntax, your tests are your system documentation. You can use it to ramp up new engineers as well as a reference for how your system works in practice.
Tests Prove That Your System Works
The code you write may be the greatest code ever and you may think it works. That being said, you never know if it actually works until it’s in production serving real customers. A green test suite is proof of your code working as expected as opposed to a guess or a good feeling that your system will work.
Get Dividends With Each Test Run
Writing automated tests is an investment that pays dividends with every test run. Once the test is written, there is no need to execute it again manually. You can run an automated test at any time on-demand, or have it running automatically on every new code change. Every time you run it, it makes sure your system behaves the way it should.
Disadvantages of Automated Testing
The main disadvantage of automated testing is that it makes a large sweeping change harder. If you’re writing a prototype, or your product didn’t achieve the product-market fit, then automated tests will get in the way of pivoting or making a full restructure change. Ironically, this is one of the most common reasons for not adopting automated testing. The teams we’ve seen tried to do it too early. After having some bad experience with it, they stopped testing altogether and never came back to it. Only discovering later that they wrote a lot of tightly coupled code that is hard to test.
Efficient Testing Strategy
Our recommendation is to double down on unit tests, write fewer integration tests, and fewer UI/API tests. Why? Because unit tests are the easiest to write, fastest to run, easiest to parallelize, and it forces you to produce a robust baseline for your modular architecture. Integration and unit tests have some great coverage. On another hand it’s hard to write integration tests, they are slower and more brittle. Let’s go over each kind of test in detail.
Anatomy of Automated Testing
Unit Tests
Unit tests test a specific unit of code. They’re usually isolated testing a specific class, a specific function, or a small group of classes and functions. Unit Tests are usually written by the engineers while the functionality is implemented. Most of the business logic is covered by unit tests. Since the unit of code under each test is relatively small, you can set up different conditions for each test. It works especially well when setting up each test with some randomized data. Unit tests are quickest to run, they are the easiest to parallelize and optimize. We recommend writing most of your test cases at the unit test level.
Integration Tests
While unit tests test each unit of code in isolation, an integration test tests how multiple units work together. Due to a higher level of complexity, these tests are usually slower to run, harder to set up, and harder to parallelize. It’s also harder to test boundary conditions because the test set up is more complicated. That’s why we recommend writing fewer cases and structuring your integration tests to cover a larger number of integration points in each test.
UI / API Tests
UI Tests drive your GUI (i.e. click or tap actual buttons on the screen, etc) and API tests drive your API (i.e. sending real HTTP or GraphQL requests to your endpoints, etc). As you probably guessed UI or API tests are one of the slowest to run and the most complicated to set up. The reason is that they operate at the UI or API level which is usually just the tip of the iceberg. That being said, UI or API tests give you the most coverage for the number of tests. Here, we recommend going with a few happy paths to make sure everything behaves as needed, all screens render properly and all requests come back as normal.
Test Suite
Since you get the main benefit of automated tests when you execute it, your test suite is your best friend. The test suite runs all necessary tests in sequence parallelizing tests where it’s possible. Depending on your situation you may have several test suites running at different stages of the process. We recommend creating a robust CI/CD pipeline which runs before merging your feature branches and when you deploy your code.
Load Tests
As the name suggests, load tests test your system under load. There are numerous tools available starting from a simple CLI tool like Apache Benchmark to more sophisticated solutions with a GUI like jMeter or Locust. We recommend performing load testing in a separate environment similar in capacity to a production environment. This way we compare apples to apples. Before running the tests, make sure your monitoring is set up so you can see the metrics before, during, and after the test. Depending on your system, you might want to monitor your database throughput, CPU / Memory / Hard Disk usage of your application servers, etc.
How To Start With Automated Testing
All it takes is just starting to do it. If you don’t have a test suite running, getting it to run and putting it on CI is the first step. There are several ways to give the team visibility into the health of your automated tests. You can enable CI notifications about the failed builds to go to Slack. Another helpful technique is exposing the CI status as a dashboard on a TV in the office. Once you have your tests running seeing your test data will become eye-opening. Even if your test suite has just a few tests.
If you’re dealing with a monolithic tightly coupled system, the main thing is to start carving out some pieces of the monolith into smaller chunks which are easier to test in isolation. Step by step you’ll be able to take your system under control.
Advanced Topics
Book References
Conclusion
Automated testing is a great way to make sure you write modular and robust code that always works. You get the proof that your system works as expected with every run. Adding more test cases to your system has a cumulative effect since you get the dividends with every run of your test suite. If writing automated tests is something new for your team, or you would like to improve your testing setup, don’t hesitate to reach out. Right Balance did it many times in the past.
Get in touch
