Are your end-to-end UI test suites flaky and taking a long time to run? Perhaps you are suffering from a combinatorial edge-case explosion.
Many developers are most comfortable writing end-to-end tests because the system behaviour from outside is easy to figure out. For example, within a test, you open a page, type into a text input, click the submit button and verify the results shown on the page are acceptable. This is a happy behavioural case, and I warmly recommend writing automated tests for all of them.
However, as developers, we are also keen to consider all possible cases where things might go wrong. Many of the test libraries and language structures even offer a convenient way for testing the edge cases. For example, you would specify an array of invalid input combinations to test a text input validation — e.g. in the context of a calculator app. Then for each combination, you would write a programmatic test case passing the input string to the test method.
...and here the troubles begin.
While the above intention is good, you launch your entire application and browser environment for each test case, thus rendering the runtime exceptionally slow. Furthermore, if you don't handle parallelization and clean up the resources adequately, you end up with a flaky test suite filled with complex and racy behaviour. As a result, you spend more time debugging your tests than writing them. Indeed, you don't want that to happen.
So, what can you do?
The higher you traverse the test pyramid, the fewer edge cases you should test for. Instead, consider your unit tests separated from the I/O. Those easily verify all the edge case behaviour in a matter of seconds — as opposed to a matter of minutes.
Yes, but you say that unit tests are difficult to write when my code is not testable. I thoroughly agree with that sentiment. Perhaps the code you're attempting to test belongs to an old legacy project, or it never was designed with testability as the first priority. Maybe, the original team who wrote the codebase were huge fans of end-to-end testing. I've been there myself, so I don't blame that.
Legacy codebases place an excruciating burden on us. To be able to test, we need to refactor the existing code to make it testable so that we can ultimately refactor it satisfying and clean. It's compelling to fall back to testing both happy and edge behaviour end-to-end in such situations, especially under tight deadlines and budgets. After all, with end-to-end tests, we can capture the exact behaviour we intend to test without spending substantial time cleaning up the codebase. For a novice, that sounds like a win-win situation until you realize you've been practising this in the same codebase for over a year and can no longer trust your tests at all. Hence, the struggle is necessary as the often visible outcome is a robust and testable architecture keeping the I/O-bound logic in its boundaries.
Don't fall in love with your end-to-end tests, but don't forget them. Invest most of your time in I/O-free tests, and your team will thank you later.
If you have problems maintaining your rigid and slow test suites, get in touch. Let's talk what you could do to improve.
Many developers are most comfortable writing end-to-end tests because the system behaviour from outside is easy to figure out. For example, within a test, you open a page, type into a text input, click the submit button and verify the results shown on the page are acceptable. This is a happy behavioural case, and I warmly recommend writing automated tests for all of them.
However, as developers, we are also keen to consider all possible cases where things might go wrong. Many of the test libraries and language structures even offer a convenient way for testing the edge cases. For example, you would specify an array of invalid input combinations to test a text input validation — e.g. in the context of a calculator app. Then for each combination, you would write a programmatic test case passing the input string to the test method.
...and here the troubles begin.
While the above intention is good, you launch your entire application and browser environment for each test case, thus rendering the runtime exceptionally slow. Furthermore, if you don't handle parallelization and clean up the resources adequately, you end up with a flaky test suite filled with complex and racy behaviour. As a result, you spend more time debugging your tests than writing them. Indeed, you don't want that to happen.
So, what can you do?
The higher you traverse the test pyramid, the fewer edge cases you should test for. Instead, consider your unit tests separated from the I/O. Those easily verify all the edge case behaviour in a matter of seconds — as opposed to a matter of minutes.
Yes, but you say that unit tests are difficult to write when my code is not testable. I thoroughly agree with that sentiment. Perhaps the code you're attempting to test belongs to an old legacy project, or it never was designed with testability as the first priority. Maybe, the original team who wrote the codebase were huge fans of end-to-end testing. I've been there myself, so I don't blame that.
Legacy codebases place an excruciating burden on us. To be able to test, we need to refactor the existing code to make it testable so that we can ultimately refactor it satisfying and clean. It's compelling to fall back to testing both happy and edge behaviour end-to-end in such situations, especially under tight deadlines and budgets. After all, with end-to-end tests, we can capture the exact behaviour we intend to test without spending substantial time cleaning up the codebase. For a novice, that sounds like a win-win situation until you realize you've been practising this in the same codebase for over a year and can no longer trust your tests at all. Hence, the struggle is necessary as the often visible outcome is a robust and testable architecture keeping the I/O-bound logic in its boundaries.
Don't fall in love with your end-to-end tests, but don't forget them. Invest most of your time in I/O-free tests, and your team will thank you later.
If you have problems maintaining your rigid and slow test suites, get in touch. Let's talk what you could do to improve.