Agile Test Automation Framework
Thomas Hamilton
⚡ Smart Summary
Agile test automation applies automated checks inside short sprints, where requirements change weekly and a suite built for a stable waterfall release quickly becomes a maintenance burden rather than a safety net.
Agile Automation Testing
Agile automation testing is the practice of using test automation inside an Agile delivery process. Its purpose is to make software development more effective and efficient while protecting quality and controlling the time and resources a release consumes. Because tests are written alongside the feature work, the practice depends heavily on coordination between developers and testers.
Ever since Agile methodology set out to do away with the laborious realities of the waterfall model, its influence has been felt in automation testing too. The two disciplines have to be combined deliberately:
.jpg)
Automation in Waterfall vs Automation in Agile
In a traditional software testing life cycle, automation testing becomes feasible once the application is stable and the requirements are settled. It assumes a considerable amount of time, highly skilled automation specialists and a noticeable set-up cost. The basic purpose is to reduce cost over the long run and to confirm that no new defects have been introduced around the existing test cases.
Automation testing is not exploratory by nature, because its main role is to save time and reduce cost. It is not designed to surface new and innovative defects. Automation testing mostly confirms behaviour that already exists.
The two settings therefore place very different demands on a test suite:
| Factor | Automation in waterfall | Automation in Agile |
|---|---|---|
| Application state | Stable and signed off before scripting starts | Changing every sprint, often while scripting |
| Time available | A dedicated automation phase | Whatever fits inside a one to four week sprint |
| Who scripts | A separate team of automation specialists | The delivery team, testers and developers together |
| Primary goal | Long-run cost reduction across a large regression suite | Fast feedback on the increment just built |
| Maintenance risk | Low, because requirements move slowly | High, because requirements move constantly |
How to Automate in Agile Methodology
By its own definition, Agile methodology does away with tedious documentation so that new ideas can be implemented quickly and people can interact freely. It favours exploratory work over paperwork:
There is a real contradiction between the fundamental philosophies of Agile methodology and automation testing. Agile teams resolve it by narrowing what they automate rather than automating less: checks are written in the same sprint as the feature, pushed down to the unit and API level where they are cheapest to maintain, and run on every build.
Fundamental Points for Agile Test Automation
Before committing sprint capacity to automation, weigh the points that decide whether a script can be finished at all:
- Design and coding time: Every script has to be designed, coded and reviewed like production code.
- Validation against test data: The finished script must be validated with the existing test data before anyone can trust it.
- Purpose of the test: Functional and regression tests carry different maintenance costs and different shelf lives.
- Sprint length: A sprint runs one to four weeks, most commonly two, which rarely leaves room for a large scripting effort.
A second factor is requirement change. Agile is, by definition, a technique for responding to customer-driven change, so it lends itself to frequent adjustment throughout development.
Automation testing, in contrast, is most useful against stable requirements. It does not lend itself well to the constant churn of an Agile methodology, which is why the choice of what to automate carries more weight than the amount automated.
Agile Automation Tools
The selection of a relevant automation tool is another important factor in adopting automation testing within an Agile methodology. Licensed automation tools, for example, impose strict security access criteria on different types and levels of user, which limits who can reach the resources belonging to that testing automation framework.
Agile methodology, by contrast, emphasises open collaboration and open-ended interaction between team members. Restrictive access policies work against that cohesion and can produce results that are neither helpful nor conducive to the success of the project.
The priority is to deliver quality automation scripts within the time an Agile process allows. Choose candidate test cases carefully, so the resulting scripts can be re-used later and still finished inside the allotted time.
Even in an Agile setting some tests still have to be covered — regression tests in particular. The next section looks at the situations where automation testing fits and how each maps onto Agile testing.
Automation Testing Concepts When Applied to Agile
The table below takes the seven classic conditions that justify automating a test and gives the Agile answer for each. Only three translate cleanly into an Agile sprint, and all three are regression-shaped:
| # | Automation testing concept | Agile methodology answer |
|---|---|---|
| 1 | The test must be repeated often. | This is where the concept of regression testing comes into picture. |
| 2 | The test’s workflow and its validation evolve and change slowly over time. | Not useful for Agile testing, as Agile testing means frequent changes in requirements. |
| 3 | The test validates a business process or workflow, rather than look and feel, colour or table layout. | This scenario can be considered as involved with manual testing. |
| 4 | The test produces results for a regulatory body that demands those results be electronically recorded and archived as formal evidence of compliance. | Not suitable for Agile methodology, as an exhaustive level of documentation is not part of Agile methodology. |
| 5 | The test is very repetitive or has many steps that must be performed exactly the same each time, where manual tester fatigue must be avoided. | Not suitable for Agile methodology. |
| 6 | The test’s pass or fail result is reasonably easy to determine and capture with the selected automation tool. | Suitable for those regression tests during Agile testing which require repetitive and laborious abilities. |
| 7 | The test needs to drive a significant amount of data into the application. | Can be incorporated as regression testing. |
