How to Select Best Automation Testing Tool?

In the article Test Process Improvement, you learned the improvement points from the project Guru99 Bank. It is using automated Testing instead of Manual Testing. But team members may ask you –

Answer to this question is challenging because there are many factors to consider

• You want to support your test activities by means of a software tool, but you don’t know tools currently available in market
• Which type of tool will best fit the requirement and the project budget?
• Who on the team has the skills to use the tool once you have purchased it

The Importance of the Software Testing Tool Selection

Success in any test automation depends on identifying the right tool for automation. Selecting the “correct” Testing Tool for your project is one of the best ways to achieve the project target. The following example will show you the benefit of the testing tool selection

In the project Guru99 Bank, to save the effort of testing, the project team decided to use an automated testing tool for the test execution. After many meetings, your team selected a suitable tool for the project.

One month later, you got the report from the project team about this tool

The results are great. The new automated tool doubled the testing productivity. It means we saved 50% cost of test execution

This is an example of the benefit of using the testing tool in the project. Selecting the right testing tool help you to improve the project productivity and save project cost.

Type of Test Tools

There’re many types of test tool, which Test Manager can consider when selecting the test tools.

Open-Source Tools

Open source tools are the program wherein the source code is openly published for use and/or modification from its original design, free of charge.

Open-source tools are available for almost any phase of the testing process, from Test Case management to Defect tracking. Compared to commercial tools Open source tools may have fewer features.

Commercial Tools

Commercial tools are the software which are produced for sale or to serve commercial purposes.

Commercial tools have more support and more features from a vendor than open-source tools.

Custom Tools

In some Testing project, the testing environment, and the testing process has special characteristics. No open-source or commercial tool can meet the requirement. Therefore, the Test Manager has to consider the development of the custom tool.

Example: You want to find a Testing tool for the project Guru99 Bank. You want this tool to meet some specific requirement of the project.

Automation Feasibility Analysis

Back to the above example, the project team decided to develop a custom tool which can meet project requirements. Suppose they have been given 100 test cases to automate and they estimated 5 days to develop a tool which can automate all of those test cases.

Here is the result of their work

As in above scenario, the issue is that the test tool cannot automate all the test cases of test specification. It means that not all application features can be thoroughly tested using the test tool.

If the functionality of application under test changes frequently or is too complicated, it is difficult to create test automation for all the application features, because every tool has its own limitations.

If you don’t want to be in such situation, before selecting the test tool, you must analyze the test cases and decide which test cases should be automated and which test cases should not. This is the Automation Feasibility Analysis activity.

Automation Feasibility Analysis is the very significant contributor in testing. In this analysis, you need to check if the application under test is qualified for automated test.

Some factors you need to consider:

To know more about these factors, you can use a checklist which is included in this article

How to Select Best Automation Testing Tool

To select the most suitable testing tool for the project, the Test Manager should follow the below tools selection process

Step 1) Identify the Requirement for Tools

How can you select a testing tool if you do not know what you are looking for?

You to precisely identify your test tool requirements. All the requirement must be documented and reviewed by project teams and the management board.

Consider the following example:

Step 2) Evaluate the Tools and Vendors

After baselining the requirement of the tool, the Test Manager should

• Analyze the commercial and open source tools that are available in the market, based on the project requirement.
• Create a tool shortlist which best meets your criteria
• One factor you should consider is vendors. You should consider the vendor’s reputation, after sale support, tool update frequency, etc. while taking your decision.
• Evaluate the quality of the tool by taking the trial usage & launching a pilot. Many vendors often make trial versions of their software available for download

Step 3) Estimate Cost and Benefit

To ensure the test tool is beneficial for business, the Test Manager have to balance the following factors:

A cost-benefit analysis should be performed before acquiring or building a tool

Example: After spending considerable time to investigate testing tools, the project team found the perfect testing tool for the project Guru99 Bank website. The evaluation results concluded that this tool could

• Double the current productivity of test execution
• Reduce the management effort by 30%

However, after discussing with the software vendor, you found that the cost of this tool is too high compare to the value and benefit that it can bring to the teamwork.

In such a case, the balance between cost & benefit of the tool may affect the final decision.

Step 4) Make the Final Decision

To make the final decision, the Test Manager must have:

• Have a strong awareness of the tool. It means you must understand which is the strong points and the weak points of the tool
• Balance cost and benefit.

Even with hours spent reading software manual and vendor information, you may still need to try the tool in your actual working environment before buying the license.

You should have the meeting with the project team, consultants to get the deeper knowledge of the tool.

Your decision may adversely impact the project, the testing process, and the business goals; you should spend a good time to think hard about it.

Things To Consider While Choosing a Test Management Tool

With the wide array of choices available, it’s easy to get confused while shopping for a test management tool. This is an unbiased guide to help you elicit the best tool as per your needs.

In this guide, I will provide you 5 most important but often ignored parameters while selecting your test management tool. But before I begin, let me iterate-

Do not compare Apple and Oranges

It does not make sense comparing a $30 per month SaaS tool with a full blow $6000 commercial tool when your budget does not permit it. It’s important that you filter your tool with a list based on BUDGET first. If on a budget, open-source is a good option. Although commercial tools are expensive, they offer great support and are easy to use with lots of training material provided.

The next thing you should calculate is the costing for your license. For that you need to estimate the duration for test completion and what method you opt for testing, this will help you to figure out what model would fit in your budget like whether you want to pay as per registered user, concurrent user, or persistent user. Usually, persistent or one-time costs are higher because of its capital expense. For certain software, you need run time license. Not all vendors support run time licenses, so you need to consider the run time license expense also if you are thinking to install into different machines.

Once, you have distilled your choices based on budget, here are 5 very important parameters to compare your test management tools-

Tip #1 Test Management Tool should Improve Productivity

Testing Teams are facing steep challenges to keep up with rapid technological changes & need for test on multiple platforms (both browsers and devices). Since manual testing occupies most of the testing time one way to deal with this challenge is to step up automation. But there are other areas to improve the efficiency of a tool which includes effective test lab/data management, release management, insightful reporting amongst others. ChoIt wosing the correct Test Management Tool can hugely assist in improving these parameters.

Let’s look into what is required from a Test Management Tool from a productivity standpoint-

Granularity of test Information

The test management tool should include detailed information reports related to the testing process. For example, in test case failure, it is always helpful to know at which step the test failed instead of knowing the test case failed as a whole. Whatever minor changes applied during the testing, the test management tool should inform about the changes to the corresponding team members. For easy follow up and quick action it should also keep an audit trail of changes, including full versioning of test cases. Also, the test management tool should save results of test as a session and must have the ability to compare the test run results. The other features that are expected from a standard test management tools are documenting test strategy, planning test execution, creation of defects, linking stories, etc. A video upload feature and API support would be an icing on the cake.

Release Management Tracking (DevOps)

Software Releases are complex with multiple activities (like tracking of build version, coordination of team and transparency of releases) all happening at the same time. A good test management tool must have robust release management tracking ability that keeps the record of the list of the software being tested and released, so it don’t conflict with other test artifacts. Also, it should be capable of managing all project artifacts into one environment.

The test management tool must provide a single view of requirements for all stakeholders. Collaboration with stakeholders guarantees that the right requirements are captured. The tool should be able to trace the links between other development assets and requirements.

For distributed teams, it should provide a single shared repository to collaborate and share requirements, evaluate linked defects and understand their relationship to tests.

For quality and consistency, your test management tool must enforce standardization.

Reporting

It is a chaos if your testing team knows how many defect are open in the release but don’t know how long it is going to take to resolve it. To monitor the test activities, test engineers highly depend on the reports generated by test management tool. A reliable test management tool should deliver following information in their reports like test plan progress, test case authoring status, test case failed and passed, test status by suites, test status by area, test case to be automated, etc.

Your test management tool should allow full visibility of developer activity and lifecycle processes from requirements management to testing and fixing defects.

A good test management tool should comprise of following attributes

• Graph and dashboard should give insight on how many of your automated tests have run or awaiting execution
• Should highlight common test attributes like the number of test cases passed/failed/skipped, user workload, etc.
• Make decision about the readiness of each release
• View your automated test runs together with manual test results

Tip #2 Agile Support

The demand for agile in the current market is increasing, and it’s getting bigger each day. It is important that your test management tool support the agile methodology. In Agile, the requirements are mutable and completed incrementally, your test management tool should be able to accommodate such requirements. The agile test management tool of your choice should promote deep partnership between developers, testers and other stakeholders (basic principle of agile).

The Agile test management tool should support

• Creating stories
• Estimation
• Sprint backlog planning
• Velocity charts and other reports etc.
• Scrum and Kanban

While in an Agile environment, test run comparison by test management tool would be productive if you are looking at changes in defect rates per iteration.Also for Agile, test management tool will need to keep the record of the product release for each iteration and backlog

Tip #3 External Integration

Automated test case execution can reduce a lot of burden from test engineers. You should select a test management tool that easily integrates with automation tools. Look for the ability to manage test scripts as well as schedule and launch test both locally and on the remote host. Also, ability to store automation test results in the test management tool is desirable. Your test management tool must support continuous integration. The main advantage of the test management tool with CI is that tests get launched automatically. The launch can be triggered by anything like a source code commit from a developer or a test case with underlying test automation or a scheduled task for a specific time. The continuous Integration differs in time, detail, direction and granularity.

For any QA team, integration of bug tracking tool makes their testing process very easy. A standard test management tool that integrate seamlessly with SDLC app or other bug tracking tools like JIRA, BugZilla, Mantis adds more value to the tool. It should allow users, to link bugs, to test case run instance, for centralized management of your software testing process. It should also allow the entire project team to work together on a single issue at the same time.

The integration of your support desk system is an important feature any test management tool should provide. A support desk software or otherwise known as ticket system helps the users to get an instant solution to their problem. It helps customers to get an overview of all open and closed issue.

API Support for custom integrations is desirable.

Tip #4 Mobile

Your Test management tool should support mobile devices and tablet.

Mobile support is extremely helpful for teams who are at the client location, not at the desk, or for managers, working with teams at different geographical locations.

The test management tool must have full (not reduced) feature set available on mobile.

Support for multiple devices and mobile O.S is important as with the increase in number of devices there are more complexities like location, gesture, accelerometers and orientation issues. After all, you never buy a phone thinking it is compatible with your test management tool!

Tip #5 Support

The support team of the test management tool vendor should be committed to help their customers in all possible ways. Many test management tool are more focused on creating a glossy websites and ads to attract more customers. But for continuous and repeated business they have to focus on delivering a best help desk support.

The best test management tool should have features like live chat, phone call, FAQ’s & online user guides, product documentation information, knowledge base, raising help desk ticket, etc. to help their customers.

Many test management tool also facilitate live video conference with tool experts to solve customer queries. If the live conference is not feasible, then demos or recorded videos related to test management tool should be always available on site and can be accessed anytime. While for developers, if the tool provides API or custom integration support, demo script, and detailed documentation must be provided.

Though all of the above support channels may not be available for all tools, you should select a tool that provides your “preferred” support channel.

One more important and often forgotten parameter is whether the tool vendor is Customer Focused? Here are a few things to consider while selecting your test management tool-

• The tool interface should be user friendly and include customer integration feature
• It should accommodate multiple languages and multiple time zones
• It should have an advanced search ability

Hope this guide helps in selecting a tool that best fits your testing needs!