With the wide array of choices available, it's easy to get confused while shopping for a test management tool. This is an unbiased guide to help you elicit the best tool as per your needs.
In this guide, I will provide you 5 most important but often ignored parameters while selecting your test management tool. But before I begin, let me iterate-
Do not compare Apple and Oranges
It does not make sense comparing a $30 per month SaaS tool with a full blow $6000 commercial tool when your budget does not permit it. It's important that you filter your tool with a list based on BUDGET first. If on a budget, open-source is a good option. Although commercial tools are expensive, they offer great support and are easy to use with lots of training material provided.
The next thing you should calculate is the costing for your license. For that you need to estimate the duration for test completion and what method you opt for testing, this will help you to figure out what model would fit in your budget like whether you want to pay as per registered user, concurrent user, or persistent user. Usually, persistent or one-time costs are higher because of its capital expense. For certain software, you need run time license. Not all vendors support run time licenses, so you need to consider the run time license expense also if you are thinking to install into different machines.
Once, you have distilled your choices based on budget, here are 5 very important parameters to compare your test management tools-
Testing Teams are facing steep challenges to keep up with rapid technological changes & need for test on multiple platforms (both browsers and devices). Since manual testing occupies most of the testing time one way to deal with this challenge is to step up automation. But there are other areas to improve the efficiency of a tool which includes effective test lab/data management, release management, insightful reporting amongst others. ChoIt wosing the correct Test Management Tool can hugely assist in improving these parameters.
Let's look into what is required from a Test Management Tool from a productivity standpoint-
Granularity of test Information
The test management tool should include detailed information reports related to the testing process. For example, in test case failure, it is always helpful to know at which step the test failed instead of knowing the test case failed as a whole. Whatever minor changes applied during the testing, the test management tool should inform about the changes to the corresponding team members. For easy follow up and quick action it should also keep an audit trail of changes, including full versioning of test cases. Also, the test management tool should save results of test as a session and must have the ability to compare the test run results. The other features that are expected from a standard test management tools are documenting test strategy, planning test execution, creation of defects, linking stories, etc. A video upload feature and API support would be an icing on the cake.
Release Management Tracking (DevOps)
Software Releases are complex with multiple activities (like tracking of build version, coordination of team and transparency of releases) all happening at the same time. A good test management tool must have robust release management tracking ability that keeps the record of the list of the software being tested and released, so it don't conflict with other test artifacts. Also, it should be capable of managing all project artifacts into one environment.
The test management tool must provide a single view of requirements for all stakeholders. Collaboration with stakeholders guarantees that the right requirements are captured. The tool should be able to trace the links between other development assets and requirements.
For distributed teams, it should provide a single shared repository to collaborate and share requirements, evaluate linked defects and understand their relationship to tests.
For quality and consistency, your test management tool must enforce standardization.
It is a chaos if your testing team knows how many defect are open in the release but don't know how long it is going to take to resolve it. To monitor the test activities, test engineers highly depend on the reports generated by test management tool. A reliable test management tool should deliver following information in their reports like test plan progress, test case authoring status, test case failed and passed, test status by suites, test status by area, test case to be automated, etc.
Your test management tool should allow full visibility of developer activity and lifecycle processes from requirements management to testing and fixing defects.
A good test management tool should comprise of following attributes
- Graph and dashboard should give insight on how many of your automated tests have run or awaiting execution
- Should highlight common test attributes like the number of test cases passed/failed/skipped, user workload, etc.
- Make decision about the readiness of each release
- View your automated test runs together with manual test results
The demand for agile in the current market is increasing, and it's getting bigger each day. It is important that your test management tool support the agile methodology. In Agile, the requirements are mutable and completed incrementally, your test management tool should be able to accommodate such requirements. The agile test management tool of your choice should promote deep partnership between developers, testers and other stakeholders (basic principle of agile).
The Agile test management tool should support
- Creating stories
- Sprint backlog planning
- Velocity charts and other reports etc.
- Scrum and Kanban
While in an Agile environment, test run comparison by test management tool would be productive if you are looking at changes in defect rates per iteration.Also for Agile, test management tool will need to keep the record of the product release for each iteration and backlog
Automated test case execution can reduce a lot of burden from test engineers. You should select a test management tool that easily integrates with automation tools. Look for the ability to manage test scripts as well as schedule and launch test both locally and on the remote host. Also, ability to store automation test results in the test management tool is desirable. Your test management tool must support continuous integration. The main advantage of the test management tool with CI is that tests get launched automatically. The launch can be triggered by anything like a source code commit from a developer or a test case with underlying test automation or a scheduled task for a specific time. The continuous Integration differs in time, detail, direction and granularity.
For any QA team, integration of bug tracking tool makes their testing process very easy. A standard test management tool that integrate seamlessly with SDLC app or other bug tracking tools like JIRA, BugZilla, Mantis adds more value to the tool. It should allow users, to link bugs, to test case run instance, for centralized management of your software testing process. It should also allow the entire project team to work together on a single issue at the same time.
The integration of your support desk system is an important feature any test management tool should provide. A support desk software or otherwise known as ticket system helps the users to get an instant solution to their problem. It helps customers to get an overview of all open and closed issue.
API Support for custom integrations is desirable.
Your Test management tool should support mobile devices and tablet.
Mobile support is extremely helpful for teams who are at the client location, not at the desk, or for managers, working with teams at different geographical locations.
The test management tool must have full (not reduced) feature set available on mobile.
Support for multiple devices and mobile O.S is important as with the increase in number of devices there are more complexities like location, gesture, accelerometers and orientation issues. After all, you never buy a phone thinking it is compatible with your test management tool!
The support team of the test management tool vendor should be committed to help their customers in all possible ways. Many test management tool are more focused on creating a glossy websites and ads to attract more customers. But for continuous and repeated business they have to focus on delivering a best help desk support.
The best test management tool should have features like live chat, phone call, FAQ's & online user guides, product documentation information, knowledge base, raising help desk ticket, etc. to help their customers.
Many test management tool also facilitate live video conference with tool experts to solve customer queries. If the live conference is not feasible, then demos or recorded videos related to test management tool should be always available on site and can be accessed anytime. While for developers, if the tool provides API or custom integration support, demo script, and detailed documentation must be provided.
Though all of the above support channels may not be available for all tools, you should select a tool that provides your "preferred" support channel.
One more important and often forgotten parameter is whether the tool vendor is Customer Focused? Here are a few things to consider while selecting your test management tool-
- The tool interface should be user friendly and include customer integration feature
- It should accommodate multiple languages and multiple time zones
- It should have an advanced search ability
Hope this guide helps in selecting a tool that best fits your testing needs! Here are my pointers again for easy reference
- Budget: Foremost, shortlist your tool based on budget.
- Productivity: Test management tool should be capable of improving overall productivity by focusing on key areas like granularity of test information, release management tracking and reporting
- Agile: Looking to current trend and demand, a test management tool must support agile testing
- External Integration: Automation test results in the test management tool are desirable, your test management tool must support Automation and Continuous Integration
- Mobile: Mobile support in the test management tool is extremely helpful
- Support: The best test management tool should have support features like live chat, phone call, FAQ's, raising help desk ticket, etc.
This article is contributed by - Sanjay Zalavadia is the VP of Client Services for Zephyr, who offers a real-time, full-featured test management system. Learn more about Zephyr right here