Continuous Testing in DevOps
Thomas Hamilton
What is Continuous Testing?
Continuous Testing in DevOps is a software testing type that involves testing the software at every stage of the software development life cycle. The goal of Continuous testing is evaluating the quality of software at every step of the Continuous Delivery Process by testing early and testing often.
The Continuous Testing process in DevOps involves stakeholders like Developer, DevOps, QA and Operational system.
How is Continuous Testing different?
Continuous Testing
The old way of testing was hand off centric. The software was handed off from one team to another. A project would have definite Development and QA phases. QA teams always wanted more time to ensure quality. The goal was that the quality should prevail over project schedule.
However, business wants faster delivery of software to the end user. The newer is the software, the better it can be marketed and increase revenue potential of the company. Hence, a new way of testing was evolved.
Continuous means undisrupted testing done on a continuous basis. In a Continuous DevOps process, a software change (release candidate) is continuously moving from Development to Testing to Deployment.
Continuous DevOps Process
The code is continuously developed, delivered, tested and deployed.
For Example, whenever a developer checks the code in the Source Code Server like Jenkins automated set of unit tests are executed in the continuous process. If the tests fail, the build is rejected, and the developer is notified. If the build passes the test, it is deployed to performance, QA servers for exhaustive functional and load tests. The tests are run in parallel. If the tests pass, the software is deployed in production.
Continuous Testing is a small cog in the Continuous Development, Integration and Deployment Cycle.
Current Testing Stack
Software development is not same as it is in the past we cut development from month to weeks. The current testing stack (see figure above) is titled towards UI testing. But the goal is to have more and more automated unit tests.
How Is Continuous Testing Different from Test Automation?
Test automation vs Continuous Testing
| Parameter | Test Automation | Continuous Testing |
|---|---|---|
| Definition | Test automation is a process where tool or software is used for automating tasks. | It is a software testing methodology which focuses on achieving continuous quality & improvement. |
| Purpose | A set of similar or repetitive tasks, a machine can execute, faster, with a fewer mistake. | The continuous testing process helps to find the risk, address them and improve the quality of the product. |
| Prerequisite | Automation in testing possible without integrating continuous testing. | Continuous testing can not be implemented without test automation. |
| Time | Software release can take a month to years. | Software release may be released weekly to hourly. |
| Feedback | Regular feedback after testing each release. | Feedback at each stage needs to be instant. |
| History | Automated testing has been done for decades to make the testing process faster. | Continuous testing is a relatively newer concept. |
How to do Continuous Testing
- Using to tools to generate test automation suite from user stories/requirements
- Create Test Environment.
- Copy and anonymize production data to create test data bed
- Use service virtualization to test API
- Parallel performance testing
Continuous Testing Tools
Here is a curated list of best Continuous Testing Tools :
1) QuerySurge
QuerySurge is the smart data testing solution that is the first-of-its-kind full DevOps solution for continuous data testing. Key features include Robust API with 60+ calls, detailed data intelligence & data analytics, seamless integration into the DevOps pipeline for continuous testing, and verifies large amounts of data quickly.
2) Jenkins
Jenkins is a Continuous Integration tool which is written using Java language. This tool can be configured via GUI interface or console commands.
Download link: https://jenkins.io/
3) Travis
Travis is continuous testing tool hosted on the GitHub. It offers hosted and on-premises variants. It provides a variety of different languages and a good documentation.
Download link: https://travis-ci.org/
4) Selenium
Selenium is open-source software testing tool. It supports all the leading browsers like Firefox, Chrome, IE, and Safari. Selenium WebDriver is used to automate web application testing.
Download link: https://www.seleniumhq.org/
Benefits of Continuous Testing
- Accelerate software delivery
- Continuous testing improves code quality
- It helps to assess exact business risk coverage.
- It seamlessly integrates into DevOps Process
- Helps to create an agile and reliable process in just hours instead of months.
- Accelerates time-to-market with a continuous feedback mechanism.
- Merges traditionally siloed teams to meet modern enterprise needs. Dissolves disconnect between development, testing, and operations teams.
- Test Automation helps to achieve consistency by maintaining the same configuration for all relevant tests.
- Emphasizes business expectations to mitigate business risks
- Providing ubiquitous test environment access with Service Virtualization
Challenges of Continuous Testing
- The traditional process is limiting cultural shift among Development & QA professionals.
- Lack of DevOps skills and right tools for testing in Agile & DevOps environments.
- Heterogenous test environments which will never reflect production environment.
- Conventional testing process and loosely defined test data management.
- Longer code integration cycles create integration issues and late defect fixes
- Insufficient and ineffective resources and test environments
- Complex application architecture and business logic that restricts DevOps adoption.
Conclusion:
- In Software Engineering, Continuous testing is a process of testing early, testing often, testing everywhere, and automate.
- The old way of testing was a handoff centric. The software is handed off from one team to another
- Jenkins, Travis, and Selenium are popular Continuous Testing and Integration tools.
- Continuous Testing delivers actionable feedback according to each stage of the delivery pipeline.
- Continuous testing helps to improve code quality
- The traditional process is limiting cultural shift among Development & QA professionals.
- Longer code integration cycles create integration issues and late defect fixes