Test Process Improvement (TPI) using PDCA Model

What is Test Process Improvement?

Test Process Improvement is the practice of measuring how a testing process performs, identifying its weak points, and applying controlled changes so that the next project delivers higher quality at a lower cost and in less time. It treats testing as a process that can be measured and tuned, instead of an activity that is repeated the same way on every release.

Imagine the Guru99 Bank project has just been completed. The management board appreciates your work, and the customer is satisfied. Even so, your boss still has questions for you.

Managers often describe software testing as a troublesome and uncontrollable process. Looking back at the Guru99 Bank project, did you face any of the following issues?

These are common issues in almost any test project. Many organizations realize that improving the test process is the only durable way to solve them, because learning from past mistakes is what prevents the same mistakes from repeating in the next release cycle.

Why Test Process Improvement?

The following scenario shows why Test Process Improvement matters. The Guru99 Bank project is complete, the quality of testing was great, and you received good feedback from the customer.

What is the lesson learned from this scenario? It is simply “always try to do better”. Even when you believe you have done a good job, there are always others doing it better, because they have found better ideas and better solutions than yours.

Every business wants the project completed with the highest quality, at the lowest cost, and in the shortest delivery time. Test Process Improvement is what helps a test team move toward all three targets at the same time.

How to Implement Test Process Improvement?

To implement Test Process Improvement on the Guru99 Bank project, the Test Manager can follow the PDCA model. PDCA (Plan-Do-Check-Act) is a four-step management method used in business for the control and continuous improvement of a process. Each pass through the loop is a single improvement cycle, and the output of Act becomes the input of the next Plan.

Step 1) Plan

The Plan phase is where the improvement is designed. It is divided into three smaller steps.

Step 1.1) Identify the problem

The first activity of a test improvement process is identifying the problems that occurred in the current project. The problems in this project may happen again in other project. Solving problems and finding out the solutions in order to avoid them in the future is the primary goal of Test Improvement.

Now back to the project Guru99 Bank website, do you find any problems or improvement point? Select below

Step 1.2) Determine the target

Understand the problem and the issues that occurred in the project. This is how you determine the improvement points and the test phases that deserve attention first.

Suppose you have identified that the test execution phase took too much time and cost to complete. Could the testing be made faster and cheaper? That question becomes the target for the cycle. A useful target is stated as a number and a deadline, for example “reduce regression execution effort by 30 percent before the next release”.

Step 1.3) Define the improvement actions

Based on the agreed target, the improvement actions are determined. These actions should be gradual and introduced bit by bit, because it is not realistic to change everything immediately.

For example, to make the testing faster and cheaper, the following actions are candidates.

In the example above, options A and B both move testing toward faster and cheaper. Option C would make testing faster, but it costs more, because a more experienced tester commands a higher salary. That trade-off is exactly why every candidate action must be judged against the target, not against intuition.

Step 2) Do

You have already defined the improvement points. It is now time to build a plan that implements them. This plan must answer the following questions.

• Which improvement points have to be implemented, and in what order?
• When must the plan be finished?
• What steps have to be completed to achieve the plan?
• Who owns each step, and how will completion be confirmed?

Perform improvement actions

Once the plan is established, it has to be implemented. Improvement activities can disturb the test work that is already in flight, so a Test Manager must pay attention to them in order to avoid unwanted consequences.

Consider the following scenario. On the Guru99 Bank project, in order to make testing faster and cheaper, you decided to use automation testing in place of a large block of manual regression tests. After the action was applied, productivity increased significantly.

Step 3) Check

In the Check step you do three things.

• Evaluate the efficiency of the test improvement actions
• Measure how effective the solution was
• Analyse whether it could be improved further

The goal of this phase is to confirm that the improvement actions were implemented successfully, and to evaluate whether the target that was set in Plan has actually been reached.

The best way to perform that evaluation is with metrics. Metrics are essential to successful organization management. The Test Manager collects data and uses it to measure parameters such as productivity, quality, and cost.

For example, before automation was applied to the project, testing productivity was 10 test cases per man-hour. After automation was applied, productivity was measured at 20 test cases per man-hour.

But an unwanted problem appeared alongside that gain.

In this case, applying automation increased the productivity of testing, but the quality of testing decreased. An improvement action can therefore cause serious consequences elsewhere. In such a scenario the testing tool has to be selected far more carefully, and the automated suite has to be reviewed with the same rigour as production code. A structured evaluation of candidate tools, such as the one described in the Selenium tutorial, prevents a tool from being adopted purely because it is popular.

Consider the same scenario again. The Guru99 project cost had overrun because team members took too much time to execute the test cases. By using the automated test tool, you saved 30 percent of the project cost. That is a good improvement, but your boss expects more.

You therefore always have to look for newer solutions that improve the testing process further. In this scenario, other options can save additional project cost.

• Manage your human resources effectively, so that skilled testers are used where they add most value
• Negotiate better commercial terms with your tool and staffing vendors
• Retire duplicate or low-value test cases instead of automating them

Step 4) Act

When the improvement actions have been implemented successfully and the target has been met, the Test Manager should complete the loop with the following activities.

• Review the improvement activities and act on the lessons learned
• Standardize the improvement point inside the test management process
• Update the policy documents, the test plan templates, and the standard process documents
• Determine when and where these changes will be applied on the next project

If the target was not met, the cycle does not stop. The unmet target is carried into a new Plan phase together with everything the Check phase revealed about why the action underperformed.

TPI NEXT, TMMi, and CMMI Compared

PDCA is the engine of improvement, but a reference model tells you what “better” looks like. Three models are commonly used, and they are frequently confused with one another.

TPI NEXT is a test-specific reference model published by Sogeti. It assesses 16 key areas, organized into three groups, against four maturity levels: Initial, Controlled, Efficient, and Optimizing. Because assessment happens key area by key area, a team can be Efficient in one area while still Controlled in another.

TMMi, maintained by the TMMi Foundation, is a staged test maturity model with five levels: Initial, Managed, Defined, Measured, and Optimization. An organization reaches a level only after satisfying the process areas of that level.

CMMI is not a testing model at all. It covers the whole development organization, and its staged representation also has five maturity levels: Initial, Managed, Defined, Quantitatively Managed, and Optimizing. TMMi was designed to complement CMMI rather than replace it.

Metrics That Prove Test Process Improvement

The Check phase collapses without numbers. A small, stable metric set collected before and after each cycle is enough, and the same definitions must be used on both sides of the comparison.

• Test execution productivity — test cases executed per man-hour
• Defect Detection Percentage (DDP) — defects found in testing as a share of all defects found, including those reported after release
• Cost per executed test case — total test effort cost divided by test cases executed
• Requirement coverage — requirements with at least one linked test case
• Defect fix turnaround — average age of a defect across the defect life cycle

Using the Guru99 Bank figures, where 180 defects were found in testing and 20 were reported by the customer after release, the calculation is straightforward.

# Defect Detection Percentage and improvement deltas
def ddp(found_in_test, found_after_release):
    return found_in_test / (found_in_test + found_after_release) * 100

def delta(before, after):
    return (after - before) / before * 100

print("Defect Detection Percentage: %.1f%%" % ddp(180, 20))
print("Productivity gain: %.1f%%" % delta(10, 20))
print("Test cost change: %.1f%%" % delta(50000, 35000))

Output:

Defect Detection Percentage: 90.0%
Productivity gain: 100.0%
Test cost change: -30.0%

A DDP of 90 percent means one defect in ten still escaped to the customer, so the quality target was not fully met even though productivity doubled and cost fell by 30 percent. That single view is what stops a team from declaring victory too early.

Common Test Process Improvement Mistakes

Most improvement programmes fail for organizational reasons rather than technical ones. The following mistakes account for the majority of abandoned initiatives.

• Improving without a baseline. If nobody measured the process before the change, no one can prove the change helped. Capture the baseline during Plan, not afterwards.
• Chasing a maturity level instead of a business driver. A certificate that does not reduce cost, defects, or lead time is an expense, not an improvement.
• Changing too much at once. When five actions land in the same release, a regression cannot be traced to any one of them.
• Automating a broken process. Automation multiplies whatever process it is applied to, including weak test design and unclear entry criteria.
• Skipping the Act phase. An improvement that is never written into the test policy and the software testing life cycle documents dies with the project team that invented it.
• Excluding the testers. People who were not consulted about a change reliably find ways to work around it.

To take these ideas further, review the phases of the software testing life cycle, tighten your test case design, formalize the defect management process, evaluate where automation testing gives a genuine return, and see how a tool such as HP ALM can hold the metrics your Check phase depends on.

FAQs