Test Monitoring & Control in Software Testing

What is Test Monitoring?

Test Monitoring in test execution is a process in which the testing activities and testing efforts are evaluated in order to track current progress of testing activity, finding and tracking test metrics, estimating the future actions based on the test metrics and providing feedback to the concerned team as well as stakeholders about current testing process.

What is Test Control?

Test Control in test execution is a process of taking actions based on results of the test monitoring process. In the test control phase, test activities are prioritized, test schedule is revised, test environment is reorganized and other changes related to testing activities are made in order to improve the quality and efficiency of future testing process.

Congratulation! We now start with Test Execution phase. While your team works on the assigned tasks, you need to monitor and control their work activity.

In the Test Management Phases tutorial, we briefly introduced Test Monitoring and Control. In this tutorial, you will learn it in detail.

Why do we monitor?

This small example shows you why we need to monitor and control test activity.

After finishing the Test Estimation and test planning, the management board agreed with your plan and the milestones are set as per the following figure.

Monitor

You promised to finish and deliver all test artifacts of the Guru99 Bank Testing project as per above milestones. Everything seems to be great, and your team is hard at work.

But after 4 weeks, things are not going as per plan. The task of “Making Test specification” is delayed by 4 working days. It has a cascading effect and all succeeding tasks are delayed.

Monitor

You missed the milestone as well the overall project deadline.

As a consequence, your project fails and your company loses the customer trust. You must take full responsibility for the project’s failure.

Monitor

Take a look the project progress, can you answer your boss’s question

Why did you miss the deadline?

I forgot to monitor and control the project progress

My team member did not work well

I don’t know why.

Correct

You miss the deadline because you forgot to monitor and control the project progress. You should review the plan and actual schedule. You will figure out that the delay happened in the very first task itself (Make Test specs). This delay accumulated in the succeeding tasks. Had you monitored the project carefully, you could have detected this issue early and find a solution to solve it.

Incorrect

No matter how much and carefully we plan, something will go wrong. We need to actively monitor the project to

  • Early detect and react appropriately to deviations and changes to plans
  • Let’s you communicate to stakeholders, sponsors, and team members exactly where the project stands and determine how closely your initial plan of action resembles reality
  • It will be helpful for the Manager to know whether the project is going on the right track according to the project goals. Allows you to make the necessary adjustments regarding resources or your budget.

Project monitoring helps you avoid disasters. Monitoring can be compared to checking the gas gauge in your car as you drive. It helps you see how much gas left in the tank, monitoring your project helps you avoid running out of gas before you reach your goal.

What do we monitor?

Monitoring will allow you to make comparisons between your original plan and your progress so far. You will be able to implement changes, where necessary, to complete the project successfully.

In your project, as the Test Manager, you should monitor the key parameters as below

Monitor

Cost

Cost

Costs are an important aspect of project monitoring and control. You have to estimate and track basic cost information for your project. Having accurate project estimates and a robust project budget is necessary to deliver project within the decided budget.

Suppose, your boss has agreed to fund the project with $100,000. You must keep an eye on the actual costs while the project is being implemented. As mentioned in Test estimation article, there’s a ton of project activities which need money. You have to monitor and manage the project budget in order to control all that activities. Without monitoring the project cost, the project will most likely never be delivered on-budget.

Schedules

How can you work without a schedule? It can be compared to driving your car but without any idea of how long it takes you get to the destination. No matter how big or small is the size & scope of your project, you must prepare a project schedule. The schedule tells you

  • When should each activity be done?
  • What has already been completed?
  • The sequence in which things need to be finished.

Here is an example of project schedule

You assigned a Team member to a Task: Executing the Integration Cases of Guru99 Bank website.

This task should be finished in one week. You can create a schedule as given below

Schedules

Resources

As mentioned in previous articles, resources are all things required to carry out the project tasks. They can be people or equipment required to complete the project activity. Lack of resources can affect the project progress.

The truth is, everything may not happen as planned, employees will leave, the project budget may be cut, or the schedule will get pushed. Monitoring resources will help you to early detect any resource crunch and find a solution to deal with it.

Quality

Quality monitoring involves monitoring the results of specific work products (like test case suite, test execution log), to evaluate whether its meets the defined quality standards. In case results do not meet quality standards, you need to identify potential resolution.

Example: Suppose that you monitored and controlled the project progress very well. Finally, you delivered the product at the deadline. The project seems to be successful.

But after delivering 2 weeks, you got this feedback from customer

Quality

What did I do wrong?

I did nothing wrong. Maybe the customer made mistake.

I forgot monitoring the quality of the project output.

I don’t know why.

Incorrect
Correct

The critical mistake you made in this project is that you forgot monitoring the quality of the project output. Because project monitoring requires monitoring not only project schedule, but also the project quality.

How to monitor?

As your project comes to life, keep these questions in mind:

  • Are you on schedule? If not, how far behind are you, and how can you catch up?
  • Are you over budget?
  • Are you still working toward the same project goal?
  • Are you running low on resources?
  • Are there warning signs of impending problems?
  • Is there pressure from management to complete the project sooner?

These are just a few of the questions you should ask yourself as you monitor the progress of your project.

It is important to monitor the progress of the project so you will know if adjustments need to be made to get it moving back in the right direction. To monitor project progress effectively, you should follow the following steps

Monitor

Step 1) Create Monitoring Plan

You cannot monitor progress unless you have a plan to monitor progress with DEFINED metrics. Similar to Test Plan, Monitoring Plan is the first and one of the most important steps in progress monitoring.

In the Monitoring Plan, you must plan carefully about

Create Monitoring Plan

  • What metrics you need to collect and measure?
  • When to collect the metrics?
  • How to evaluate the project’s progress via metrics?

What metrics need to collect and measure?

In the monitoring plan, you should clearly define what metrics you need to collect and measure. As mention in previous section, the metrics which you need to collect

  • The cost (time, money) spent for the project so far
  • How much resource (employees, equipment) are used for the project
  • The status of the task (on schedule, behind or before the schedule)
  • The quality of the work product (Run rate/pass rate, defect metrics)

When to collect the data?

Now decide when or how often you are going to collect the data for monitoring in the monitoring plan –Weekly or monthly? Or just at the start and end of the project?

As the Plan, the Guru99 Bank project will be completed in one month. In that case, we recommend you monitor the project progress weekly or daily basis.

How to evaluate the project’s progress via metrics?

In the monitoring plan, you should define the methods to evaluate the project’s progress via collected metrics. Some methods you can refer are

  • Compare the progress in plan with the actual progress that the team has made
  • Define the criteria which are used to evaluate the project’s progress. For example, if the effort to complete a task took more than 30% effort than planed a project delay.

You can refer the template of monitoring plan here. Here is a sample monitoring plan for Guru99 Bank project

Step 2) Update progress record

With time, your team member will be making progress on their project task. You must track their activity as per schedule and ask them frequently update the progress information such as time spent, task status…etc. By checking these records, you can immediately see the impact on the project plan.

One of the best methods to track the member progress is holding regular meetings.

Update Progress Record

In the meeting, all members report their current status and issues if any. If a team member or members have fallen behind or have run into obstacles, formulate a plan for identifying and solving the problem.

Let’s practice with following scenario

As defined in monitoring plan, you assigned a task “Setting up Test environment” for testing website Guru99 bank to a member in your Team. His role is a Test Administrator. He has to set up the Test Environment in 6 days. You required him to report the current status in every team meeting. Here is an example of his record of current progress

Update Progress Record

Step 3) Analyze record and make the adjustment

There’re 2 sub-steps in the steps

Analyze Record And Make The Adjustment

Step 3.1) Analyze

In this step, you compare the progress you defined in plan with the actual progress that the team has made. By analyzing the record, you can also see how much time has been spent on individual task and the total time spent on the project overall.

Let’s move back to the report, which the Test Administrators sent you, in the previous section.
In that report, what issue did you figure out?


Nothing wrong, it’s still good

The task progress seems to be delayed

I could not find any issue in that report

Incorrect
Correct

As per the plan, the Test Administrator must finish 100% his task on 6th day. However, as per current status, it is 3rd day (50% time schedule), but he just finished 20% of the task. As a result the task may be delayed and miss the deadline. Therefore, this issue can affect to the overall project.

By tracking and analyzing the project progress, you can early detect any issue which may happen to the project, and you can find out the solution to solve that issue.

Step 3.2) Adjustment

Make the necessary adjustments keep your project on track. Reassign tasks, modify schedules, or reassess your goals. This will help you keep moving toward the finish line.

In the above example, you found issues in the task “Setting up Test Environment”
What should you do?


Modify the schedules

Do nothing

Modify project goals

Ask support from other team member to speed up the task

Incorrect
Correct

As a Test Manager, you may face similar situation several times. There are a ton of solutions to solve this issue, and you may have your own solution.
In above case, we recommend you choose the solution – Ask support from other team member to speed up the task. Some other the solution such as modify the schedules or do nothing may affect the overall project. They are not the best solution

Step 4) Produce the report

Produce the Report

If your boss asks you about the current project progress, whether the progress is behind or ahead the schedule, what will you answer? You need to prepare progress report of the project.

Using the report is a good option to share the overall project progress with team members or the Management Board. It is also a useful way to show your boss whether the project is on track.

You can use some template reports to ensure the progress data is presented consistently and clearly. This article includes the report template which you can refer. Also, check a sample report for Guru99 Banking project as reference

Best Practices in Test Monitoring and Control

  • Follow the standards: One important consideration of project planning is to ensure standardization. It means that all the project activities must follow the standard process guideline. Standardized processes, tools, templates, and measurement values make analysis easy, facilitate easy communications, and help the project team members understand the situation better.

  • Documentation: What will happen if you do not write down any discussion or decision in a document? You may forget them and lose many things. You should write down discussions and decisions at the appropriate place, and establishing a formal documentation procedure for meetings. Such documentation helps you to resolve issues of miscommunication or misunderstandings among the project team.

  • Proactivity: Issues occur in all projects. The important thing is that you have to adopt a proactive approach to solve issues and problems that arise during project execution. Such issues could be budget, scope, time, quality, and human resources