What is Test Monitoring?
Test Monitoring in test execution is a process in which the testing activities and testing efforts are evaluated in order to track current progress of testing activity, finding and tracking test metrics, estimating the future actions based on the test metrics and providing feedback to the concerned team as well as stakeholders about current testing process.
What is Test Control?
Test Control in test execution is a process of taking actions based on results of the test monitoring process. In the test control phase, test activities are prioritized, test schedule is revised, test environment is reorganized and other changes related to testing activities are made in order to improve the quality and efficiency of future testing process.
Congratulation! We now start with Test Execution phase. While your team works on the assigned tasks, you need to monitor and control their work activity.
In the Test Management Phases tutorial, we briefly introduced Test Monitoring and Control. In this tutorial, you will learn it in detail.
Why do we monitor?
This small example shows you why we need to monitor and control test activity.
After finishing the Test Estimation and test planning, the management board agreed with your plan and the milestones are set as per the following figure.
You promised to finish and deliver all test artifacts of the Guru99 Bank Testing project as per above milestones. Everything seems to be great, and your team is hard at work.
But after 4 weeks, things are not going as per plan. The task of “Making Test specification” is delayed by 4 working days. It has a cascading effect and all succeeding tasks are delayed.
You missed the milestone as well the overall project deadline.
As a consequence, your project fails and your company loses the customer trust. You must take full responsibility for the project’s failure.
Take a look the project progress, can you answer your boss’s question
I forgot to monitor and control the project progress
My team member did not work well
I don’t know why.
You miss the deadline because you forgot to monitor and control the project progress. You should review the plan and actual schedule. You will figure out that the delay happened in the very first task itself (Make Test specs). This delay accumulated in the succeeding tasks. Had you monitored the project carefully, you could have detected this issue early and find a solution to solve it.
No matter how much and carefully we plan, something will go wrong. We need to actively monitor the project to
- Early detect and react appropriately to deviations and changes to plans
- Let’s you communicate to stakeholders, sponsors, and team members exactly where the project stands and determine how closely your initial plan of action resembles reality
- It will be helpful for the Manager to know whether the project is going on the right track according to the project goals. Allows you to make the necessary adjustments regarding resources or your budget.
Project monitoring helps you avoid disasters. Monitoring can be compared to checking the gas gauge in your car as you drive. It helps you see how much gas left in the tank, monitoring your project helps you avoid running out of gas before you reach your goal.
What do we monitor?
Monitoring will allow you to make comparisons between your original plan and your progress so far. You will be able to implement changes, where necessary, to complete the project successfully.
In your project, as the Test Manager, you should monitor the key parameters as below
Costs are an important aspect of project monitoring and control. You have to estimate and track basic cost information for your project. Having accurate project estimates and a robust project budget is necessary to deliver project within the decided budget.
Suppose, your boss has agreed to fund the project with $100,000. You must keep an eye on the actual costs while the project is being implemented. As mentioned in Test estimation article, there’s a ton of project activities which need money. You have to monitor and manage the project budget in order to control all that activities. Without monitoring the project cost, the project will most likely never be delivered on-budget.
How can you work without a schedule? It can be compared to driving your car but without any idea of how long it takes you get to the destination. No matter how big or small is the size & scope of your project, you must prepare a project schedule. The schedule tells you
- When should each activity be done?
- What has already been completed?
- The sequence in which things need to be finished.
Here is an example of project schedule
You assigned a Team member to a Task: Executing the Integration Cases of Guru99 Bank website.
This task should be finished in one week. You can create a schedule as given below
As mentioned in previous articles, resources are all things required to carry out the project tasks. They can be people or equipment required to complete the project activity. Lack of resources can affect the project progress.
The truth is, everything may not happen as planned, employees will leave, the project budget may be cut, or the schedule will get pushed. Monitoring resources will help you to early detect any resource crunch and find a solution to deal with it.
Quality monitoring involves monitoring the results of specific work products (like test case suite, test execution log), to evaluate whether its meets the defined quality standards. In case results do not meet quality standards, you need to identify potential resolution.
Example: Suppose that you monitored and controlled the project progress very well. Finally, you delivered the product at the deadline. The project seems to be successful.
But after delivering 2 weeks, you got this feedback from customer
I did nothing wrong. Maybe the customer made mistake.
I forgot monitoring the quality of the project output.
I don’t know why.
The critical mistake you made in this project is that you forgot monitoring the quality of the project output. Because project monitoring requires monitoring not only project schedule, but also the project quality.
How to monitor?
As your project comes to life, keep these questions in mind:
- Are you on schedule? If not, how far behind are you, and how can you catch up?
- Are you over budget?
- Are you still working toward the same project goal?
- Are you running low on resources?
- Are there warning signs of impending problems?
- Is there pressure from management to complete the project sooner?
These are just a few of the questions you should ask yourself as you monitor the progress of your project.
It is important to monitor the progress of the project so you will know if adjustments need to be made to get it moving back in the right direction. To monitor project progress effectively, you should follow the following steps
Step 1) Create Monitoring Plan
You cannot monitor progress unless you have a plan to monitor progress with DEFINED metrics. Similar to Test Plan, Monitoring Plan is the first and one of the most important steps in progress monitoring.
In the Monitoring Plan, you must plan carefully about
What metrics need to collect and measure?
In the monitoring plan, you should clearly define what metrics you need to collect and measure. As mention in previous section, the metrics which you need to collect
- The cost (time, money) spent for the project so far
- How much resource (employees, equipment) are used for the project
- The status of the task (on schedule, behind or before the schedule)
- The quality of the work product (Run rate/pass rate, defect metrics)
When to collect the data?
Now decide when or how often you are going to collect the data for monitoring in the monitoring plan –Weekly or monthly? Or just at the start and end of the project?
As the Plan, the Guru99 Bank project will be completed in one month. In that case, we recommend you monitor the project progress weekly or daily basis.
How to evaluate the project’s progress via metrics?
In the monitoring plan, you should define the methods to evaluate the project’s progress via collected metrics. Some methods you can refer are
- Compare the progress in plan with the actual progress that the team has made
- Define the criteria which are used to evaluate the project’s progress. For example, if the effort to complete a task took more than 30% effort than planed a project delay.
Step 2) Update progress record
With time, your team member will be making progress on their project task. You must track their activity as per schedule and ask them frequently update the progress information such as time spent, task status…etc. By checking these records, you can immediately see the impact on the project plan.
One of the best methods to track the member progress is holding regular meetings.
In the meeting, all members report their current status and issues if any. If a team member or members have fallen behind or have run into obstacles, formulate a plan for identifying and solving the problem.
Let’s practice with following scenario
As defined in monitoring plan, you assigned a task “Setting up Test environment” for testing website Guru99 bank to a member in your Team. His role is a Test Administrator. He has to set up the Test Environment in 6 days. You required him to report the current status in every team meeting. Here is an example of his record of current progress
Step 3) Analyze record and make the adjustment
There’re 2 sub-steps in the steps
Step 3.1) Analyze
In this step, you compare the progress you defined in plan with the actual progress that the team has made. By analyzing the record, you can also see how much time has been spent on individual task and the total time spent on the project overall.
In that report, what issue did you figure out?
Nothing wrong, it’s still good
The task progress seems to be delayed
I could not find any issue in that report
As per the plan, the Test Administrator must finish 100% his task on 6th day. However, as per current status, it is 3rd day (50% time schedule), but he just finished 20% of the task. As a result the task may be delayed and miss the deadline. Therefore, this issue can affect to the overall project.
By tracking and analyzing the project progress, you can early detect any issue which may happen to the project, and you can find out the solution to solve that issue.
Step 3.2) Adjustment
Make the necessary adjustments keep your project on track. Reassign tasks, modify schedules, or reassess your goals. This will help you keep moving toward the finish line.
What should you do?
Modify the schedules
Modify project goals
Ask support from other team member to speed up the task
As a Test Manager, you may face similar situation several times. There are a ton of solutions to solve this issue, and you may have your own solution.
In above case, we recommend you choose the solution – Ask support from other team member to speed up the task. Some other the solution such as modify the schedules or do nothing may affect the overall project. They are not the best solution
Step 4) Produce the report
If your boss asks you about the current project progress, whether the progress is behind or ahead the schedule, what will you answer? You need to prepare progress report of the project.
Using the report is a good option to share the overall project progress with team members or the Management Board. It is also a useful way to show your boss whether the project is on track.
You can use some template reports to ensure the progress data is presented consistently and clearly. This article includes the report template which you can refer. Also, check a sample report for Guru99 Banking project as reference
Best Practices in Test Monitoring and Control
Follow the standards: One important consideration of project planning is to ensure standardization. It means that all the project activities must follow the standard process guideline. Standardized processes, tools, templates, and measurement values make analysis easy, facilitate easy communications, and help the project team members understand the situation better.
Documentation: What will happen if you do not write down any discussion or decision in a document? You may forget them and lose many things. You should write down discussions and decisions at the appropriate place, and establishing a formal documentation procedure for meetings. Such documentation helps you to resolve issues of miscommunication or misunderstandings among the project team.
Proactivity: Issues occur in all projects. The important thing is that you have to adopt a proactive approach to solve issues and problems that arise during project execution. Such issues could be budget, scope, time, quality, and human resources