Risk Based Testing: Approach, Matrix, Process & Examples

Risk Based Testing

Risk Based Testing (RBT) is a software testing type which is based on the probability of risk. It involves assessing the risk based on software complexity, criticality of the business, frequency of use and the areas most likely to contain a defect. Risk Based Testing prioritizes testing of the features and functions of the software application which are more impactful and more likely to have defects.

Risk is the occurrence of an uncertain event with a positive or negative effect on the measurable success criteria of a project. It could be an event that occurred in the past, a current event, or something that could happen in the future. These uncertain events can have an impact on the cost, business, technical and quality targets of a project.

Risks can be positive or negative.

• Positive risks are referred to as opportunities and help in business sustainability. Examples include investing in a new project, changing business processes, and developing new products.
• Negative risks are referred to as threats, and recommendations to minimize or eliminate them must be implemented for project success.

Because the technique allocates effort rather than adding a new test level, it sits on top of the other types of software testing rather than replacing any of them.

When to Implement Risk Based Testing

Risk Based Testing can be implemented in

• Projects having time, resource or budget constraints.
• Projects where risk based analysis can be used to detect vulnerabilities to SQL injection attacks.
• Security testing in cloud computing environments.
• New projects with high risk factors, such as a lack of experience with the technologies used or a lack of business domain knowledge.
• Incremental and iterative delivery models.

Risk Management Process

Let us now understand the steps involved in the Risk Management Process.

Risk Identification

Risk identification can be done through risk workshops, checklists, brainstorming, interviewing, the Delphi technique, cause and effect diagrams, lessons learnt from previous projects, root cause analysis, and contacting domain experts and subject matter experts.

A Risk Register is a spreadsheet which holds the list of identified risks, potential responses, and root causes. It is used to monitor and track the risks (both threats and opportunities) throughout the life of the project. Risk response strategies can be used to manage positive and negative risks.

A Risk Breakdown Structure plays an important role in risk planning. It helps in identifying the risk prone areas and supports effective evaluation and risk monitoring over the course of the project. It helps in allocating sufficient time and resources for risk management activities, and in categorizing the many sources from which project risks may arise.

The sample below shows how a Risk Breakdown Structure groups project risks into categories so that no source of risk is overlooked.

Risk Analysis (Includes Quantitative and Qualitative Analysis)

Once the list of potential risks has been identified, the next step is to analyze them and to filter the risks by significance. One of the qualitative risk analysis techniques is the Risk Matrix (covered in a later section). This technique is used to determine the probability and impact of the risk.

Risk Response Planning

Based on the analysis, we can decide whether the risks require a response. For example, some risks will require a response in the project plan, some require a response in project monitoring, and some will not require any response at all.

The risk owner is responsible for identifying options to reduce the probability and impact of the assigned risks.

Risk mitigation is a risk response method used to lessen the adverse impacts of possible threats. This can be done by eliminating the risks or reducing them to an acceptable level. The diagram below places risk response planning within the wider risk management cycle.

Risk Contingency

Contingency can be described as the possibility of an uncertain event whose impact is unknown or unpredictable. A contingency plan is also known as the action plan or back-up plan for worst case scenarios. In other words, it determines what steps could be taken when an unpredictable event materializes.

Risk Monitoring and Control

The risk control and monitoring process is used to track the identified risks, monitor residual risks, identify new risks, update the risk register, analyze the reasons for any change, execute the risk response plan, and monitor risk triggers. Their effectiveness in reducing risk is then evaluated.

This can be achieved by risk reassessments, risk audits, variance and trend analysis, technical performance measurement, status update meetings and retrospective meetings.

The table below gives information about the inputs, tools and outputs of risk monitoring and control.

We need to remember that risk increases with changes in technology, the size of the project, the length of the project (a longer project timeframe), the number of sponsoring agencies, project estimates, effort, and a shortage of appropriate skills.

Risk Based Testing Approach

The management process above feeds the test approach below. Each numbered step produces an input that the next step consumes.

1. Analyze the requirements.
   • Documents (SRS, FRS, use cases) are reviewed. This activity is done to find and eliminate errors and ambiguities.
   • Requirements sign-off is one of the risk-reduction techniques for avoiding the introduction of late changes into the project. Any change to a requirement after the document is baselined involves a change control process and subsequent approvals.
2. Assess the risks by calculating the likelihood and impact each requirement could have on the project, taking the defined criteria such as cost, schedule, resources, scope, technical performance, safety, reliability and complexity into consideration.
   • Identify the probability of failure and the high-risk areas. This can be done using a risk assessment matrix.
   • Use a risk register to list the set of identified risks. Update, monitor and track the risks periodically at regular intervals.
   • Risk profiling needs to be done at this stage to understand the risk capacity and risk tolerance levels.
3. Prioritize the requirements based on the rating.
   • The risk-based test process is defined.
   • Highly critical and medium risks can be considered for mitigation planning, implementation and progress monitoring. Low risks can be kept on a watch list.
   • Risk data quality assessment is done to analyze the quality of the data.
4. Plan and define the tests according to the rating.
   • Apply an appropriate testing approach and test design techniques so that the highest risk items are tested first. High-risk items can be tested by a resource with good domain knowledge and experience.
   • Different test design techniques can be used — for example, the decision table technique on high-risk test items, and only equivalence partitioning for low-risk test items.
   • Test cases are also designed to cover multiple functionalities and end to end business scenarios.
   • Prepare the test data, test conditions and test bed.
5. Review the test documentation — the test plans, test strategy, test cases, test reports and any other document created by the testing team.
   • Peer review is an important step in defect identification and risk reduction.
6. Perform dry runs and quality checks on the results.
   • Test cases are executed according to the priority of the risk item.
   • Maintain traceability between risk items, the tests that cover them, the results of those tests, and the defects found during testing. All testing strategies executed properly will reduce quality risks.
   • Risk Based Testing can be used at every level of testing — component, integration, system and acceptance testing.
   • At the system level, we need to focus on what is most important in the application. This can be determined by looking at the visibility of functions, the frequency of use and the possible cost of failure.
   • Evaluation of exit criteria: all high-risk areas fully tested, with only minor residual risks left outstanding.
7. Report the risk-based test results and analyze the metrics.
   • Reassess existing risk events and new risk events based on Key Risk Indicators.
   • Update the risk register.
   • Contingency plans work as a fallback or emergency plan for the high exposure risks.
   • Defect analysis and defect prevention are used to eliminate the defects.
   • Retesting and Regression Testing validate the defect fixes based on the pre-calculated risk analysis, and high-risk areas should be the most intensively covered.
   • Risk-based automation testing, if feasible.
   • Residual risk calculation.
8. Monitor and control the risks.
   • Exit criteria or completion criteria can be defined separately for different risk levels. All key risks have been addressed with appropriate actions or contingency plans, and risk exposure is at or below the level agreed as acceptable for the project.
   • Risk profiling reassessment and customer feedback.

Risk Based Testing Approach to the System Test

1. Technical System Test — This is referred to as the environment test and integration test. The environment test includes testing in the development, testing and production environments.
2. Functional System Test — Testing of all functionalities, features, programs and modules. The purpose of this test is to evaluate whether the system meets its specified requirements.
3. Non-functional System Test — Testing the non-functional requirements: performance, load tests, stress tests, configuration tests, security tests, backup and recovery procedures, and documentation (system, operation and installation documentation).

The diagram below gives a clear overview of the above-mentioned process.

System testing includes both functional tests and non-functional tests.

Functional testing ensures that the product or application meets customer and business requirements. On the other hand, non-functional testing is done to verify whether the product stands up to the customer’s expectations in terms of quality, reliability, usability, performance and compatibility.

How to Do Risk Based Testing: Complete Process

This section covers the risk based test process, which runs in five phases.

1. Risk Identification
2. Risk Analysis
3. Risk Response
4. Test Scoping
5. Test Process definition

The five phases feed into one another as shown below.

1. In this process, the risks are identified and categorized, a draft register of risks is prepared, and risk sorting is done to identify the significant risks.
2. Risk response involves formulating the test objectives from the risks, and selecting appropriate techniques so that the test activity or test technique meets those test objectives.
3. Documented dependencies, requirements, cost and the time required for software testing are considered in order to calculate the test effectiveness score.
4. Test scoping is a review activity that requires the participation of all stakeholders and technical staff. It is important to adhere to the agreed scope of risks. These risks need to be addressed by testing, and all members must agree with the responsibilities assigned to them and the budget allocated for these activities.
5. After the scope of testing has been finalized, the test objectives, assumptions and dependencies for each test stage have to be compiled in the standard format.

The worked example below maps each requirement to its associated risk and to the test objective that addresses it.

Let us consider the functional requirements F1, F2 and F3, and the non-functional requirements N1 and N2.

F1 — Functional Requirement, R1 — Risk associated with F1

• Test Objective 1 — Demonstrate using a test that the expected features and functionalities of the system work correctly, and that the risk R1 can be addressed by functional testing.
• Test — Browser page testing is done to execute important user tasks and verify that R1 (the risk associated with F1) could be addressed in a range of scenarios.

F2 — Functional Requirement, R2 — Risk associated with F2

• Test Objective 2 — Demonstrate using a test that the expected features and functionalities of the system work correctly, and that the risk R2 can be addressed by functional testing.
• Test — Browser page testing is done to execute important user tasks and verify that R2 could be addressed in a range of scenarios.

F3 — Functional Requirement, R3 — Risk associated with F3

• Test Objective 3 — Demonstrate using a test that the expected features and functionalities of the system work correctly, and that the risk R3 can be addressed by functional testing.
• Test — Browser page testing is done to execute important user tasks and verify that R3 could be addressed in a range of scenarios.

N1 — Non-functional Requirement, NR1 — Risk associated with N1

• Test Objective N1 — Demonstrate using a test that the operational characteristics of the system work correctly, and that the risk NR1 can be addressed by non-functional testing.
• Test — Usability testing is a technique used to assess how easy user interfaces are to use, and to verify that NR1 could be addressed by usability testing.

N2 — Non-functional Requirement, NR2 — Risk associated with N2

• Test Objective N2 — Demonstrate using a test that the operational characteristics of the system work correctly, and that the risk NR2 can be addressed by non-functional testing.
• Test — Security testing is a technique used to check whether the application is secure or vulnerable to attack, whether there is any information leakage, and to verify that NR2 could be addressed by security testing.

Specific test objectives: the risks and test objectives listed are specific to the test types, as summarised below.

Procedure to Design the Risk Based Test Process

• Prepare a risk register. This records the risks derived from a generic risk list, an existing checklist and brainstorming sessions.
• Include the risks associated with the system’s functional and non-functional requirements (usability, security, performance).
• Each risk is allocated a unique identifier.

Columns 1 and 2 of that register hold the identifier and the risk description. The remaining columns are described below.

The probability (1 low, 5 high) and consequences (1 low, 5 high) of each risk are assessed, as the two register extracts below show.

• Test exposure is computed.
• The tester analyzes each risk and evaluates whether the risk is testable or not.
• Test objectives are defined for the testable risks.
• The tester specifies the test activity that should be carried out in a planned way to meet the test objective (static reviews, inspections, system tests, integration tests, acceptance tests, HTML validation, localization testing and so on).
• These testing activities can be classified into stages (component testing or unit testing, integration testing, system testing, acceptance testing).
• At times, a risk might be addressed by more than one test stage.
• Identify the dependencies and assumptions (availability of skills, tools, test environments and resources).
• Test effectiveness is computed. Test effectiveness relates to the confidence level of the tester that the risk will be definitively addressed through testing. The test effectiveness score is a number between one and five (5 = high confidence, 1 = low confidence).
• Estimate the effort, the time required and the cost to prepare and execute these tests.

The next two extracts show the remaining register columns and the test effectiveness score in place.

• The test priority number is calculated. It is the product of the probability, consequences and test effectiveness scores.
• 125 (maximum) — a very serious risk that could be detected with testing.
• 1 (minimum) — a very low risk that would not be detected with testing.
• Based on the test priority number, the test importance can be classified as High (red), Medium (yellow) and Low (green). The highest risk items are tested first.
• Allocate the test activities to the test stages. Designate the group that will perform testing for each objective in the different test stages (unit testing, integration testing, system testing, acceptance testing).

The allocation across the test stages is shown below.

What is in scope and out of scope for testing is decided in the test scoping phase.

• For each stage, the test objectives, component under test, responsibility, environment, entry criteria, exit criteria, tools, techniques and deliverables are defined.

Generic test objectives — these generic objectives are applicable to multiple projects and applications.

• The component meets the requirement and is ready for use in larger subsystems.
• The risks associated with the specific test types are addressed, and the test objectives are accomplished.
• Integrated components are correctly assembled, and interface compatibility among the components is ensured.
• The system meets the specified functional and non-functional requirements.
• Product components satisfy end user needs in their intended operating environment.
• A risk management strategy is used to identify, analyze and mitigate risks.
• The system meets industry regulation requirements.
• The system meets contractual obligations.
• Institutionalization and the achievement of other specific objectives such as cost, schedule and quality objectives.
• Systems, processes and people meet business requirements.

Generic test objectives can be defined for the different test stages.

• Component Testing
• Integration Testing
• System Testing
• Acceptance Testing

Let us consider the system test stage.

1. G4 and G5 demonstrate that the system meets the functional requirements (F1, F2, F3) and the non-functional requirements (N1, N2).
2. Demonstrate using tests that the expected features and functionalities of the system work correctly, and that the risks associated with F1, F2 and F3 can be addressed by functional testing.
3. Demonstrate using tests that the operational characteristics of the system work correctly, and that the risks associated with N1 and N2 can be addressed by non-functional testing.
4. Based on the test priority number, the test importance can be classified as High (red), Medium (yellow) and Low (green).

Prioritization and Risk Assessment Matrix

The risk assessment matrix is the probability-impact matrix. It provides the project team with a quick view of the risks and the priority with which each of these risks needs to be addressed.

Risk rating = Probability x Severity

Probability is the measure of the chance that an uncertain event will occur, based on exposure in terms of time, proximity and repetition. It is expressed as a percentage.

This can be classified as Frequent (A), Probable (B), Occasional (C), Remote (D), Improbable (E) and Eliminated (F).

• Frequent — Expected to occur several times in most circumstances (91 – 100%).
• Probable — Likely to occur several times in most circumstances (61 – 90%).
• Occasional — Might occur sometime (41 – 60%).
• Remote — Unlikely to occur, though it could occur sometime (11 – 40%).
• Improbable — May occur in rare and exceptional circumstances (0 – 10%).
• Eliminated — Impossible to occur (0%).

Severity is the degree of impact of the damage or loss caused by the uncertain event. It is scored 1 to 4 and can be classified as Catastrophic = 1, Critical = 2, Marginal = 3 and Negligible = 4.

• Catastrophic — Harsh consequences that make the project completely unproductive and could even lead to project shutdown. This must be a top priority during risk management.
• Critical — Large consequences which can lead to a great amount of loss. The project is severely threatened.
• Marginal — Short term damage that is still reversible through restoration activities.
• Negligible — Little or minimal damage or loss. This can be monitored and managed by routine procedures.

The priority is classified into four categories, which are mapped against the severity and probability of the risk as shown in the image below.

• Serious
• High
• Medium
• Low

Serious: The risks that fall in this category are marked in amber. The activity must be stopped, and immediate action must be taken to isolate the risk. Effective controls must be identified and implemented. Further, the activity must not proceed unless the risk is reduced to a low or medium level.

High: The risks that fall in this category are marked in red and require immediate action or a risk management strategy. Immediate action must be taken to isolate, eliminate or substitute the risk and to implement effective risk controls. If these issues cannot be resolved immediately, strict timelines must be defined to resolve them.

Medium: The risks that fall in this category are marked in yellow. Reasonable and practical steps must be taken to minimize the risks.

Low: The risks that fall in this category are marked in green and can usually be accepted, as they do not pose any significant problem. A periodic review is still a must to ensure the controls remain effective.

Generic Checklist for Risk Based Testing

The matrix decides how a risk is rated. The checklist below decides which candidates go into the matrix in the first place.

• Important functionalities in the project.
• User visible functionality in the project.
• The functionality having the largest safety impact.
• Functionalities that have the largest financial impact on users.
• Highly complex areas of source code and error prone code.
• Features or functions that can be tested early in the development cycle.
• Features or functionalities that were added to the product design at the last minute.
• Critical factors of similar or related previous projects that caused problems.
• Prime factors or issues of similar or related projects that had a huge impact on operation and maintenance expenses.
• Poor requirements that lead to poor designs and tests, which could have an impact on the project goals and deliverables.
• In the worst case scenario, a product may be so defective that it cannot be reworked and must be completely scrapped, which would cause serious harm to the company’s reputation. Identify what kind of problems are crucial to the product objectives.
• Situations or problems that would cause persistent customer service complaints.
• End to end tests that could easily focus on multiple functionalities of the system.
• The optimal set of tests that can maximize the risk coverage.
• Which tests will have the best high-risk-coverage to time-required ratio.

Risk Based Testing Results Reporting and Metrics

1. Test report preparation. Reporting test status is about effectively communicating the test results to the project stakeholders, giving a clear understanding and showing the comparison of test results against the test objectives.
   • Number of test cases planned versus executed.
   • Number of test cases passed or failed.
   • Number of defects identified, and their status and severity.
   • Number of critical defects still open.
   • Environment downtimes, if any.
   • Showstoppers, if any.
   • Test summary report and test coverage report.
2. Metrics preparation. A metric is a combination of two or more measures used to compare software processes, projects and products.
   • Effort and schedule variation.
   • Test case preparation productivity.
   • Test design coverage.
   • Test case execution productivity.
   • Risk identification efficiency %.
   • Risk mitigation efficiency %.
   • Test effectiveness %.
   • Test execution coverage.
   • Test execution productivity.
   • Defect leakage %.
   • Defect detection efficiency, and defect density.
   • Requirement stability index.
   • Cost of quality.

Those measures are then read back against the risks:

• Analyze the risks in non-functional categories (performance, reliability and usability) based on defect status and the number of test pass or fail results, in relation to the risks.
• Analyze the risks in functional categories using testing metrics, defect status and test pass or fail status, in relation to the risks.
• Identify key lead and lag indicators and create early warning indicators.
• Monitor and report on lead and lag risk indicators (Key Risk Indicators) by analyzing the data patterns, trends and interdependencies.

Inherent Risk vs. Residual Risk Assessment

Risk identification and analysis should also include inherent risks, residual risks, secondary risks and recurrent risks.

• Inherent Risk: The risks that were identified or already present in the system before the controls and responses were implemented. Inherent risks are also known as gross risks.
• Residual Risk: The risks that are left over after the controls and responses have been implemented. Residual risks are known as net risks.
• Secondary Risk: The new risk caused by the implementation of the risk response plan.
• Recurrent Risk: The likelihood that the initial risks will occur again.

Test result measurement based on risk helps the organization to know the residual level of quality risk during test execution, and to make informed release decisions.

Risk Profiling and Customer Feedback

Risk profiling is a process for finding the optimal level of investment risk for the client, considering the risk required, the risk capacity and the risk tolerance.

1. Risk required is the level of risk the client needs to take in order to obtain a satisfactory return.
2. Risk capacity is the level of financial risk the client can afford to take.
3. Risk tolerance is the level of risk which the client would prefer to take.

Customer feedback: gather customer feedback and reviews to improve the business, product, service and experience.

Benefits of Risk Based Testing

The benefits of Risk Based Testing are given below.

• Improved productivity and cost reduction.
• Improved market opportunity (time to market) and on-time delivery.
• Improved service performance.
• Improved quality, as all of the critical functions of the application are tested.
• Clear information on test coverage. Using this approach, the team knows what has and has not been tested.
• Test effort allocation based on risk assessment is the most efficient and effective way to minimize the residual risk upon release.
• Test result measurement based on risk analysis enables the organization to identify the residual level of quality risk during test execution, and to make informed release decisions.
• Optimized testing with clearly defined risk evaluation methods.
• Improved customer satisfaction, due to customer involvement and good reporting and progress tracking.
• Early detection of potential problem areas, so effective preventive measures can be taken.
• Continuous risk monitoring and assessment throughout the project’s entire lifecycle helps in identifying and resolving risks, and in addressing the issues that could endanger the achievement of overall project goals.

FAQs