Conformance Testing is a software testing technique used to certify that the software system complies with the standards and regulations as defined by IEEE, W3C or ETSI. The purpose of conformance testing is to determine how a system under test confirms to meet the individual requirements of a particular standard. Conformance Testing is also called Compliance Testing.
It may deal with some technical aspect but intentionally it includes:
- Behavior of system
In this tutorial, you will learn-
- What is Conformance Testing?
- Types of Conformance Testing
- Why do we need Conformance Testing?
- What do we need to test?
- When and how to perform Conformance Testing?
- Conformance Testing Process/Program:
- Advantages of Conformance Testing:
- Disadvantages of Conformance Testing:
- Conformance Testing for Mobile System:
Conformance Testing can be logical or physical, and it comprises following types of testing;
- To check for the system’s requirements fulfillment
- To check whether the system documentation is complete with needful
- To check the development, design and evaluation as per specifications
- The standards through which the implementation takes place
- The call of the system that is to be developed
- Scope of specifications
- Specification objectives
Conformance Testing is initiated by the management with total assurance about the team and their understanding of standards, specifications, and procedures.
To build an efficient application, standards and specifications should be clearly mentioned to avoid ambiguities. If not then conformance testing is itself useful to take the necessary steps to make it relevant and reliable.
- When we need to check the system’s overall consistency, completeness and correctness with respect to requirement specifications at that point conformance testing are performed
- There are several methodologies for performing it not much different from our Development Lifecycle Stages such as;
- Analyzing requirements specifications
- Preparing test plan
- Preparing test cases and its purpose
- Create documentation for prepared test designs
- We need to focus on some additional aspects of specifications such as subsets, which are generally of three types such as ‘Profile’, ‘Level’ and ‘Modules’.
- Profile is a subset of specifications which includes system’s functionality to satisfy requirements and dedicated to a particular group of users
- Levels are nested subset of specifications where Level1 indicates basic specifications and Level2 indicates Level1 + additional/new functionalities
- Modules are collection of significantly correlated standards and specifications
Conformance Testing Process or Compliance Testing Process is necessary to ensure efficiency, performance and to avoid future risks for a software application. An effective conformance testing includes following step by step process :
- Analysing Standards and Specifications
- Selecting Test tools and Test suite
- Designing Testing Procedures
- Checking necessary validations
- Adapting relevant testing policies and certifications
Let us have a quick look at diagrammatic representation of Conformance/Compliance Testing Process for better understanding :
With the help of Use Case, this process will be easier to understand;
It is mentioned above that Conformance Testing occasionally called Compliance Testing but still there are some basic facts which make Conformance Testing different from Compliance Testing.
We will review the compare and contrast between both;
|Conformance Testing||Compliance Testing|
- Assures the proper implementation of specifications
- Assures portability and interoperability
- Provides appropriate utilization of standards
- Makes possible to assure that interfaces and functions are working as expected
- Helps to identify the areas that are to be conformed with those which are not to be conformed such as syntax and semantics
Like other types of testing, Conformance Testing also has some tasks that are quite difficult to perform and may become a challenge. Some of them are listed below;
- To perform testing for conformance efficiently, it needs to identify the class of the system need to be tested along with the suitable methodology
- Categorizing specifications into Profiles, Levels, and Modules
- Predefining values
- Designing extensions, options and approaches to run the testing process successfully.
Conformance Testing: Critical Views:
Conformance Testing has some areas that need extra attention and knowledge for successful implementation;
- Testing methodologies that are best suited
- Selection of test tool that will automate the testing process
- Clear understanding of Conformance Testing that shows only non-conformance instead of conformance
- Designing and Implementing Conformance Testing Program
- Defining how formally it should be performed
- Policies and regulation
Just like other types of Software Testing, Conformance Testing can also be done for Mobile systems. Once the system is assembled, it is checked for the use with prior to its deployment through Conformance test.
According to the type of networks such as GSM or CDMA, the tests are performed to achieve conformance and interoperability.
These tests may include;
- Protocol Testing
- Safety/Security Testing
- SIM card Testing
- Radio Frequency(RF) Testing
- Audio Tests
- Specific Absorption Tests
Example Use Case Diagram
Following Use Case Diagram shows how the conformance testing is performed for deployed mobile systems;
Furthermore, this can elaborated in the following manner for SIM Card Testing in two stages;
- Password Validation
- Making Call
There can be two cases for a system such as
1. Call Succeed
2. Call Failed
Similarly, for Protocol Testing consider following Use Case;
Following Test Case Template will help you to understand how we can write test cases for Protocol Testing
|Test ID||Test Scenario||Test Steps||Test Data||Expected Result||Actual Result||Status||Remarks|
|PT001||Protocol has no deadlock||Start with the communication process||_||Should establish communication successfully||Communication established successfully||Pass||_|
|PT002||“Ping” Command operates correctly for Timeout||Enter Ping command as ‘Ping -w’ in command prompt||Ping 1000||Should show message “Transmit Successful.”||The message shows “Transmit Successful.”||Pass||_|
|PT003||Count the no. of ‘echo’ requests||Enter Ping command as ‘Ping -n’ in command prompt||Ping 4||Should Start pinging 0.0.0.4 with 32 bytes data and Should show the message “Transmit Successful.”||Starts pinging 0.0.0.4 with 32 bytes data but shows the message “Transmit Failed.”||Fail||Semantic/Functional error|
|PT004||Any state can be reached from any state||Exit from a state and return to previous or next||_||Should successfully change the state||Successfully changes the state||Pass||_|
These terms show that Conformance Testing is also useful to test the implementation of technical standards.
To bring uniformity into the test process, the test cases are being defined. Depending upon the type of networks, hundreds of test cases may be defined and executed to ensure the working of the mobile system as per requirements specification.
- Conformance Testing is also called as Type Testing which is a formal way of testing
- It is used to determine whether the system follows specification of requirements
- It includes Stress Testing, Load Testing, Volume Testing and Compliance Testing
- It is also considered as Compliance Testing but there basic differences between both
- Conformance Testing Program, Testing Methodologies and Test Tools are some most critical areas of Conformance Testing
- Assurance for standard utilization and interoperability is achieved through Conformance Testing in Software Engineering.
- Conformance Testing is performed for Mobile Systems too.