Compliance Testing (Conformance Testing) in Software Testing

By : Thomas Hamilton Thomas Hamilton
Hours Updated

⚡ Smart Summary

Conformance Testing certifies that a software system meets the standards and regulations set by bodies such as IEEE, W3C or ETSI, covering performance, functions, robustness, interoperability and overall system behavior.

  • 🔘 Also called: Compliance Testing or Type Testing, a formal verification against a published standard.
  • 📚 Coverage: Includes Compliance, Load, Stress and Volume Testing within one conformance program.
  • 🧭 Subsets: Specifications are grouped into Profiles, Levels and Modules before test design starts.
  • 🔄 Process: Analyze standards, select tools, design procedures, validate, then adopt certification policies.
  • 📱 Mobile: GSM and CDMA devices add protocol, SIM, RF, audio and absorption tests.
  • ⚖️ Distinction: Conformance is formal and full-support; compliance is informal and partial-support.
Read More

Conformance Testing and Compliance Testing process in software testing

Conformance Testing

Conformance Testing is a software testing technique used to certify that a software system complies with the standards and regulations defined by bodies such as IEEE, W3C or ETSI. It determines how well a system under test conforms to the individual requirements of a particular standard. Conformance Testing is also called Compliance Testing.

It may deal with technical aspects, but by design it also covers:

  • Performance
  • Functions
  • Robustness
  • Interoperability
  • Behavior of system

Types of Conformance Testing

Conformance Testing can be logical or physical, and it comprises the following types of testing:

The diagram below shows how these testing types sit together inside a single conformance program.

Types of conformance testing grouped as compliance, load, stress and volume testing

Why do we need Conformance Testing?

  • To check the fulfillment of the system requirements
  • To check whether the system documentation is complete
  • To check the development, design and evaluation against the specifications

What do we need to test?

  • The standards through which the implementation takes place
  • The call of the system that is to be developed
  • Scope of specifications
  • Specification objectives

Conformance Testing is initiated by management, with assurance that the team understands the applicable standards, specifications and procedures.

Standards and specifications should be stated clearly to avoid ambiguity. Where they are not, conformance testing is itself useful in identifying the steps needed to make them relevant and reliable.

When and how to perform Conformance Testing?

Conformance testing is performed when the overall consistency, completeness and correctness of the system has to be checked against the requirement specifications. The methodologies used are not very different from the ordinary development lifecycle stages:

  • Analyzing requirements specifications
  • Preparing the test plan
  • Preparing test cases and stating their purpose
  • Creating documentation for the prepared test designs

Some additional aspects of the specifications need attention, in particular subsets, which are generally of three types — Profile, Level and Modules:

  • Profile is a subset of the specifications that includes the system functionality required to satisfy a particular group of users
  • Levels are nested subsets of the specifications, where Level1 indicates the basic specifications and Level2 indicates Level1 plus additional or new functionality
  • Modules are collections of significantly correlated standards and specifications

Conformance Testing Process

The Conformance Testing Process, or Compliance Testing Process, is necessary to ensure efficiency and performance and to avoid future risk for a software application. An effective conformance test follows this step-by-step process:

  1. Analyzing standards and specifications
  2. Selecting test tools and the test suite
  3. Designing testing procedures
  4. Checking necessary validations
  5. Adopting relevant testing policies and certifications

The diagrammatic representation of the Conformance and Compliance Testing Process below makes the sequence easier to follow.

Five stage conformance testing process from analyzing standards to certification

Expressed as a use case, the same process shows which actor performs each stage.

Use case diagram of the conformance testing process with tester and certification actors

RELATED ARTICLES

Conformance Testing vs Compliance Testing

Conformance Testing is occasionally called Compliance Testing, yet a few basic facts separate the two. The table below compares them.

Conformance Testing Compliance Testing
A formal and precise way of testing against standards An informal and less precise way of testing against standards
Certification of conformance applies only to an operating system holding official Certification Authority approval An operating system that provides a single API of POSIX (Portable Operating System Interface) is said to be compliant
Used for testing a system that provides full support for the given standards Used for testing a system that provides support for only some of the given standards

Advantages of Conformance Testing

  1. Assures the proper implementation of the specifications
  2. Assures portability and interoperability
  3. Provides appropriate utilization of standards
  4. Makes it possible to assure that interfaces and functions work as expected
  5. Helps identify the areas that must conform and those that need not, such as syntax and semantics

Disadvantages of Conformance Testing

Like other types of testing, Conformance Testing carries tasks that are difficult to perform and may become a challenge. Some of them are listed below:

  1. Efficient conformance testing requires the class of the system under test to be identified along with a suitable methodology
  2. Categorizing specifications into Profiles, Levels and Modules
  3. Predefining values
  4. Designing the extensions, options and approaches needed to run the testing process successfully

Conformance Testing: Critical Views

Conformance Testing has areas that need extra attention and knowledge for successful implementation:

  1. Testing methodologies that are best suited
  2. Selection of a test tool that will automate the testing process
  3. Clear understanding that conformance testing shows only non-conformance, never conformance itself
  4. Designing and implementing the Conformance Testing Program
  5. Defining how formally it should be performed
  6. Policies and regulation

Conformance Testing for Mobile System

Like other types of software testing, Conformance Testing can also be done for mobile systems. Once the system is assembled, it is checked for use prior to deployment through a conformance test.

According to the type of network, such as GSM or CDMA, the tests are performed to achieve conformance and interoperability. These tests may include:

Example Use Case Diagram

The following use case diagram shows how conformance testing is performed for deployed mobile systems.

Use case diagram for conformance testing of a deployed mobile system

This can be elaborated for SIM card testing in two stages. The first stage is password validation, shown below.

SIM card password validation use case with correct and incorrect PIN paths

The second stage is making a call, which has two outcomes, call succeeded and call failed.

Making a call use case showing call succeeded and call failed outcomes

Protocol testing follows the same pattern, as this use case shows.

Protocol testing use case for a mobile system under conformance test

The test case template below shows how test cases are written for protocol testing.

Test ID Test Scenario Test Steps Test Data Expected Result Actual Result Status Remarks
PT001 Protocol has no deadlock Start with the communication process _ Should establish communication successfully Communication established successfully Pass _
PT002 “Ping” command operates correctly for timeout Enter Ping command as ‘Ping -w’ in command prompt Ping 1000 Should show message “Transmit Successful.” The message shows “Transmit Successful.” Pass _
PT003 Count the number of ‘echo’ requests Enter Ping command as ‘Ping -n’ in command prompt Ping 4 Should start pinging 0.0.0.4 with 32 bytes data and show the message “Transmit Successful.” Starts pinging 0.0.0.4 with 32 bytes data but shows the message “Transmit Failed.” Fail Semantic or functional error
PT004 Any state can be reached from any state Exit from a state and return to the previous or next one _ Should successfully change the state Successfully changes the state Pass _

Templates like this bring uniformity to the test process. Depending on the type of network, hundreds of test cases may be defined and executed to confirm the mobile system works as the requirement specification demands.

FAQs

IEEE, W3C, ETSI and 3GPP are the ones named most often, alongside ISO. Each publishes the specification and, in many cases, an official test suite a laboratory runs before granting certification.

Models read a long specification and map each clause to a candidate test case, which shortens the traceability work. A qualified reviewer still signs off, because certification evidence must be defensible to an auditor.

Yes, for the repetitive harness code and assertions around a published test suite. Agentic runners can also re-execute the suite on every build, though the pass criteria must come from the standard, not the assistant.

An accredited certification authority or test laboratory recognized by the standards body, not the development team. Self-declared results are usually described as a declaration of conformity rather than a certificate.

Whenever the standard is revised or the implementation changes in a way that touches a certified interface. Many certificates also carry a validity period, after which the suite is re-executed.

Conformance proves one implementation matches the specification. Interoperability testing then proves two independently built implementations actually work together. Passing the first does not guarantee the second.

Predominantly black box. The test suite exercises the published interface and compares observed behavior with the standard, without depending on how the implementation was built internally.

A traceability matrix linking every clause of the standard to a test case, the executed results, and a statement of any claimed profile or level. Auditors read that package rather than the code.

Summarize this post with: