What is Configuration Testing? Example Test Cases

Configuration Testing

Configuration Testing is a software testing technique in which the application is tested with multiple combinations of software and hardware, in order to evaluate the functional requirements and find the optimal configurations under which the application works without defects or flaws.

A configuration is any combination the product must support: an operating system version, a browser, a database release, a driver, a memory size or an attached peripheral. It is worth separating this from compatibility testing, which asks whether the product coexists with other software and platforms. Configuration testing asks a narrower question: does the same build still behave correctly once its own supported setup changes?

Configuration Testing Example

Consider a desktop application as a worked example.

Desktop applications are usually built in 2 tier or 3 tier form. Take a 3 tier desktop application developed in ASP.NET, consisting of a Client, a Business Logic Server and a Database Server, where each component supports the platforms listed below.

• Client platform – Windows XP, Windows 7, Windows 8, and so on
• Server platform – Windows Server 2008, Windows Server 2008 R2, Windows Server 2012 R2
• Database – SQL Server 2008, SQL Server 2008 R2, SQL Server 2012, and so on

The tester has to exercise the client, server and database together across those platform and database versions, to confirm that the application functions correctly and does not fail on any supported combination.

Configuration testing is not restricted to software. It applies to hardware as well, which is why the hardware side is called hardware configuration testing: printers, scanners, web cams and similar devices that the application under test has to support. The matrix below shows how those combinations are laid out before test execution starts.

Pre-requisites for Configuration Testing

Before the configuration test begins on any project, three pre-requisites have to be in place.

• Creation of a matrix listing the various combinations of software and hardware configurations
• Prioritising those configurations, because testing every one of them is not realistic
• Testing each configuration in the order set by that prioritisation

Objectives of Configuration Testing

Configuration testing sets out to achieve the following.

• Validate the application against its configurability requirements
• Cause failures deliberately to surface defects that ordinary testing misses, for example by changing regional settings such as time zone, language or date format
• Determine the optimal configuration of the application under test
• Analyse system performance while hardware resources change, such as adding load balancers, increasing or reducing memory, or connecting different printer models
• Analyse system efficiency against the prioritisation, judging how well the tests used the resources available to reach that optimal configuration
• Verify the system in a geographically distributed environment, for example with the server in one location and clients in another, where the system should work regardless of local system settings
• Verify how easily defects reproduce when the configuration changes
• Confirm that application items stay traceable through proper documentation and clearly identifiable version records
• Confirm that application items remain manageable throughout the software development life cycle

How to do Configuration Testing

The strategy depends on which of the two configuration testing types is in scope.

• Software Configuration Testing
• Hardware Configuration Testing

Software Configuration Testing

Software configuration testing runs the application under test against multiple operating systems, software updates and dependency versions. It is time consuming, because every round means installing and uninstalling the software involved.

A common way to cut that cost is to test on virtual machines. A virtual machine is an environment installed in software that behaves like physical hardware, so the tester works as though on a real machine while the configuration itself is disposable. Virtual machines simulate real configurations closely enough for most functional checks.

Rather than installing and uninstalling on several physical machines, the application is installed on a virtual machine and testing continues from there. Running several virtual machines in parallel, each restored from a snapshot, simplifies the work considerably.

Software configuration testing can typically begin when

• The configurability requirements to be tested are specified
• The test environment is ready
• The testing team is trained in configuration testing
• The released build has passed unit and integration testing

The typical test strategy is to run the functional test suite across each software configuration and verify that the application behaves as intended, without flaws or errors. A second strategy is to fail test cases deliberately and check how efficiently the system copes.

Example:

Take a banking application that has to be tested across multiple browsers. Hosted in an environment where every prerequisite is present, it may well pass unit and integration testing in the test lab.

Installed at a client site, the same application may fail, because those machines are missing software updates or the dependency versions the application relies on directly or indirectly. Deliberately failing the tests by removing some configurability requirements, then testing again, is what exposes that class of defect before the customer finds it. The screen below shows such a configuration-dependent failure being reproduced in a controlled environment.

Hardware Configuration Testing

Hardware configuration testing is generally performed in a lab holding physical machines with different hardware attached to them.

Whenever a build is released, the software is installed on every one of those machines and the test suite is run on each, to confirm the application works with the attached device.

That task takes significant effort: installing the software on each machine, attaching the hardware, and then running the suite manually or automating it first.

The type of hardware to be tested also has to be specified. Computer hardware and peripherals exist in such variety that covering all of them is impossible, so the tester analyses which devices the user base actually relies on and tests according to that prioritisation.

Sample Test Cases

Consider a banking scenario tested for hardware compatibility. A banking application connected to a note counting machine has to work with several models, such as Rolex, Strob, Maxsell and StoK.

Sample test cases for the note counting machine include the following.

• Verify the connection between the application and the Rolex model when the prerequisites are NOT installed
• Verify the connection between the application and the Rolex model when the prerequisites are installed
• Verify that the system counts the notes correctly
• Verify how the system reports a miscount
• Verify the handling of tampered notes
• Verify the response times
• Verify that fake notes are detected

Those cases cover a single model, and each remaining model on the market has to be set up in a test lab and tested the same way, which is rarely practical in-house. Outsourcing hardware configuration testing to an organisation that specialises in it is often the more realistic option.

FAQs