Game Testing: How to Test Mobile/Desktop Apps

Game Development Lifecycle

Pre- Production: During this stage Game idea, storyboard, features, requirement analysis, and documentation is done. This phase includes technical design document and feature specifications, game architecture, frame overlay, animation. Following items are considered

• Music, Camera (zoom in and out, replay, cinematic view,) player and action attributes
• Game flow logic, Rules, and condition to attain the next level
• Object & event triggers, scores, player movement and positioning, player statistics,
• Non-Interactive sequence, Special effects, title screens, multi-button actions
• Gamepad, movie clip, shock/vibration effects, legal texts, use of button functions, use of analog &digital mode.

Production: During this stage, actual coding is done. This phase includes coding, integration of the various modules.

Testing and Deployment: During this stage Functional tests, regression testing, Alpha, Beta, Gold are performed. Testing of coverage and flows, data integrity, algorithm-specific testing, path testing, incremental testing is also done.

In this tutorial, you will learn

• Game Development Lifecycle
• How Game Testing is different from other Software Testing.
• Types of Game Testing
• Assistive gaming using adaptive technology
• Game Metrics That a Tester should Know.
• Key Risks in Game Testing

How Game Testing is different from other Software Testing.

Testing games is a repetitive process every new build may have bugs and must be tested thoroughly.

All game testing follows a basic structure irrespective of the size of the game & time required for producing the game.

The QA needs to study game rules & requirements. Understand the overall game component architecture and file architecture, the flow, and the file structures and the dependencies associated with the game. With every new prototype of the game, the tests documents need to be frequently revisited to update any change in the specifications, new test cases & new configuration support. The tester should ensure that no new issues were introduced.

A Game Tester needs to

• Classify the requirements based on the intended purpose and target audience.
• Identify the user and system requirement and should be classified into functional, nonfunctional, domain requirements
• Identify Testable items, Non-testable items, targets and measures for functional and nonfunctional requirements
• Check if the functional requirements are complete, consistent & comprehensible
• Identify customizable requirements, conflicting requirements
• Identify interdependent requirements.
• Prioritize the requirements based on the uniqueness, complexity, criticality
• Identify the theme of the game, Characters, Animation, AI, Cinematic, Camera view, gameplay

Types of Game Testing

1) Functional Testing

Functionality testers look for the generic problems within the game or its user interface & graphics, such as game mechanic issues, stability issues, and game asset integrity. User interface testing ensures user-friendliness of the game

Example: Checking colors and backgrounds, menu structure, screen orientation and screen resolution, font size, alignment errors, usability, system navigation such as loading time, timeout and display, sorting, confirmation messages, sequences, animations and audio elements aspects of the game, instructions, and dialogue messages. User Interactions, User Interfaces, Transactions testing, Calibration and accuracy testing of mobile phone cameras, Screen resolutions, Mobile responsive design testing, Audio quality Testing

2) Compatibility Testing

Checking if the game is compatible across different devices, and on different configurations of hardware and software.

Example: Install and Uninstall the game on all supported consoles/desktops/mobiles.

3) Performance Testing

The overall performance of the Game is checked. Performance tuning is performed to optimize game speed.

Importance parameters checked during performance testing

• Response time on client and servers, Transaction completion time(s), Peak load performance, Longevity, network coverage, Memory leakage, low memory, low battery, Time taken to download applications, simultaneous(Multiple users) access to application's server, speed, throughput, reliability, scalability, etc.
• Battery Consumption and graphics performance: Measure the battery consumption of the mobile game. Battery Consumption must be optimum over long hours, and game responses should be satisfactory under varying heavy loads across different devices
• Processor and memory constraints: Performance counters are used to measure the CPU and memory consumption of the application.
• Network connectivity: Measures the response time of the mobile games on different network types (Wi-Fi, 2G, 3G, 4G), It gives an overall insight into how well the game will perform on unreliable networks. It also checks connectivity between mobile devices, data centers or the cloud. The whole Peak Times, Jittery Connections, Duplication of Data, Packet loss, Fragmentation of Data are monitored.
• Testing game performance especially MMO

4) Conformance /Compliance Testing

Marketplace guidelines compliance (e.g., Apple App Store policies), Enterprise policy compliance (e.g., prohibited content. Compliance may also refer to regulatory bodies such as PEGI and ESRB. The game targets a particular content rating. If there is an objectionable content that is inappropriate for the desired rating, then they are identified and reported. Even a single violation in submission for license approval may have the game rejected, incurring additional costs in further testing and resubmission.

Example: If the game is to be published in European countries, test for PAL conversion if the game is produced for Northern America, test for NTSC conversions.

5) Localization testing

Localization testing becomes essential important when a game is targeted for the global markets. Game titles, content, and texts need to be translated and tested with devices in multiple languages. These types of tests can be performed quickly (with the help of cloud-based device access and test automation).

Example: Localization needs specific to MENA region (Middle East/North Africa), Arabic localization( Right-to-Left text support, Bi-Directional displays), Pseudo-localization testing, double-byte characters (for East Asian languages), local time/date, currency, address formats, and other local requirements.

6) Soak testing

This testing involves leaving the game running for a prolonged period in various modes of operation. For example, idling paused, or at the title screen. Soaking can identify memory leaks or rounding errors.

Example: Game has begun, and the character is made to stand idle for 24 hours. This technique is used to detect crashes brought on by memory leaks and other faults in the game engine.

7) Recovery testing

In software, recovery testing checks how well the application can be recovered from crashes, hardware failures, and other similar failures. The application is forced to fail, and later it will be observed how it recovers from the failure conditions and the environment.

Example: While a gaming application is running, suddenly restart the gaming console, & check the validate the data integrity

8) Security testing

It is done to check how safe the software works from external threats. Data protection from external threats, uncontrolled system access restrictions, data breach, operating system flaws, communication system flaws and weak encryption algorithms.

Example: Changing a URL from /login to /play on a gaming site should not allow direct access to the games.

9) Other Games testing

Testing of real or virtual characters. In multiplayer games, connectivity to the server and synchronization of game status are two critical areas to be tested.

Example: Multiplayer 3D racing games.

Testing new features such as game status updates, friend invitations, premium gift sharing, etc. This ensures rich gaming experience to the user.

Example: Facebook, Blogs

Sound Testing

Testing if there is an error in loading the files, listening to sound files for errors or distortions, CC profiler to analyze the color commentary

Database and game statistics

Database verification using debugging to investigate if the game is using the data correctly. Ensure that the data is loaded in the right place and shows the correct information

White-box testing

White Box Testing for Games focuses on the architectural, integration and system aspects of the mobile game.

1. Code Inspection: Source code is reviewed, Program logic &common programming errors, compliance with coding standards are analyzed.
2. Focus Testing: Chunks of codes are fed to the isolated modules, and the output is analyzed.
3. Data Analysis: Data usage, interpretation, and manipulation are analyzed and validated for the different modules.
4. Path and Flow Testing: Correct sequence of objects are executed.
5. Algorithm-specific testing: Testing a particular game scenario or feature by setting data variables, data values to the code and executing it in the runtime environment.
6. Artificial Intelligence Analysis: The run statistic of the programmable moves and plays of the AI component is generated. The result is validated to check if all the programmable moves are used. Example: side grip on the snowboard and plays(combination punch/kick in multidirectional action) are used.

Assistive gaming using adaptive technology

Assistive Gaming is also known as accessibility gaming. Features are designed using adaptive technology for individuals with various disabilities like low vision, blurred vision, blindness, inability to distinguish colors, speech, hearing, cognitive, motor & mobility impairments.

Cardinal Direction (CD), Tower of London (TOL) are two popular games that have been modified for the visually challenged users. In these games, Visual stimuli are replaced with audio inputs.

A tester should note the following while testing such a game

1. The colors should flash in a pattern, and the tones would play for each color.
2. Each color should be accompanied by an audible tone.
3. Visual data need to be described in words so that visually impaired have not face any issues receiving through the use of screen readers.
4. The player should hear sounds in the game in three dimensions and must navigate the word using the touchscreen 3D Audio and Spatialized Sound

Game Metrics That a Tester should Know.

DAU/MAU (Daily active users/monthly active users): The ratio of active users who play each day over the number of monthly active users. Also commonly referred to as the stickiness factor.

Session: Every time any user opens the app, that counts as a session. Here the focus is on the average number of sessions per DAU.

Download Rank: The rank of a game in a particular app store(iOS, Android Play) by monthly game downloads.

Retention: Very important metric in a free to play the game. To calculate retention, separate the users into cohorts based on the day the application was downloaded.

Performance metrics: This is for tracking the performance of the online games or persistent games. Frame rate at which a game executes on a client hardware platform, or in the case of a game server, its stability and Performance metrics can be used to monitor changing features and updates.

Key Risks in Game Testing

1. The game does not create compelling experiences for the targeted audience.
2. The game does not have a player-centric design
3. The fun factor and addictive gameplay missing in the games.
4. Game not unique, competitive, fast paced.
5. The game fails because of technical issues, broken features, critical bugs, bad music sound, and poor video.
6. Game development cost goes over budget
7. The game should have simple aesthetic design and the gameplay.

Summary:

• Game development life-cycle consists of three stages of Pre-production, production, testing and deployment
• Testing games is a repetitive process every new build may have bugs, and so it must be tested thoroughly
• The different types of testing are 1) Functional Testing, 2) Compatibility Testing, 3) Performance Testing, 4) Conformance /Compliance Testing, 5) Localization testing, 6) Soak testing, 7) Recovery testing, 8) Security testing
• White Box Testing for Games focuses on the architectural, integration and system aspects of the mobile game which includes 1) code inspection 2) focus testing 3) Data analysis 4) Path and flow Testing 5) Algorithm-specific testing 6) Artifical Intelligence analysis
• Assistive Gaming is also known as accessibility gaming. Features are designed using adaptive technology for individuals
• Some important Game Metrics are DAU/MAU, Session, Download Rank, Retention, and Performance Metrics
• The key risk of game testing is that it does not create compelling experiences for the target audience