Getting started with iOS testing

Why iOS testing ?

First released on June 29, 2007, iOS is the name of Apple’s platform for mobile applications. Unlike Android, Apple does not license iOS for installation on non-Apple hardware. iOS and iOS applications are only installed on Apple devices. Your app must be compatible with 4 types of devices and iOS versions.

This is the common question when developer create an iOS application

It doesn't matter how much time you invest in design and implementation, mistakes are inevitable, and bugs will appear. There are some common bugs on iOS application. As shown in below figure.

1. Application crashing

     One of the most frustrating problems, when using Apple's devices, is that an application may crash frequently during execution. Many times the app crashes because there's some bugs or memory leak in apps.

2. Application incompatibilities

     Your iOS application may run perfectly on current iOS version, but if iOS is upgraded it may not work due to incompatibility issues.

3. Security vulnerability

     Security vulnerability in iOS allow the hacker to attack your iOS devices, steal your private information. Until now, severe iPhone security vulnerabilities are discovered in different iOS versions.

4. Memory leaks

     Memory leaks are blocks of allocated memory that the program no longer uses. Memory leaks cause your iOS application to crash. They are bugs and should always be fixed.

iOS testing MindMap

As shown in the above figure, iOS Testing MindMap shows all the items which tester should consider when conduct testing on iOS.

iOS Testing Strategy

The below figure introduces some common types of iOS testing strategy.

Automated testing

Automated testing is the most advantages of iOS testing. It enables you to detect the bug and performance issues quickly. The benefits of automated testing as shown below:

• Automated testing can run on multiple devices, saving your time
• Automated testing can target SDKs. You can run test on different SDK versions
• Automated testing increase your productivity of testing, save your cost of software development
• There's many open source Testing frameworks support automated testing on iOS

Unit testing with OCUnit

When the original iOS SDK was released, it lacked Unit Testing capabilities. So Apple has brought back the OCUnit unit test solution in iOS SDK version 2.2.

OCUnit is a testing framework for C- Objective in the Mac OS. The biggest advantages of OCUnit framework are the tight integration into XCode development environment as shown below

Some of benefits of OCUnit are shown in below figure.

UI Testing with UIAutomation

UI Automation is a JavaScript library provided by Apple Inc, which can be used to perform automated test on real devices and on iOS Simulator. This framework is added to iOS SDK4.0. Using UI Automation, you can automate testing the application not only on simulator but also real device.

UIAutomation bring you these benefits:

• Reduce effort on Manual Testing
• Use less your memory to execute all your tests
• Simplify your UI testing procedure ( just push one or three buttons and run full your test suites)

UIAutomation instrument works from scripts, which are written in JavaScript. It simulates user events on target iOS application.

UIAutomation Cons vs. Pros

The above figure represents some common classes in UIAutomation framework.

• The UIAElement class is the super class for all user interface elements in the context of the Automation
• The UIATarget class represents the high-level user interface elements of the system under test
• The UIALogger class provides test and error information on retrieval functionality
• The UIAActivityViewclass allows access to, and control of, activity views within your app.
• The UIAActionSheet class allows access to, and control of, action sheets within your app.
• User Event Action
  • UISlider class
  • UIAButton class
  • UIAKey class
  • UIAKeyboard class

Other Automated testing frameworks

• MonkeyTalk: A tool for automated testing of iOS, Android, HTML5 and Adobe applications. It is an integrated environment for managing and running test suites
• Frank: Automated Acceptance Test framework for iPhone and iPad
• KIF : is an iOS integration test framework. It allows for easy automation of iOS apps by leveraging the accessibility attributes that the OS makes available for those with visual disabilities.

Manual testing

Exploratory Testing

It is a testing without a formal test plan. Exploratory Testing is low cost testing method, but it can miss the potential bugs in your iOS application.

Exploratory Testing Cons vs. Pros

User Testing

User testing is a type of Manual Testing on iOS. The purpose of this testing is to create a better apps, not only bug-free apps. Below figure shows 4 types of User Testing

Concept testing:

Evaluate user response to an application idea before release to the market. Procedures of concept testing on iOS are described as below

Usability testing

Usability Testing is a test how easy to use your iOS application. In iOS testing, usability test could be recorded to remember or to share with others.

There is some tools support usability testing on iOS.

Magitest, a simple iOS usability testing for sites and apps.

Delight.io, this tool can capture real user interaction on your iOS apps

Beta testing

Beta Testing is the integration testing with real data to get final feedback from users. To distribute your apps for beta testing, you have to follow steps below.

-Pre-condition: If you are beta testing a final candidate for a release, be sure to validate the app before distributing it to testers.

-Find tester via service: you collect device IDs from testers and add them to Member Center

-Create ad-hoc distribution: Ad Hoc distribution allows the tester to run your app on their device without need Xcode. This step includes 2 sub-steps

• Create distribution certificates
• Create Ad hoc provisioning profiles

-Solicit feedback from tester: Tester conduct testing and send bug reports to you. After your app is released, you can get the reports from iTunes connect.

A/B testing

A/B testing is one of the most powerful ways to evaluate the effectiveness of your iOS app. It uses randomized experiments with two devices, A and B.

A/B testing includes 3 main steps

• Configure a test: Prepared 2 versions of your iOS app (A & B) and test metric
• Test: Test 2 iOS apps versions above on devices simultaneously.
• Analyze: Measure and select better version to release

Following tools support A/B testing on iOS.

• Arise : A/B testing for both iOS and Android. It can be integrated into your iOS app and make the testing process more quickly.

Best practices for A/B testing

• Define the target of your test. Any test is useless without a target.
• Watch end users use your app the first time
• Run one test only per update. It saves your time when conduct testing
• Monitor your test carefully. You can learn experiences from your test by monitoring it.

iOS testing Best practise

Here are some tips you should know when organizing the testing of your iOS application

1. Test the application on a real device to get real about performance
2. Improve your testing methods, because traditional testing methods are no longer sufficient to coverage all tests on iOS testing
3. Using console log to test iOS application. This is an iOS feature includes information from every application on the device.
4. Document application bugs using built-in screen short command. It helps developer understand how the bugs occur.
5. Crash reporting is useful tools when test your application. They can detect crashes and log detail so you can investigate the bugs easily.

MYTHS about iOS testing

This section examines a few popular myths and realities of iOS testing

Testing application on iOS and Android are the same thing

iOS and Android are two platforms were developed by Apple Inc and Google. They are totally different. Etc test environments, test frameworks, programming languages.

Test application on iOS Simulator is enough

iOS Simulator is not strong enough for testing an app. Because iOS Simulator has some limitations:

• Hardware limitations (Camera, Microphone input, Sensor)
• Your app's UI may appear to run both faster and smoother than on a device
• API Limitations
• Some frameworks unsupported (Media Player, Store Kit, Message UI..)

Everyone will download my apps on app store because it has many features

The more features your application has, the more bugs you could get. No user will download your application if it still has many defects.