Why iOS testing ?
First released on June 29, 2007, iOS is the name of Apple’s platform for mobile applications. Unlike Android, Apple does not license iOS for installation on non-Apple hardware. iOS and iOS applications are only installed on Apple devices. Your app must be compatible with 4 types of devices and iOS versions.
This is the common question when developer create an iOS application
It doesn't matter how much time you invest in design and implementation, mistakes are inevitable, and bugs will appear. There are some common bugs on iOS application. As shown in below figure.
- Application crashing
One of the most frustrating problems, when using Apple's devices, is that an application may crash frequently during execution. Many times the app crashes because there's some bugs or memory leak in apps.
- Application incompatibilities
Your iOS application may run perfectly on current iOS version, but if iOS is upgraded it may not work due to incompatibility issues.
- Security vulnerability
Security vulnerability in iOS allow the hacker to attack your iOS devices, steal your private information. Until now, severe iPhone security vulnerabilities are discovered in different iOS versions.
- Memory leaks
Memory leaks are blocks of allocated memory that the program no longer uses. Memory leaks cause your iOS application to crash. They are bugs and should always be fixed.
iOS testing MindMap
As shown in the above figure, iOS Testing MindMap shows all the items which tester should consider when conduct testing on iOS.
iOS Testing Strategy
The below figure introduces some common types of iOS testing strategy.
Automated testing is the most advantages of iOS testing. It enables you to detect the bug and performance issues quickly. The benefits of automated testing as shown below:
- Automated testing can run on multiple devices, saving your time
- Automated testing can target SDKs. You can run test on different SDK versions
- Automated testing increase your productivity of testing, save your cost of software development
- There's many open source Testing frameworks support automated testing on iOS
Unit testing with OCUnit
OCUnit is a testing framework for C- Objective in the Mac OS. The biggest advantages of OCUnit framework are the tight integration into XCode development environment as shown below
Some of benefits of OCUnit are shown in below figure.
UI Testing with UIAutomation
UIAutomation bring you these benefits:
- Reduce effort on Manual Testing
- Use less your memory to execute all your tests
- Simplify your UI testing procedure ( just push one or three buttons and run full your test suites)
UIAutomation Cons vs. Pros
|1.||Good support for gesture and rotation||It's not open source, less support from the developer|
|2.||Can run UIAutomation tests on the device, not only simulator.||Can’t integrate with other tools extremely well|
The above figure represents some common classes in UIAutomation framework.
- The UIAElement class is the super class for all user interface elements in the context of the Automation
- The UIATarget class represents the high-level user interface elements of the system under test
- The UIALogger class provides test and error information on retrieval functionality
- The UIAActivityViewclass allows access to, and control of, activity views within your app.
- The UIAActionSheet class allows access to, and control of, action sheets within your app.
- User Event Action
- UISlider class
- UIAButton class
- UIAKey class
- UIAKeyboard class
Other Automated testing frameworks
- MonkeyTalk: A tool for automated testing of iOS, Android, HTML5 and Adobe applications. It is an integrated environment for managing and running test suites
- Frank: Automated Acceptance Test framework for iPhone and iPad
- KIF : is an iOS integration test framework. It allows for easy automation of iOS apps by leveraging the accessibility attributes that the OS makes available for those with visual disabilities.
It is a testing without a formal test plan. Exploratory Testing is low cost testing method, but it can miss the potential bugs in your iOS application.
Exploratory Testing Cons vs. Pros
Less preparation is needed, early detect serious bugs
Requires high skill of tester
Don't need Test Plan speed up the bug detection
Test coverage is low. It does not guarantee that all your requirements are tested
Most bugs are discovered early by some sort of exploratory testing
Lack of testing documentation
User testing is a type of Manual Testing on iOS. The purpose of this testing is to create a better apps, not only bug-free apps. Below figure shows 4 types of User Testing
Evaluate user response to an application idea before release to the market. Procedures of concept testing on iOS are described as below
Usability Testing is a test how easy to use your iOS application. In iOS testing, usability test could be recorded to remember or to share with others.
There is some tools support usability testing on iOS.
Magitest, a simple iOS usability testing for sites and apps.
Delight.io, this tool can capture real user interaction on your iOS apps
Beta Testing is the integration testing with real data to get final feedback from users. To distribute your apps for beta testing, you have to follow steps below.
-Pre-condition: If you are beta testing a final candidate for a release, be sure to validate the app before distributing it to testers.
-Find tester via service: you collect device IDs from testers and add them to Member Center
-Create ad-hoc distribution: Ad Hoc distribution allows the tester to run your app on their device without need Xcode. This step includes 2 sub-steps
- Create distribution certificates
- Create Ad hoc provisioning profiles
-Solicit feedback from tester: Tester conduct testing and send bug reports to you. After your app is released, you can get the reports from iTunes connect.
A/B testing is one of the most powerful ways to evaluate the effectiveness of your iOS app. It uses randomized experiments with two devices, A and B.
A/B testing includes 3 main steps
- Configure a test: Prepared 2 versions of your iOS app (A & B) and test metric
- Test: Test 2 iOS apps versions above on devices simultaneously.
- Analyze: Measure and select better version to release
Following tools support A/B testing on iOS.
- Arise : A/B testing for both iOS and Android. It can be integrated into your iOS app and make the testing process more quickly.
Best practices for A/B testing
- Define the target of your test. Any test is useless without a target.
- Watch end users use your app the first time
- Run one test only per update. It saves your time when conduct testing
- Monitor your test carefully. You can learn experiences from your test by monitoring it.
iOS testing Best practise
Here are some tips you should know when organizing the testing of your iOS application
- Test the application on a real device to get real about performance
- Improve your testing methods, because traditional testing methods are no longer sufficient to coverage all tests on iOS testing
- Using console log to test iOS application. This is an iOS feature includes information from every application on the device.
- Document application bugs using built-in screen short command. It helps developer understand how the bugs occur.
- Crash reporting is useful tools when test your application. They can detect crashes and log detail so you can investigate the bugs easily.
MYTHS about iOS testing
This section examines a few popular myths and realities of iOS testing
Testing application on iOS and Android are the same thing
iOS and Android are two platforms were developed by Apple Inc and Google. They are totally different. Etc test environments, test frameworks, programming languages.
Test application on iOS Simulator is enough
iOS Simulator is not strong enough for testing an app. Because iOS Simulator has some limitations:
- Hardware limitations (Camera, Microphone input, Sensor)
- Your app's UI may appear to run both faster and smoother than on a device
- API Limitations
- Some frameworks unsupported (Media Player, Store Kit, Message UI..)
Everyone will download my apps on app store because it has many features
The more features your application has, the more bugs you could get. No user will download your application if it still has many defects.