Complete Guide to Android Testing & Automation

Details: Last Updated: 29 August 2017

Android is the largest operating system in the world. At the same time, Android is fragmented. there are tons of devices and Android versions that your app must be compatible with

Why Android Testing?

Android is the largest operating system in the world. At the same time, Android is fragmented. there are tons of devices and Android versions that your app must be compatible with.

It doesn't matter how much time you invest in design and implementation , mistakes are inevitable, and bugs will appear.

Myths of Android Testing

Many enterprises develop android Testing strategies that are based on common misconceptions. This section examines a few popular myths and realities of Android testing.

Myth #1:All Android devices are the same... test on emulators is enough

Let's start with a simple example. An application works perfectly on emulator s but on some real devices, it crashes during execution

Application crashes during execution on real device

Emulators are not sufficient for your mobile testing. You must test your app on real devices.

Myth #2:Testing on some common devices is enough

On different devices, your application looks different because different devices have different hardware, screen sizes, memory etc. You must test your application on different devices, OS versions, carrier networks and locations.

Myth#3:Exploratory testing just before launch is enough

Generally in all testing, we design the test cases then execute them. But in Exploratory testing, test design and execution all will be done together.
In exploratory testing, there's no plan and no preparation, then tester would do tests that he wants to do. Some functions will be tested repeatedly, while some functions will not be tested altogether.

Myth#4:If there some bugs in application, users will understand

If application doesn't work and has bugs, users uninstall your app
Quality issues are the first reason for bad review in Google Play. It affects to your reputation and you lose customer's trust.

Therefore its essential to have a proper android testing strategy in place

Android Testing Strategy

A correct android testing strategy should include the following

Unit Test
Integration Test
Operational Test
System Test

Unit tests

Unit Tests include sets of one or more programs which are designed to verify an atomic unit of source code, such as a method or a class.

Android platform comes pre-integrated Junit 3.0 framework. It's open source framework for automating Unit Testing. Android Testing Framework is powerful tool for developer to write the effective unit test program.

The integration of Android and JUnit framework

An addition to Unit testing is User Interface (UI) tests. These tests relate to UI components of your target application. UI tests ensure that your application return the correct UI output in response to sequence of user actions on device.

Common user UI actions on application

The common way to performance UI tests on device is Android Instrumentation. But this has performance issues. One of the best tools to conduct UI testing on Android is Robotium .

Integration tests

In Integration Testing, all unit tested modules, are combined and verified. In Android, integration tests often involve checking integration withAndroid components such as Service testing, Activity testing, Content Provider testing, etc

Types of integration test on Android

There's many testing frameworks are used to conduct integration test for Android such as Troyd, Roboelectric, Robotium.

Operational tests

Operational are also called Functional Tests or Acceptation Tests. They are high level tests designed to check the completeness and correctness of application.
In Android, FitNesse is open-source framework that makes it easy to conduct operational tests for target application.

System tests

In System Testing the system is tested as a whole and the interaction between the components, software and hardware is checked.

In Android, System Testing normally includes

GUI tests
Usability tests
Performance tests
Stress tests

In the above list, Performance Testing is given more focus. You can use tools like Traceview to conduct performance test on Android .This tool can help you debug your application and profile its performance.

Automated ANDROID TESTING

As android is fragmented, testing on multitude of devices is necessary. But this will also cost you money. Automated Android Testing can help reduce costs

Benefits of automated android testing

Reduce time for executing test cases
Increase productivity of your development process
Early bug detection, save cost on software maintenance
Quickly found and fix the bugs on implementation
Ensure the quality of software

We will study the following 3 frameworks

Android Testing framework
Robotium Testing framework
Roboelectric Testing framework

Android testing framework

One of the standard testing frameworks for Android application is Android testing framework. It is a powerful and easy-to-use testing framework that is well integrated with the Android SDK tools.

Android testing framework Architecture

Application package is your target application which needs to be tested
InstrumentationTestRunner is the Test Case runner that executes test case on target application. It includes:

2a) Test tools: A SDK tools for building test. They are integrated in Eclipse IDE or run as command line.

2b) MonkeyRunner: A tool that provides APIs for writing program which control an Android device or emulator outside of Android code.

Test package are organized into test projects. This package follows naming convention. If the application under test has a package name of "com.mydomain.myapp" than Test package should be "com.mydomain.myapp.test" .Test package includes 2 objects as below:

3a) Test case classes:include test methods to executed on target application.

3b) Mock objects : includes mock data that will be used as sample input for test cases.

Android Test Case Classes

AndroidTestCase class diagram

TestCase includes JUnit methods to run JUnit test
TestSuite is used to run set of test cases
InstrumentationTestSuite is a TestSuite that injects Instrumentation into InstrumentationTestCase before running them.
InstrumentationTestRunner is the test case runner that execute test case on target application.
AndroidTestCase extends JUnit TestCase. It contains methods for accessing resources like Activity Context.
ApplicationTestCase verifies the Application classes in a controlled environment.
InstrumentationTestCase verifies a particular feature or behavior of target application , for e.g., verify UI output of application.
ActivityTestCase is base class that supports testing the Application Activities.
ProviderTestCase is class for testing single ContentProvider.
ServiceTestCase is used to test Service classes in testing environment. It also supports Service's life cycle.
SingeLauchActivityTestCase is used to test single Activity with an InstrumentationTestCase.
ActivityUnitTestCase <Activity> is used to test single isolated activity.
ActivityInstrumentationTestCase2<Activity> extends the JUnit TestCase class. It connects you to target application with instrumentation. With this class, you can access application's GUI component and send UI event (keystroke or touch event) to the UI.

Below is an example of ActivityInstrumentationTestCase. It verifies the UI operation of Calculator application, check the correctness of the UI outputs.

ActivityInstrumentationTestCase2 testing example

Robotium testing framework

Standard Android testing framework has some limitation as below

Unable to handle multiple activities
Test execution performance is slow
Test cases are complex & hard to implement

Robotiumframework is the better choice to conduct testing on Android application

Robotium is open source framework and is considered an extension of Android test framework. Using Robotium, developer can create robust automatic GUI test cases for Android applications. Moreover, developer can write functional, system and acceptance test scenarios, spanning multiple Android activities.

Advance features of Robotium

Robotium Test Case Classes

Robotium uses set of classes (com.jayway.android.robotium.solo) for testing. This class supports test cases that span over multiple activities. Solo is integrated with the ActivityInstrumentationTestCase2.

Integration Robotium and ActivityInstrumentationTestCase2

Tester can write test cases without knowledge of application design (black box testing) by using Robotium test case classes. It is an outstanding feature compare to Android test case classes.

To use Robotium in your Android test project, you need follow the steps below

Using Robotium to conduct testing on Android application

Roboelectric testing framework

Testing using Android Testing framework with device or emulator is difficult. Building and running test is slow and take much development effort. To fix this problem, there's another choice - Roboelectric testing framework.

Roboelectric framework allows you to run Android tests directly on JVM without the need for a device or an emulator.

Advance features of Roboelectric

Roboelectric Test Case Classes

Operation of Roboelectric

As shown above, Roboelectric can perform following actions:
Register and create a Shadow class
Intercept the loading of Android class
Uses javaassist to override the method bodies of Android class
Bind Shadow object to Android class
This allows the code under test to execute without Android environment.

Others testing framework

Besides testing frameworks which were mentioned above, there are many other testing frameworks such as:
Calculon, a Java DSL for Android Activity Testing which runs on Dalvik Virtual Machine
Android Mock, a framework for mocking interfaces and classed on Dalvik VM
Roboguide, a dependency injection in Android application
Android Junit Report, a custom instrumentation test runner for Android that generates XML reports for integration with other tools.

Best practices in Android Testing

Application developers should create the test cases at the same time when they are writing the code
All test cases should be stored in version control-together with source code
Use continuous integration and run tests every time the code is changed
Avoid using emulators and rooted devices