As seasoned app developers, we instinctively know when our applications need to undergo testing. Maintaining stability across different releases is often a business requirement. Ideally, we aim to automate the build and publishing process. To achieve this, we require reliable Android testing tools to ensure our build functions as expected.

Tests provide an additional layer of confidence in our creations. Building a flawless, bug-free product is nearly impossible. Our objective, therefore, should be to increase our chances of market success. This can be achieved by implementing a test suite that can quickly identify newly introduced bugs within our application.

Testing Android and other mobile platform apps can be tricky. Implementing unit tests and adhering to principles like test-driven development can feel counterintuitive. However, testing remains crucial and should never be underestimated or neglected. David, Kent, and Martin have explored the advantages and disadvantages of testing in a series of dialogues. These conversations have been compiled into an article entitled “Is TDD dead?.” The article also features links to the video recordings of these discussions, providing further insights into incorporating testing into your development process and determining the extent to which it aligns with your needs.

This Android testing tutorial will guide you through unit, acceptance, and regression testing on the Android platform. We will delve into the abstraction of test units on Android. This will be followed by examples of acceptance testing, with an emphasis on simplifying and accelerating the process to minimize delays between developers and QA.

Is This Tutorial For Me?

This tutorial explores various possibilities for testing Android applications. Developers or project managers seeking a deeper understanding of current Android testing options can use this tutorial to determine if the approaches discussed here align with their requirements. However, it’s important to remember that testing is not a one-size-fits-all solution. The ideal approach varies depending on factors like product specifics, deadlines, existing code quality, system coupling, developer preferences in architectural design, and the anticipated lifespan of the feature being tested.

Thinking in Units: Android Testing

Ideally, we strive to test each logical unit or component of an architecture independently. This ensures each component functions correctly with its expected inputs. Mocking dependencies enables us to write fast-executing tests. Additionally, we can simulate various system states based on the input provided to the test, encompassing even unusual cases.

“The goal of Android unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits.” —Wikipedia

Robolectric

Robolectric is an Android unit testing framework that allows you to run tests within the JVM on your development machine. Robolectric accomplishes this by rewriting Android SDK classes as they load, allowing them to function within a standard JVM environment. This results in significantly faster test execution times. Moreover, it handles view inflation, resource loading, and other tasks typically performed by native C code on Android devices. This eliminates the need for emulators or physical devices to run automated tests.

Mockito

Mockito is a mocking framework that facilitates clean and efficient test writing in Java. It simplifies the creation of “test doubles” (mocks), which replace actual dependencies of a component or module during testing. A StackOverflow answer offers a clear and concise explanation of about the differences between mocks and stubs, which you may find helpful.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// you can mock concrete classes, not only interfaces
 LinkedList mockedList = mock(LinkedList.class);

// stubbing appears before the actual execution
when(mockedList.get(0)).thenReturn("first");

// the following prints "first"
System.out.println(mockedList.get(0));

// the following prints "null" because get(999) was not stubbed
System.out.println(mockedList.get(999));

Mockito also allows us to verify method calls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// mock creation
List mockedList = mock(List.class);

// using mock object - it does not throw any "unexpected interaction" exception
mockedList.add("one");
mockedList.clear();

// selective, explicit, highly readable verification
verify(mockedList).add("one");
verify(mockedList).clear();

As you can see, we can define action-reaction pairs that dictate what happens when a specific action is performed on the mocked object or component. This enables us to mock entire modules of our application. For each test case, we can manipulate the mocked module to react differently, reflecting various potential states of the tested component and the mocked component pair.

Unit Testing

In this section, we’ll assume an MVP (Model View Presenter) architecture. Activities and fragments represent the views, models act as the repository layer for database or remote service calls, and the presenter serves as the “brain.” The presenter connects everything, implementing logic to manage views, models, and the flow of data within the application.

Abstracting Components

Mocking Views and Models

In this example, we’ll mock the views, models, and repository components to unit test the presenter. This constitutes one of the simplest tests, focusing on a single component within the architecture. We will use method stubbing to establish a testable chain of reactions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

@RunWith(RobolectricTestRunner.class)
@Config(manifest = "app/src/main/AndroidManifest.xml", emulateSdk = 18)
public class FitnessListPresenterTest {

	private Calendar cal = Calendar.getInstance();

	@Mock
	private IFitnessListModel model;

	@Mock
	private IFitnessListView view;

	private IFitnessListPresenter presenter;

	@Before
	public void setup() {
		MockitoAnnotations.initMocks(this);

		final FitnessEntry entryMock = mock(FitnessEntry.class);

		presenter = new FitnessListPresenter(view, model);
		/*
			Define the desired behaviour.

			Queuing the action in "doAnswer" for "when" is executed.
			Clear and synchronous way of setting reactions for actions (stubbing).
			*/
		doAnswer((new Answer<Object>() {
			@Override
			public Object answer(InvocationOnMock invocation) throws Throwable {
				ArrayList<FitnessEntry> items = new ArrayList<>();
				items.add(entryMock);

				((IFitnessListPresenterCallback) presenter).onFetchAllSuccess(items);
				return null;
			}
		})).when(model).fetchAllItems((IFitnessListPresenterCallback) presenter);
	}

	/**
		Verify if model.fetchItems was called once.
		Verify if view.onFetchSuccess is called once with the specified list of type FitnessEntry

		The concrete implementation of ((IFitnessListPresenterCallback) presenter).onFetchAllSuccess(items); 
		calls the view.onFetchSuccess(...) method. This is why we verify that view.onFetchSuccess is called once.
	*/
	@Test
	public void testFetchAll() {
		presenter.fetchAllItems(false);
		// verify can be called only on mock objects
		verify(model, times(1)).fetchAllItems((IFitnessListPresenterCallback) presenter);
		verify(view, times(1)).onFetchSuccess(new ArrayList<>(anyListOf(FitnessEntry.class)));
	}
}

Mocking the Global Networking Layer with MockWebServer

Mocking the global networking layer can be quite beneficial. MockWebServer lets us queue responses for specific requests made during our tests. This allows us to simulate unusual server responses that are difficult to reproduce, ensuring comprehensive test coverage with minimal code.

MockWebServer’s code repository provides a clear example that you can reference to gain a deeper understanding of this library.

Custom Test Doubles

You have the option to create your own model or repository component and inject it into the test. This is achieved by providing a different module to the object graph using Dagger (http://square.github.io/dagger/). You can then verify if the view state updates correctly based on the data provided by the mocked model component:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

/**
	Custom mock model class 
*/
public class FitnessListErrorTestModel extends FitnessListModel {

	// ...

	@Override
	public void fetchAllItems(IFitnessListPresenterCallback callback) {
		callback.onError();
	}

	@Override
	public void fetchItemsInRange(final IFitnessListPresenterCallback callback, DateFilter filter) {
		callback.onError();
	}

}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

@RunWith(RobolectricTestRunner.class)
@Config(manifest = "app/src/main/AndroidManifest.xml", emulateSdk = 18)
public class FitnessListPresenterDaggerTest {

    private FitnessActivity activity;
    private FitnessListFragment fitnessListFragment;

    @Before
    public void setup() {
        /*
            setupActivity runs the Activity lifecycle methods on the specified class
        */
        activity = Robolectric.setupActivity(FitnessActivity.class);
        fitnessListFragment = activity.getFitnessListFragment();
        
        /*
            Create the objectGraph with the TestModule
        */
        ObjectGraph localGraph = ObjectGraph.create(TestModule.newInstance(fitnessListFragment));
        /*
            Injection
        */
        localGraph.inject(fitnessListFragment);
        localGraph.inject(fitnessListFragment.getPresenter());
    }

    @Test
    public void testInteractorError() {
        fitnessListFragment.getPresenter().fetchAllItems(false);

        /*
            suppose that our view shows a Toast message with the specified text below when an error is reported, so we check for it.
        */
        assertEquals(ShadowToast.getTextOfLatestToast(), "Something went wrong!");
    }

    @Module(
            injects = {
                    FitnessListFragment.class,
                    FitnessListPresenter.class
            }, overrides = true,
            library = true
    )
    static class TestModule {
        private IFitnessListView view;

        private TestModule(IFitnessListView view){
            this.view = view;
        }

        public static TestModule newInstance(IFitnessListView view){
            return new TestModule(view);
        }

        @Provides
        public IFitnessListInteractor provideFitnessListInteractor(){
            return new FitnessListErrorTestModel();
        }

        @Provides public IFitnessListPresenter provideFitnessPresenter(){
            return new FitnessListPresenter(view);
        }
    }

}

Running Tests

Android Studio

Right-clicking on a test class, method, or entire package in Android Studio allows you to run tests directly from the IDE’s options dialog.

Terminal

Running Android app tests from the terminal generates reports for the tested classes, which are placed in the “build” folder of the target module. The terminal approach is particularly useful if you’re setting up an automated build process. In Gradle, you can run all debug-flavored tests by executing the following command:

1

gradle testDebug

Accessing Source Set “test” from Android Studio Version

Android Studio 1.1 and later, along with the Android Gradle plugin, introduced support for unit testing. You can find more details in their excellent documentation on it. While this feature is still experimental, it’s a valuable addition. It lets you switch between unit test and instrumentation test source sets directly within the IDE, similar to switching between flavors.

Simplifying the Process

Writing tests might not be as enjoyable as developing the application itself. However, some tips can make writing tests easier and help you avoid common setup issues.

AssertJ Android

As the name implies, AssertJ Android provides a collection of helper functions specifically designed for Android testing. It extends the popular AssertJ library. AssertJ Android offers a range of functionalities. These range from simple assertions like “assertThat(view).isGone()” to more complex ones like:

1
2
3
4

assertThat(layout).isVisible()
    .isVertical()
    .hasChildCount(4)
    .hasShowDividers(SHOW_DIVIDERS_MIDDLE)

AssertJ Android’s extensibility provides a straightforward and convenient starting point for writing Android application tests.

Robolectric and Manifest Path

When using Robolectric, you might need to specify the manifest location and set the SDK version to 18. You can achieve this using the “Config” annotation:

1

@Config(manifest = "app/src/main/AndroidManifest.xml", emulateSdk = 18)

Running Robolectric tests from the terminal can sometimes lead to new challenges. For example, you might encounter exceptions like “Theme not set.” If your tests run correctly in the IDE but fail from the terminal, it could be due to an unresolved manifest path. The hardcoded manifest path might be incorrect from the command’s execution point. Custom runners can resolve this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

public class RobolectricGradleTestRunner extends RobolectricTestRunner {
	public RobolectricGradleTestRunner(Class<?> testClass) throws InitializationError {
		super(testClass);
	}

	@Override
	protected AndroidManifest getAppManifest(Config config) {
		String appRoot = "../app/src/main/";
		String manifestPath = appRoot + "AndroidManifest.xml";
		String resDir = appRoot + "res";
		String assetsDir = appRoot + "assets";
		AndroidManifest manifest = createAppManifest(Fs.fileFromPath(manifestPath),
		    Fs.fileFromPath(resDir),
		    Fs.fileFromPath(assetsDir));
		return manifest;
	}
}

Gradle Configuration

Here’s a sample Gradle configuration for unit testing. You may need to adjust dependency names and versions as needed for your project:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

// Robolectric
testCompile 'junit:junit:4.12'
testCompile 'org.mockito:mockito-core:1.9.5'
testCompile 'com.squareup.dagger:dagger:1.2.2'
testProvided 'com.squareup.dagger:dagger-compiler:1.2.2'

testCompile 'com.android.support:support-v4:21.0.+'
testCompile 'com.android.support:appcompat-v7:21.0.3'

testCompile('org.robolectric:robolectric:2.4') {
	exclude module: 'classworlds'
	exclude module: 'commons-logging'
	exclude module: 'httpclient'
	exclude module: 'maven-artifact'
	exclude module: 'maven-artifact-manager'
	exclude module: 'maven-error-diagnostics'
	exclude module: 'maven-model'
	exclude module: 'maven-project'
	exclude module: 'maven-settings'
	exclude module: 'plexus-container-default'
	exclude module: 'plexus-interpolation'
	exclude module: 'plexus-utils'
	exclude module: 'wagon-file'
	exclude module: 'wagon-http-lightweight'
	exclude module: 'wagon-provider-api'
}

Robolectric and Play Services

If your application utilizes Google Play Services, you’ll need to define a custom integer constant for the Play Services version. This ensures Robolectric functions correctly:

1
2
3
4

<meta-data
	android:name="com.google.android.gms.version"
	android:value="@integer/gms_version"
	tools:replace="android:value" />

Robolectric Dependencies to Support Libraries

Another interesting challenge is that Robolectric sometimes struggles to reference support libraries properly. A workaround is to create a “project.properties” file within the module containing your tests. For instance, for Support-v4 and AppCompat libraries, the file should include:

1
2

android.library.reference.1=../../build/intermediates/exploded-aar/com.android.support/support-v4/21.0.3
android.library.reference.2=../../build/intermediates/exploded-aar/com.android.support/appcompat-v7/21.0.3

Acceptance/Regression Testing

Acceptance/Regression testing automates a portion of the final testing phase, running tests in a real, complete Android environment. Unlike unit tests, we do not use mocked Android OS classes at this level. Instead, tests are executed on actual devices or emulators.

The variety of physical devices, emulator configurations, device states, and feature sets makes this testing phase less predictable. Additionally, factors like the operating system version and screen size can significantly impact how content is displayed, further complicating matters.

Creating tests that consistently pass across a wide range of devices is complex. However, it’s best to start small and gradually expand. Creating tests with Robotium is an iterative process that can be streamlined with a few tricks.

Robotium

Robotium](https://code.google.com/p/robotium/) is an open-source Android test automation framework that has been around since January 2010. It’s worth noting that while Robotium is a paid solution, it offers a free trial.

To expedite the process of writing Robotium tests, we can transition from manually writing tests to recording them. This approach prioritizes speed over code quality. If your user interface undergoes frequent changes, test recording can be highly beneficial as it allows for quick creation of new tests.

Testdroid Recorder is a free test recorder that automatically generates Robotium tests as you interact with the user interface. Installation is simple, and described in their documentations offers a step-by-step video guide.

While Testdroid Recorder is an Eclipse plugin and this article focuses on Android Studio, compatibility is not a concern. You can use the plugin directly with an APK to record tests.

Once the tests are created, you can easily copy and paste them into Android Studio along with any required Testdroid Recorder dependencies. A recorded test might resemble the class shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

public class LoginTest extends ActivityInstrumentationTestCase2<Activity> {

private static final String LAUNCHER_ACTIVITY_CLASSNAME = "com.toptal.fitnesstracker.view.activity.SplashActivity";
private static Class<?> launchActivityClass;
static {
try {
	launchActivityClass = Class.forName(LAUNCHER_ACTIVITY_CLASSNAME);
		} catch (ClassNotFoundException e) {
			throw new RuntimeException(e);
		}
	}
	private ExtSolo solo;

	@SuppressWarnings("unchecked")
	public LoginTest() {
		super((Class<Activity>) launchActivityClass);
	}

	// executed before every test method
	@Override
	public void setUp() throws Exception {
		super.setUp();
		solo = new ExtSolo(getInstrumentation(), getActivity(), this.getClass()
				.getCanonicalName(), getName());
	}
	
	// executed after every test method
	@Override
	public void tearDown() throws Exception {
		solo.finishOpenedActivities();
		solo.tearDown();
		super.tearDown();
	}

	public void testRecorded() throws Exception {
		try {
			assertTrue(
					"Wait for edit text (id: com.toptal.fitnesstracker.R.id.login_username_input) failed.",
					solo.waitForEditTextById(
							"com.toptal.fitnesstracker.R.id.login_username_input",
							20000));
			solo.enterText(
					(EditText) solo
							.findViewById("com.toptal.fitnesstracker.R.id.login_username_input"),
					"user1@gmail.com");
			solo.sendKey(ExtSolo.ENTER);
			solo.sleep(500);
			assertTrue(
					"Wait for edit text (id: com.toptal.fitnesstracker.R.id.login_password_input) failed.",
					solo.waitForEditTextById(
							"com.toptal.fitnesstracker.R.id.login_password_input",
							20000));
			solo.enterText(
					(EditText) solo
							.findViewById("com.toptal.fitnesstracker.R.id.login_password_input"),
					"123456");
			solo.sendKey(ExtSolo.ENTER);
			solo.sleep(500);
			assertTrue(
					"Wait for button (id: com.toptal.fitnesstracker.R.id.parse_login_button) failed.",
					solo.waitForButtonById(
							"com.toptal.fitnesstracker.R.id.parse_login_button",
							20000));
			solo.clickOnButton((Button) solo
                    .findViewById("com.toptal.fitnesstracker.R.id.parse_login_button"));
            assertTrue("Wait for text fitness list activity.",
                    solo.waitForActivity(FitnessActivity.class));
			assertTrue("Wait for text KM.",
					solo.waitForText("KM", 20000));

			/*
				Custom class that enables proper clicking of ActionBar action items
			*/
            TestUtils.customClickOnView(solo, R.id.action_logout);

            solo.waitForDialogToOpen();
            solo.waitForText("OK");
            solo.clickOnText("OK");

            assertTrue("waiting for ParseLoginActivity after logout", solo.waitForActivity(ParseLoginActivity.class));
            assertTrue(
                    "Wait for button (id: com.toptal.fitnesstracker.R.id.parse_login_button) failed.",
                    solo.waitForButtonById(
                            "com.toptal.fitnesstracker.R.id.parse_login_button",
                            20000));
		} catch (AssertionFailedError e) {
			solo.fail(
					"com.example.android.apis.test.Test.testRecorded_scr_fail",
					e);
			throw e;
		} catch (Exception e) {
			solo.fail(
					"com.example.android.apis.test.Test.testRecorded_scr_fail",
					e);
			throw e;
		}
	}
}

As you can see, much of the code is quite straightforward.

When recording tests, use “wait” statements liberally. Wait for dialogs, activities, and text to appear before interacting with them. This ensures the activity and view hierarchy are fully loaded and ready for interaction, preventing unexpected behavior. Simultaneously, incorporate screenshots into your tests. Since automated tests typically run unattended, screenshots provide valuable insights into what occurred during those tests.

Whether tests pass or fail, reports are invaluable. You can find these reports in the “module/build/outputs/reports” directory within the build folder:

In theory, the QA team could take on the responsibility of recording and optimizing tests. This could be achieved by establishing a standardized test case optimization model. Typically, recorded tests require some tweaking for optimal performance.

Finally, to run these tests from Android Studio, simply select and run them as you would unit tests. From the terminal, it’s a single command:

1

gradle connectedAndroidTest

Performance of Testing

Android unit testing with Robolectric is incredibly fast because it executes directly within the JVM on your local machine. Conversely, acceptance testing on emulators and physical devices is significantly slower. Depending on the complexity of the flows being tested, individual test cases can take anywhere from a few seconds to several minutes to complete. Ideally, the acceptance test phase should be integrated into an automated build process on a continuous integration server.

Parallelization across multiple devices can enhance speed. Explore this excellent tool from Jake Wharton and the team at Square, http://square.github.io/spoon/. It offers impressive reporting capabilities as well.

In Conclusion

A variety of Android testing tools are available, and as the ecosystem continues to evolve, setting up testable environments and writing tests will likely become more manageable. Numerous challenges still lie ahead. However, with a large community of developers tackling these challenges daily, there’s ample opportunity for constructive dialogue and rapid feedback.

Use the approaches outlined in this tutorial as a guide for overcoming your testing challenges. If you encounter issues, revisit this article or the linked references for solutions to known problems.

In a future post, we’ll delve deeper into parallelization, build automation, continuous integration, Github/BitBucket hooks, artifact versioning, and best practices for managing large-scale mobile application projects.