Running mobile performance regressions in CI is hard, so we built a dev-friendly tool capable of:

Provisioning real devices to install and run the app
Running simple UI tests over the app on pre-release stages
Gathering device resource consumption metrics and transaction times
Defining baselines to pass-fail the pipeline
Running tests fast, in just a few minutes

If you are looking to add mobile performance tests to your CI pipeline, this is an excellent way to start. In this guide, you will discover how fast this can be achieved using Apptim CLI and a popular CI tool: Github Actions.

Let’s get started!

1 - Setting up a test

For this, you will need:

Your app file (apk / ipa) to install in a real device
Your apptim key: APPTIM_API_KEY
AWS keys and AWS project ID: AWS_ACCESS_KEY_ID , AWS_SECRET_ACCESS_KEY and PROJECT_ARN

If you don’t have an Apptim account, just create one here and login to the cloud to get your API Key. AWS credentials are provided by Apptim, as part of a Team Plan (contact us at [email protected]). Otherwise, you can use your own AWS keys from any IAM user that has privileges to run test on AWS Device Farm.

You can set up your keys using Github secrets, as show below:

github secrets configuration in repository

1.1 Creating the test spec

Create a YAML file named test.yml where you will be declaring the path to your app and which devices you want to use for the test. Apptim will take care of provisioning the devices for you using AWS Device Farm.

This example shows how to a startup time test of apptim-demo.apk on a Google Pixel 3 with Android 10.

name: Startup time test - Android 10
app-file: apptim-demo.apk
package-name: "com.apptim.android"
test-runner: 
  name: startup-time
thresholds-file: thresholds.yml
timeout-minutes: 5
test-devices:
  - device:
    name: "Google Pixel 3"
    os: "10"
device-farm:
	project-arn: "AWS project ARN"  # Apptim Android

For Android apps you can specify how many times to launch the app in the device (iterations) and the type of launch (cold, hot or warm), as detailed here.

This type of tests also support simulating different network characteristics, for example, to test the startup time in a 3G network vs 4G.

1.2 Setting the baseline

In order to define the baseline to fail the startup time test, create a thresholds.yml file with the main KPIs you want to measure, overriding the value with a specific number for your app:

startup_time_top:   # First startup time (in ms) 
  warning:
    operator: ">"
    value: 1500

app_size:           # App Size (in bytes)
  warning:
    operator: ">"
    value: 60000000

⚠️ Using these thresholds, your pipeline will fail only if launching the app takes more than 1.5 seconds, or if the app size is bigger than 60MB.

That’s all, now you are ready to launch your tests!

📘 You can read more about defining custom thresholds here. If you want to learn more about how startup time tests work on each platform, go to Android | iOS

2- Running a test in CI

Apptim CLI can be installed on windows, macOS and linux. The zip file contains the executable and some libraries. To start a test, you can use the following command:

macOS \| Linux:	Windows
`./apptim run --config test.yml`	`apptim.exe run --config test.yml`

2.1 Running a test using Github CI

When running with Github CI, you can use the apptim-cli-action available in the Marketplace. This action takes care of installing the latest apptim-cli version and running it using the YAML file as argument.

To use apptim-cli-action, make sure you have the app (ipa or apk) and both YAML files created on the previous steps. For example, you can store them on a git repository and do a check out inside the pipeline.
As with any Github workflow, you must create a YAML file under .github/workflows/ folder. For example .github/workflows/run-tests.yml

name: Run tests workflow using apptim-cli-action
on:
  push:
    tags:        
      - 'v[0-9]*'         # Match any tag starting with v
    branches:
      - 'release/[0-9]*'  # Match any branch starting with release/

jobs:
  launch-tests-action:
    name: Run startup time test
    runs-on: ubuntu-20.04
    steps:  
			# Get the app, test.yml and thresholds.yml
      - name: Checkout code  
        uses: actions/[email protected]
      # Run the tests
      - name: Run tests      
        uses: apptim/[email protected]
        env: 
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} 
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
          APPTIM_API_KEY: ${{ secrets.APPTIM_API_KEY }}
          CONFIG_PATH: test.yml

This will launch the test using apptim-cli, which will install the app and run the test on all devices selected in the test.yml file. After the test runs, it will check the results against the thresholds defined, and will fail or pass your pipeline based on the criteria used. You will also get the detailed output for each KPI on the console log of your pipeline and an HTML report with more details and logs.

Example scenario:

Run a startup time test with a fail criteria of 250ms (defined in a thresholds.yml file). After the test run, the following output is shown:

The startup time took 1765 ms and the threshold was at 250ms, so the pipeline failed showing the following warning:

And a link to an HTML report is generated:

A public report can be found here.

2.2 Running a test in any CI pipeline

Apptim CLI can be easily added to any CI pipeline using tools like Jenkins, Azure DevOps, TravisCI, CircleCI, or more.

For example, on Azure DevOps you can use the bash task. You will only need to download apptim-cli and use it on a bash step, as shown below:

Need support?

If you have any questions about this guide or need support, please reach out to [email protected]. Our team will be happy to help!