When running your first test with Apptim, default thresholds will be used, as explained here. But after you have run a few tests, you will probably want to set your own thresholds for your specific app/devices used (see Benefits).

How can I set my own thresholds?

Mobile performance thresholds are set inside your Apptim Cloud account, in a specific workspace and app. This way, the thresholds can be accessed and used by different teammates when running tests inside the same workspace. The only pre-requisite to define your own thresholds is to have at least 1 report published to a cloud workspace.

Once a first report is published, you need to:

  1. Log in to your Apptim Cloud account

  2. Navigate to a workspace / app

  3. Click on SET UP THRESHOLDS FOR YOUR APP option

  4. Write a name for your thresholds file, like "Low-end devices"

  5. Define the performance metrics and thresholds for each using specific keys (see Keys)

  6. Save the file

This will automatically create a .yaml file with a key for each performance metric and a moderate/warning threshold. For example:

# CPU
cpu_avg:
  moderate:
    operator: ">"
    value: 20
  warning:
    operator: ">"
    value: 40

means that for the CPU consumption measured during the test, if the average exceeds 20%, a moderate limit alert (yellow mark 🟡) will be shown in the report. If the average exceeds 40%, a warning limit alert (red mark 🔴) will be shown instead. Otherwise, you will see a low limit result (green mark 🟢) in the report.

Notes:

  • Operators allowed are "<", "<=", "==", ">" and ">="

  • Be aware that .yaml files need to be validated and are sensitive to padding / whitespaces, make sure to respect the file structure provided

Keys available for each performance metric

Android

# CPU
cpu_avg | Average CPU usage in %
cpu_max | Maximum CPU usage in %

# MEMORY
memory_pss_avg | Average memory footprint in MB
memory_pss_max | Maximum memory footprint in MB

# NETWORK
net_transfer_total | Total network transfer (upload) in KB
net_receive_total | Total network receive (download) in KB

# STORAGE
storage_used | Total amount of local storage used in MB

# FPS
fps_avg | Average frame rate (in FPS)
render_frame_jank | Maximum discarded (janked) frames

# ACTIVITIES
activities_count | Number of activities declared on the app
activities_backstack | Number of activities that are stored on the backstack

# LAYOUT
controls_per_screen_max | Maximum UI controls allowed on a certain activity

# LAYOUT DEEP
layout_deep_max | Maximum nested level of layouts on a certain activity

iOS

# CPU
cpu_avg | Average CPU usage in %
cpu_max | Average CPU usage in %

# MEMORY
memory_usage_avg | Average memory footpring in MB
memory_usage_max | Maximum memory footpring in MB

# NETWORK
net_transfer_total | Total network transfer (upload) in KB
net_receive_total | Total network receive (download) in KB

# STORAGE
storage_used | Total amount of local storage used in MB

# FPS
fps_avg | Average frame rate (in FPS)

How can I use my own thresholds in a test?

Once you have defined and saved your new thresholds, you can start a new test in Apptim Desktop (supported on Apptim Desktop v0.6.4 or later).

Make sure you've selected the same workspace where you set the thresholds before (by clicking the folder icon the top right menu) and the same app to test. The new thresholds will appear in a drop-down menu, as seen in the following screen:

Benefits of using custom mobile performance thresholds

Once you have run a few tests with Apptim Desktop using your own devices, you will have enough data on how your app performance is on a specific device. This can be used as a baseline to set your own thresholds according to the results obtained and identify performance regressions in future versions of your app (before it reaches the end user 😉).

Did this answer your question?