Test checklist for iOS mobile applications

I am not a huge fan of test cases (as defined e.g. by ISTQB). Large suites full of unmaintained step-by-step scenarios is not my cup of tea.

However, it is definitely good to have at least a list of things that you really want to test, high level scenarios, a checklist that you can go thru and at the end of the day rest well with good “I have it covered” feeling.

Below is my “brain dump” of things that I test on mobile devices. This one is iOS specific, but I think most of it can be used for other platforms as well.

(You can also download the list as PDF or .xls at the end of the post)

UX/UI

Test case
Description
Notes
Splash screen The splash screen is present long enough.
User is able to read the information on it
Check if not too long (>~3 sec). If necessary, suggest adding spinner / progress indicator
Portrait / Landscape mode If supported, check each screen of the app in both display modes and verify all components are visible / accessible Landscape mode with keyboard displayed does not leave too much space (even less with iPhone status bars)
Status bars iOS displays status bar (in call, microphone used for recording, device connected to hotspot …). Check that UI is not broken by the status bar
Related image
Accessible controls All UX elements are accessible, buttons / hyperlinks can be pressed, input fields clicked (in order to enter text) Especially check if there  are e.g. buttons near sides or in corners of the app – those are really hard to press and/or can interfere with gestures (swipe left / right)
Content fits the screen Check that images / text is fully visible on the screen, no trimming or cropping. There may be exceptions – e.g. partial images indicating horizontal scroll. Check for unintentional cropping
Soft Keyboard toggle The keyboard should appear if user is about to enter text (click on a input field). It should disappear if user click outside the input or confirms the form. The UI should be designed to deal with (approx.) half of the screen is covered by the keyboard. Make sure the entered text is visible. Also, in case of e.g. Search/Filter functionality that affects the content of the same page,  make sure that at least part of the page is visible (so user sees the results)

Try different keyboards as well (default, custom – e.g. Swift Keyboard)

Keyboard context The keyboard should be appropriate for the given input field (based on expected input format) – e.g. numerical keyboard for pure numeric entry (phone number, PIN code, etc.) Check  / suggest using special keys like “@” for email entry, or “.com” key for website input fields
Unmapped keys Check if the keyboard does not contain keys that do nothing Brute force required here … Simply tap all the buttons in the soft keyboard
Form fields navigation It is possible to navigate thru form fields using the soft keyboard (e.g. a “next” key or “→”) It should not be required to hide and show the keyboard each time, just to switch between form fields.
Optional form fields There should be an indication of mandatory vs. optional field Typically it is an asterisk for mandatory fields, but it can differ  – check specifically for your app under test.
Gestures Perform all the supported iOS gestures. Check that the gesture has expected effect. If not directly supported by the app, it should not break it or have other unexpected results. The gestures are: tap, drag, flick, swipe, pinch, double tap, touch and hold, shake

See: Apple Developer Guidelines

Interrupts

(info) Try those at different phases of the application workflow (e.g. in the middle of the login process, during payment processing etc.)

Test case
Description
Notes
Incoming call If the application is interrupted by incoming call, it should be able to resume from the same point Test both accepting and rejecting the call
Incoming SMS Incoming text message should not have any unexpected impact on the app (crash, freeze etc.). User should be able to resume after reading the SMS Test just the display of SMS notification, as well as clicking on it and penning the Messages app.
Incoming push notification from other application Push notification should not have any negative impact, the app can resume at any point. E.g. Facebook / Twitter notification, event in Google calendar. Try just reading it or clicking on it in order to open the other app and going back to app under test.
Low battery notification The application should not be affected by low battery message and switching to battery saver mode Prepare for the exciting work of waiting for the battery to discharge.
First message appears at 20% battery level, second on 10%
Battery fully charged Check that the notification about battery fully charged (and exiting battery saver mode) does not affect the application under tests
Sleep mode Check that the application resumes correctly when exiting from sleep mode
Lock screen Lock the device and check that the app resumes correctly after unlocking
Switching to another application Switch to another running app or start a new one while minimizing the application under test. Check that it resumes correctly upon returning back. In order to switch between running apps, double tap the Home button (or swipe up from the bottom of the screen in case of iPhone X)

Unavailable Services

(info) Try disabling the services required by the application – network connectivity, camera, location services etc.

Test case
Description
Notes
Network not available Disable network connectivity and check if the application handles the missing connection gracefully – e.g. notification to the user, request timeout, ability to continue and eventually syncing with the server later (depends on the app). Test ideally during ongoing data transfer (like in case of login process or fetching data from server)

Tip: to quickly disable the network, use Airplane mode.

Denied access to services Verify how the application handles if it has restricted access to required services. Ideally, in case the service is not allowed, the application should inform the user and provide the link to Settings where it can be enabled. The app should not crash of display some unexpected behavior.

(Check which of these services is required by the app under tests. Also check why it is required and if it is necessary)

  • Contacts
  • Microphone
  • Calendars
  • Camera
  • Reminders
  • HomeKit
  • Photos
  • Health
  • Motion activity and fitness
  • Speech recognition
  • Location Services
  • Bluetooth sharing
  • Media Library
  • Social media accounts, such as Twitter and Facebook

=> create test cases based on the supported / required permissions for your app

The app should ask for the first time you try to access the service (e.g. take a picture with the camera).

After that, it can be enabled / disabled in the iPhone’s Settings

App specific features

(info) Make sure you know what features the application under tests supports

Test case
Description
Notes
Test case
Description
Notes
Widgets In case your application supports widgets, add the widget to the screen and check the data consistency between the app and the widget. In fact the widget itself can be tested as a “mini app” with almost all the test cases applicable (especially UX/UI, assess to services etc.)
Payments Check that the payment gateway works as expected and that it requires authentication (PIN code, touch-id, face recognition…) Be extra careful when dealing with payments. Make sure you understand how the payments are implemented, and that a test mode is enabled.

Also make sure that there is a plan of switching to production mode as the app is released.

Sharing If there are sections in the app that can be shared (upload photo on Facebook, send report to email or slack, save to Notes etc.), check that the result is in expected format.

Also verify that the app handles e.g. missing connection to Facebook account correctly, without crashing or other unexpected result.

Voice recognition If the application supports voice recognition as input, test that it is processed correctly. Try also in combination with disabled microphone
Localization If there is multi-language support, verify that the languages can be switched without errors. Also test that the content is displayed correctly in all languages Remember the very long German words? They can easily overflow…
Security check If the application provides any form of security check (PIN code, touch id, password, …), make sure you test both positive and negative scenarios and the way how the app deals with attempts for unauthorized access. Also check the “tone” – it should be more “seems like you have problems, may I help you with resetting the password” rather then “you are criminal, banned from this app forever” after you type in the wrong password.

Performance

(info) There are no general “optimal” values for performance parameters. Always evaluate based on the character of the application. Watch for obvious signs of trouble (memory leaks, suspicious CPU usage or battery consumption)

Test case
Description
Notes
Response times Check the response times of the application – how long does it take to respond after user e.g. clicks a button or submits a form. Use both “gut feeling” (does the app feel fast and smooth?) as well as solid numbers (debugging tools for time measurements, or simple stop watches). The optimal value is hard to tell, usually ~ 1 sec is considered to be the upper bound. If there must be some sort of delay, make sure the app keeps the user’s eyes occupied with verification of action – a loading spinner, progress bar …
Memory Check the memory consumption of the app and make sure there are no memory leaks.

It is not possible exactly to tell some “optimal” value, it always depends on the type of the application.

Try e.g. comparing with similar types of apps. Ask developers in case of suspicious numbers.

See this SO answer for general boundary values (the max that the app can handle without crashing, usually it is about 60% of the total memory)

CPU Check the CPU usage of the device with / without the application running. Check for suspicious peaks (cross reference it with the activity you do in the application) Use any of the system monitoring apps that offers real time monitoring with graphs (e.g. System Status Pro)
Battery Check for excessive battery consumption – use different modes of work: intensive (use the app continuously, click on everything (TM) ), normal (use it as you would expect default user to work with the app), idle (leave the app running on the background) iPhone offers basic statistics of battery consumption per app, which is quite sufficient lead. If there are some doubts, use specialized monitoring app
Network traffic Verify that the app has expected amount of network traffic – check for suspicious large data transfers (especially when no action was triggered by user), excessive API calls etc Monitoring apps usually offer statistics of Wi-Fi / cell data usage and graphs (e.g. System Status Pro).

If possible, check the logs of the backend server for incoming requests from the client device.

Device & OS

Test case
Description
Notes
Various devices Test that the app behaves correctly on all supported devices
  • Try smallest / largest device when checking UI
  • Use cloud services (e.g. Browserstack or AWS Device Farm [1]) to check devices you do not own physically
Various OS Test on older versions of iOS
  • Check with client the required backwards compatibility
  • Check with developers if there are any known OS version limitations
  • Use cloud service to check OSes you do not have on local devices
  • Check iOS coverage statistics

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s