How to perform good testing of mobile apps
Quality is that what differs mobile apps in the store from those created by non-professionals. If you do not keep it on right level shop rates will go down and people will not use it anymore due to crashes or lack of usability and slow performance. But how to assure the quality when there is so much devices and versions of mobile operating systems?
Below you could find few tips how to make sure that your app will go out to the market with right quality.
Define quality strategy
In order to perform tests in repeatable way you need to set up the process of testing so there won’t be a chaos when the app will be ready for development. Those steps could be end of sprint/milestone or end of project. Sooner is better as every bug that you will find out on early stage won’t cost you much to fix. We test also always before we send app to a client. We don’t want to make testers out of our clients as freelancers tend to do.
Use right people
Self-confidence was my first real surprise when I was at my first development job. I went to the CEO of the company with very confident approach to show him my almost done project. After two or three clicks he told me “What a crap! I cannot use it at all, nothing is working”... Yes. He clicked magic [Cancel] button which I never clicked before assuming that it’s not necessary when the flow is right. Lesson from that history shown me at the very beginning that however motivated the developer is and how ever hard he is trying he will always forget about something or will be blind for things that he have in front of him. That’s why the app you develop should be always checked by third person, who was not involved in programming it. When we hire people for QA department we spent lot of energy figuring out their profile. We look for business oriented, focused on details and having analytic skills individuals.
Specify requirements for end product
Make sure that you know what you want to achieve and where your app will be used. Testing is much better if you know on what exact devices app should work and with what mobile operating system version. Make sure that you or your supplier has all necessary devices to run the app. From our experience only testing on real devices will show you real issues. To avoid late surprises it's better to know if the app will be used on tablets and if it should have separated horizontal view. When it comes to Android devices the variety is big but you should at least define:
- Most used mobile phones. Ex. Samsung s4,s5,s6, HTC one, LG g4,g5,
- Operating system Ex. Android > 4.2
- Most desired screen resolutions which will also indicate ratio. Ex. 720x1280, 1080x1920
With Apple it’s much less to decide. Phones that are not more than 4 years old and two operating system, the current one and the previous one.
Connect testing phones to development environment
If you read my article about productivity in software house you know that developers time usage it’s the most important part of the process. Every time when QA engineer find a bug he need to describe it to the developer. The most important part for the developer are steps to reproduce an issue but even with that he need to spent time to reproduce it by himself. When tester’s phone is connected to development environment like Android Studio or xCode you could see the console stack trace of the issue. That log could be pasted into issue description and lead to even 10x faster issue resolution time.
Analyze crashes of apps by using third party software
While your app is still in development or beta mode you could add special part of code into it to enable sending all the reports about crash of the app to external service. That way developers could see all incidents from plenty of phones and their crashes reports. That kind of report often include data that is hard to extract from users such as: phone version, precise operating system version, state of the application and value of all settings. We are using Crashlytics for both iOS and Android native development.
Test on multi devices configuration
Simply run the app on few devices same time to be sure that it works and look the same on all of them. You need to differ devices in particular way:
- Different screen sizes
- Different phone vendors (ex. Samsung’s Android add-on could cause you a lot of pain)
- Different operation systems
- Different phone performance - if the app works smooth and fast on old and cheap phone it means that it will work on top ones as well. Testing only on good phones won’t tell you much
Do not perform functional and regression tests only
When we did our first mobile applications we were struggling with plenty of bugs there. After we established QA team to test applications according to designs and limited specification we thought that we are now all set up and heading for right quality. We were so wrong! We realized how important for quality UX check was. At the end some of the features that we developed according to client specification and design were so wrong. The best example was the same design for the Android and iOS app. There is simply no back button in iPhone and there was not possible to use the app at the end. Lucky for us few of our developers had their own experience with developing apps and deploying them. We then learn the lesson, gather all inputs and knowledge we could find about the subject. We created internal document with 37 pages of hints and screenshots what to look for. Nowadays we are able to provide Valuable feedback for our clients regarding their app design and ideas.
Above in article I mentioned about hiring right and create right documents for UX testing. After having it done we created series of internal training sessions for QA engineers to assure that they will be able to find certain issues for UX, app crashes, app logic failures and that they will be able to pass that information to the developers or client in the right way. Each time when we perform training we create internal presentations and document. Every time new person is on board we use them to share the knowledge we already have. Important part is also to ask QA engineers to be proactive to share the knowledge with colleagues and update training materials each time there found out something new.
Use right software to communicate with client and manage issues
It’s a good idea to use apps like TestFlight or HockeyApp for test deployment of your app for all stakeholders. Despite fact what kind of software you are using to distribute app for the tester and client’s feedback. We recommend using also ticket management system to be able to track and handle the tickets by developers. We use JIRA for internal issues and JIRA Service Desk for clients as it’s much easier to use.
Your objection is to have app with best possible quality but it’s not always needed to have an app with all features on board in first release. In my opinion the most important part is to have it with the right scope and the right quality for the moment. Sometimes it’s better not to have some features or have some workarounds inside app than struggling few more weeks trying to fix them. From our experience not all features are needed there in first place. It’s much more important to have client’s feedback, see how they are using the app and what they really need to have there.
If you are want to read more about that topic - my colleague just wrote an article: How to make good app.