Beta Testing

Almost half of the small businesses these days have their own mobile app.

A custom app is an opportunity to generate business leads, build customer loyalty, and connect with customers.

The path from idea to app covers a lot of ground. A key stopping point along with the way is the beta testing stage, where the app gets worked over by real users to ensure the final product works the way it should.

In this article, we take a look at the importance of beta testing and the steps involved in making your test a success.

Beta testing is the last step you take before launching your application to the world. It is a chance to distribute the app to a big audience of general users who try out the app for a set period of time and report back with any issues, bugs, or feedback.

Testing with actual users allows you to validate the idea and features of your app and confirm it meets their needs as you expected. It is a chance to resolve any flaws or issues before releasing it out to everyone.

You might have a lot of confidence in your mobile app development team, but even the best programmers can miss issues in the code. While they will have done rigorous testing of the app during an alpha phase, their perspective and experience do not accurately reflect that of your end-users.

Beta testing gives you feedback in a real-world environment with users who do not have any particular expectations about the app. Their feedback can help you improve the features you have and uncover any issues before launch.

Beta is also a time to bring in a lot more users than you would have at the alpha stage. The larger group can run through more scenarios and use the app for a longer period than a smaller in-house group. They will be working with your app on real platforms and different devices, some of which you might not have had access to during early testing.

Your team might have known issues that they have accepted and learned to live with. But mobile app users are an impatient bunch with a low tolerance for mobile apps that crash or use too much battery. More than half will uninstall an app with serious issues, and 80 per cent only give you three tries at most to impress them.

At the end of the day, beta testing is about improving the chances of your app finding a happy home on user devices. You want the product to be a good reflection of you and your brand and serve your customers’ needs well.

To get the most out of your beta test phase, you need a smart plan rooted in your mobile app strategy. With mobile app development, timing is critical and you want to start your beta as soon as you can. Some key points that need to be in your plan include:

The basis for your plan should be the business objectives of the app. The plan should provide as much detail on the scope, schedule, and tools you will be using.

Two additional parts of the plan should relate to when you start the test and how to know when you are done. Entry criteria include being done with alpha testing and having the beta version ready to launch. User manuals and logged known issues should be available and the tools for capturing bugs and feedback ready.

With your plan in place, you are ready to start with your beta test. The process begins with some planning decisions before you launch and moves through to collecting feedback at the end.

Figuring out how many testers to have is more art than science.

Only about 20 per cent of your group will really test the app and give valuable feedback. So you need a large enough pool to get a good number that completes the process, but not so many that managing the pool becomes onerous for you.

A frequently recommended rule of thumb is 100-300 testers. That gets you enough feedback without being overwhelming.

You can also decide on numbers based on the type of app you are making. You can get away with just five-10 testers for small projects and 25-40 for casual-use applications. A gaming app definitely calls for 200 or more testers due to the more complex nature of the program.

The type of tester you need depends in part on the tester persona you choose. There are two general types of tester – technical ones, who look for bugs and give constructive feedback, and marketing ones, who work to increase awareness before the app is launched.

We are mostly talking about technical testers in this article, but a beta test can be used to begin your mobile app marketing as well. You can do a split test with testers placed into different groups based on the feedback you are looking for from them.

The right testers affect the quality and outcome of your test. With a closed test, you can be selective in choosing testers with a relationship to your company or product or a track record serving as beta testers.

Just how long should your beta test be? Too long of a test risks burning out testers leading to decreasing participation over time. Too short of a test might not give you enough time to reach the test goals.

A good rule of thumb is no shorter than two weeks and no longer than 12. Most beta tests tend to run between six to 10 weeks. Your timeframe should be decided based on your goals, resources, number of phases, and tester limitations.

Whatever timeframe you decide on, definitely have an end date in mind. You do not want the beta stage to go on indefinitely. There should be just enough time for your testers to test all the features of your app and check for any issues.

Finding great beta testers is critical to success. So just where do you go to find them? Your testers become your early adopters and future evangelists, so you want enthusiastic participants.

One good way to find technical testers is by posting your test on specialized platforms. Some respected ones that can connect you specifically with testers include:

You can also put out an open call on Twitter, Reddit, or Quora. When you post, use an appropriate hashtag like #testmyapp or #betatesters so people can find you. You can share your call on specific subreddits – /r/TestMyApp, /r/alphaandbetausers and /r/AndroidAppTesters to start.

Look for communities on Facebook and LinkedIn where you can share your call as well as through your own channels. Use your recruitment posts to talk about the app and the solution it provides.

It is time to get to the meat of the test! Installation packages for the app need to be distributed to your test group. You also need to be able to share user manuals, guides, known issues lists, and other testing information.

The best way to go about all of this is to work with a management platform. It becomes the single place for your testers to get what they need and for you to collect bug reports and feedback.

In choosing a platform, you can go with the official ones from Apple and Google or select a third-party option.

Google Play Console tests your Android apps and allows you to run an open beta or a closed beta by email address. TestFlight is the official Apple platform. You run a test on the site by submitting a build for review, then either add external testers or share a public link.

There are downsides to using one of the official platforms, however. They require special accounts and only support apps meant for their operating system. A third-party choice should allow you to test across all platforms and devices.

Some well-known third-party distribution platforms include this shortlist:

Whatever platform you choose, you want it to allow for easy distribution of your files and the ability to add release notes. Crash reporting, report management, and integration with bug-tracking systems will also prove useful during the test period.

Getting feedback from your users involves two different facets. First, you have to get the testers to use your app regularly and enough to try most of the features. Second, you need to get them to provide you with comments and feedback.

A common way to motivate testers is to provide incentives. These could be offers for signing up, completing certain tasks, and providing feedback. Companies often give away copies of the final app and offer to include testers in future tests.

You can also consider offering codes for premium features, which encourages testers to spend more time in the app. Gift cards for in-app purchases are also a nice incentive that does not cost you anything but keeps them in your app more.

You will have a certain amount of fall off of testers as time goes on. Decide early on how much time and effort you will spend interacting with testers to encourage them to spend the time needed to make a solid evaluation of your app.

If you have analytics or other monitoring tools in your app’s backend, use them to track how your testers interact with the app. Consider a plan to turn features off and on remotely so you can conduct A-B split tests.

Using backend features to track activity also gives you a more accurate view of how well features are doing than just relying on self-reporting and surveys.

If you do use follow-up surveys, make sure you structure your questions to improve the quality of feedback you receive. These include questions that ask for specific details, offer multiple-choice answers, and use scales of one to five rather than 10. Make at least one question an open-ended catch-all to keep the door open for random thoughts.

Keep in mind that you are actually working to get negative feedback. You want testers to try to break the app and probe the dark corners. The questions should be looking to get answers that point to your app’s weak spots so you can fix them before launch.

Trust your testers and take into consideration what they tell you about the app, even if you do not like it. If you selected well, you will get honest feedback that might be difficult to hear at times.

Trusting does not mean you need to make every change they request, though. And do not forget you need to add time to the schedule to test changes you do choose to make.

You want your mobile app to put your company’s best foot forward right from launch. Beta testing ensures you get actionable feedback to make your project a success from day one. Following these steps for testing gives you a solid plan for reaching that goal and making sure the entire process delivers the information you need.

Check out our other articles about mobile development or contact us to talk about how we can help you with planning, building, and testing a mobile application.