Pick the Right Process: A/B Testing vs Prototyping

For anyone involved in building a product, you know the importance of going fast while minimizing risk. You need to move fast to get your ideas out to customers, to solve your users’ problems, and to beat the competition. But, you also need to carefully minimize the risk of building the wrong product or of spending too much time on ineffective features.

By testing your ideas frequently and vigorously with users, you can gather data on what works and what doesn’t, and then move forward with the most significant improvements.

Should You Use A/B Testing to Test Your Mobile App?

The most typical way to do test your mobile app is with A/B testing: release two variations (A and B) and see how they perform compared to each other. Especially with feature flag management as a service today, we’re seeing more and more adoption among enterprise companies undergoing agile transformations.

Limitations to A/B Testing

Sounds great and easy enough, right? It can be, but it's important to remember that the process isn't perfect. Here are a few limitations you might run into with A/B testing:

• Limited variations in each version: As with any good experiment, you want to minimize the number of changes when you compare them against each other, or you may not know which of the changes in each version was responsible for how it performed.
• How it performed, not why: While you’ll get data on which version users clicked more, you won’t know why. And while those results may match your hypothesis, it’s hard to know why, especially if your question isn’t narrowed down enough.
• Building different versions can be expensive: Unless the variations are pretty similar, designing and developing each version for release can be expensive and time-consuming.
• Global vs. local maximum: This is really the underlying issue here. If you’re making tweaks to a process or have several similar versions you want to test, A/B testing is the way to go. However, this method is not ideal when you want to make larger changes or are still exploring different directions.

Utilizing Prototyping to Create Your App

This is where prototyping and user testing can be more helpful in your process. User research is the qualitative equivalent of A/B testing’s quantitative focus. Instead of showing a fully developed version to millions of users and collecting the data, show a prototype and learn through user research why certain approaches are (or are not) effective for certain users. And unlike traditional approaches to user research, you can now quickly and easily build a full-fledged app instead of having users click through static wireframes. Through a working prototype, you can:

• Use real data: Real data matters to users’ perceptions of their experience. When you ask users what they think about something, they react very differently when it’s personally meaningful to them. Not only will they tell you what they think, but they’ll start engaging with your mobile in ways that you may not have anticipated. At a minimum you’ll find some friction points in your flow, but you may find additional paths and use cases you hadn’t even considered.
• Identify edge cases early: Not only does using real data change user perceptions, but it can affect the design process as well. Data is invariably messy, meaning it's not always evenly distributed. Your data may not be structured correctly, and it may not even be accurate in places. Playing around with the data in the actual user experience is a lot easier to understand in context than viewing the data in JSON or a spreadsheet. And it’s a lot cheaper to make changes now than after you’ve finished (or even started!) your mobile app development.
• Experience your ideas in context: When Jeff Hawkins was developing an early handheld device (pre-iPhone days), he carried a block of wood as a prototype to meetings to simulate what it’d be like to hold and use the device in everyday situations. Especially with mobile apps, using the app on a device (not a web browser on a laptop) out and about is a different experience than in a lab environment.
• Get feedback with minimal investment: Oftentimes, you’ll have a grand vision of what re-hauling UX could look like, but it’s hard to get the buy-in for big changes. A working prototype allows you to get feedback on what those changes would mean without having to change your production app. Even if you don’t get the data necessary to make an eventual large investment, you’ll likely find insights on what aspects of that vision are most beneficial.

Tips for Getting Started with Prototyping

In situations where a prototype would be most helpful, here are some tips to get you started:

• Develop a hypothesis: Like with A/B testing, it’s important to know what questions you want answered from the prototype. Setting up the right questions means you can prioritize how “full-fledged” of a prototype to build and which functionalities to include or exclude.
• Build and iterate: Building a prototype is like a mini version of app development but faster and with fewer constraints. Pick a tool to help you quickly get from vision to app faster and still allows you to easily make changes. Even before you start showing users your mobile app, you’ll likely want to make adjustments; be prepared to do that early and often.
• Show, not tell: Prototypes are helpful not just in getting feedback from users or coworkers but also as a way of sharing your thoughts. Wireframes or even a sketch on the back of a napkin can do a better job of illustrating your ideas than just the written word. Providing a working app gives everyone a clearer idea of your vision.

Final Thoughts

It doesn’t matter if your product has mass adoption or is still in its infancy, you’ll need to constantly test and evolve. Gaining a firm understanding of your users’ workflows, desires, and goals is what separates elite app developers from the rest. The secret to cracking that code is through thoughtful and frequent user testing. Utilizing rapid prototyping can help improve iteration cycles, adoption rates, and key success metrics. So, use this guide when determining which type of methodology is right for you.

When the time is right, Thunkable is here to help you design, test, and iterate quickly.