User Testing Your Shopify App: Public App Use Cases You Should Test

So you’re building a Shopify app—that’s amazing! Apps are a critical part of the Shopify ecosystem and fuel countless merchant stores every day.

Building a Shopify app is the mark of a great developer, because not only are you building an app for a merchant, you’re building an app for the merchant’s customers as well.

It can be tricky to consider all the nuances of Shopify and commerce as a whole. You might think you’ve completed the project after finishing that last line of code—but really, your journey is just beginning.

User testing your app

Before you submit your app to the Shopify App Store or to merchants directly, you need to remember to test it rigorously. On the App Review team at Shopify, we see lots of in-beta or incomplete initial app submissions. While we appreciate the enthusiasm and eagerness, you do need to ensure you create feature-complete products that are ready for our eyes and merchant eyes.

This means that you have to test your app as thoroughly as possible because not only does your app impact your business, it impacts merchants’ businesses as well.

"Test your app as thoroughly as possible because not only does your app impact your business, it impacts merchants’ businesses as well."

In theory, testing sounds pretty simple. You build an app, test installation, verify the main functionality of the app, and you’re good to go, right? Wrong! Shopify is a complex platform, and every one of our merchants is unique. So while you will build your app for the many, there are some specific use cases that will come up that you should be testing, too. 

That might sound stressful, but not to worry—in this article, we share a quick list of testing scenarios. This information is designed to give you necessary context for some specific situations within the Shopify ecosystem. It will also help ensure your app looks great for merchants, and will help you pass the review process when you submit your app.

Throughout the post, we’ll get into some common use cases and some basic foundations including core online store components, installation, products, orders, billing, and general principles. Let’s begin!

You might also like: How to Build a Shopify App: The Complete Guide.

Testing the online store

If you’re building an app that impacts the online store, you are in good company. For many merchants, the online store is their main—or only—sales channel. As such, when building an app for the online store, you must ensure you are building a product that will make the main business channel look good.

Some important things to consider here are:

Impact on the cart

The cart is how merchants’ customers reach checkout, which means it’s your job to ensure merchants don’t miss a sale. If your app impacts the cart in any way, you have to make sure that you test it across multiple themes, with multiple apps (using partner-friendly apps), and with multiple different product combinations in the cart. 

Page and drawer type carts should both be tested. When in doubt, have a contingency plan and ensure merchants have all necessary setup instructions, uninstall instructions, app limitations, and requirements.

Shop domain

While some merchants do wish to keep their initial .myshopify domain, many merchants do not and opt for custom domains. Your app should be able to handle the merchant’s correct shop information and brand the shop correctly. 

Mobile responsiveness

If you’re building an app that impacts the storefront, you should be building mobile-first. Mobile-first commerce is projected to grow to $3.56 billion in sales in 2021, according to Statista. Mobile commerce is projected to comprise almost three-quarters of ecommerce sales, so your app needs to be as mobile responsive as our merchants’ themes and follow industry-standard accessibility principles.

Multiple themes

So many apps produce errors on themes or don’t have expected outcomes on certain themes. While Shopify themes follow a similar pattern and have become more organized since they were sectioned off, all themes are still different. This is why as developers building for the storefront, you can’t rely on testing just one theme (such as Debut). You have to test with different themes. You can also fill in theme sections with text and placeholder images made easier through our Burst integration.

Testing different order and product scenarios

You can’t just test for starter shops. While many of the merchants who join the platform are fresh entrepreneurs looking to start their new businesses, Shopify powers businesses of all different sizes, and your app should be tested to accommodate their various needs.

Some important things to consider here are:

Test on multiple locations

Many merchants operate using inventory from different locations, and your app needs to be able to accomodate. 

Test on shops with sold out stock

I can’t tell you the number of times, as a reviewer, an app has tried to sell me a sold out product and allowed me to add it to cart, only to be rejected by the checkout. There are also many times that apps that work in the back end for merchants end up producing inventory or sales order issues for shops with sold out products. The best way to avoid this? Set up some sold out products and see how your app interacts with them.

Test on shops that do not track inventory

Some merchants opt to not track inventory in Shopify. This could be anything from grocers that sell a variety of different foods based on weight, to a dropshipping merchant that doesn’t need to worry about inventory. 

Your app should be able to handle inventory that is not tracked as well as inventory that is. A key piece of advice: do not treat “N/A” stock as “out of stock.” It could cost a merchant a sale.

Test on products with “compare at prices” 

Many merchants take advantage of compare at prices to make their sales more visible to customers and to also better document marked down items for reporting. Your app must be able to successfully handle the discrepancy and use the appropriate price where relevant.

Testing on product variants

When shopping you’ll often notice the same product is sold in various different sizes, colours, editions, etc. To support this, Shopify merchants take advantage of the product variant functionality, offering merchants a different set of properties for the different variations of their main product. You must take variants into consideration when building your app.

"You must take variants into consideration when building your app."

Process various test orders 

If your app impacts any post checkout process (i.e. order status, inventory adjustments, post-purchase marketing, shipping and fulfillment), you need to be performing your due diligence as the app developer and performing multiple different test orders. These test orders should include controlled groups of app-affiliated resources, non app-affiliated resources, and a combination of both.

Test on different order statuses

Merchants use many different payment methods, from payment gateways to cryptocurrency to cash on demand. As a result of these different payment methods, not all orders will be marked “paid.” The same can be said about fulfillment (i.e. fulfilled and unfulfilled). You must be mindful of these different payment and fulfillment statuses. 

If your app operates in fulfillment, either physical or digital, a crucial piece to test for is whether your app automatically fulfills pending orders (not paid). This could put merchants at a loss should payment fail and in doing such, hurts your business as well as the merchant’s.

You might also like: Everything You Need to Know About Development Stores.

Testing installation

Although I mentioned this as a simple step earlier on this article, you’d be surprised at the number of apps that get rejected per month based on failure to properly install according to our standards.

Here are some key things to remember about install:

Cross check your code to your app set up info

In your Partner Dashboard under your app name and app setup, you’ll find some URLs and some token info. These are the URLs and tokens you have (hopefully) plugged in to your app’s code. If they’re not, your app will likely fail to install or attempt to install another app to your test shop.

Test direct install from the Partner Dashboard 

This will predominantly apply to those apps that want to submit to the Shopify App Store. It’s one thing to use a direct install URL and plug in your .myshopify domain, but it’s another to go directly to the OAuth screen when Shopify gives you the shop properties for install. 

To ensure your app is moving to the OAuth screen first after hitting Add app, test the install on a development store in your Partner Dashboard. More info on how to do this can be found in our docs—or, even better, consider using session token authentication. 

Double check your embedded app

Embedding your app in Shopify gives you a lot of great resources and the possibility to add new functionality. But embedded apps also breathe a lot more complexity into your process. You need to make sure that the content you load is hosted in the same place, and that all content is SSL secured, otherwise it will not display. You also need to make sure that your app is properly redirecting out to content if you do choose to display other content. Rule of thumb: click every single button and link in the embedded view.

Test reinstall

There are two main ways to test reinstall. You can test another install about a minute after you uninstall the app in the same session, or you can test reinstall after you’ve cleared your cache. Both of these scenarios are likely to happen, and your app needs to handle both.

Test the app on another shop with the same shop email

It should be no surprise to you at this point that Shopify merchants have different business models and structures, meaning some merchants choose to operate multiple shops under the same email. 

If they’ve fallen in love with your app, it’s a good thing! To keep their affections, you need to test your app on multiple shops with the same email address to grant that merchant the functionality they need, error-free.

You might also like: How to Make Your Embedded Apps Load Quickly and Reliably.

Testing your billing

If you’re building a public app you want to submit to the Shopify App Store, you are required to use the Billing API unless your app and service is completely free. In implementing the Billing API, you’re also required to test how your app is using it, including running through a test of accepting its charge. Learn more about how to do this in our docs.

Here are some scenarios that you should test:

Upgrade, downgrade, upgrade, downgrade

Merchants’ businesses are sometimes seasonal, meaning there are high traffic points and low traffic points. Merchants may therefore require lower tier plans at some points in the year, and higher tier plans at others. You should test your app to make sure that it’s able to handle fluctuating plan selections.

Billing cancellations

I hate to be the bearer of bad news, but not everyone will accept your billing charge and as a result, they’ll hit cancel. Shopify tells merchants when they have cancelled a charge. That doesn’t mean that your job is over. 

Merchants may revisit the app or try to choose another plan. You have to test your app to see not only whether cancelling a charge produces errors, but also if your app is able to POST another charge for the merchant to accept and for you to activate.

Check the Partner Dashboard after accepting a charge

If you’ve accepted a charge and are redirected back to the app’s user interface, that’s great. But you can’t forget to activate that charge in order to get paid. The simplest way to check activation is by looking through your Partner Dashboard app history.

Uninstall and reinstall

Merchants are business owners, but so are you. A common error I see is an app that forgets to charge merchants for a reinstall after the initial billing cycle is over. You need to get paid, so don’t forget to check this use case, because it is bound to come up in the future.

You might also like: Getting Paid: An Overview of Shopify App Billing Cycles.

General testing guidelines

To sum up everything we’ve covered in this article, below are some general guidelines to keep in mind when testing your Shopify app.

Test in a live environment

This is potentially one of the most important testing instructions and ties everything together. If you have a great app locally that breaks in production, you’ve failed. You need to test like your end user would. This means testing the in-production app on a Shopify store with some products, orders, customers, discounts, and maybe a theme.

Test how your app handles errors

Users are unpredictable. User actions are unpredictable. If you’ve ever closed all your windows by accidentally hitting a button on your keyboard, you’ll know what I mean. This unpredictability needs to be taken into account, and you need to ensure you purposefully create errors to see how your app handles them. 

Try to access a 404, try to plug in an invalid param, and try to access things that other users shouldn’t as a user. It will give you great insights about how to mitigate these possible user errors.

Try to break your own app

All those sleepless nights you spent making sure your code compiles means you know your app inside and out. But you can’t let that stop you from testing critically. Consider asking a friend to try and break your app for you. 

Merchants are not the experts on your app and will be prone to making mistakes. When this happens, they may uninstall the app if they are too confused. Mitigating the potential downfalls of the app as much as possible and trying to break it before someone else does is a great way to future proof your app.

Avoid the “wait and see” mentality

You only get one chance at a first impression with your app. Rather than shipping an unfinished or beta project that might land you a rejection with the app review team, or get you complaints and negative reviews once live, be very meticulous with your testing. The stronger your app is at launch, the higher your install retention rate will be.

"You only get one chance at a first impression with your app."

There’s something really rewarding about growing the businesses of merchants while growing your own business, and we want you to experience that feeling to the fullest. Like with any good business, you have to release a great product. This doesn’t happen by accident. But by combining your creativity, your development skills, and some hard work by user testing your app, we know you’ll be successful.

What does your app testing process look like? Let us know in the comments below!