How to Use Relative Pagination In Your Application

Loading all the records for a particular model from Shopify is one of the most time consuming operations an application can perform, whether they are orders, products, variants, or something else. This is especially true for large shops that have millions of records.

To help address this problem, we’ve scrapped the page parameter and added support for relative cursor-based pagination via link header for some select endpoints in the 2019-07 REST Admin API version. In this article, we look at what this change means for you as a Shopify app developer, explain how to employ two different forms of relative cursor pagination in your application, and go over how to use the new bulk operations API to fetch all records.

Relative cursors vs the page parameter

Before we dive into relative pagination, let’s look at what came before. The classic way to load a shop’s records from Shopify is by using a page parameter, where starting from page one, each subsequent request has a page parameter one higher than the previous. The problem with this approach is that the larger the page number, the larger the offset in the query. The larger the offset, the slower the request is, up until the point that it times out. 

There is an alternative the provides better performance no matter how far into the records you go: relative cursors. This basically means remembering the point where the previous page ended and the next page begins. Think of it like using a bookmark: instead of flipping through pages in a book to find the place you last stopped reading, you can jump right to the correct location. More information about the implementation and performance differences between the two can be found on the Shopify Engineering Blog.

Introducing relative cursor-based pagination to Shopify

In the 2019-07 API version, we removed the page parameter and added support for relative cursor-based pagination via link header on select endpoints. The remaining endpoints will be updated in the 2019-10 REST Admin API version. Older API versions that support page will continue to work for nine months from the day of the version release that replaces them. The endpoints updated in the 2019-07 version will continue to work with page by using an older API version until April 2020, and those updated in the 2019-10 version will continue to work with page in an older API version until July 2020.

You might also like: Introducing API Versioning at Shopify.

Types of relative pagination

Inside of Shopify, we support two types of relative pagination: using a since_id and following URLs from the Link header in the response. In the following sections we’ll give examples of how to use each one and explain the tradeoffs of both.

Pagination with since_id

This is the simplest form of relative cursor pagination. Each request will include a since_id parameter with the id of the last record from the previous page. This does require the records to be sorted by id ascending, but this happens automatically when a since_id parameter is present. To ensure the first page is sorted by id, you can include a since_id parameter with a value of 0. This form of pagination already existed on all endpoints prior to the 2019-07 API version.

For example, if you’re requesting pages of products from your shop, the request for the first page might look like the following:

https://shop-domain.myshopify.com/admin/products.json?limit=5&fields=id,title&since_id=0

This will come back with the following response:

Taking the id of the last record, the request to get the next page would be:

https://shop-domain.myshopify.com/admin/products.json?limit=5&fields=id,title&since_id=45678

Using a since_id is the fastest solution and should be the first choice for pagination when order doesn’t matter.

Pagination with a link header

What about when order does matter? Maybe you want to load the first 1,000 products with the most inventory, but with the 250 upper bound on limit, this would mean making at least four requests. You also wouldn’t want to use since_id, because you’d have to fetch all of the products before determining the 1,000 with the highest inventory. Depending on the size of the store, this could mean loading a lot more records from Shopify. Furthermore, the inventory total is not exposed as a single field on products, so it would also mean adding up the inventory quantity of the individual variants and sorting on that.

There is an easier way: using the Link header. In the newer API versions where the page parameter is removed, every request that has multiple pages will include a response header called Link. This header contains the URLs for the previous and next pages, if they exist, respecting the sort order asked for. All of the data needed to load the next page is embedded in the URL inside of a page_info parameter. 

However, there is a downside: our tests showed that using the URLs from the link header is 10 to 30 percent slower than using since_id. But still, they are both faster than using a page parameter, particularly when there are thousands of records or more.

Using the above example of loading products sorted by total inventory quantity descending, the first request would look like:

https://shop-domain.myshopify.com/admin/api/2019-07/products.json?order=inventory_total+desc&limit=250

The response will contain a Link header that would look like this if there are more pages:

<https://shop-domain.myshopify.io/admin/api/2019-07/products.json?limit=250&page_info=eyJsYXN0X2lkIjoxOTg1ODQxNzk0LCJsYXN0X3ZhbHVlIjoiQWNxdWlyZWQgQ2FudmFzIFBlbiIsImRpcmVjdGlvbiI6Im5leHQifQ%3D%3D>; rel="next"

And the URL for the next page would look like this: 

https://shop-domain.myshopify.io/admin/api/2019-07/products.json?limit=250&page_info=eyJsYXN0X2lkIjoxOTg1ODQxNzk0LCJsYXN0X3ZhbHVlIjoiQWNxdWlyZWQgQ2FudmFzIFBlbiIsImRpcmVjdGlvbiI6Im5leHQifQ%3D%3D

This URL can be parsed out of the header and, after following it, the subsequent request will have another header in the response that could look like the following:

 <https://shop-domain.myshopify.com/admin/api/2019-07/products.json?limit=50&page_info=eyJkaXJlY3Rpb24iOiJwcmV2IiwibGFzdF9pZCI6MTk4NTgyMTYzNCwibGFzdF92YWx1ZSI6IkFjcm9saXRoaWMgQWx1bWludW0gUGVuY2lsIn0%3D>; rel="previous", <https://shop-domain.myshopify.com/admin/api/2019-07/products.json?limit=50&page_info=eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6MTk4NTgzNjU0NiwibGFzdF92YWx1ZSI6IkFoaXN0b3JpY2FsIFZpbnlsIEtleWJvYXJkIn0%3D>; rel="next

Since this request wasn’t for the first page, this time there is a next and a previous page URL in the header. Parsing out the next page URL and following it can be repeated until the desired number of records have been retrieved, or until there is no next page URL in the header, at which point all the matching records have been retrieved.

Note that after the first request, the limit parameter is the only parameter in the URL besides page_info. This is because it is safe to modify the limit in between requests. Parameters that could break the request if they were to change, such as sort order and filters, are embedded directly inside of page_info. Trying to add or modify filters or order parameters on URLs from the link header will cause the request to fail. If a different sort order or filtering is needed, then you must restart on the first page. It’s also important to note that the link header URLs are temporary and should not be saved for later use.

You might also like: The Shopify GraphQL Learning Kit.

Migrating your app to use relative pagination

Based on our API release schedule, support will be removed in April 2020 for the 2019-04 API version, which is the last version that will support the page parameter on those endpoints updated in the 2019-07 version. Similarly, support will be removed in July 2020 for the 2019-07 API version, at which point no endpoints will support the page parameter. It is important to have all applications migrated over to using relative pagination by then, or they will stop working past the first page.

To make it easier to update, the Shopify API gem now supports relative pagination. Using it to get the first 1,000 products sorted by inventory total descending is fairly simple:

For applications not written in Ruby, the change that added support to the Shopify API gem can be used as a blueprint. It’s available on GitHub.

There’s a big benefit to making this switch: speed! Both methods of relative pagination are faster than using a page parameter, particularly with large page numbers. In our experiments on a large test shop, loading the 100,000th product with a relative cursor was hundreds of times faster than using a page parameter.

One common case for using page right now is to make concurrent requests on a series of page numbers all at once. With relative cursors this will no longer be possible, as it will require getting one page to know where the next page starts. Relative cursor-based pagination is faster for getting all the records on large shops, and this increase in speed will make up for the lack of concurrent requests on these shops.

Using bulk operations in the Admin GraphQL API

At Shopify Unite 2019, we introduced bulk operations in GraphQL, which allow apps to fetch large amounts of data in a completely new way. Instead of making a series of requests for successive pages of records, the app makes a single request to kick off an operation which will run asynchronously. The app periodically polls the status of the operation and, once it’s complete, will get a link to download a file containing the full set of results.

Bulk operations use relative cursors internally, and so has the same performance benefits as using relative cursor-based pagination. It is a good choice to replace concurrent requests using a page parameter.

Bulk operations are now available for our Admin GraphQL API as of 2019-10. Examples of how they can be used can be found in our docs. 

Migrate your app for better functionality 

All apps will eventually have to move away from pagination using the page parameter, but between the increased speed of using relative cursors and the simplicity of using the bulk operations API, these changes will help your app have a faster overall performance, particularly when dealing with shops with large amounts of data. If you have questions about pagination or the new bulk operations API, head over to our forums.

Did you miss the announcements from Shopify Unite? Check out our new APIs or watch the track sessions on YouTube.