An Insider's Look at the Technology That Powers Shopify

[Editor’s note: This article is based on a talk that Chief Technology Officer Jean-Michel Lemieux gave at Shopify Unite 2019.]

Two weeks ago on the Shopify Unite mainstage, we announced launches and features that will continue to bring commerce to everyone, everywhere. These launches are exciting, and will further serve to democratize commerce. But there’s one aspect of these launches we didn’t discuss, the quality that makes it all possible: the technology inside our retail operating system, Shopify’s technology platform.

With the exception of the Shopify Engineering Blog, we don’t often go behind the scenes of this technology platform, or look at the philosophy that guides our decision making. At Shopify Unite, our Chief Technology Officer Jean-Michel Lemieux gave a talk that dove into the technology in our platform, how it’s built, and what we’re building next. In this post, we take a look at what he shared. 

If you’d like to watch the talk, check out the video below.

Shopify’s technology platform

Before we get into the specifics of our platform, let’s look at what came before. Thousands of years ago, the Silk Road connected Europe, Asia, and North Africa with massive trade networks that carried goods, ideas, and people across borders. This trade network flourished for hundreds of years.

When we look at Shopify, we see countless parallels with the Silk Road. Building and maintaining the Silk Road involved technological challenges that we still face: advanced infrastructure, civil engineering, and physical security innovations all served to protect travelers on the Silk Road, as they serve to protect our merchants today.

The lessons learned by studying the history of the Silk Road apply to how we build and maintain Shopify’s infrastructure. Like the real Silk Road, our platform is composed of different key technologies. When combined, they give you a solid foundation on which to build your business. These are the building blocks of our retail operating system. Let’s take a closer look at each one.

You might also like: Here’s Everything We Announced at Shopify Unite 2019.

1. An open internet

We’ll start our journey by talking about the open internet and the role Shopify is playing in keeping it that way. 

Commerce needs roads that are accessible so that products and knowledge can continue to flow through borders. In some parts of the world, and on some parts of the internet, these roads are in danger as people, states, and companies try to put up walls and borders. But Shopify’s digital highway has been built on the promise of the open internet.

The internet was built on open standards like DNS, BGP, TCP/IP, and HTTP. These need to be protected and enhanced, because there is a real danger of reverting to the dark ages if we allow the creation of closed systems.

That’s why we’re leading the fight on open standards more than any other company of our size. Let’s quickly talk about a few of our open internet initiatives.

Payments

When the internet was first being built, commerce wasn’t considered a building block. The first browsers had a back button, a forward button, but no pay button.

We joined the W3C’s Web Payment Working Group as a founding member in 2016, with the goal of playing an active role in standardizing payment actions across the web, and testing their impact on checkout conversion.This new payments specification will allow customers to pay within any browser using the payment methods that are secured on your device. 

The 1.0 release of the Payment Request API is launching very soon, and is available in all major browsers. When you see that pay button in your browser, you’ll know that Shopify helped make that happen.

3D models

We all know that 3D models will eventually replace images on the internet, but to fully embrace them, we need a file format that’s optimized for speed. We also need to figure out how to efficiently store all the associated variant information. With regular product images, it’s cheap to have one image per variant, but that won’t work with large 3D models that have a ton of geometric data, scenes, materials, etc.

That’s why we’re working with the Khronos Groups, a prominent open standards body, to advance and deploy 3D content for commerce on the web. And you can see this already—right now, five of the six shops highlighted by Apple as examples of stores using excellent 3D models are running on Shopify.

These are just two examples of our investments in an open internet, and we will continue to invest and advocate for our merchants and partners to keep the internet open.

2. Security and privacy

Next, let’s look at what we’re doing around security and privacy. Just as there were bandits on the Silk Road, there are threats to merchants and buyers today that need to be considered. For any open commerce network to succeed, merchants and partners need to trust it. The integrity of a merchant’s relationship with their customers and their data need to be protected throughout the entire commerce journey, and it’s our role to provide that type of protection. 

At Shopify, a merchant’s data belongs to the merchant. It’s their state secret. It’s their business advantage. Merchants hold explicit control over their data and who has access to it. 

Merchants also play a big part in their own security. We have the tools and support in the platform to keep merchants safe, but they have to use them. The security of a merchant’s store is the sum of all who influence it, including their staff and the apps they use. Making careful decisions for the safety of their livelihood is an aspect of the merchant’s job.

Here are some of the tools we are working on to help merchants protect their interests.

Identity

One of the biggest threats to a merchant’s data is unauthorized access to credentials: compromised password or phishing attacks. The average person receives 16 emails a month that have the malicious intent of stealing credentials. Often a password is all that lies between a bad user and your business’ bank accounts.

This month, we launched a new identity vault as a key part of further securing our platform. Merchants and partners will now have one account, one password, and one set of security settings that apply by default to all their shops and properties.

The identity vault also includes support for the new Web Authentication API created by the W3C, which allows our servers to integrate with strong authenticators now built into devices, like Windows Hello or Apple’s Touch ID.

The identity vault is rolling out to all shops now. If you have multiple logins, you’ll be brought through a simple step to merge your accounts. This is also a great time to turn on two-factor authentication to make your account even more secure.

Shopify Partners and security

Partners also have a big role in security and privacy.

Last year, we created additional support around data ownership and privacy by developing different webhooks and requests for data deletion. We created greater transparency, especially around business information and auth scopes, so merchants know what they’re opting into and what data goes to what apps. We now rank and monitor apps to ensure they implement and respect the data they collect on behalf of the merchant.

Taking security seriously allows us to keep merchant data safe.

Security hackers

Every day, we process around 10 billion events and safeguard over 10 PB of data. We’re able to handle all of this because we understand the importance of creating products built on a foundation on trust.

We don’t expect you to just take our word about security. We’ve also invested in making Shopify one of the most popular targets for the community of security hackers around the world. We have a network of nearly 3,000 security researchers participating in our programs worldwide, and we’ve rewarded more than $1 million in bounties.

But there’s still more to do. Moving forward, you’ll see an increasing investment in metafields on our platform. We want to make it easier for apps to avoid storing data or personally identifiable information at all. Stay tuned over this year for more news.

These are just some of the ways that we’re improving and innovating the security of our platform.

You might also like: Every API We Announced at Shopify Unite 2019.

3. Extensibility

One of the successes of the Silk Road was the variety of goods that traveled it—a huge array of products were available to buyers along the road. In a similar way, every store on Shopify sells different products, with each merchant offering something unique.

When you build on Shopify, we give you the tools that make these different experiences possible. The key to this creativity are the APIs and extensions available on Shopify.

Five years ago, the API that powered Shopify products wasn’t the same as what powered the apps in our ecosystem. We had an API for Shopify, and a separate API for apps. Those days are over. Our products are now powered with the exact same GraphQL API you build your apps on. There also used to be a lag between when we’d ship features, and the APIs that powered them. Now, APIs ship with our products. In fact, the new POS is built with apps as a core feature.

At Shopify Unite, we gave an overview of all the new features in our developer platform. The takeaway is clear: we’re investing heavily in APIs so you can help make Shopify the most creative platform on which to build new commerce experiences.

4. Global infrastructure

Global infrastructure was a key component of the Silk Road. Its scale empowered vast regions to participate in trade and was one of the main reasons it thrived for 2,000 years. It’s one one of the reasons we value our global infrastructure so highly.

If we were to pop open the hood of Shopify, you would see infrastructure that was intentionally designed with global scale and internationalization in mind. When a business opens on Shopify, they plug into a global superhighway of cross-border selling. Data from the World Bank shows important growth in trade happening globally, and we’re seeing the same global trend on our platform. Cross border sales for Shopify merchants outside of our primary English speaking markets has grown 500 percent in the last three years. And to keep the momentum going, we’re continuing to invest in our global scale.

This year, we scaled from two to 180 Shopify points of presence (POPs) around the world, connecting shops to buyers. These are the blue dots in the above image. POPs have decreased latency for buyers and merchants by almost 50 percent for regions outside of North American. We also have two compute regions in North America, and we’ll be launching four new regions in the next 18 months. By bringing servers closer to buyers, we’re creating a faster consumer experience.

Handling scale

As we go global, our scale changes. To look closer at an example of this, let’s zoom in to Canada on October 17, 2018. On this day, Canada ended 95 years of cannabis prohibition. Shopify worked with governments and licensed sellers across the country to provide a safe, reliable, and scalable commerce platform for this new industry.

So what happened on launch day?

When the doors opened at midnight, traffic increased in the Canada region, reaching an obvious peak. But when you zoom out, you can see the impact this had on the Shopify platform as a whole.

This is traffic across all of Shopify on that same day—you’ll notice that the scale moved from thousands to millions. Traffic from Canada can barely be seen. That’s the scale of the commerce happening on our platform. 

But that’s just one example. Let’s take a look at another.

When most merchants are just starting out, they average one order per day. When merchants start to gain traction, they average anywhere between five and 20 orders per day. On the day that cannabis legalization launched in Canada, cannabis sales averaged around 100 orders a minute. That’s a lot—but let’s compare it to some of our flash sale merchants.



The image above shows 8,000 orders per minute for some of our flash sale merchants. To really understand the scale of this, that’s equal to what the entire platform averaged on Black Friday/Cyber Monday 2018.

Take a second for this to sink in. This chart shows that a brand new business, a business on its way to IPO, and a government ending a nationwide prohibition all have access to the same technology and the same power of our platform.

Handling speed

We know that it’s not just scale that matters—speed is important, too. We’ve been working on keeping Shopify fast for years, but there’s always more to do, more performance wins to squeeze out of the platform, and big step changes to invest in.

In addition to the important latency impact of our new global points of presence, we’ve recently had two other big performance wins. 

Image delivery service

The first is an upgrade to our image delivery service. WebP will soon roll out to all merchants, resulting in images that are 30 percent smaller, meaning faster load times for online stores.

New Liquid renderer

We also have an entirely new online store renderer in the labs, running on a few shops. This is a rebuild of the Liquid engine that takes your Liquid templates and generates your online stores.

Our themes are optimized for speed out of the box, but as they’ve grown and gotten more complex, the rendering engine has been in need of a performance boost. This new version is seven times faster than the current renderer. Every merchant will see impact to page load speed as it rolls out later this year.

Performance dashboards

As Shopify Partners, you play a critical role keeping shops fast.

We’ve provided a lot of flexibility and power for you to customize the merchant and buyer experience. With great power, comes great responsibility. Soon, we will begin highlighting site speed ranking on your Partner Dashboard, so that you have more visibility and accountability into how you can build to help make the platform fast. We’ve collected all the performance data across the platform and will make it available to you.

This may sound intimidating, but we have to band together to constantly push for a fast web experience for our merchants. We will continue to provide platform improvements that everyone gets for free, but we need your help to double down on performance in the areas that you as partners have control over.

Building for all businesses

That was a lot of information, so let’s recap our main points:

• We’re keeping the internet open with our work with the W3C and Khronos Group
• We’re investing in security and privacy for our merchants and partners, by releasing the identity vault and continuing to invest in a large research community
• We’re doubling down on APIs and the extra layer of opportunity they create in the long term
• We’re investing in scale and speed across our global infrastructure

Together, we are building together for businesses of every size, anywhere.

Want to see more of the track sessions from Shopify Unite 2019? Head over to our YouTube channel.