How the Developer of Shopify App Mechanic is Scaling with EventBridge

If you develop apps for Shopify, you’re probably well aware that scaling your app to accept webhook traffic directly from Shopify can be challenging, especially if your users conduct flash sales. To manage these traffic loads, you might have noticed that auto-scaling doesn’t always happen fast enough to keep up, and larger servers get expensive quickly.

This year, we introduced a better solution: an integration with Amazon EventBridge, which allows you to receive Shopify webhooks using an event bus. In theory, EventBridge should enable you to reduce infrastructure costs while building a scalable system.

But what does that look like in practice?

We sat down with Isaac Bowen, creator of Shopify apps Mechanic and Locksmith, to see how EventBridge has impacted his experience building with Shopify.

The challenge of webhooks

Webhooks are a tool that allows you to execute code after an event happens on a shop, by sending you an alert after an event happens on a shop. Webhooks allow you to connect events on a user’s shop to specific actions your app takes in response.

Webhooks work on a subscription basis—you subscribe to a certain category of events, and register an HTTP endpoint to receive webhooks related to that event. When that specific event occurs on a user’s shop, you receive a webhook with a JSON payload that contains a copy of the relevant object.

Problems can arise, however, when large volumes of events occur and overwhelm your app infrastructure. This is particularly common when large merchants start using your app, or if any of your users host flash sales. That firehose of incoming data means your app infrastructure might not be able to keep up with the onslaught of traffic, which can result in a broken app experience. To combat this, many app developers typically run powerful servers to ensure they can handle sudden bursts in traffic, or build complex queueing systems.

The implicit assumption with using webhooks successfully is that your infrastructure can keep up with Shopify’s. Choosing to use webhooks means your app has to keep up with the potential load of Shopify activity—a difficult task, especially for newer apps.

This is where Amazon EventBridge comes in. EventBridge is a serverless, event-driven alternative to receiving webhooks via standard HTTP. Integrating with EventBridge allows you to send event data to AWS directly, instead of needing to handle all that traffic yourself. 

Event-driven architecture like EventBridge puts an infinitely-large buffer between you and the massive scale of Shopify. It allows you to process events at the rate that works for your tech stack without getting overwhelmed. It’s highly scalable and resilient, and allows you to accept more webhook traffic while reducing your infrastructure cost and complexity.

"Event-driven architecture like EventBridge puts an infinitely-large buffer between you and the massive scale of Shopify."

Losing sleep over webhooks

Isaac Bowen has been working with the Shopify platform in one way or another for a decade.

“Just over 10 years ago, I wrote the first lines for an app called Gatekeeper. That was 2010—Gatekeeper was one of the first 100 listings on the Shopify App Store,” Isaac says.

Since then, Isaac’s involvement with Shopify has only grown. Lightward, the company he founded, now has two apps in the Shopify App Store: Gatekeeper’s successor Locksmith, and Mechanic. 

“I’ve always loved understanding how things work, and I’ve always delighted in seeing things become whatever they want to become,” Isaac explains. “That’s an underlying source of inspiration in everything that I make and do. Mechanic is a good example of this. The core concept for it is four years old, and I released the app two years ago. The entire process has been one of listening to what this thing really wants to be, and how it wants to work. Today, Mechanic is a tool for improving the developer-merchant relationship, by giving developers the ability to rapidly solve specific automation problems in a way that lets them leverage their skills with code, while making it easy to hand off a final product that’s easily merchant-configurable without code. We also have nearly 300 pre-built ‘tasks’ that merchants can pull off the shelf.”

When Isaac was brainstorming ideas to come up with Mechanic, the only path available to automate workflows in Shopify was to build an app—a big undertaking even for seasoned developers, let alone non-technical merchants who were looking to make their processes a little easier.

Isaac realized that anyone who worked with Shopify had likely come into contact with Liquid, Shopify’s templating language. And so, for maximum flexibility, Mechanic brings Liquid to event processes, allowing users to treat incoming events as Liquid variables and use Liquid templates to render the definitions of actions to perform.

The benefit of doing so means that developers can build whatever automations they can imagine, as long as they have an understanding of Liquid. It also allows merchants—most of whom have never interacted with a programming language—to harness the power of automation. And for those merchants who are ready to explore Liquid, the code behind their automation is a click away.

Mechanic works by automatically performing work based on specific events in a user’s store. 

“While there are a few event domains in Mechanic, the busiest one is Shopify,” Isaac explains. “Any webhook in Shopify can become a Mechanic event. On a normal day, we process around 500 Shopify webhooks per minute.”

While the webhook approach worked for Mechanic’s purposes, increasing scale left its creator with a problem.

“Mechanic use is spreading at a steady clip. The more large stores use it, the more opportunities there are for random, massive spikes of events,” Isaac says. “Traditional webhooks made it annoyingly hard to solve that problem well.”

With traditional webhooks, downtime is always a possibility.

“Webhooks are set-it-and-forget-it, until your web servers are suddenly saturated with a random 10x spike in webhook traffic. They weren’t impacting my day-to-day, but they were absolutely affecting how soundly I was sleeping at night.”

You might also like: How App Challenge Winner Tilo Mitra is Building for Success on the Shopify App Store.

Introducing EventBridge 

It turned out that EventBridge was just the solution Isaac was looking for.

“My initial impression of EventBridge was, oh thank God someone finally solved this properly,” Isaac says.

My initial impression of EventBridge was, oh thank God someone finally solved this properly.

As an added bonus, introducing EventBridge was a smooth process.

“Looking back at the original commit in which I introduced EventBridge, we had 10 application files changed, with 147 insertions and 54 deletions. It was a hilariously trivial update. And it was more code than it could have been, because we left ourselves the ability to switch stores from traditional webhooks to EventBridge one at a time.”

In addition to being easy to implement, EventBridge also saved Isaac time and resources on work he had been planning to undertake to help ensure his app could handle the traffic it was receiving.

“I was about to split our web infrastructure into two pieces: one for normal app traffic, and one literally just for Shopify webhooks. EventBridge meant that I didn’t have to. I was living with the knowledge that any app downtime meant triggering Shopify webhook retries, at best, or dropping Shopify webhooks, at worst. Delays are bad, lost data is really bad. EventBridge, piped into SQS, meant that our guarantees around event ingress were suddenly backed by the reliability of AWS itself.”

Now that Mechanic is integrated with EventBridge, Isaac’s energy can be spent growing the app, instead of worrying about how it will continue to handle growing traffic.

“EventBridge means that we can consume any scale of events without a second thought. And it means that the time it takes to boot new capacity is no longer time that web requests are waiting, or being dropped: all those requests are now just events waiting in SQS. Our SQS pollers run more efficiently than our web servers, and since the vast majority of our web traffic was webhooks, our overall operation is tighter now that EventBridge is in the mix.”

In the end, it simply means a more sound foundation for Isaac’s app to grow.

“Webhooks were already reliable,” he says. “Using EventBridge has allowed us to match that reliability. Also, I’m sleeping much better now.”

Webhooks were already reliable. Using EventBridge has allowed us to match that reliability. Also, I’m sleeping much better now.

You might also like: From a Leap of Faith to Building New Futures: The Chatdesk Vision of Creating New Jobs.

Looking forward with Mechanic

For Isaac, tools like EventBridge are enabling him to build his vision of the company he wants to see in the world.

“Lightward’s entire point is to be a running experiment, asking if we can really run a viable operation where our main heuristic for decision-making is just, ‘Does this feel like a toward-the-light kind of decision or not?’ Can we run a business like this? It seems that we can. :)”

That philosophy extends into his approach to app development too. Isaac shares the following advice with other developers considering whether to launch or expand businesses:

“Build the thing that wants to exist. Not because it’s profitable, or easy (though it may be those things too). Build the thing that belongs here in the world. Get connected with people and platforms that are very obviously doing that too. That’s why I started doing Shopify work in the first place. There aren’t many companies that are paying attention the way that Shopify is, and consequently building things the way they are. The introduction of an EventBridge integration is just one in a long, long chain of decisions that reflect this kind of process, affirming the decision that I’ve made every day to stick around. Make choices like that, in your code and in your business and in everything else, and that’s pretty much the whole deal.”

Build the thing that wants to exist. Not because it’s profitable, or easy.

And a more tactical piece of advice:

“Also, set up age monitoring on your SQS queues. That’s important too.”

Using EventBridge to scale

With EventBridge’s server-less, event-driven architecture, Isaac has been able to benefit from a foundation that allows for easier scaling with reduced infrastructure costs—and the peace of mind that comes from not needing to worry about whether his app would go down.

As an app developer, integrating with EventBridge can help you grow your app and business in a more scalable and cost-efficient way so you can continue to provide an excellent experience to your users—no matter how much traffic they drive.