Webhooks are a great way to add integration with external services and systems. I’ve used webhooks from Stripe, Dropbox and other systems to tell my app when something happens on the external service, making it easy for my system to respond to whatever the event was.
While webhooks are generally easy to handle, there are a few challenges I’ve run into when it comes to handling them in large volume or running a long process in response.
A Basic Webhook Handler
If you’ve every built an Express app with an HTTP “GET” or “POST” route handler, you already know how to build a web hook handler. A webhook is really nothing more than an HTTP GET or POST that sends some information to you. It works the same as any other HTTP request, in this manner.
The typical response from the webhook is going to be a “200 OK” status message, as well. Sometimes you’ll send data back, but not often.
So, your typical webhook handler can look as simple as this:
There’s nothing terribly special, here. It’s just an Express router that handles a POST and responds. It doesn’t do anything, but it does technically handle the webhook.
The real challenge, though, is handling the webbhook quickly.
You Must Respond Quickly
Speed is the name of the game.
Many of the most popular services require your HTTP request handler for the webhook to respond with a “200 OK” within a few seconds.
Dropbox, for example, requires you to respond in 10 seconds:
Your app only has ten seconds to respond to webhook requests.
This sounds easy at first – and usually is. When you only have a few webhooks coming in, and you don’t have much code in place, responding in 10 seconds or less is not a challenge.
When you start growing, though, and you start adding more business process to the back-end systems; or when you realize that your back-end system needs to access external resources and services that may be down, or you have a long-running business process for the web hook data – this is when things get “fun” with that 10 second limit.
What’s the solution, then? How do you build an Express app that will always respond quickly, no matter the number of webhooks coming in, and no matter the length and complexity of the process being run?
Don’t Process The Webhook Immediately
Typically, an HTTP API needs to send a response. If you have an API to get users, or get a user by id, for example, the software making the request will expect a list of users or a single user to be returned through the HTTP stream.
With a webhook, however, the only response required is usually HTTP status “200 OK” – just to say, “hey, I got the message. Thanks!” Thankfully, this need for a simple response gives you plenty of options for handling a large volume of webhooks, and having long running processes.
Whatever tools you choose, the primary mechanism for handling volume and lengthy processes will be sending the webhook data to another back-end service somewhere.
Publish A Message, Respond Quickly
RabbitMQ is my current choice for message queueing systems. With Express and RabbitMQ, you can write very simple code for a webhook handler.
This example (using a well encapsulated call to publish a message to RabbitMQ) will create a JSON document and publish it The web server then responds with “200 OK”, and the webhook publisher knows that the message was received.
Then, on the back-end of the system, another process can pick up the message from RabbitMQ and run whatever code it needs.
Handle The Message
Once the message is in a queue, your options for handling it in the background are nearly endless. You don’t even have to stick with Node, at this point – any language with a RabbitMQ library could pick up the message and run your back-end code.
Whatever language you choose, and whatever code you need to run, you can handle it with all the time you need. There’s no need to performance optimize everything down to the nth degree, immediately, because you don’t have to worry about that 10 second response window.
Sure, you may have to deal with your own system performance needs. That, at least, will be in your control and not a mandate from the external service that provided the webhook data.
Improve Architecture and Performance
One of the best side-effects of handling webhooks in this manner, is the architecture and performance improvement you can gain.
Architecture improves when we split out monolithic applications into a system of smaller parts. RabbitMQ makes this easy, allowing us to push messages between applications without having the various parts know about each other.
Improvements in architecture and splitting apart a monolith also provide performance enhancements. Users and external systems will receive responses faster. Additionally, you can scale up and out individual parts of the system, as needed.
These, and many other benefits, go hand in hand with messaging architectures.