One of the patterns found in the Enterprise Integration Patterns book covers the idea of a “selective consumer” – that is, a message consumer that uses some criteria to determine which messages it receives and processes, from a given channel (queue).
Years ago, when I was first working with queueing in WebsphereMQ, I used this pattern to ensure my message consumers only got messages they could handle. It worked well. Fast forward to more recent days, though, and when I tried to do the same thing in RabbitMQ, I found it to be painful and fraught with problems. The pattern of selective consumer turns out to be an anti-pattern in RabbitMQ!
That doesn’t mean we can’t be selective about what messages are consumed. Instead of using selection criteria, though, we have to turn to the features and capabilities of RabbitMQ to accomplish the same thing – something that is easier and more natural than you might think.
The Core Problem: Selection
RabbitMQ doesn’t have any kind of selection criteria that you can apply to a consumer and a queue. Once a consumer is attached to a queue, it will potentially receive any message from that queue. If you have multiple consumers on a given queue, you don’t know which one will get the message but there’s a possibility that any of the consumers could get a specific message. You have no way of ensuring a specific message goes to a specific consumer, once that message is in a queue.
To simulate the idea of a selective consumer, you can write code that examines a message once it is received from a queue. If the message cannot be handled by the consumer, the consumer can ‘nack’ the message and send it back to the queue.
This works … to a point. But it introduces some potentially serious performance and thrashing problems in your application.
The Problem, Applied To Airport Baggage Claim
Imagine for a moment, that you’re standing around an airport baggage claim carousel – the kind of carousel where bags ride conveyor belts around and around, waiting for people to grab them.
You stand there, hoping to find your bags quickly – but they never show up. After a few minutes of seeing other people’s luggage go by, you look toward the opening from which the luggage is pouring out. What you see is utterly dumbfounding – but it explains why you haven’t seen your luggage in front of you.
Every time a piece of luggage comes out of the conveyor belt, some one looks at it to see if it belongs to them. If it doesn’t, they don’t let the bag continue to ride around the carousel. Instead… THEY THROW IT BACK! Down in to the depths of the luggage loading system it goes – to be put at the end of the line, waiting to be stuck on the conveyor belt again!
This is MADNESS! Why would anyone toss these bags back to the end of the line, instead of letting them ride the conveyor belt?!
WAIT! Is that your bag, there? YES! … NO! SOMEONE GRABBED IT AND TOSSED IT BACK!!! Now you have to wait for it it come back out and hope no one else checks it before it gets to you again. AAARRRGGGHHH!!!!
Selective Consumer: The Airport Luggage Anti-Pattern
Using code to selectively check a message, determine whether or not the current consumer can handle it and ‘nack’ it back to the end of the queue is exactly like having your luggage thrown back to the end of the line in the baggage claim.
If you have multiple consumers attached to a queue, you are potentially at the whim of every other attached consumer. If one of the other consumers consistently picks up your message and tosses it back to the end of the queue, you’ll never get your message and never be able to process it. The queue will continuously thrash as the message is received, examined and rejected … over and over and over again.
Don’t do this.
Solving The Selection Problem With Routing
RabbitMQ doesn’t allow you to apply selection criteria to a queue, for a consumer. Instead, you have to assume that any message consumer attached to a given queue can handle any message delivered to that queue. To prevent consumers from receiving the wrong message, then, use routing keys to only send messages to a queue with consumers that can handle the message.
If you have a consumer that knows how to handle “enroll student” messages, then you should have a queue for “student.enroll” (or similar). Each consumer that attaches to this queue must be able to handle “enroll student” messages appropriately. If the consumer cannot handle that message, it should not attach to that queue.
Patterns And Practices: Apply With Care
The Enterprise Integration Patterns book is still the source of knowledge for working with messaging systems and message based architectures. But that doesn’t mean you can carte blanche apply these patterns as-is with every messaging system and message broker out there. RabbitMQ, for example, does a great job of including many of the patterns and practices found in this book. That doesn’t mean RabbitMQ applies all of these patterns in the way the authors of the book describe, as you’ve already seen.
To get a better understanding of how the most common and most important of the Enterprise Integration Patterns apply to RabbitMQ, check out my RabbitMQ Patterns email course / ebook. In here, you’ll find a wealth of experience in applying messaging architecture and patterns to RabbitMQ and application design – enough knowledge to help you get up and running, applying these patterns effectively with RabbitMQ.