A question was asked on StackOverflow about best practices for RabbitMQ exchanges, queues and bindings. While this question was technically “off topic” for StackOverflow, I answered it anyways because it’s a common set of questions and offers insight in to a few points of confusion when starting out with RabbitMQ.
One Exchange, Or Many?
The core parts of the question include:
I’am looking for best practices for the design of the system regarding topics/queues etc. One option would be to create a message queue for every single event which can occur in our system, for example:
I think it is not the right approach to create hundreds of message queues, isn’t it?
Isn’t it better to just have something like “user-service-notifications” as one queue and then send all notifications to that queue? I would still like to register listeners just to a subset of all events, so how to solve that?
My second questions: If I want to listen on a queue which was not created before, I will get an exception in RabbitMQ. I know I can “declare” a queue with the AmqpAdmin, but should I do this for every queue of my hundreds in every single microservice, as it always can happen that the queue was not created so far?
It’s a difficult set of questions to answer, but also a common set. There are a lot of options and possibilities for which type of exchange is used, and how the bindings are configured for routing messages.
One Or Many Exchanges?
I generally find it is best to have exchanges grouped by object type / exchange type combinations. In the example of user events, you could do a number of different things depending on what your system needs.
Exchange Per Event
In one scenario, it might make sense to have an exchange per event as you’ve listed. you could create the following exchanges
| exchange | type |
| user.deleted | fanout |
| user.created | fanout |
| user.updated | fanout |
This would fit the pub/sub pattern of broadcasting events to any listeners, with no concern for what is listening.
With this setup, any queue that you bind to any of these exchanges will receive all messages that are published to the exchange. this is great for pub/sub and some other scenarios, but it might not be what you want all the time since you won’t be able to filter messages for specific consumers without creating a new exchange, queue and binding.
Exchange Per Object Type
In another scenario, you might find that there are too many exchanges being created because there are too many events. you may also want to combine the exchange for user events and user commands. this could be done with a direct or topic exchange:
| exchange | type |
| user | topic |
With a setup like this, you can use routing keys to publish specific messages to specific queues. For example, you could publish user.event.created as a routing key and have it route with a specific queue for a specific consumer.
| exchange | type | routing key | queue |
| user | topic | user.event.created | user-created-queue |
| user | topic | user.event.updated | user-updated-queue |
| user | topic | user.event.deleted | user-deleted-queue |
| user | topic | user.cmd.create | user-create-queue |
With this scenario, you end up with a single exchange and routing keys are used to distribute the message to the appropriate queue. notice that i also included a “create command” routing key and queue here. this illustrates how you could combine patterns through.
Registering Messages For Specific Consumers
Later in the question set, this person asks:
I would still like to register listeners just to a subset of all events, so how to solve that?
If you need specific messages to go to specific consumers, you do this through the routing and bindings. Trying to filter once the message is in the queue, is an anti-pattern in RabbitMQ.
This leaves you with options for how you would set up the pre-filtering of messages, using routing.
By using a fanout exchange, you would create queues and bindings for the specific events you want to listen to. each consumer would create it’s own queue and binding.
By using a topic exchange, you could set up routing keys to send specific messages to the queue you want, including all events with a binding like
user.events.#. This binding uses a wildcard to send all events to the queue for this binding.
There are still other options and configurations, of course. Without knowing the specifics of a given scenario, it is hard to give a solid answer.
When To Declare Queues And Bindings
The last question in this set deals with when to create your queues.
If I want to listen on a queue which was not created before, I will get an exception in RabbitMQ. I know I can “declare” a queue with the AmqpAdmin, but should I do this for every queue of my hundreds in every single microservice, as it always can happen that the queue was not created so far?
There are two ways to do this:
- Pre-define the exchanges, queues and bindings
- Define them at runtime
In my experience, pre-defining exchanges queues and bindings becomes difficult. RabbitMQ and the AMQP specification allow and often require configuration to be done through the communication protocol itself.
You don’t need a third party application to configuration the layout of your RabbitMQ installation. You can use the Web Administration plugin for this if you want. But, in my experience, this leads to problems with large applications. You will run in to a scenario where the queue name and binding cannot be pre-determined.
There is some use for this, though. There will be times when you need an exchange to always be around or you want to do some quick testing to make sure things are set up correctly. The Web Admin plugin is very useful, here.
Dynamically Defined Exchanges
Because of the way AMQP works, and the need to define queues and bindings dynamically, I find it best to have each message consumer declare the queues and bindings it needs before trying to attach to it. This can be done when the application instance starts up, or you can wait until the queue is needed. Again, this depends on what your application needs.
I tend to prefer to wait until the application needs it… but there are some cases where the queue and binding should be there before the code runs, to ensure all possible messages are caught. In that scenario, having the queues pre-defined at application startup can help.
Learn Through Others’ Experiences
I know the answers I’m providing are rather vague and full of options, rather than real answers. Ultimately, there is no right or wrong answer for which exchange type and configuration to use without knowing the specifics of each system’s needs.
Truthfully, you could use any exchange type for just about any purpose. There are tradeoffs with each one, and that’s why each application will need to be examined closely to understand which one is correct.
If you’re interested in learning more about the tradeoffs and how to make decisions around these questions, I’ve written a small eBook that covers these topics. This book takes a rather unique perspective of telling stories to addresses many of these questions, though sometimes indirectly.
Step inside the mind of another developers and learn how to make decisions about RabbitMQ Structure and Layout.