Introduction
In modern cloud architecture, a message queue(MQ) is a middleware that allows applications and services to send and retrieve messages that do not require an immediate response. The process is called asynchronous messaging. MQs can receive, hold, and deliver messages for application components in a distributed system. These messages are usually small packets of data that some components created for some other components to consume. For example, it could be a request from one component to another to start a task, it could contain the result of a completed task, or just be a plain message. The components that create messages are called producers. The components that receive messages are called consumers. There can be multiple producers and consumers using the queue, but each message will be delivered only once to a single consumer. In the event that no consumer is available to retrieve the message, the message can wait safely in the queue until the receiving component is ready. MQs are typically used to decouple heavyweight processing, to buffer or batch work, and to smooth spiky workloads.
The problem that MQs address is to make it easier to integrate applications and services on diverse platforms by providing a single, robust and secure shared messaging backbone while decoupling the producers and consumers.
By decoupling the producers and consumers, MQs help to build Event-Driven Architecture(EDA) where components will perform specified action based on the occurrence of events while enabling components to continue to function even in the case of connection failures between them.
In this post, I will discuss the benefits and drawbacks of MQs with a focus on cloud MQ services such as AWS Simple-Queue-Service(SQS).
Benefits of Using MQs
Better Performance
MQs enable asynchronous communication, which means the producers and consumers only interact with the queue, not each other. Producers can add requests to the queue without waiting for them to be processed. Consumers process messages only when they are available. No component in the system is ever stalled waiting for another, optimizing data flow.
Reliability
Messages are persistently stored in the queue until they are retrieved by a consumer. This means if one part of the system goes offline, none of the messages for it will be lost. After the system recovers, it can pick up from where it left off and continue to process the messages.
Simplified Decoupling
Decoupling is the process of reducing the degree of dependency between application components. In a fully decoupled architecture, the individual components can remain completely autonomous and unaware of each other. This makes it easier to change different parts of the application because a change in one usually doesn't require a change in the other. Also, decoupled components are easier to scale because they are less complex and are often implemented independently as microservices on the cloud using technologies such as FaaS.
Potential Drawbacks of MQs
Extra Latency
The messaging between the producers/consumers and the queue may introduce latency into the system due to limited network speed. This means if two components can communicate locally(e.g. on the same host), then it might be better to use a local MQ solution such as RabbitMQ or Redis instead for higher system throughput.
One Recipient
MQs use a point-to-point messaging pattern which means messages can only be received by a single consumer. If you are trying to broadcast a message to multiple parties, MQs might not be the best solution for your problem. Instead, you should consider using a publish/subscribe(pub/sub) model for broadcasting needs.
Case Study(Optional)
Let’s say you are building a payment service where there are two basic features: the checkout and the wallet features. The checkout feature creates transactions when users check out their orders. The wallet feature checks and reduces account balances. In order for a transaction to be processed successfully, it must be created by the checkout component and approved by the wallet component. The approval of a transaction may take a long time(e.g. a day). At any time, users can only interact with the checkout component.
This service can be implemented in two ways, either synchronously as a single service or asynchronously as two independent services. Given the fact that approving a transaction may take a long time(e.g. a day), the synchronous approach would not work because it would lead to users waiting hopelessly for transactions to be approved while constraining their abilities to do other things. As such, the asynchronous approach is preferred for a better user experience.
We can now place an MQ between the checkout component and the wallet component. In this example, the checkout component becomes the producer, and the wallet component becomes the consumer. The message is the payment transaction created by the checkout system. Whenever users checkout their order, a payment transaction will be sent to the queue, allowing the wallet component to process them in the background. This allows the checkout component to directly respond to the user without being stalled by the wallet component, telling the user that the payment is pending and asking them to check back later.
By using an MQ, we have implemented an asynchronous workflow for our payment service to provide a better user experience by processing heavyweight tasks(e.g. approve transactions) in the background. Then imagine that the business and workload are growing and the system needs to be scaled up. All that needs to be done is to add more consumers to work off the queues faster.
How to Get Started Using MQs
In order to start using cloud MQs, you must pick a MQ service provider. Most major cloud providers such as Amazon and Microsoft offer managed cloud messaging as a service (MaaS). The primary difference between them is how many extra features are supported. You may choose a provider that meets most of your service requirements. I strongly recommend AWS Simple-Queue-Service(SQS) because it offers a simple HTTP API for sending and receiving messages that meet most application needs. It is also a popular choice because SQS messages can act as the event source for AWS Lambda--a popular FaaS implementation. The best way to learn about AWS SQS is by following this official tutorial. Alternatively, you may consider other popular options such as Redis and CloudAMQP. For a more comprehensive list of available MQ services, please refer to this article for more detail.
Summary
A Message Queue(MQ) is a middleware that allows applications and services to send and retrieve messages that do not require an immediate response. MQs are typically used to decouple heavyweight processing, to buffer or batch work, and to smooth spiky workloads. By enabling asynchronous processing, MQs separate the producers from consumers and enable individual components to evolve independently. Some of the benefits of using MQs are simpler coding for decoupled application components, improved performance, reliability, and scalability. If you want to start using MQs for your next project, I recommend AWS SQS because it provides a simple API and works well with AWS Lambda.
References
Bruns, Ralf, and Dunkel Jürgen. “Event-Driven Architecture: Softwarearchitektur für Ereignisgesteuerte Geschäftsprozesse.” Amazon, Springer, 2010, aws.amazon.com/event-driven-architecture/?trk=ps_a134p000006ByZUAA0.
“Message-Queues.” IBM, www.ibm.com/cloud/learn/message-queues.
Sharma, Mohit. “What Is Message Queue.” Medium, Data-Driven Investor, 19 Oct. 2020, medium.com/datadriveninvestor/what-is-message-queue-b5468ff6db50.
System Design - Asynchronism. massivetechinterview.blogspot.com/2017/04/system-design-asynchronism.html.
“What Is Message Queuing?” CloudAMQP, www.cloudamqp.com/blog/2014-12-03-what-is-message-queuing.html.
“What Is a Message Queue?”Amazon, aws.amazon.com/message-queue/.
“Why Is Application Decoupling a Good Thing.” CloudAMQP, www.cloudamqp.com/blog/2016-10-12-why-is-application-decoupling-a-good-thing.html.
Comments