What an asynchronous queue is
In the synchronous model the sender makes a request and waits for a response: if the receiver is slow or temporarily unavailable, the whole flow hangs. An asynchronous message queue breaks that dependency. The sender places a message into a broker (Kafka, RabbitMQ, SQS) and carries on immediately. The receiver — the queue consumer — picks up and processes messages at its own pace.
Producer -> [queue: view_event, cart_event, purchase_event] -> Consumer
^ buffers the spike ^ processes independently
Why it matters in e-commerce
In e-commerce, traffic often grows 10–20 times during sale periods. With a synchronous integration the personalization platform can become the bottleneck: if it cannot accept events fast enough, they are either lost or they slow the main site down.
A queue solves that in three ways:
- Buffering peaks. Every event during a spike lands in the queue — the site does not wait for it to be processed.
- Reliable delivery. The broker holds the message until the consumer confirms it has been processed.
- Decoupling. If the personalization platform is unavailable for a second, events are not lost; they simply accumulate and are processed once it is back.
Key broker characteristics
| Parameter | Kafka | RabbitMQ | AWS SQS |
|---|---|---|---|
| Throughput | Very high | Medium | High |
| Message order | Within a partition | FIFO within a queue | Standard: not guaranteed; FIFO queue: guaranteed |
| Message retention | Long-term (replay) | Temporary | Up to 14 days |
| Operations | Self-managed | Self-managed | Managed |
Important: when personalization events are processed asynchronously (view, add to cart, purchase) it is critical to preserve the order of events per user. Kafka handles this by partitioning on user_id.
Delivery guarantees
Brokers support different guarantee models:
- At-most-once — the message is delivered no more than once, but it can be lost.
- At-least-once — delivery is guaranteed, but duplicates are possible. Requires idempotent processing on the receiving side.
- Exactly-once — strict single delivery; implemented in Kafka Transactions and SQS FIFO, at the cost of extra overhead.
For events such as a purchase, use at-least-once with deduplication on a unique event identifier.
Common mistakes
- A blocking call with no timeout. If the queue is saturated and the sender waits for room in the buffer, you are back to a synchronous scenario. Set hard timeouts on publish.
- No lag monitoring. Consumer lag — how far the consumer trails the producer — is the key health metric of a queue. Ignoring it means data arrives with a long delay.
- Idempotency in theory only. The processing logic must handle the same event twice correctly — without duplicating purchases or crediting points twice.