Sync, Async, Message Queue, PubSub, Kafka

Async vs Sync

• Synchronous tasks are high-priority tasks that require immediate execution and user feedback. They are generally associated with user actions that need immediate system response.
• Asynchronous tasks can be processed in the background and are not time-sensitive. They don't need immediate user feedback and often involve long-running operations that can be offloaded to background systems.

Message Queues:

• Asynchronous Communication: Message queues enable asynchronous communication between different services, meaning one service doesn't need to wait for another service to complete its task before moving on to its next task.
• Load Balancing: They can also help distribute the load evenly among different services or instances of a service.
• Controlling Throughput: By adjusting the rate at which messages are sent or received, you can control the throughput of the system.
• Decoupling Components: Message queues decouple the services, meaning the services do not need to interact with each other directly.
• Scaling: As the load increases, more queues or services reading from the queues can be added to scale the system.
• Buffering & Throttling: Queues can act as a buffer, holding messages when the processing service is not ready. Throttling can be implemented to control the rate of message processing based on the current load on the system.

Distributed Message Queue vs Non-Distributed Message Queue

	Non-Distributed Message Queue	Distributed Message Queue
Availability	Lower: Since the system isn't distributed, a single point of failure can cause the entire service to be unavailable.	Higher: Distributed queues are designed to avoid single points of failure. If one node fails, the system can still continue to operate.
Message Persistence	Depends on the specific queue technology and its configuration. Some may support persistent messaging, but may not be as robust as distributed systems.	More Robust: Messages in a distributed queue can be replicated across multiple nodes, ensuring that no data is lost in case of a node failure.
Scalability	Limited: The capacity is limited by the resources of the single machine where it operates.	Higher: Since the system is distributed, it can be easily scaled up by adding more nodes to the system.
Throughput	Lower: Being limited to a single machine's resources, the throughput might be limited compared to distributed systems.	Higher: As you can distribute the load across multiple machines, you can achieve much higher throughput.

| Geographical Distribution | Limited: All the data resides on a single machine, which might be located in one geographic location. | Enabled: Nodes can be spread across different geographical locations which can help in reducing latency and enhancing data locality. |

| Reliability | Lower: Since there's only a single machine, if it fails, the service becomes unavailable. | Higher: The distributed nature of these systems allows for built-in redundancy. If one node fails, others can take over its load. | | Resilience | Lower: A single machine's failure can disrupt the whole system. | Higher: Even when individual nodes fail, the system as a whole can continue functioning, making it highly resilient to faults. |

Producer/Consumer vs Publish/Subscribe

Producer/Consumer (One-to-one communication): In this pattern, a producer sends messages to a queue, and a consumer reads from that queue. The key characteristics are:

• The producer and consumer are decoupled.
• The producer adds messages to the queue without knowing about the consumer's state.
• The consumer can consume messages from the queue at its own pace.

Example: Each order on an e-commerce platform (Amazon, for instance) can be seen as a message produced by the Order Management Service. The Delivery Service, which is responsible for processing these orders, acts as the consumer. It takes orders from the queue and processes them for delivery.

Tools: RabbitMQ, Apache Kafka, Amazon SQS.

Publish/Subscribe (One-to-many communication): In this pattern, a publisher sends messages to a topic, and multiple subscribers can receive those messages. The key characteristics are:

• It involves one-to-many communication, where one publisher sends messages to multiple subscribers.
• Subscribers express interest in receiving specific types of messages by subscribing to relevant topics.

Example: When a customer places an order, the Order Management Service publishes a message (order details). Multiple services like Delivery Service (to process delivery) and Receipt Service (to generate a receipt) are interested in this message. They subscribe to this topic and receive the message.

Tools: Google Pub/Sub, Apache Kafka, RabbitMQ, AWS SNS.

These two communication patterns serve different purposes and the choice between them depends on the specific use case. The Producer/Consumer pattern is used when you need to distribute tasks among different workers (like processing orders). The Publish/Subscribe pattern is used when you want to broadcast messages to multiple receivers (like notifying different services about a new order).

Kafka

• Distributed streaming platform for real-time data streaming and processing.
• Ideal for high-throughput, responsive applications, surpassing the capabilities of JMS, RabbitMQ, and AMQP.

Key Kafka Concepts in E-commerce Example

• Producer: Generates and pushes records (messages) into topics. E.g., Order Management Service creates order messages.
• Consumer: Reads data from Kafka topics. E.g., Delivery Service processes order messages.
• Topic: A category for records where multiple consumers can subscribe. E.g., "Orders" topic for order messages.
• Broker: Servers storing and managing data in a Kafka cluster.
• Cluster: A set of brokers, scalable without downtime.
• Partition: Divides topics for organization and scalability. Hosted on different servers.
• Offset: Unique record identifier in a partition.
• Replica: Copies of partitions for fault tolerance.
• Consumer Group: A group of consumers that collaboratively process data.

Message Structure

• Key (optional): Used for partitioning topics.
• Value: Event details (e.g., string/object).
• Timestamp.
• Compression type.
• Headers (optional).
• Partition and offset ID (assigned once written to a topic).

How Consumer Consumes and Tracks

• Kafka consumers maintain their position using "offsets."
• Periodic heartbeat updates to Kafka with the latest offset.
• Kafka does not track whether a message is consumed by all consumers.
• Multiple consumers are organized into consumer groups.
• Each consumer group reads from specific partitions, ensuring each message is delivered to only one consumer.

Replication of Partition

• Kafka replicates each partition across multiple brokers for data reliability and fault tolerance.
• One broker is the "leader," handling data requests, while others are "followers" duplicating the leader's data.
• Replicas are distributed across brokers, ensuring data availability during broker failures.

Zookeeper and Its Evolution in Kafka

• Zookeeper is a coordination service used for maintaining configuration, synchronization, and group services in distributed systems.
• Zookeeper was initially essential for Kafka to manage metadata and cluster status.
• Maintaining Zookeeper added complexity and potential single points of failure.
• Since Kafka 2.8.0, Kafka introduced its internal metadata management system, reducing the dependency on Zookeeper.

Kafka's internal system, known as KRaft mode, simplifies Kafka's architecture, improves performance, and enhances reliability.

• Kafka can now operate independently without Zookeeper, making it more manageable and robust.

Kafka vs RabbitMQ

info

Note that the choice between Kafka and RabbitMQ depends on specific use-case requirements, and each has its strengths and weaknesses.