In the last decade, organisations have become reliant on multiple systems and applications to fulfil their business needs. To work effectively, these systems and applications must be able to communicate with each other in a secure and efficient way. Messaging frameworks have become a critical part of the big data stack for these data-driven organisations, although it is difficult to choose which platform will suit their needs.
There are currently three types of messaging frameworks:
Messaging Queue Frameworks – The traditional message queue paradigm, which is to be used only when there is a fixed end-to-end messaging system to support it.
Distributed Messaging Pub-Sub Frameworks – Publish–subscribe is a sibling of the message queue paradigm. This pattern provides greater network scalability and a more dynamic network topology, with a resulting decreased flexibility to modify the publisher and the structure of the published data.
Distributed Stream Processing Frameworks – Stream processing frameworks are runtime libraries which help developers write code to process streaming data, without dealing with lower level streaming mechanics.
In this blog we give an in-depth overview of these three types of messaging frameworks and a comparison of the specific platforms available in today’s market.
Messaging Queue Frameworks
Active MQ / RabbitMQ / ZeroMQ / RocketMQ
These are earlier traditional message brokers with more emphasis on queuing rather than streaming.
They are built over point to point messaging models.
These are recommended only when there is a fixed end to end communication system.
Distributed Messaging Pub-Sub Frameworks
Apache Kafka
Apache Kafka is more mature and stable distributed and scalable publish-subscribe data streaming platform with simple producer-consumer, distributed broker, message topics, append only logs and distributed partitions modal.
Apache Pulsar
Similarly to Kafka, Apache Pulsar is also an open-source distributed and scalable pub-sub messaging system - originally created at Yahoo and now part of the Apache Software Foundation.
Distributed Stream Processing Frameworks
Apache Samza
Apache Samza is a distributed and scalable real time stream processing framework. Samza allows you to build stateful applications that process data in real-time from multiple sources including Apache Kafka.
Apache Flink
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
Apache Spark
Apache Spark is a unified analytics engine for large-scale data processing. It achieves high performance for batch and streaming data engine, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine.
Apache Storm
Apache Storm is an open source distributed real time computation system. Apache Storm makes it easy to reliably process unbounded streams of data, doing for real time processing what Hadoop did for batch processing.
Distributed Messaging Broker platform (Kafka) is actively evolved in the market as a nervous connection network for any data platforms or any type of data engines.
If you would like to find out how to bring best practice in your Kafka deployment and optimise the performance and scalability of your Kafka clusters, then give us a call on +44 (0)203 475 7980 or email us at Salesforce@coforge.com
We are a global digital services and solutions provider, who leverage emerging technologies and deep domain expertise to deliver real-world business impact for our clients. A focus on very select industries, a detailed understanding of the underlying processes of those industries, and partnerships with leading platforms provide us with a distinct perspective. We lead with our product engineering approach and leverage Cloud, Data, Integration, and Automation technologies to transform client businesses into intelligent, high-growth enterprises. Our proprietary platforms power critical business processes across our core verticals. We are located in 23 countries with 30 delivery centers across nine countries.
WHAT WE DO.
Explore our wide gamut of digital transformation capabilities and our work across industries.
