Over the years, organisations have invested heavily in IT and, consequently, accumulated large portfolios of IT systems comprising of hundreds, possibly even thousands, of separate IT applications. A single organization might use separate systems, developed either in-house or licensed from a third party vendor, to manage their supply chain, customer relationships, employee information, and business logic. This modularization is often desirable. In theory, breaking the task of running a business into multiple smaller functionalities allows for easy implementation of the best and newest technological advancements in each area, and quick adaptation to changing business needs.
The appropriateness of the integration architecture largely depends upon the business requirements for integration, and on what can be achieved with the available technology. Each kind of integration architecture has its own capabilities and limitations.
Batch Process: The simplest type of integration architecture, IT applications communicates asynchronously with each other and there is no business requirement for real-time processing. Batch integration architecture is often suitable for organisations that perform high-volume back-end processing that is able to take place overnight, weekly, or on a scheduled basis.
Point-to-Point Integration: In a point-to-point integration model, a unique connector component is implemented for each pair of applications or systems that must communicate. This connector handles all data transformation, integration, and any other messaging related services that must take place between only the specific pair of components it is designed to integrate.
Broker Model: In a broker approach to EAI, a central integration engine, called the broker, resides in the middle of the network, and provides all message transformation, routing, and any other inter-application functionality. All communication between applications must flow through the hub, allowing the hub to maintain data concurrency for the entire network.
SOA: The key principle of SOA is to design and build enterprise IT architecture around services rather than complete applications. In an architecture like this, the idea is to create components called services — small, discrete units of software that provide a specific functionality and, importantly, are reusable in every application. An example of an application created with SOA principles might be a bank loan application; it would be composed of credit status check services, interest rate services, and customer data services.
ESB: A different way to approach the problems SOA was meant to solve is with a standalone enterprise service bus (ESB) or integration platform as a service (iPaaS) instead of a full proprietary stack. ESBs can power the creation and orchestration of services without requiring an application server or other infrastructure component, eliminating the high upfront costs of implementing SOA. This enables developers to build reusable interfaces with APIs while also establishing a core framework for integrating a governance model for the future.
Integration Platform and API management: Some organizations remain unconvinced of the need for an application-programming interface (API) management solution. This is especially true of those that already use an integration platform as a service (iPaaS) to integrate their applications and data. Nevertheless, the fact is API management and iPaaS solutions perform unique functions that are equally critical to succeeding in today’s digital economy. Modern organizations need both.
An iPaaS lets you exchange data between different systems. It enables data to flow between cloud applications, data warehouses, IoT devices, data lakes, and other systems in your technology stack.API creation and API management are two different things. If you don’t manage the APIs you create, you run the risk of data breaches, chronic system malfunctions, and operational inefficiencies — the origins of which can be hard to pinpoint. What’s more, you fail to capitalize on the revenue potential of your APIs.
If you want to turn the APIs you create with your iPaaS into a profit-generating digital ecosystem, then you must manage your API.
Today’s technology and business professionals have several options when it comes to integrating their application data to other applications.
IBM, Oracle, TIBCO, WebMethods (Software AG) and Microsoft BizTalk are examples of enterprise integration solutions that have extensive background and experience in the integration market with their ability to scale performance and functionality in complex, heterogeneous environments.
Mulesoft, Snaplogic, Boomi and Jitterbit are examples of newer companies that have offered their products in the last 5 to 8 years. Their premise is to leverage newer technologies and frameworks to offer simpler yet rich-enough and robust-enough products that are best suited for quicker and smaller integration projects.
We could break up the enterprise-wide centralized ESB component into smaller, more manageable, dedicated components. Perhaps even down to one integration runtime for each interface we expose. This makes each individual integration easier to change independently, and improves agility, scaling, and resilience.
Agile integration architecture requires that the integration topology be deployed very differently.
The main three parts that matters most in selecting Lightweight platforms are covered here and should be sufficient to come to conclusion:
Any traditional ESB covers lot of features that not everyone would be using at a time. There in, we need get over the pitfalls of complex and heavy platforms:
Connectivity and communication: Any integration initiative starts with a question, what has to be integrated? If standard protocols only are used it is relevant to look towards lightweight ESBs. One can think of combining asynchronous messaging with lightweight ESBs to get best of both the worlds, either it can call a distant broker or can embed in itself.
Messaging patterns: Messaging patterns have to be chosen in view of business flows to implement. Some patterns are overly complex most of them are fundamental to any integration solution. Lightweight ESBs bring simplicity and efficiency by configuring only the required messaging patterns.
Transformation: Lightweight ESBs have the same mapping capabilities as traditional ones. The general notion for years have been ‘No business rule in middleware’, therefore same issues apply here as well: whether to put the mapping logic on applications side or on the middleware side, whether to use Canonical Data Model or not, etc.
Deployment: Robustness and availability are greatly eased by embedding a lightweight ESB into an application: the integration system is not a bottleneck neither a single point of failure (SPOF) anymore. And over all there is no additional system in the infrastructure. The integration framework as embed in the application then acts as the integration module of a software package. This is typically the positioning of Spring Integration.
One limitation with embedding the middleware into an application could be the impossibility to implement flows with a many-to-many cardinality which in real life are uncommon.
This may not be a problem anymore with lightweight integrations as most of the products we talked in detailed above now offer good capabilities around performance their internal architecture consists of less layers and that these layers are more integrated together, they offer good performances.
Architectural initiative: Rightly or wrongly, the choice of an ESB is often made at Enterprise level, for strong infrastructure rationalization purpose. Even if the ESB is centralized, try at least not to centralize the integration teams too much. Keep teams autonomous and reactive instead.
Technical competences: The cost of access to the technology is lowered as any JEE (Spring ideally) developer can do the job: developers can use their preferred development environment and capitalize on their JEE knowledge, without learning all the specificities of a vendor solution.
Below picture depicts market leaders as per report published by Forrester & Gartner in early 2019.
“Developed integrations with Boomi in a quarter of the time it took us before, and those integrations run three or four times faster than they did on our previous platform, handled complex business use case and application silos that needed to exchange data”
The Wells Fargo API Gateway, powered by WSO2 API Manager, has over 20 APIs in production for various data services and payment functions such as account aggregation, account balance, foreign exchange, and wire payments. Wells Fargo has built up its transactional APIs to be re-used, allowing customers to move from one experience to another with minimal changes to their resources underlying the APIs.