Evolution of Integration Architecture

As long as computer systems existed, integration between them has been one of hardest challenges that architects face. In the old days with the very first computers data was transferred to another system by printing and loading paper punched cards, nowadays it goes all automatically. But as over the years we have seen the types of integrations evolve, still some of the older technologies remain, just because of its characteristics and reliability. 

In this chapter we review the integration architecture evolution by each step with the following pictured layers:  

1. Data Transfer

Data transfer between two computer applications is the very first evolution step how systems integrate. This was in the old days through punched cards or magnetic tapes, bringing them to the other systems to upload it. Nowadays, we use technologies such as Extraction, Transformation and Load Tools (ETL) to transfer same data sets to other systems. Characteristics to use this type of integration are:

• (Extremely) High volumes of transactions, 
• No realtime requirements to respond or process them,
• Used when there is Processing time takes time, could be up to days, or for Replication between Databases 
• Typical usage: Data loads into Data Warehouses or Replication between databases 
• Typical Tools: ETL, CDC   

2. Point to Point Integration

The second generation is the direct integration point-to-point between two systems. In practice we talk here about the gross of interfaces you find in systems. On a small scale, this works excellently, but with more systems, the complexity grows exponentially as you can see in the digram below: 

Imagine that we have 2 systems with each its own language. With just 2 systems, only one (bi-directional) interface is required. But with 6 systems, already 15 integrations are needed. This high number of integrations not only complicates the design in translation issues, but also impacts the maintainability as for each interface, it needs to cover error handling, recovery capabilities such as how to react when an interface does not work. So that is where an Enterprise Service Bus comes in. 

Characteristics of Point-to-point interfaces:

• Simple Application Landscapes with few applications and low number of integrations, 
• Realtime requirements,
• Needs error handling and recovery mechanisms defined for each interface,
• No central visibility when an interface fails,
• Typical usage: Integrations in small companies 
• Typical Tools: Dependent on specific applications

3. Enterprise Service Bus

As we have seen in the previous section, as enterprise landscape grows, the number of interfaces increments exponentially impacting design and maintainability of such architectures. Therefore, companies started to use Enterprise Service Buses (ESB). An ESB facilitates the common functions that interfaces need for its operations such as error handling, robustness, recovery, without systems to focus on those capabilities for each interface. 

An ESB is focused on integration between back-end systems, typically systems of record. And as most integrations go through the ESB, this platform is one of the most robust platforms in the company. The ESB has the following characteristics:

• Interoperability Adapters, inter-connect different systems with each other independent whether it is mainframe, java, .Net or any other system, 
• Message Mediation, enabling to match incompatible protocols, data formats and interaction patterns across different applications. Data can be split, cloned, aggregated and enriched, allowing ESB to match the different capabilities of services. It also allows rich transformations on the messages. 
• Process Abstraction, making it easier and transpartent for developers to orchestrate processes with services and data entities, 
• Data Translation, translating between different data formats, 
• Logging, providing centralized logging capabilities, 
• Error Handling, providing centralized error handling, such as when errors happen, how to respond and recover,   
• Governance, centralizing the governance of integrations with a repository to add, modify and delete interfaces connecting through ESB, 
• Auditing, providing an audit trail for any transaction passing ESB, 
• Security, ensuring authentication and authorization for each of the services provided and exposed. 

 The following diagram shows the ESB providing this centralized integration platform:

SOA & BPM

While ESB take care of integrating systems, it is still a technical platform with technical interfaces. Businesses still did not obtain the flexibility and agility to change and orchestrate building blocks. That is where Service Oriented Architecture (SOA) comes into play. 

SOA is an architecture paradigm that provides a functional building blocks that can be used to orchestrate business processes over systems and applications. The services that are referenced here are actually software components that provide ready-to-use business functionalities such as:

• Discount Calculation, 
• Order Decomposition and 
• Payment Validation
• etc 

The services are standardized, shared, using SOAP "data" standard and can be used in any business process. To arrange a business flow, it can be managed by putting the blocks behind each other through programming, a workflow or Business Process Management (BPM) platform. The following picture shows how processes can be orchestrated from different interfaces:

API Management

With the popularity of digital channels another integration platform became more popular, using Application Programming Interfaces (API´s) to integrate. These API´s are different than the API´s from the earlier years of programming, as these specific interfaces are using REST "data" standard to transfer data. It is a very light and easier protocol than SOAP as it only uses the data that is needed to interact between systems. In case of SOAP, it sends over the complete data tree of a specific product as in REST it only sends over the product number to request the price. Another difference with SOAP is that APIs use http as a transport protocol, that is that with a simple GET http://application.com/GetPrice, you can obtain the price from a system. 

Examples of APIs are like web or mobile application business functionalities such as login, GetPrice, BuyProduct. And very popular public APIs are like Paypal and Facebook button one get insert in another website.   

These APIs are used mostly to interact with digital channels such as websites and mobile applications, but also with third party external systems where interaction need to be simple and fast. And as with these sources it is difficult to predict the volumes of usage, an integration platform is required to control traffic and security: the API management platform. 

An API platform provides the following features:

• Exposure to Digital and External Parties, ability to to be used by consumers and third party websites for instance that can leverage the API to directly obtain data without leaving their own website. 
• Scalability, allows to ramp up usage for certain APIs as usage can be unpredictable so the platform can provide additional resources to ramp the performance easily of one of the APIs, 
• Traffic Control, as clients can hit the system any time of the day, its traffic can be controlled by source, such as limiting the usage of a specific API by a third party, while internal usage can be without limits, 
• Security, digital channels and third parties provide additional vulnerability exposure, the platform provides security features to ensure that only authorized users and systems can use the services, 
• Proxy Management, provides an additional layer of security before entering companies premises, decoupling external traffic from internal traffic.   

If you want to know the difference between API Gateway and ESB, its use and when to use it, please read the following post "Difference between ESB and API".   

Microservices

As microservices is a hot topic today, we will talk about it in our next post Microservices in more depth.