The Enterprise Service Bus (ESB) has become the hot and hyped product of 2004 to the extent that the established integration vendors are attempting to release one, claiming that they already have one, or arguing that an ESB is nothing new at all. But can it deliver the promise of cost-effective integration? Or will it go the way of web services on which standards, among others, it is built and end up primarily used in 'nice-to-have' applications rather than core business projects?
The adoption hurdle
The core rationale for investment in integration technologies is to support new business processes with already existing technology. This is reflected in two primary requirements for any integration project:
- Flexibility to work with current systems and processes quickly and inexpensively.
- Efficiency in using existing skills and software wherever possible.
Web services promised to address these requirements for a broad range of enterprise integration projects both within and across organizations. But what was and remains striking is the limited adoption of classic web services for high-value projects of this type.
The original, Internet-centric definition of web services followed the path of simplicity, defining only the bare minimum required for delivery of a service-oriented architecture: a service definition in WSDL and the expected messages and responses in SOAP. To facilitate consumers of these services, a UDDI registry stores the service definitions, allowing consumers to discover service providers.
In my opinion, this classic variety of web services failed to achieve widespread adoption in high value projects largely because vendors failed to answer the two questions that any CIO will ask of a new technology:
- How do I get this adopted by my organization?
Over the last few years, a multitude of 'web services' standards have appeared, and often disappeared. The lack of adoption of these standards is telling and reflects their failure to address the central concerns of CIOs: for the most part these standards defined replacement technologies where none was wanted. Clearly at some point in the future, when standards become mature, established products may support them. Until then it is vital to work with what is already in place. The early web services vendors largely ignored this basic truth in their quest to re-invent the integration wheel.
- How easily can I realize the business benefits?
First-generation web service tools delivered an immediate benefit. They allowed point-to-point connections between previously hard-to-connect technologies. Unfortunately, this in itself had little business benefit: it didn’t support more complex integrations or deliver true flexibility. The problem was one of abstraction. The benefits of service orientation are only easily realised if the ‘consuming’ application is new and has been written specifically to use the service. Unfortunately, integration projects tend to involve existing applications, which are not able to use the service without expensive and disruptive change.
These barriers are overcome using two related but subtly different activities: orchestration, and what I call mediation: all those activities required to get the right message into the right format. Let's look at each of these activities in turn.
Orchestration involves the sequencing of web service calls, with simple decision logic and some data transformation occurring between calls, in order to complete a business process. The BPEL standard is becoming accepted as the appropriate way to orchestrate web services.
However, orchestration as supported by BPEL is limited to situations where there are web services and definitions available, and those definitions are appropriate for the process being orchestrated. Experience tells us that in many cases neither is true, which is why mediation is essential.
Mediation is the missing piece of the jigsaw. Simply put, it involves the transformation, routing, validation and processing of messages, which in turn enables differences in information models between web service providers and users to be accounted for and overcome when creating applications. This is essential when integrating newly defined web services with existing infrastructure.
Some vendors have attempted to embed mediation within the BPEL definition with the BPELJ (or 'BPEL for Java') standard. While this does cover some simple mediation, it does not handle the more complex integration problems encountered in the real world without excessive 'blobs' of code. The alternative (proposed by many vendors) is to code your own integration layer a little like buying an expensive car and being told to build your own engine. A new alternative is needed.
Enter the ESB (perhaps)
The Enterprise Service Bus (ESB) enables web services to integrate with existing enterprise infrastructure, and in doing so it provides answers to those two key CIO questions discussed above. ESB products support messaging systems, J2EE application servers, commercial applications such as SAP and PeopleSoft, relational databases and SNMP management platforms. As such, they provide seamless and guaranteed integration with the existing infrastructure meaning that investment in SOA and web services need not involve replacing working systems.
Unfortunately, whether the second question ('delivering the benefits') is answered successfully depends on whether mediation is considered as part of an ESB or not.
Many vendors are viewing the ESB primarily as queuing with a few extra features. Such a minimalist approach will not meet the complex integration challenges facing the CIO in any large organization. Mediation is absolutely fundamental to the successful integration of multiple services within new and existing applications.
The ESB is at a crossroads. One route is the partial dead-end already explored by classic web services. The other is delivery of an SOA genuinely capable of delivering real-world integration solutions, both on a tactical and a strategic basis. Organizations shopping for an ESB to meet their integration needs should carefully consider the direction in which their proposed choice will lead them.
More on this topic
An ESB is something you build for your enterprise or organization ...
As web services moves into the area of application integration ...
We name today's top seven ESB vendors ...