Category Archives: ESB

You Ain’t Gonna Need ESB

Bobby Wolf posted a great article about a wide-spread problem plaguing many SOA implementations: over-engineering of SOA infrastructure, meaning that people rollout products that are not particularly required to implement their business services. He specifically talks about ESBs, but I would say that “you ain’t gonna need it” principle should be applied to any component of a SOA stack. For example, why implement a super-expensive BPM suite (or a BPEL engine) when an organization is simply trying to build some data services? Or why pay for a registry if there are only a handful of services in place?

What many people don’t realize is just how many SOA-related features are already provided by application servers. Open-source Glassfish server supports JAXR Web services repositories, XSLT mediations, JBI integration and provides extensive Web services monitoring capabilities. It can be augmented by several open-source JBI-based ESBs, such as OpenESB. Other application servers provide similar set of features, although using different technologies (e.g., SCA instead of JBI in IBM products). With an application server being so powerful, there should be a very good reason and business rationale for upgrading to commercial SOA/integration products.

In my mind the right strategy for any SOA is to first implement some services that provide immediate value to the business stakeholders. These services should be implemented using the products already owned and used by an organization (or by using open source ones). After the initial success of this implementation, the organization can evaluate commercial products to see how these products will help implement next set of business requirements/goals more efficiently.

Comparison of SOA Suites

Several SOA vendors are trying to put together comprehensive suites of SOA products that in theory should be capable of
all aspects of SOA, including governance, integration, business process management and others.

Formation of SOA suites is having a tremendous impacts on how SOA products are selected
as many organizations
are being tempted to settle for “one stop shop” approach as opposed to doing proper product evaluation
within each SOA product category. (Interesting discussion about SOA suites is available at

So what is a SOA suite and how offerings from different vendors support different aspects of SOA?
The table below attempts to answer this question.

Read the rest of this post »

Is ESB The Mediation Infrastructure of Web Services Platform?

Lori MacVitte from F5 had a few comments about my post on using XML appliances in ESB capacity. While I don’t completely agree with some of her specific points about what ESB capabilities are missing from XML appliances (for example, parallel processing is employed by most appliance vendors, transactionality is also supported for relevant protocols (although no XML appliance can act as an XA transaction coordinator)), the overall observation is correct – there is (and there always will be) a big difference between software-based ESBs and hardware-based XML appliances.

Software ESBs are implemented on top of application servers or messaging products. As such, they are infinitely flexible. Even if certain feature (e.g., some fancy transformation) is not supported by the ESB product out of the box, it can always be custom-developed using Java or other language supported by the product. This custom code can use all the facilities of the underlying application server, such as transaction management, connection pooling and so forth.

Read the rest of this post »

Is XML Appliance the Ultimate ESB?

IBM recently announced that DataPower XI50 appliance now supports transformations developed using WebSphere Transformation Extender design studio. This is the same technology used in the Message Broker product. Additionally, support for WebSphere Registry and Repository has been added. From the announcement:

New offerings being announced today include the WebSphere DataPower Integration Appliance XI50, which now supports direct database connectivity including IBM’s DB2 data server extending the XI50’s enterprise service bus capabilities. It also adds support for WebSphere Transformation Extender design studio providing common data transformation tooling across IBM’s ESB portfolio. The XI50, along with a new version of the WebSphere DataPower XML Security Appliance XS40, now integrates with the IBM WebSphere Service Registry and Repository (WSRR) to offer enhanced governance capabilities and improve service interoperability, reuse and connectivity.

So why is it significant?

Read the rest of this post »

Finally, Somebody is Thinking about Batch Processing

Batch processing (a.k.a. “bulk processing”) is dull and boring compared to the new world of SOA, Software as a Service and Web 2.0. It’s hardly ever mentioned these days, so one can get an impression that batch processing all but disappeared from an enterprise and got replaced by “enterprise mashups”, or, at the very least, Web services.

Of course, nothing can be further from the truth. While Web services indeed begin playing a key role in many organizations, batch files still remain one of the most widespread form of system integration due to a large number of “legacy” systems that rely on it. At the same time, batch processing presents certain challenges to developers; these challenges include high volume/high throughput requirements since batch processing is inherently “spiky”, ability to deal with failures, which usually requires checkpoint/restart capabilities and many others.

The way batch files are processed varies greatly from system to system; in most cases developers end up creating their own custom frameworks to handle batch processing requirements.

Read the rest of this post »

Yahoo Pipes – A Great Way to Create Composite Applications

Yahoo Pipes web site was launched last week and almost immediately drew the attention of a large crowd – I think the site actually went down for a few hours after the launch.

Yahoo Pipes makes it extremely easy to “mash” different Web sources together – without any programming, using drag-and-drop AJAX UI. The UI is actually very slick, it loads fast and provides very intuitive environment. Basically, the UI allows users to create a “message flow” or a pipe using predefined customizable operations, which is a paradigm familiar to any enterprise developer.

I literally took me twenty minutes to put together a simple “pipe” aggregating different SOA-related feeds and allowing user to filter feeds (title or body) using keywords – you can check it out here (you don’t even need a Yahoo account to run it). Learning time was close to zero (perhaps five minutes).

This experience got me thinking. Why is it so easy to create composite applications on the Web (it was easy enough before and with Yahoo pipes it’s just gotten easier) and why is it so hard to do it in an enterprise SOA environment? Why ESB tools don’t have the same level of ease of use and the same degree of “zero administration”? Visual message flow and workflow editors for ESBs are nothing new, but they still come with steep learning curve, tricky configuration requirements and hefty price tags. Of course it’s not fair to compare a simple RSS aggregation/filtering function with typical enterprise tasks (e.g., try to reconcile three different customer XML schemas with different field semantics), but I still think that we have plenty of room for improvement in the enterprise SOA space.

Is ESB the Starting Point for SOA?

This blog entry discusses
Forrester Wave report on ESB market. The report endorses ESB products and suggests that
ESB is “the most straightforward way to get started with service-oriented integration today”.

And I’ve always thought that the most straightforward way is to start
implementing services instead of infrastructure products (however useful these products might be).
As I blogged before, ESB is not
a magic wand that will make SOA happen with a flick of a switch.
SOA is about implementing services that provide some value to their consumers. Whether these services
are mediated by an ESB is completely secondary.

In my opinion, an ESB should come into play later, after certain critical mass of services is designed (
and potentially implemented) and
a need for mediations and transformation provided by an ESB becomes more obvious.
In certain environments with
a relatively homogeneous application set (e.g., an organization which is 100% J2EE shop),
ESB may not be required at all, assuming that all applications can speak Web services and
canonical data model is well defined.

Even in heterogeneous environments, there are options. For example, ESBs can be used to provide
mainframe integration capabilities (conversion to copybook format, etc.), but a
different approach could be to use CICS 3.1
which support SOAP/XML natively.

The bottom line is that the need for ESBs must be driven by specific business and technical requirements,
not by some kind of perceived goodness of ESB products in general. That’s why I recommend
analysing requirements, designing services (and, perhaps, implementing certain set of services) before
implementing ESB infrastructure.

The Most Important Characteristic of an ESB

ESB and other SOA middleware can provide many important capabilities but at the
end of the day what really counts is whether these products make developers and
other stakeholders more productive.

There is nothing “magic” about ESB products. They are all based on well-known
application servers and/or messaging products and so they really don’t change
the underlying basic characteristics of these platforms. Some ESBs may provide a
leg up if advanced Web services specifications, such as WS-Security and WS-ReliableMessaging,
must be used; however, their use is still relatively rare even for internal
integration, let alone for integrating with external system outside of an
enterprise. As far as basic Web services go, ESBs don’t really have any
advantage, since any application server and many open-source products can
provide equally good Web service run-time platforms.

So the true value-add of ESBs is transformation and mediation capabilities that
I described in one of my previous posts. However, there is nothing “magic”
about these capabilities either. Any mediation flow or a BPEL flow can be
implemented in a regular language, such as Java. There are many open-source and
relatively light-weight

workflow implementations
(some of them even support
BPEL) that can be used in place of a full-blown commercial ESB.
There are also many adapters and converters for a variety of different applications and formats that can be purchased separately (and many ESB vendors do not package adapters with their products).

So it all boils down to whether an ESB product’s tools help create and modify
mediation flows and BPEL flows quicker and easier than competitors. This is why
all ESB vendors put great emphasis on visual modeling and development tools. And
this is where an evaluation of ESB products must focus on. Ideally, users should
be able to measure a productivity gain when creating a mediation flow/process
and when making changes, such as adding an activity or re-wiring a flow. This
gain could even be measured in absolute time it takes from the start of
implementation to when changes are fully deployed in a target environment. Of
course, any evaluation should include multiple products, potentially using
different approaches (e.g., visual tool versus API-based workflow library).

My suspicion is that ESBs demonstrate productivity gains primarily in a case of
simple to medium complexity flows. As I blogged

, BPEL and many proprietary
mediation flow languages could be inadequate when dealing with complex flows
with a very high number of activities and complex business logic. So an ideal
product should also provide an easy-to-use APIs to allow developers to switch
from “visual” mode to “textual” development environment where all features of
Java or, perhaps, a scripting language, can be used (so that the appropriate
reusable components can be created).

Limits of Visual Service Orchestration

Visual tools for designing message or process flows are part of many ESB
products. BPM products supporting BPEL or BPMN also heavily rely on visual tools.
Flowchart-like visual notation used by these tools represents a specialized
language that includes all elements of structural programming (“if-then-else”, “while” and
so on).

Unfortunately, these languages have very limited support for reuse. Not all such
languages support sub-processes (i.e., subroutines). There is no inheritance or
extension, in other words, I can’t take an existing process flow or message flow
and change some of its logic without duplicating the logic of an existing flow.
Lack of these facilities makes it difficult to implement complex and reusable
process and message flows.

From my experience any real-life process flow is going to be pretty complex.
Three-step loan approval process widely used as an example in many different
BPEL products does not exist in reality. Real-life business processes include
many different activities and involve many different roles. They also have to
take into account various exceptions, not just a “happy path” scenario. For
example, if a loan application is rejected because of a credit history, a lender
can try to offer a loan under different terms.

Also note that there is a difference between using visual tools for modeling and
for coding. A model does not have to include each and every detail, its main
goal is to convey key ideas; details can be taken care of during implementation.
Not so if we want to visually create executable code; in this case all details
have to be accounted for.

I am also not convinced that structural programming approach that BPEL and other
“flow” languages support is best suited for visual tools. Structural
programming constructs replace explicit “flow” represented by “go to”. In a
visual flow-oriented language, however, “go to” (i.e., explicitly connecting
nodes) might be a better alternative.

I also doubt that visual approach works well for code with high cyclomatic
complexity (and without subroutines high complexity is almost guaranteed),
especially with high degree of nesting. Text-based representation of nested code
blocks using indentation could be very compact. This is not the case when each
branch has to be presented by its own “flow” of connected nodes.

So I think that text-based representation using a high-level specialized
language could be a much better alternative for expressing process and message
flows in SOA. This language could be a domain specific language (DSL) or a
general purpose dynamically typed language with built-in XML and Web services
support and other constructs required for supporting process flow. Syntax of
this language should be simple enough and at the same time it needs to have good
support for reuse (this may or may not require full OO support).

Visual representation of a program written in such language could still be
supported; however I see it as a “report” that could be used to facilitate
discussion with business users. Developers should not be required to use visual
environment for programming; I don’t think it makes anybody more productive
relative to text-based approach (unless a flow is really simple). And I’m not
talking about XML-based serialization of a visual flow, such as in BPEL where
XML is so unwieldy that it is almost impossible to hand-edit it. We need a true
powerful “service orchestration” language that will make developers (and
business users along with them) more productive.

What is an ESB?

ESB products are touted by vendors as key infrastructure component of an SOA.
ESB product selection is a great challenge because ESB as a product category is
still very new. Unlike in JEE application server space, there are no standards
that define ESB capabilities. So vendors are free to use ESB moniker for,
essentially, any integration middleware that has some Web services support.

I’ve looked at several ESB products trying to understand exactly what these
products are about and what the key features that characterize an ESB are. Here
is the list of these features:

  • Data transformation. Supports bi-directional transformation for XML and non-XML
    formats, such as CSV, EDI, COBOL copybooks. XML-to-XML transformation is done
    with the help of XSLT or XQuery; non-XML transformation is implemented using
    various proprietary mechanisms. Multiple transformations can be “chained”
    together to implement a complex transformation logic.
  • Message routing and service orchestration flows. This facility allows ESB to
    create a flow that supports transformations, branching (“if” logic), and
    services invocation. Some vendors use BPEL for routing and orchestration (CapeClear,
    IBM ProcessServer), some use proprietary mechanisms (BEA, IBM Message Broker)
    and some use both (Fiorano).
  • Support for multiple protocols. Almost all vendors support HTTP, JMS, FTP and
  • GUI tools (typically, Eclipse-based) for developing data transformation and
    service orchestration flows. All vendors promote ESBs as a way to improve “agility
    ” of IT, in other words, allowing IT respond to business requirements quicker.
    Easy-to-use tools are the key piece of this vision; in theory they should allow
    for designing flows and transformations in a visual manner with minimal or no
    programming (I have to admit that I’m somewhat skeptical of this view).

This pretty much describes “core” ESB capabilities. Of course, many (if not all)
ESB products go above and beyond this basic feature set and implement many other
features, such as service management, security, additional WS-* specs (such as
WS-ReliableMessaging), services registry, connectors to enterprise applications
(SAP, etc.) and others. But I would argue that all these extra amenities are not
really what ESBs are about. They can be supported either with other product
categories (such as SOA management tools) or by application servers, especially
since many ESB products run on top of existing application servers (e.g., BEA,

So when selecting an ESB for your project, focus on the core capabilities and
move non-core stuff down the list. By the way, ESB
conducted by Network computing magazine provide a good starting
point for ESB selection (although their review is a bit too superficial in my