Bobby Wolf posted a great article about a wide-spread problem plaguing many SOA implementations: over-engineering of SOA infrastructure, meaning that people rollout products that are not particularly required to implement their business services. He specifically talks about ESBs, but I would say that “you ain’t gonna need it” principle should be applied to any component of a SOA stack. For example, why implement a super-expensive BPM suite (or a BPEL engine) when an organization is simply trying to build some data services? Or why pay for a registry if there are only a handful of services in place?

What many people don’t realize is just how many SOA-related features are already provided by application servers. Open-source Glassfish server supports JAXR Web services repositories, XSLT mediations, JBI integration and provides extensive Web services monitoring capabilities. It can be augmented by several open-source JBI-based ESBs, such as OpenESB. Other application servers provide similar set of features, although using different technologies (e.g., SCA instead of JBI in IBM products). With an application server being so powerful, there should be a very good reason and business rationale for upgrading to commercial SOA/integration products.

In my mind the right strategy for any SOA is to first implement some services that provide immediate value to the business stakeholders. These services should be implemented using the products already owned and used by an organization (or by using open source ones). After the initial success of this implementation, the organization can evaluate commercial products to see how these products will help implement next set of business requirements/goals more efficiently.

Lori MacVitte from F5 had a few comments about my post on using XML appliances in ESB capacity. While I don’t completely agree with some of her specific points about what ESB capabilities are missing from XML appliances (for example, parallel processing is employed by most appliance vendors, transactionality is also supported for relevant protocols (although no XML appliance can act as an XA transaction coordinator)), the overall observation is correct – there is (and there always will be) a big difference between software-based ESBs and hardware-based XML appliances.

Software ESBs are implemented on top of application servers or messaging products. As such, they are infinitely flexible. Even if certain feature (e.g., some fancy transformation) is not supported by the ESB product out of the box, it can always be custom-developed using Java or other language supported by the product. This custom code can use all the facilities of the underlying application server, such as transaction management, connection pooling and so forth.

Read the rest of this post »

IBM recently announced that DataPower XI50 appliance now supports transformations developed using WebSphere Transformation Extender design studio. This is the same technology used in the Message Broker product. Additionally, support for WebSphere Registry and Repository has been added. From the announcement:

New offerings being announced today include the WebSphere DataPower Integration Appliance XI50, which now supports direct database connectivity including IBM’s DB2 data server extending the XI50’s enterprise service bus capabilities. It also adds support for WebSphere Transformation Extender design studio providing common data transformation tooling across IBM’s ESB portfolio. The XI50, along with a new version of the WebSphere DataPower XML Security Appliance XS40, now integrates with the IBM WebSphere Service Registry and Repository (WSRR) to offer enhanced governance capabilities and improve service interoperability, reuse and connectivity.

So why is it significant?

Read the rest of this post »

Batch processing (a.k.a. “bulk processing”) is dull and boring compared to the new world of SOA, Software as a Service and Web 2.0. It’s hardly ever mentioned these days, so one can get an impression that batch processing all but disappeared from an enterprise and got replaced by “enterprise mashups”, or, at the very least, Web services.

Of course, nothing can be further from the truth. While Web services indeed begin playing a key role in many organizations, batch files still remain one of the most widespread form of system integration due to a large number of “legacy” systems that rely on it. At the same time, batch processing presents certain challenges to developers; these challenges include high volume/high throughput requirements since batch processing is inherently “spiky”, ability to deal with failures, which usually requires checkpoint/restart capabilities and many others.

The way batch files are processed varies greatly from system to system; in most cases developers end up creating their own custom frameworks to handle batch processing requirements.

Read the rest of this post »

Yahoo Pipes web site was launched last week and almost immediately drew the attention of a large crowd – I think the site actually went down for a few hours after the launch.

Yahoo Pipes makes it extremely easy to “mash” different Web sources together – without any programming, using drag-and-drop AJAX UI. The UI is actually very slick, it loads fast and provides very intuitive environment. Basically, the UI allows users to create a “message flow” or a pipe using predefined customizable operations, which is a paradigm familiar to any enterprise developer.

I literally took me twenty minutes to put together a simple “pipe” aggregating different SOA-related feeds and allowing user to filter feeds (title or body) using keywords – you can check it out here (you don’t even need a Yahoo account to run it). Learning time was close to zero (perhaps five minutes).

This experience got me thinking. Why is it so easy to create composite applications on the Web (it was easy enough before and with Yahoo pipes it’s just gotten easier) and why is it so hard to do it in an enterprise SOA environment? Why ESB tools don’t have the same level of ease of use and the same degree of “zero administration”? Visual message flow and workflow editors for ESBs are nothing new, but they still come with steep learning curve, tricky configuration requirements and hefty price tags. Of course it’s not fair to compare a simple RSS aggregation/filtering function with typical enterprise tasks (e.g., try to reconcile three different customer XML schemas with different field semantics), but I still think that we have plenty of room for improvement in the enterprise SOA space.

This blog entry discusses
Forrester Wave report on ESB market. The report endorses ESB products and suggests that
ESB is “the most straightforward way to get started with service-oriented integration today”.

And I’ve always thought that the most straightforward way is to start
implementing services instead of infrastructure products (however useful these products might be).
As I blogged before, ESB is not
a magic wand that will make SOA happen with a flick of a switch.
SOA is about implementing services that provide some value to their consumers. Whether these services
are mediated by an ESB is completely secondary.

In my opinion, an ESB should come into play later, after certain critical mass of services is designed (
and potentially implemented) and
a need for mediations and transformation provided by an ESB becomes more obvious.
In certain environments with
a relatively homogeneous application set (e.g., an organization which is 100% J2EE shop),
ESB may not be required at all, assuming that all applications can speak Web services and
canonical data model is well defined.

Even in heterogeneous environments, there are options. For example, ESBs can be used to provide
mainframe integration capabilities (conversion to copybook format, etc.), but a
different approach could be to use CICS 3.1
which support SOAP/XML natively.

The bottom line is that the need for ESBs must be driven by specific business and technical requirements,
not by some kind of perceived goodness of ESB products in general. That’s why I recommend
analysing requirements, designing services (and, perhaps, implementing certain set of services) before
implementing ESB infrastructure.

ESB and other SOA middleware can provide many important capabilities but at the
end of the day what really counts is whether these products make developers and
other stakeholders more productive.

There is nothing “magic” about ESB products. They are all based on well-known
application servers and/or messaging products and so they really don’t change
the underlying basic characteristics of these platforms. Some ESBs may provide a
leg up if advanced Web services specifications, such as WS-Security and WS-ReliableMessaging,
must be used; however, their use is still relatively rare even for internal
integration, let alone for integrating with external system outside of an
enterprise. As far as basic Web services go, ESBs don’t really have any
advantage, since any application server and many open-source products can
provide equally good Web service run-time platforms.

So the true value-add of ESBs is transformation and mediation capabilities that
I described in one of my previous posts. However, there is nothing “magic”
about these capabilities either. Any mediation flow or a BPEL flow can be
implemented in a regular language, such as Java. There are many open-source and
relatively light-weight

workflow implementations (some of them even support
BPEL) that can be used in place of a full-blown commercial ESB.
There are also many adapters and converters for a variety of different applications and formats that can be purchased separately (and many ESB vendors do not package adapters with their products).

So it all boils down to whether an ESB product’s tools help create and modify
mediation flows and BPEL flows quicker and easier than competitors. This is why
all ESB vendors put great emphasis on visual modeling and development tools. And
this is where an evaluation of ESB products must focus on. Ideally, users should
be able to measure a productivity gain when creating a mediation flow/process
and when making changes, such as adding an activity or re-wiring a flow. This
gain could even be measured in absolute time it takes from the start of
implementation to when changes are fully deployed in a target environment. Of
course, any evaluation should include multiple products, potentially using
different approaches (e.g., visual tool versus API-based workflow library).

My suspicion is that ESBs demonstrate productivity gains primarily in a case of
simple to medium complexity flows. As I blogged

before, BPEL and many proprietary
mediation flow languages could be inadequate when dealing with complex flows
with a very high number of activities and complex business logic. So an ideal
product should also provide an easy-to-use APIs to allow developers to switch
from “visual” mode to “textual” development environment where all features of
Java or, perhaps, a scripting language, can be used (so that the appropriate
reusable components can be created).

ESB products are touted by vendors as key infrastructure component of an SOA.
ESB product selection is a great challenge because ESB as a product category is
still very new. Unlike in JEE application server space, there are no standards
that define ESB capabilities. So vendors are free to use ESB moniker for,
essentially, any integration middleware that has some Web services support.

I’ve looked at several ESB products trying to understand exactly what these
products are about and what the key features that characterize an ESB are. Here
is the list of these features:

• Data transformation. Supports bi-directional transformation for XML and non-XML
  formats, such as CSV, EDI, COBOL copybooks. XML-to-XML transformation is done
  with the help of XSLT or XQuery; non-XML transformation is implemented using
  various proprietary mechanisms. Multiple transformations can be “chained”
  together to implement a complex transformation logic.
• Message routing and service orchestration flows. This facility allows ESB to
  create a flow that supports transformations, branching (“if” logic), and
  services invocation. Some vendors use BPEL for routing and orchestration (CapeClear,
  IBM ProcessServer), some use proprietary mechanisms (BEA, IBM Message Broker)
  and some use both (Fiorano).
• Support for multiple protocols. Almost all vendors support HTTP, JMS, FTP and
  email.
• GUI tools (typically, Eclipse-based) for developing data transformation and
  service orchestration flows. All vendors promote ESBs as a way to improve “agility
  ” of IT, in other words, allowing IT respond to business requirements quicker.
  Easy-to-use tools are the key piece of this vision; in theory they should allow
  for designing flows and transformations in a visual manner with minimal or no
  programming (I have to admit that I’m somewhat skeptical of this view).

This pretty much describes “core” ESB capabilities. Of course, many (if not all)
ESB products go above and beyond this basic feature set and implement many other
features, such as service management, security, additional WS-* specs (such as
WS-ReliableMessaging), services registry, connectors to enterprise applications
(SAP, etc.) and others. But I would argue that all these extra amenities are not
really what ESBs are about. They can be supported either with other product
categories (such as SOA management tools) or by application servers, especially
since many ESB products run on top of existing application servers (e.g., BEA,
CapeClear).

So when selecting an ESB for your project, focus on the core capabilities and
move non-core stuff down the list. By the way, ESB
roundup conducted by Network computing magazine provide a good starting
point for ESB selection (although their review is a bit too superficial in my
opinion).