Bobby Wolf posted a great article about a wide-spread problem plaguing many SOA implementations: over-engineering of SOA infrastructure, meaning that people rollout products that are not particularly required to implement their business services. He specifically talks about ESBs, but I would say that "you ain't gonna need it" principle should be applied to any component of a SOA stack. For example, why implement a super-expensive BPM suite (or a BPEL engine) when an organization is simply trying to build some data services? Or why pay for a registry if there are only a handful of services in place?
What many people don't realize is just how many SOA-related features are already provided by application servers. Open-source Glassfish server supports JAXR Web services repositories, XSLT mediations, JBI integration and provides extensive Web services monitoring capabilities. It can be augmented by several open-source JBI-based ESBs, such as OpenESB. Other application servers provide similar set of features, although using different technologies (e.g., SCA instead of JBI in IBM products). With an application server being so powerful, there should be a very good reason and business rationale for upgrading to commercial SOA/integration products.
In my mind the right strategy for any SOA is to first implement some services that provide immediate value to the business stakeholders. These services should be implemented using the products already owned and used by an organization (or by using open source ones). After the initial success of this implementation, the organization can evaluate commercial products to see how these products will help implement next set of business requirements/goals more efficiently.
Several SOA vendors are trying to put together comprehensive suites of SOA products that in theory should be capable of addressing all aspects of SOA, including governance, integration, business process management and others.
Formation of SOA suites is having a tremendous impacts on how SOA products are selected as many organizations are being tempted to settle for "one stop shop" approach as opposed to doing proper product evaluation within each SOA product category. (Interesting discussion about SOA suites is available at ZDNet).
So what is a SOA suite and how offerings from different vendors support different aspects of SOA? The table below attempts to answer this question. (more…)
Lori MacVitte from F5 had a few comments about my post on using XML appliances in ESB capacity. While I don't completely agree with some of her specific points about what ESB capabilities are missing from XML appliances (for example, parallel processing is employed by most appliance vendors, transactionality is also supported for relevant protocols (although no XML appliance can act as an XA transaction coordinator)), the overall observation is correct â€“ there is (and there always will be) a big difference between software-based ESBs and hardware-based XML appliances.
Software ESBs are implemented on top of application servers or messaging products. As such, they are infinitely flexible. Even if certain feature (e.g., some fancy transformation) is not supported by the ESB product out of the box, it can always be custom-developed using Java or other language supported by the product. This custom code can use all the facilities of the underlying application server, such as transaction management, connection pooling and so forth. (more…)
IBM recently announced that DataPower XI50 appliance now supports transformations developed using WebSphere Transformation Extender design studio. This is the same technology used in the Message Broker product. Additionally, support for WebSphere Registry and Repository has been added. From the announcement:
New offerings being announced today include the WebSphere DataPower Integration Appliance XI50, which now supports direct database connectivity including IBM's DB2 data server extending the XI50's enterprise service bus capabilities. It also adds support for WebSphere Transformation Extender design studio providing common data transformation tooling across IBM's ESB portfolio. The XI50, along with a new version of the WebSphere DataPower XML Security Appliance XS40, now integrates with the IBM WebSphere Service Registry and Repository (WSRR) to offer enhanced governance capabilities and improve service interoperability, reuse and connectivity.
So why is it significant? (more…)
Batch processing (a.k.a. "bulk processing") is dull and boring compared to the new world of SOA, Software as a Service and Web 2.0. Itâ€™s hardly ever mentioned these days, so one can get an impression that batch processing all but disappeared from an enterprise and got replaced by â€œenterprise mashupsâ€, or, at the very least, Web services.
Of course, nothing can be further from the truth. While Web services indeed begin playing a key role in many organizations, batch files still remain one of the most widespread form of system integration due to a large number of "legacy" systems that rely on it. At the same time, batch processing presents certain challenges to developers; these challenges include high volume/high throughput requirements since batch processing is inherently â€œspikyâ€, ability to deal with failures, which usually requires checkpoint/restart capabilities and many others.
The way batch files are processed varies greatly from system to system; in most cases developers end up creating their own custom frameworks to handle batch processing requirements. (more…)
Yahoo Pipes web site was launched last week and almost immediately drew the attention of a large crowd - I think the site actually went down for a few hours after the launch.
Yahoo Pipes makes it extremely easy to "mash" different Web sources together - without any programming, using drag-and-drop AJAX UI. The UI is actually very slick, it loads fast and provides very intuitive environment. Basically, the UI allows users to create a "message flow" or a pipe using predefined customizable operations, which is a paradigm familiar to any enterprise developer.
I literally took me twenty minutes to put together a simple "pipe" aggregating different SOA-related feeds and allowing user to filter feeds (title or body) using keywords - you can check it out here (you don't even need a Yahoo account to run it). Learning time was close to zero (perhaps five minutes).
This experience got me thinking. Why is it so easy to create composite applications on the Web (it was easy enough before and with Yahoo pipes it's just gotten easier) and why is it so hard to do it in an enterprise SOA environment? Why ESB tools don't have the same level of ease of use and the same degree of "zero administration"? Visual message flow and workflow editors for ESBs are nothing new, but they still come with steep learning curve, tricky configuration requirements and hefty price tags. Of course it's not fair to compare a simple RSS aggregation/filtering function with typical enterprise tasks (e.g., try to reconcile three different customer XML schemas with different field semantics), but I still think that we have plenty of room for improvement in the enterprise SOA space.
This blog entry discusses Forrester Wave report on ESB market. The report endorses ESB products and suggests that ESB is “the most straightforward way to get started with service-oriented integration today”.
And I’ve always thought that the most straightforward way is to start implementing services instead of infrastructure products (however useful these products might be). As I blogged before, ESB is not a magic wand that will make SOA happen with a flick of a switch. SOA is about implementing services that provide some value to their consumers. Whether these services are mediated by an ESB is completely secondary.
In my opinion, an ESB should come into play later, after certain critical mass of services is designed ( and potentially implemented) and a need for mediations and transformation provided by an ESB becomes more obvious. In certain environments with a relatively homogeneous application set (e.g., an organization which is 100% J2EE shop), ESB may not be required at all, assuming that all applications can speak Web services and canonical data model is well defined.
Even in heterogeneous environments, there are options. For example, ESBs can be used to provide mainframe integration capabilities (conversion to copybook format, etc.), but a different approach could be to use CICS 3.1 which support SOAP/XML natively.
The bottom line is that the need for ESBs must be driven by specific business and technical requirements, not by some kind of perceived goodness of ESB products in general. That’s why I recommend analysing requirements, designing services (and, perhaps, implementing certain set of services) before implementing ESB infrastructure.
ESB and other SOA middleware can provide many important capabilities but at the end of the day what really counts is whether these products make developers and other stakeholders more productive.
There is nothing “magic” about ESB products. They are all based on well-known application servers and/or messaging products and so they really don’t change the underlying basic characteristics of these platforms. Some ESBs may provide a leg up if advanced Web services specifications, such as WS-Security and WS-ReliableMessaging, must be used; however, their use is still relatively rare even for internal integration, let alone for integrating with external system outside of an enterprise. As far as basic Web services go, ESBs don’t really have any advantage, since any application server and many open-source products can provide equally good Web service run-time platforms.
So the true value-add of ESBs is transformation and mediation capabilities that I described in one of my previous posts. However, there is nothing “magic” about these capabilities either. Any mediation flow or a BPEL flow can be implemented in a regular language, such as Java. There are many open-source and relatively light-weight workflow implementations (some of them even support BPEL) that can be used in place of a full-blown commercial ESB. There are also many adapters and converters for a variety of different applications and formats that can be purchased separately (and many ESB vendors do not package adapters with their products).
So it all boils down to whether an ESB product’s tools help create and modify mediation flows and BPEL flows quicker and easier than competitors. This is why all ESB vendors put great emphasis on visual modeling and development tools. And this is where an evaluation of ESB products must focus on. Ideally, users should be able to measure a productivity gain when creating a mediation flow/process and when making changes, such as adding an activity or re-wiring a flow. This gain could even be measured in absolute time it takes from the start of implementation to when changes are fully deployed in a target environment. Of course, any evaluation should include multiple products, potentially using different approaches (e.g., visual tool versus API-based workflow library).
My suspicion is that ESBs demonstrate productivity gains primarily in a case of simple to medium complexity flows. As I blogged before, BPEL and many proprietary mediation flow languages could be inadequate when dealing with complex flows with a very high number of activities and complex business logic. So an ideal product should also provide an easy-to-use APIs to allow developers to switch from “visual” mode to “textual” development environment where all features of Java or, perhaps, a scripting language, can be used (so that the appropriate reusable components can be created).
Visual tools for designing message or process flows are part of many ESB products. BPM products supporting BPEL or BPMN also heavily rely on visual tools. Flowchart-like visual notation used by these tools represents a specialized language that includes all elements of structural programming ("if-then-else", "while" and so on).
Unfortunately, these languages have very limited support for reuse. Not all such languages support sub-processes (i.e., subroutines). There is no inheritance or extension, in other words, I can't take an existing process flow or message flow and change some of its logic without duplicating the logic of an existing flow. Lack of these facilities makes it difficult to implement complex and reusable process and message flows.
From my experience any real-life process flow is going to be pretty complex. Three-step loan approval process widely used as an example in many different BPEL products does not exist in reality. Real-life business processes include many different activities and involve many different roles. They also have to take into account various exceptions, not just a "happy path" scenario. For example, if a loan application is rejected because of a credit history, a lender can try to offer a loan under different terms.
Also note that there is a difference between using visual tools for modeling and for coding. A model does not have to include each and every detail, its main goal is to convey key ideas; details can be taken care of during implementation. Not so if we want to visually create executable code; in this case all details have to be accounted for.
I am also not convinced that structural programming approach that BPEL and other "flow" languages support is best suited for visual tools. Structural programming constructs replace explicit "flow" represented by "go to". In a visual flow-oriented language, however, "go to" (i.e., explicitly connecting nodes) might be a better alternative.
I also doubt that visual approach works well for code with high cyclomatic complexity (and without subroutines high complexity is almost guaranteed), especially with high degree of nesting. Text-based representation of nested code blocks using indentation could be very compact. This is not the case when each branch has to be presented by its own "flow" of connected nodes.
So I think that text-based representation using a high-level specialized language could be a much better alternative for expressing process and message flows in SOA. This language could be a domain specific language (DSL) or a general purpose dynamically typed language with built-in XML and Web services support and other constructs required for supporting process flow. Syntax of this language should be simple enough and at the same time it needs to have good support for reuse (this may or may not require full OO support).
Visual representation of a program written in such language could still be supported; however I see it as a "report" that could be used to facilitate discussion with business users. Developers should not be required to use visual environment for programming; I don't think it makes anybody more productive relative to text-based approach (unless a flow is really simple). And I'm not talking about XML-based serialization of a visual flow, such as in BPEL where XML is so unwieldy that it is almost impossible to hand-edit it. We need a true powerful "service orchestration" language that will make developers (and business users along with them) more productive.
ESB products are touted by vendors as key infrastructure component of an SOA. ESB product selection is a great challenge because ESB as a product category is still very new. Unlike in JEE application server space, there are no standards that define ESB capabilities. So vendors are free to use ESB moniker for, essentially, any integration middleware that has some Web services support.
I’ve looked at several ESB products trying to understand exactly what these products are about and what the key features that characterize an ESB are. Here is the list of these features:
This pretty much describes “core” ESB capabilities. Of course, many (if not all) ESB products go above and beyond this basic feature set and implement many other features, such as service management, security, additional WS-* specs (such as WS-ReliableMessaging), services registry, connectors to enterprise applications (SAP, etc.) and others. But I would argue that all these extra amenities are not really what ESBs are about. They can be supported either with other product categories (such as SOA management tools) or by application servers, especially since many ESB products run on top of existing application servers (e.g., BEA, CapeClear).
So when selecting an ESB for your project, focus on the core capabilities and move non-core stuff down the list. By the way, ESB roundup conducted by Network computing magazine provide a good starting point for ESB selection (although their review is a bit too superficial in my opinion).