What many people don't realize is just how many SOA-related features are already provided by application servers. Open-source Glassfish server supports JAXR Web services repositories, XSLT mediations, JBI integration and provides extensive Web services monitoring capabilities. It can be augmented by several open-source JBI-based ESBs, such as OpenESB. Other application servers provide similar set of features, although using different technologies (e.g., SCA instead of JBI in IBM products). With an application server being so powerful, there should be a very good reason and business rationale for upgrading to commercial SOA/integration products.

In my mind the right strategy for any SOA is to first implement some services that provide immediate value to the business stakeholders. These services should be implemented using the products already owned and used by an organization (or by using open source ones). After the initial success of this implementation, the organization can evaluate commercial products to see how these products will help implement next set of business requirements/goals more efficiently.

Formation of SOA suites is having a tremendous impacts on how SOA products are selected as many organizations are being tempted to settle for "one stop shop" approach as opposed to doing proper product evaluation within each SOA product category. (Interesting discussion about SOA suites is available at ZDNet).

So what is a SOA suite and how offerings from different vendors support different aspects of SOA? The table below attempts to answer this question.

One thing to keep in mind is that SOA products within the same category can differ substantially in terms of their feature sets. Definitions of ESB, registry and other SOA product categories are not standardized and so vendors are free to categorize their products as they wish. Detailed analysis and evaluation is still a requirement when selecting SOA products.

IBM, BEA and Oracle are emerging as leaders in terms of completeness of their SOA suites. Microsoft's products are not as comprehensive, nevertheless Microsoft's marketshare makes it a signifcant player - Micorsoft's shops tend to be very loyal to the vendor even if Microsoft's SOA story is not as compelling.

Other vendors, including Software AG/webMethods and HP have interesting offerings too, however, they still have some gaps in their capabilities and so they are not covered here (perhaps I'll add more vendors in the future).

Software ESBs are implemented on top of application servers or messaging products. As such, they are infinitely flexible. Even if certain feature (e.g., some fancy transformation) is not supported by the ESB product out of the box, it can always be custom-developed using Java or other language supported by the product. This custom code can use all the facilities of the underlying application server, such as transaction management, connection pooling and so forth.

Extensibility of hardware appliances is typically very limited. This is the cost of having closed-down low-maintenance device. Allowing running arbitrary Java code on it would simply defeat the purpose of going the appliance route in the first place. There is obviously no way to run business logic on the appliance.

In many cases, organizations implement ESBs to serve very simple needs, such as authentication/authorization, management/monitoring, schema validation and, perhaps some relatively simple transformations dealing with cleansing data formats and reconciling version incompatibilities. The actual business logic involving business process management, service orchestration and other aspects is very often pushed down to the application tier. In other words, ESB in this case is treated as a low-level integration and service management infrastructure as opposed to a container for Web services. Appliance-based ESB would fit very well to this environment.

On the other hand, if there is a desire to use ESB as a platform and container for Web services, appliances are certainly not appropriate. Such a Web services platform acts as a next generation application server. It allows developers to implement Web services using technologies supported by the ESB (such as mediation flows and BPEL) and/or using technologies supported by the underlying application server (Java EE). Product such as WebSphere Proces Server, and, to some degree BEA AquaLogic Bus (it does not support BPEL) fit into this category.

As a side note, I believe that ESB capabilities will be moving down the stack to the base application server products. It becomes more and more difficult for vendors to charge for application server licenses as the competition in the open source space intensifies (just look at the buzz generated by the latest version of Glass Fish). At the same time, more and more enterprise applications are being built using Web services and more and more applications become Web service consumers. This means that soon most applications will require using ESB-like integration technologies that would allow them to orchestrate and mediate services easily. So software ESB products will be used more and more as platforms for service-based applications whereas XML appliances will be playing a key role in the Web services integration/management space. ]]> http://myarch.com/is-esb-mediation-infrastructure-of-web-services-platform/feed 1 http://myarch.com/is-xml-appliance-an-ultimate-esb http://myarch.com/is-xml-appliance-an-ultimate-esb#comments Wed, 23 May 2007 04:26:20 +0000 Alexander Ananiev http://myarch.com/is-xml-appliance-an-ultimate-esb IBM recently announced that DataPower XI50 appliance now supports transformations developed using WebSphere Transformation Extender design studio. This is the same technology used in the Message Broker product. Additionally, support for WebSphere Registry and Repository has been added. From the announcement:

New offerings being announced today include the WebSphere DataPower Integration Appliance XI50, which now supports direct database connectivity including IBM's DB2 data server extending the XI50's enterprise service bus capabilities. It also adds support for WebSphere Transformation Extender design studio providing common data transformation tooling across IBM's ESB portfolio. The XI50, along with a new version of the WebSphere DataPower XML Security Appliance XS40, now integrates with the IBM WebSphere Service Registry and Repository (WSRR) to offer enhanced governance capabilities and improve service interoperability, reuse and connectivity.

So why is it significant? XML appliances that were previously used mostly as firewalls are morphing into full-blown ESB products that compete with software ESB offerings. DataPower XI50 has always provided a good set of ESB-like features; however its transformation capabilities were limited mostly to XSLT. Non-XML formats were supported but required a third-party tool for developing transformation logic. With this newly added transformation support, the product's feature set becomes more complete.

I think this trend will continue and, most likely, will accelerate. More and more SOA implementations are going to use appliances as their primary (and only) ESB product. The idea of off-loading all integration functionality into a low-maintenance, highly secure device that is also optimized for performance is extremely appealing to developers and IT departments. Implementing "proper" software-based ESBs involves complicated installation and configuration. Configuration of a self-contained device is a much easier task. It is also easier to find people with the necessary skills to support these devices; these people don't have to be developers with in-depth knowledge of Java EE or .NET (although some development skills are certainly desirable).

The beauty of a good XML appliance is that it encapsulates functions of several different products: XML gateway/firewall, XML accelerator, crypto accelerator and now even ESB. This is pretty powerful.

Another thing that is currently happening is that people begin to realize that SOA doesn't make IT infrastructure any simpler. On the contrary, SOA, being a distributed environment, introduces more complexity and require much more sophistication to manage it. XML appliances can help reduce this complexity. ]]> http://myarch.com/is-xml-appliance-an-ultimate-esb/feed 7 http://myarch.com/finally-somebody-is-thinking-about-batch-processing http://myarch.com/finally-somebody-is-thinking-about-batch-processing#comments Sun, 20 May 2007 23:08:20 +0000 Alexander Ananiev http://myarch.com/finally-somebody-is-thinking-about-batch-processing Batch processing (a.k.a. "bulk processing") is dull and boring compared to the new world of SOA, Software as a Service and Web 2.0. It’s hardly ever mentioned these days, so one can get an impression that batch processing all but disappeared from an enterprise and got replaced by “enterprise mashups”, or, at the very least, Web services.

Of course, nothing can be further from the truth. While Web services indeed begin playing a key role in many organizations, batch files still remain one of the most widespread form of system integration due to a large number of "legacy" systems that rely on it. At the same time, batch processing presents certain challenges to developers; these challenges include high volume/high throughput requirements since batch processing is inherently “spiky”, ability to deal with failures, which usually requires checkpoint/restart capabilities and many others.

The way batch files are processed varies greatly from system to system; in most cases developers end up creating their own custom frameworks to handle batch processing requirements. In many cases, ETL tools become part of the solution since most ETL tools have built-in batch processing capabilities. As a result, batch processing has become the area of proprietary solutions and expensive tools. In spite of the batch processing being part of IT landscape for tens of years, there has been no framework or platform that would introduce some notion of uniformity in batch processing and save developers from having to reinvent the wheel (I'm talking about distributed systems; batch processing on mainframe is a different story).

Spring batch framework is aiming to change that. It builds upon the “template” pattern widely used in Spring to support JDBC, JMS, transactions and other APIs. Spring batch supports processing of batches and also provides “retry” capabilities. Details are pretty sketchy at this point (there is almost no documentation), some information is available in this blog entry.

I think this is a great news. Spring is very widely adopted, it is even claimed to become "new Java EE". So Spring has enough clout to become the new "batch processing container" which can be embedded in any modern application server.

In general, I think that we should treat batch processing as an integral part of SOA and enterprise architecture. Batch processing is not going away any time soon and, let's face it, some batch processes will never be replaced. We can even see some new sources of batch processing, for example, an RSS feed, if it's big enough, is no different than traditional batch file in terms of its processing requirements. Batch feed is just another source of events for an enterprise; these events can be processed same way events from other sources are processed. From SOA prospective it should not matter whether a service was invoked directly by its client or by a batch file processing routine (that can generate events processed by SOA). In reality, however, this concept has some challenges; order of records in a batch file being one of them since often times this order is significant. Performance/throughput is another important consideration; translating a batch record into "canonical" XML representation and invoking a remote service for each record could be a costly proposition.

This is why we need frameworks like Spring batch that could help us deal with these problems. It would also be nice if ESB vendors incorporated batch processing support into their products, after all dealing with different transports/data formats is one of the key capabilities that ESB is supposed to provide.

Yahoo Pipes makes it extremely easy to "mash" different Web sources together - without any programming, using drag-and-drop AJAX UI. The UI is actually very slick, it loads fast and provides very intuitive environment. Basically, the UI allows users to create a "message flow" or a pipe using predefined customizable operations, which is a paradigm familiar to any enterprise developer.

I literally took me twenty minutes to put together a simple "pipe" aggregating different SOA-related feeds and allowing user to filter feeds (title or body) using keywords - you can check it out here (you don't even need a Yahoo account to run it). Learning time was close to zero (perhaps five minutes).

This experience got me thinking. Why is it so easy to create composite applications on the Web (it was easy enough before and with Yahoo pipes it's just gotten easier) and why is it so hard to do it in an enterprise SOA environment? Why ESB tools don't have the same level of ease of use and the same degree of "zero administration"? Visual message flow and workflow editors for ESBs are nothing new, but they still come with steep learning curve, tricky configuration requirements and hefty price tags. Of course it's not fair to compare a simple RSS aggregation/filtering function with typical enterprise tasks (e.g., try to reconcile three different customer XML schemas with different field semantics), but I still think that we have plenty of room for improvement in the enterprise SOA space.

And I’ve always thought that the most straightforward way is to start implementing services instead of infrastructure products (however useful these products might be). As I blogged before, ESB is not a magic wand that will make SOA happen with a flick of a switch. SOA is about implementing services that provide some value to their consumers. Whether these services are mediated by an ESB is completely secondary.

In my opinion, an ESB should come into play later, after certain critical mass of services is designed ( and potentially implemented) and a need for mediations and transformation provided by an ESB becomes more obvious. In certain environments with a relatively homogeneous application set (e.g., an organization which is 100% J2EE shop), ESB may not be required at all, assuming that all applications can speak Web services and canonical data model is well defined.

Even in heterogeneous environments, there are options. For example, ESBs can be used to provide mainframe integration capabilities (conversion to copybook format, etc.), but a different approach could be to use CICS 3.1 which support SOAP/XML natively.

The bottom line is that the need for ESBs must be driven by specific business and technical requirements, not by some kind of perceived goodness of ESB products in general. That’s why I recommend analysing requirements, designing services (and, perhaps, implementing certain set of services) before implementing ESB infrastructure.

There is nothing “magic” about ESB products. They are all based on well-known application servers and/or messaging products and so they really don’t change the underlying basic characteristics of these platforms. Some ESBs may provide a leg up if advanced Web services specifications, such as WS-Security and WS-ReliableMessaging, must be used; however, their use is still relatively rare even for internal integration, let alone for integrating with external system outside of an enterprise. As far as basic Web services go, ESBs don’t really have any advantage, since any application server and many open-source products can provide equally good Web service run-time platforms.

So the true value-add of ESBs is transformation and mediation capabilities that I described in one of my previous posts. However, there is nothing “magic” about these capabilities either. Any mediation flow or a BPEL flow can be implemented in a regular language, such as Java. There are many open-source and relatively light-weight workflow implementations (some of them even support BPEL) that can be used in place of a full-blown commercial ESB. There are also many adapters and converters for a variety of different applications and formats that can be purchased separately (and many ESB vendors do not package adapters with their products).

So it all boils down to whether an ESB product’s tools help create and modify mediation flows and BPEL flows quicker and easier than competitors. This is why all ESB vendors put great emphasis on visual modeling and development tools. And this is where an evaluation of ESB products must focus on. Ideally, users should be able to measure a productivity gain when creating a mediation flow/process and when making changes, such as adding an activity or re-wiring a flow. This gain could even be measured in absolute time it takes from the start of implementation to when changes are fully deployed in a target environment. Of course, any evaluation should include multiple products, potentially using different approaches (e.g., visual tool versus API-based workflow library).

My suspicion is that ESBs demonstrate productivity gains primarily in a case of simple to medium complexity flows. As I blogged before, BPEL and many proprietary mediation flow languages could be inadequate when dealing with complex flows with a very high number of activities and complex business logic. So an ideal product should also provide an easy-to-use APIs to allow developers to switch from “visual” mode to “textual” development environment where all features of Java or, perhaps, a scripting language, can be used (so that the appropriate reusable components can be created).

Unfortunately, these languages have very limited support for reuse. Not all such languages support sub-processes (i.e., subroutines). There is no inheritance or extension, in other words, I can't take an existing process flow or message flow and change some of its logic without duplicating the logic of an existing flow. Lack of these facilities makes it difficult to implement complex and reusable process and message flows.

From my experience any real-life process flow is going to be pretty complex. Three-step loan approval process widely used as an example in many different BPEL products does not exist in reality. Real-life business processes include many different activities and involve many different roles. They also have to take into account various exceptions, not just a "happy path" scenario. For example, if a loan application is rejected because of a credit history, a lender can try to offer a loan under different terms.

Also note that there is a difference between using visual tools for modeling and for coding. A model does not have to include each and every detail, its main goal is to convey key ideas; details can be taken care of during implementation. Not so if we want to visually create executable code; in this case all details have to be accounted for.

I am also not convinced that structural programming approach that BPEL and other "flow" languages support is best suited for visual tools. Structural programming constructs replace explicit "flow" represented by "go to". In a visual flow-oriented language, however, "go to" (i.e., explicitly connecting nodes) might be a better alternative.

I also doubt that visual approach works well for code with high cyclomatic complexity (and without subroutines high complexity is almost guaranteed), especially with high degree of nesting. Text-based representation of nested code blocks using indentation could be very compact. This is not the case when each branch has to be presented by its own "flow" of connected nodes.

So I think that text-based representation using a high-level specialized language could be a much better alternative for expressing process and message flows in SOA. This language could be a domain specific language (DSL) or a general purpose dynamically typed language with built-in XML and Web services support and other constructs required for supporting process flow. Syntax of this language should be simple enough and at the same time it needs to have good support for reuse (this may or may not require full OO support).

Visual representation of a program written in such language could still be supported; however I see it as a "report" that could be used to facilitate discussion with business users. Developers should not be required to use visual environment for programming; I don't think it makes anybody more productive relative to text-based approach (unless a flow is really simple). And I'm not talking about XML-based serialization of a visual flow, such as in BPEL where XML is so unwieldy that it is almost impossible to hand-edit it. We need a true powerful "service orchestration" language that will make developers (and business users along with them) more productive.

I’ve looked at several ESB products trying to understand exactly what these products are about and what the key features that characterize an ESB are. Here is the list of these features:

• Data transformation. Supports bi-directional transformation for XML and non-XML formats, such as CSV, EDI, COBOL copybooks. XML-to-XML transformation is done with the help of XSLT or XQuery; non-XML transformation is implemented using various proprietary mechanisms. Multiple transformations can be “chained” together to implement a complex transformation logic.
• Message routing and service orchestration flows. This facility allows ESB to create a flow that supports transformations, branching (”if” logic), and services invocation. Some vendors use BPEL for routing and orchestration (CapeClear, IBM ProcessServer), some use proprietary mechanisms (BEA, IBM Message Broker) and some use both (Fiorano).
• Support for multiple protocols. Almost all vendors support HTTP, JMS, FTP and email.
• GUI tools (typically, Eclipse-based) for developing data transformation and service orchestration flows. All vendors promote ESBs as a way to improve “agility ” of IT, in other words, allowing IT respond to business requirements quicker. Easy-to-use tools are the key piece of this vision; in theory they should allow for designing flows and transformations in a visual manner with minimal or no programming (I have to admit that I’m somewhat skeptical of this view).

This pretty much describes “core” ESB capabilities. Of course, many (if not all) ESB products go above and beyond this basic feature set and implement many other features, such as service management, security, additional WS-* specs (such as WS-ReliableMessaging), services registry, connectors to enterprise applications (SAP, etc.) and others. But I would argue that all these extra amenities are not really what ESBs are about. They can be supported either with other product categories (such as SOA management tools) or by application servers, especially since many ESB products run on top of existing application servers (e.g., BEA, CapeClear).

So when selecting an ESB for your project, focus on the core capabilities and move non-core stuff down the list. By the way, ESB roundup conducted by Network computing magazine provide a good starting point for ESB selection (although their review is a bit too superficial in my opinion).