evolving through ...

Applications architecture and the S.O.A. alternative

2009-03-13T15:34:00.001+02:00

Observing the developments on IT areas the last few years (no more than 10), one will conclude that the definition of ‘application' is rapidly changing. As the industry is moving from using an application-centric architecture to a Service Oriented Architecture (S.O.A.), the focus for building functionality is moving to Service Oriented Business Applications ( a nice description about that, except from the various posts in “evolving through …” you can find in SOBA more concentrated).

The above view, could provide the alternative to view applications as a set of services and components working together in fulfilling a certain business need. The technology, specifications, and standards for specifying these components and services may vary, and the choice is up to you, and of coarse your environment as a whole.

In various post here, I propose and expose from time to time this kind of approach, creating therefore any needed components and services in a model driven way and always having in mind an S.O.A. extension. However, even in that case you'll need some kind of programming model that defines interfaces, implementations, and references. The pros of this alternative is that when building an application or solution based on services, these services can be both created specifically for the application and reused from existing systems and applications. The discussions behind these, is known as Service Component Architecture (SCA), meaning the programming model that specifies how to create and implement services, how to reuse them and how to assemble or compose services into solutions.

The reason for this post, was an interesting project that an involved-in friend, asked for my opinion. The project is the AspireRFID an (one of the very-very few) RFID implementations that run over the Open Source community and therefore a candidate new niche for that domain. Although the existing approach is very interesting and very well analysed, I couldn't stop myself on telling that a more Service Oriented approach should be taken. This alternative might seem a little obscure, from the point of view that the technicalities and area of RFIDs might relate with a different business model than the well known explored ‘B’ in the SOBA. Actually, this is not the case. And the reason is that this area is promising candidate for that ‘B’ because the limited relations toy ERPs, CRMs or what ever, are the until know the well defined areas and developments of the various alternatives of open source that could be used on (aka OSGi, ServiceMix, FUSE, Camel, Newton, Jini, Tuscany etc). So, in that case the solution might not be a S.O.A. a S.O.A. towards one.

who’s and why’s of Service Component Architecture (SCA)

As has been stated in here and here, mentioning of an alternative is a crucial concern when there is the involvement of the meaning that one gives to S.O.A.. The SCA models the "A" in S.O.A. (and therefore not SOA). It provides a model for service construction, service assembly, and deployment. The SCA supports components defined in multiple languages, deployed in multiple container technologies, and with multiple service access methods. This means that with the SCA approach, we have a set of specifications, addressing both runtime v endors and enterprise developers/architects, for building service-oriented enterprise applications.

The SCA specifications try to define a model for building Service-Oriented (Business) Applications (SOBA). These specifications are based on the following design principles:

Independent of programming language or underlying implementation.
Independent of host container technology.
Loose coupling between components.
Metadata vendor-independence.
Security, transaction and reliability modelled as policies.
Recursive composition.

The first draft of SCA was published in November 2005 by the Open SOA Collaboration (OSOA). OSOA is a consortium of vendors and companies interest in supporting or building a SOA. Among them are IBM, Oracle, SAP, Sun, and Tibco. In March 2007 the 1.0 final set of specifications was released. In July 2007 the specifications were adopted by OASIS and they are developed further in the OASIS Open Composite Services Architecture (Open CSA) member section.

Concluding - Current SCA Specifications at OpenCSA

The current set of SCA specifications hosted on the OSOA site includes,
SCA Assembly Model
SCA Policy Framework
SCA Java Client & Implementation
SCA BPEL Client & Implementation
SCA Spring Client & Implementation
SCA C++ Client & Implementation
SCA Web Services Binding
SCA JMS Binding

The important element within the SCA approach, is that for the first time we have a set of specifications that address how S.O.A. can be built rather than what S.O.A. is about. Lets have a bird's eye view of SCA as a technology.

The four main elements of the SCA, aka the related specifications to start with

The set of specifications can be split into four main elements: the assembly model specification, component implementation specifications, binding specifications, and the policy framework specification. Therefore, these might be a good starting/guiding point upon the need of analysis and/or re-engineer needed for the project or the solution-application at hand.

Assembly model specification: this model defines how to specify the structure of a composite application. It defines what services are assembled into a SOBA and with which components these services are implemented. Each component can be a composite itself and be implemented by combining the services of one of more other components (i.e. recursive composition). So, in short, the assembly model specifies the components, the interfaces (services) for each component, and the references between components, in a unified, language independent-way. The assembly model is defined with XML files.

SCA Assembly Model V1.00 (PDF)
SCA Core XML Schema V1.0 (XSD file)

Using the SCA assembly model, we have to define firstly the recursive assembly of components into services. Recursive assembly allows fine-grained components to be wired into what in SCA terms is called composites. These composites than can be used as components in higher-level composites and applications. The type of a component generally defines the services exposed by the component, properties that can be configured on the components and dependencies the component has on services exposed by other components.

An example of an SCA assembly model, is shown in the diagram bellow:

Diagram 1: SCA assembly model

In the diagram above Composite X is composed of two components Component Y and Component Z, which themselves are composites. Composite X exposes its service using web services.

Composite Y is composed of a Java component, a Groovy component and a Jython component. The service offered by Java component is promoted outside the composite. The Java component depends on services offered by the Groovy and Jython components. In SCA terminology these dependencies are called references. All the references for the Java component are satisfied by services offered by other components within the same composite. However, the reference on the Groovy component has been promoted outside the composite and will have to be satisfied in a context within which Composite Y is used as a component. In the diagram above this is achieved using the service promoted by Composite Z.

Composite Z is composed of a Java component, a BPEL component and a Spring component, the service offered by the BPEL component is promoted to be used when the composite is used as a component in a higher level composite.

As you can see SCA allows building of compositional service-oriented applications using components built on heterogeneous technologies and platforms. SCA assembly model is generally expressed in an XML based language called Service Component Description Language (SCDL).

Component implementation specifications: these specifications define how a component is actually written in a particular programming language. Components are the main building blocks for an application build using SCA. A component is a running instance of an implementation that has been appropriately configured. The implementation provides the actual function of the component and can be defined with a Java class, BPEL process, Spring bean, and C++ or C code. Several containers implementing the SCA standard (meaning that they can run SCA components) support additional implementation types like .Net, OSGI bundles, etc. In theory a component can be implemented with any technology, as long as it relies on a common set of abstractions, e.g. services, references, properties, and bindings.

The following component implementation technologies are currently described:

SCA Java Component Implementation V1.00 (PDF), XSD file
SCA Spring Component Implementation V1.00 (PDF)
SCA BPEL Client and Implementation V1.00 (PDF)
SCA C++ Client and Implementation V1.00 (PDF), XSD file
SCA COBOL Client and Implementation V1.00 (PDF)
SCA C Client and Implementation V1.00 (PDF)

Binding specifications: these specifications define how the services published by a component can be accessed. Binding types can be configured for both external systems and internal wires between components. The current binding types described by OSOA are bindings using SOAP (web services binding), JMS, EJB, and JCA. Several containers implementing the SCA standard support additional binding types like RMI, Atom, JSON, etc. An SCA runtime should at least implement the SCA service and web service binding types. The SCA service binding type should only be used for communication between composites and components within an SCA domain. The way in which this binding type is implemented is not defined by the SCA specification and it can be implemented in different ways by different SCA runtimes. It is not intended to be interoperable. For interoperability the standardized binding types like web services have to be used.

Officially described binding types:

SCA Web Services Binding V1.00 (PDF), XSD file
SCA JMS Binding V1.00 (PDF), XSD file
SCA EJB Session Bean Binding V1.00 (PDF), XSD file
SCA JCA Binding V1.00 (PDF)

Policy framework specification: for describing how to add non-functional requirements to services. Two kinds of policies exist: interaction and implementation policies. Interaction policies affect the contract between a service requestor and a service provider. Examples of such policies are message protection, authentication, and reliable messaging. Implementation policies affect the contract between a component and its container. Examples of such policies are authorization and transaction strategies.

Summary

SCA provides a set of specifications, addressing both runtime vendors and enterprise developers/architects, for building service-oriented enterprise applications. For the first time we have a set of specifications that address how SOA can be built rather than what SOA is about. I have provided a bird's eye view of SCA as a technology.

The related concepts in detail

As stated before the main building block of a Service-Oriented Business Application is the component. Figure 1 exhibits the elements of a component. A component consists of a configured piece of implementation providing some business function. An implementation can be specified in any technology, including other SCA composites. The business function of a component is published as a service. The implementation can have dependencies on the services of other components, we call these dependencies references. Implementations can have properties which are set by the component (i.e. they are set in the XML configuration of a component).

Figure 1 - SCA Component diagram (taken from the SCA Assembly Model specification)

Components can be combined into assemblies, thereby forming a business solution. So from this point we get the meaning of ‘B’ for in what Business may refer to. The SCA calls these assemblies composites. As shown in figure 2 a composite consists of multiple components connected by wires. Wires connect a reference and a service and specify a binding type for this connection. Services of components can be promoted, i.e. they can be defined as a service of the composite. The same holds for references. So, in principle a composite is a component implemented with other components and wires. As stated before, components can in their turn be implemented with composites thereby providing a way for a hierarchical construction of a business solution, where high-level services (often indicated as composite services) are implemented internally by sets of lower-level services.

Figure 2 - SCA Composite diagram (taken from the SCA Assembly Model specification)

The service (or interface if you like) of a component can be specified with a Java interface or a WSDL PortType. Such a service description specifies what operations can called on the composite, including their parameters and return types. For each service the method of access can be defined. As seen before, the SCA calls this a binding type.

Figure 3 - Example XML structure defining a composite

Figure 3 exhibits an example XML structure defining an SCA composite. It's not completely filled in, but it gives a rough idea what the configuration looks like. The implementation tag of a component can be configured based on the chosen implementation technology, e.g. Java, BPEL, etc. In case of Java the implementation tag defines the Java class implementing the component.

Deployment and the Service Component Architecture contributions, clouds, and nodes

SCA composites are deployed within an SCA domain. An SCA Domain (as shown in Figure 4) represents a set of services providing an area of business functionality that is controlled by a single organization. A single SCA domain defines the boundary of visibility for all SCA mechanisms. For example, SCA service bindings (recall the earlier explained SCA binding types) do only work within a single SCA domain. Connections to services outside the domain must use the standardized binding types like web service technology. The SCA policy definitions do also only work within the context of a single domain. In general, external clients of a service that is developed and deployed using SCA should not be able to tell that SCA was used to implement the service, it is an implementation detail.

Figure 4 - SCA Domain diagram (taken from the SCA Assembly Model specification)

An SCA domain is usually configured using XML files. However, an SCA runtime may also allow the dynamic modification of the configuration at runtime.

An SCA domain may require a large number of different artifacts in order to work. In general, these artifacts consists of the XML configuration of the composites, components, wires, services, and references. We of course also need the implementations of the components specified in all kinds of technologies (e.g. Java, C++, BPEL, etc.). To bundle these artifacts the SCA defines an interoperable packaging format for contributions (ZIP). SCA runtimes may also support other packaging formats like JAR, EAR, WAR, OSGi bundles, etc. Each contribution at least complies to the following characteristics:

It must be possible to present the artifacts of the packaging to SCA as a hierarchy of resources based off of a single root.
A directory resource should exist at the root of the hierarchy named META-INF.
A document should exist directly under the META-INF directory named ‘sca-contribution.xml' which lists the SCA Composites within the contribution that are runnable.

A goal of the SCA approach to deployment is that the contents of a contribution should not need to be modified in order to install and use the contents of the contribution in a domain.

An SCA domain can be distributed over a series of interconnected runtime nodes, i.e. it is possible to define a distributed SCA domain. This enables the creation of a cloud consisting of different nodes each running a contribution within the same SCA domain. Nodes can run on separate physical systems. Composites running on nodes can dynamically connect to other composites based on the composite name instead of its location (no matter in which node the other composite lives), because they all run in the same domain. This also allows for dynamic scaling, load balancing, etc. Figure 5 shows an example distributed SCA domain, running on three different nodes. In reality you don't need different nodes (and even different components) for this example, but it makes the idea clear.

Figure 5 - A distributed SCA domain running on multiple nodes (taken from the Tuscany documentation)

Implementations of the Service Component Architecture

The SCA is broadly supported with both commercial and open source implementations.

Open source implementations:

Apache Tuscany (official reference implementation). The current version is 1.4, released in January 2009. They are also working on a 2.0 version which they aim to run in an OSGi enabled environment.
The Newton Project.
Fabric3. A platform for developing, assembling and managing distributed applications on SCA standards.Fabric3 leverages SCA to provide a standard, simplified programming model for creating services and assembling them into applications.

Commercial implementations:

Concluding…

Concluding with the analysis of this kind of alternative, concerning the SCA in combination with the overall and therefore holistic Reference Architecture for S.O.A., we have expose a step toward for a complete Architectural Design and a Guide to implement the proper scheme concerning the case that should be applicable in the case at hand. Both the concerns covered in the Reference Architecture approach and the SCA approach here, in hand with the strategic decisions that the organisation or the project has in ‘mind’ would be able to provide specific specifications (in the technical, business and conceptual analysis) about the meanings and approaches that the approach is aiming to take. The definition of ‘B’s and ‘A’s is therefore the mission of these approaches and the clearance of the meaning for the entities involved from a context based view.

So, as far as concerning the S.O.A. ‘infects’ that have to do with the proper definition of Architecture, we may have a logical solution from divide and conquer. Having as a next step after the Reference Architecture’s decisions, the Service Component Architecture (SCA) which provides the means for analysing and decomposing the Service Oriented Business Applications (SOBA) involved, we have a clearer decomposition and definition of the ‘B’ (– business meaning in case at hand) from the ‘A’ (– of the whole architecture and organisation or project).

notation & listings: The background information for this post, comes from SOBA (nice post), What every architect should now about sca (well form structure and the initiation point for enhancing the thoughts), AspireRFID (as the reason for expressing the need for alternative SOA case), the (SCA) Standard post, the SCA Raining on the JBI parade (for JBI relation with SCA as well for the movement behind SCA) side by side with the JBI?SCA and the what’s wrong & right with SCA (specially the second post for the useful features of SCA). Of coarse the Apache Tuscany, TIBCO, IBM, OSOA and OASIS links mentioned in the post.

The Class Casting Exception in OSGi Environments

2009-02-05T10:47:00.001+02:00

In previous posts, i have deal with the OSGi approach on building services, either the general approach or using iPOJO. These are ok, if one is willing to implement a startup of services bundles or some project from the start, or to develop a solution with a nature of separation or independency on existing IT assets. Unfortunately, this is rarely the case. Usually, there is either the need to consume some objects from existing applications or to just the other way around, to consume objects created in the OSGi environment. And real life, demands both. The first issue is normally easy to deal with. Actually, this is the way you should go on. Having in mind the S.O.A. enabling that your project, solution,architecture, code etc, should comply with, there is an issue in the horizon.

Demystifying the Issue one level down

Consider the common condition: You have a non-OSGi Java Application in which an OSGi-based application is embedded. You want to consume objects created in the OSGi-based application (an OSGi Service for example).

The above condition will provide us a problem issue: When you try to consume (invoking methods, assigning to local variables, etc.) the objects created inside the OSGi Environment from the non-OSGi application you get Class Casting Exception.

The reason for getting this exception, is that any class loaded inside the OSGi Environment is loaded from a special class loader provided by the OSGi Framework which is different from the class loader used by the non-OSGi application. So even if the class of the non-OSGi application is exactly the same as the one of the OSGi-based application, the objects will still be treated as of different types since their classes were loaded using different class loaders.

Getting around

If you want to perform on the object a single method invocation or two without passing parameters of conflicting types (the classes that loaded from different class loaders) just invoke the method using reflection, if you have a more complex case then use utilities that would allow you to bridge the classes.Here comes another key resource place for the OSGi’s: DynamicJava. You can overcome the above exception with the help of API-Bridge for instance, which provides such capabilities. Actually, this API was created to address this problem specifically.

Bridge objects act as their type is of the target class loader, so when the non-OSGi application works with the bridged object it will seem to it that it was loaded from the same class loader. Lets see an example that clarifies how API-Bridge functions.

API-Bridge

Below is a code snippet that loads the same class using two different class loaders and then bridges them using the API-Bridge provided by DynamicJava resources.

        1: public void testApiBridge() throws Exception {  

       2:      /// Class A which implements interface AInterface is loaded by two   

       3:      /// different class loaders so they are treated as different examples  

       4:      Class aClass1 = getClassLoader1().loadClass("org.test.A");  

       5:      Class aClass2 = getClassLoader2().loadClass("org.test.A");  

       6:      Class aInterface1 = getClassLoader1().loadClass("org.test.AInterface");  

       7:      Class aInterface2 = getClassLoader2().loadClass("org.test.AInterface");  

       8:        

       9:      /// Objects of AImplementation class loaded from Class Loader 1 are not  

      10:      /// assignable from objects loaded from Class Loader 2.  

      11:      assertFalse(aInterface1.isAssignableFrom(aInterface2));  

      12:        

      13:      Object a2 = aClass2.newInstance();  

      14:        

      15:      /// The object is not castable since it's an instance of a class  

      16:      /// that was loaded using a different class loader.  

      17:      assertFalse(aInterface1.isInstance(a2));  

      18:        

      19:      /// We create an API Bridge that bridges classes (their fields,  

      20:      /// method parameters, etc.) of the "org.test" package to  

      21:      /// the class loader 1.  

      22:      ApiBridge apiBridge = ApiBridge.getApiBridge(getClassLoader1(), "org.test");  

      23:        

      24:      Object bridgedA2 = apiBridge.bridge(a2);  

      25:      /// The bridged a2 object is now castable  

      26:      assertTrue(aInterface1.isInstance(bridgedA2));  

      27:  }

Although the example doesn't look very clear at first glance, it's important to note that we wrote only 2 lines of code to bridge the API. What we did up there is that we bridged an object and verified the fact is that it's castable to the AInterface of a different class loader.

While API-Bridge supports classes when used as implementations, super classes and method parameters, casting is only supported for interfaces.

Real World

An example of this problem is a Java Web Application run in an OSGi-oblivious Servlet Container (Tomcat for example), the Web Application has an OSGi-based application embedded inside of it. The OSGi-based application registers an OSGi Service that implements the javax.servlet.Servlet interface. The Web Application has a Delegator Servlet which delegates requests to the registered OSGi Service. The problem with this application is that the Web Application can't consume the Servlet object since it's loaded by a class loader of the OSGi Environment, the fact that causes ClassCastException to be thrown. To overcome this problem, I used the API-Bridge to bridge the Servlet API. Below is the ServletDelegator class which delegates requests to the OSGi Service and uses the API-Bridge to bridge the Servlet objects

       1: public class DelegatorServlet extends HttpServlet {  

       2:       

       3:     @Override  

       4:     protected void doGet(HttpServletRequest request, HttpServletResponse response)  

       5:             throws ServletException, IOException {  

       6:         BundleContext bundleContext = getBundleContext();  

       7:         if (bundleContext != null) {  

       8:             ServiceReference servletServiceRef = bundleContext.getServiceReference(  

       9:                     Servlet.class.getName());  

            if (servletServiceRef != null) {  

                /// We can't write a line like:  

                /// Servlet servletObject =  

                ///     (Servlet)bundleContext.getService(servletServiceRef);  

                /// because a class casting exception will be thrown since the  

                /// Servlet class in this code is loaded by Tomcat class loader   

                /// while the retrieved Servlet by the OSGi Class Loader.  

                Object servletObject = bundleContext.getService(servletServiceRef);  

                  

                ApiBridge apiBridge = ApiBridge.getApiBridge(  

                        Thread.currentThread().getContextClassLoader(),  

                       "javax.servlet", "javax.servlet.http");  

                Servlet servlet = (Servlet)apiBridge.bridge(servletObject);  

                /// Note that method parameters will be bridged too by API Bridge,  

                /// so they could be passed to the OSGi Environment.  

                servlet.service(request, response);  

            } else {  

                printHtml("Error: A Servlet OSGi Service was not found.",  

                        response);  

            }  

        } else {  

            printHtml("Error: Bundle Context was not found in the ServletContext.",  

                    response);  

        }  

    }  

      

    protected BundleContext getBundleContext() {  

        /// Here we retrieve the Bundle Context of the OSGi-based application  

    }  

      

    /// ... Other supportive code  

      

}

Alternative on Building OSGi service: build using iPOJO

2009-01-23T17:20:00.001+02:00

Continuing from a previous post on creating OSGi services, this posting, is making thinks more adaptable. The adaptation key here, is in iPOJO. iPOJO stands for injected POJO. It’s a component model from Apache on top of OSGi. It is very flexible, extensible and easy to adapt as already mentioned. It is a try to remove the overhead of developer’s handling services in OSGi. So, as before, let’s build a simple service using iPOJO and consume it.

OSGi is about modularizing your application. In this demo, we will have three projects (or three bundles) that will be deployed on Apache Felix. The needed ingredients are:

Problem Statement

The problem is very simple: Our service is a simple OSGi service that will display the string “Hi There!”.

Getting Started

Like any OSGi service, our service is represented as an interface:

package hi.service;
public interface HiService {
       public String sayHi();
}
This will be our first project or first bundle. The bundle will have just the interface. Please note that you will have to export hi.service package when building the bundle. To make this task easy, use the bnd tool to create the jar. I have used Ant script to compile and package the projects. Once bundle is ready, you will use it as a dependency library for compiling other projects.

Service Implementation

The next step is to implement our service. We will create a new project with previous projects jar file as a dependency. The service implementation is also POJO and there is no mention of OSGi service in the code. Let’s have a look at our implementation:
package hi.component;
import hi.service.HiService;
public class HiComponent implements HiService {
       String message = "Hi There!";      
       public String sayHi() {
         return message;
       }
}
Now we define that we have a iPOJO component. This is done through a xml that is placed along with the project. The xml defines that, we have a component of the class hi.component.HiComponent. The example below is a very simple one, you can have call backs, properties set etc in this xml using different xml elements.
<ipojo>
<component classname="hi.component.HiComponent">
 <provides/>
</component>
<instance component="hi.component.HiComponent"/>
</ipojo>
We will compile this project into another bundle that will be deployed in the runtime.

Using the Service

This is the final project. This project will be consuming the service we created above. The client can be a POJO or a Activator class. Like the service implementation code, we do not code for a service. Instead we go about and code as if the implementation is available to us. iPOJO framework would take care of the rest. Here is our "Hi there!" client:
package hi.client;
import hi.service.HiService;

public class HiClient {
    
     private HiService m_hi;

    public HiClient() {
        super();
        System.out.println("Hi There Client constructor...");
    }

    public void start() {
        System.out.println("Starting client...");
        System.out.println("Service: " + m_hi.sayHi()); 
    }
    
    public void stop() {
        System.out.println("Stoping client...");
    }
}
I have some sys outs to see how our client work. Just like the service project, we have a xml that will define what is the required service, callback functions etc. Have a look at the xml:
<ipojo>
<component classname="hi.client.HiClient">
 <requires field="m_hi"/>
 <callback transition="validate" method="start"/>
 <callback transition="invalidate" method="stop"/>
</component>
<instance component="hi.client.HiClient"/>
</ipojo>
The xml specifies that HiClient requires m_hi to execute. m_hi is a instance of our service. So as long as the service is not available, the HiClient component does not get executed. The callback xml elements specify which methods to execute when the state of the component changes.

Once compiling and packaging of this project is done, we are ready to deploy our example into a runtime and see it working. When you have Felix and iPOJO framework downloaded, you have to configure Felix to load the bundles when it’s started.

Felix configurations are placed in config.properties under conf folder.You will have to modify the entires for felix.auto.start.1 variable. Here is how it looked like after I modified:
felix.auto.start.1= \
 file:bundle/org.apache.felix.shell-1.0.1.jar \
 file:bundle/org.apache.felix.shell.tui-1.0.1.jar \
 file:bundle/org.apache.felix.bundlerepository-1.0.3.jar \
 file:bundle/org.apache.felix.ipojo-1.0.0.jar \
 file:bundle/hi.service.jar \
 file:bundle/hi.component.jar \
 file:bundle/hi.client.jar
I have put all my jars in felix/bundle folder. You may place them in different location but you must specify the correct paths above.

We are ready now. Run your felix runtime to see the results!

The OSGi and the Services issue

2009-01-23T16:59:00.001+02:00

Few months ago, i had introduce the Open Service Gateway initiative in a post, as trying to make a solid ground for the association of posts concerning the ServiceMix related posts, for introducing a simplistic usage of open source tools on S.O.A. related projects. From that time and on, we were introduced to a wide adoption of OSGi methodology and approach by developers and vendors (both legacy and open source). This fact gives the option and possibility to consider OSGi technology as technology that will transform Java development. Specially on the enterprise side.

To establish a point of useful OSGi, i am giving a quick example on how to build a simple service and consume it. This is quite useful in the S.O.A. development, considering it as a step in the development tasks of the reference architecture that one should follow in a S.O.A. enabled solution and approach. Specially when in a small team the object is cheap (cost and time) development considering services.

You may start and work with Equinox and Eclipse IDE. You may also use other OSGi runtime like Apache Felix or Knopflerfish. Knopflerfish even gives you are good GUI to work with. I will not be explaining fundamentals of OSGi technology.The goal here is to be a little bit more simplistic and independent, in order to be able to work with any other IDE, approach or tools you might prefer. So here we are concerned with a simple application (bundle). Lets start with.

What's OSGi service?

In very general terms, conforming with S.O.A. re-use, a service is a … repeatable task. When it comes to business, any repeatable task in your business process is a service. Similarly in a application, you can have generic tasks (even specific tasks) that are repeatedly used and therefore, can be represented as service. Representing and using these tasks as services is what SOA is all about! But that' at an enterprise level. When it comes to OSGi services, it is the same concept but applied at JVM level.
In OSGi, a service is a plain java object which is published to a registry. A consumer can consume the registered service through lookup. A service a be registered and unregistered at any point of time. Service is built using interface-based programming model. To implement or build a service you basically provide implementation to a interface. To consume, you only need the interface for the lookup and there is no need to know about the implementation. The service registry is the "middle man" who help producers and consumers to get in touch with each other.

Building STARTUPOSGISRVservice

The first step would be to create our interface or "front end" of the service. For our service, we will have a simple interface named IStartupOSGISRV:

package org.my.service.startupservice;

public interface IStartupOSGISRV {
 public String sayHi();
}
And here is our service implementation.
package org.my.service.startupservice;

public class StartupOSGISRV implements IStartupOSGISRV {
 public String sayHi() {
  return "Hi There!";
 } 
}
That's it! Our service is ready for use. But, we need to inform consumers that the service is ready to serve. For this, we will have to register our service with the OSGi service registry.

OSGi framework provides us with standard APIs to register and unregister service with the registry. We will use the registerService method to register as shown below:
serviceRegistration = context.registerService(IStartupOSGISRV.class.getName(),startupservice,null);
I am sure for beginners this is not enough. Let's explain the stuff little further.To register our new service, we will build a simple bundle that will call registerService method.
package org.my.service.startupservice;

import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.ServiceRegistration;

public class Activator implements BundleActivator {

 private ServiceRegistration serviceRegistration;
 private IStartupOSGISRV startupservice;
 
 public void start(BundleContext context) throws Exception {
  System.out.println("Starting StartupServiceBundle..");
  startupservice= new StartupService(); 
  serviceRegistration = context.registerService(IStartupOSGISRV .class.getName(),startupservice,null);
  
 }
 
 public void stop(BundleContext context) throws Exception {
  serviceRegistration.unregister();
 }

}
Our Activator class implements BundleActivator. Basically, its a simple OSGi bundle with start and stop methods. We will register our service with the bundle starts up and unregister when the bundle is uninstalled from the framework.

Now lets have a closer look at start method. We create a instance of our service and then use registerService method. The first argument is service name which is obtained using InterfaceName.class.getName(). Its a best practice to use this method instead of specifying the name as string (org.my.service.startupservice.IStartupOSGISRV). The second argument is the instance of the service itself. And the final argument is Map wherein developers can pass additional properties to the service.

To unregister the service, you simple call unregister method when we stop the bundle. So now we have a running service on our OSGi runtime. Now we must consume it.

Consuming a service

To consume a service, we first create serviceReference object form the BundleContext. This can be achieved by calling getServiceReference method. The method takes the class name as a argument. Once you have the serviceReference object, we will use getService method to finally get the service. We will have to typecast the object returned by getService method before using it.
startupserviceRef = context.getServiceReference(IHelloService.class.getName());
IStartupOSGISRV serviceObjectStartupService = (IStartupOSGISRV)context.getService(startupserviceRef);
System.out.println("Service says: " + serviceObjectStartupService.sayHi());
Implementing the service and consumer is the same package is easy. Because, the interface is available. When you have your service and consumer bundle separate, there are some important points to note. OSGi provides the capability of specifying the packages they can be exported or imported. With this facility you can expose your service interface and hide its implementation from the public. The configuration details are specified in the MANIFEST file. Have a look at our StartupService's MANIFEST file:
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: StartupService Plug-in
Bundle-SymbolicName: StartupService
Bundle-Version: 1.0.0
Bundle-Activator: org.my.service.startupservice.Activator
Bundle-ActivationPolicy: lazy
Bundle-RequiredExecutionEnvironment: JavaSE-1.6
Import-Package: org.osgi.framework;version="1.3.0"
Export-Package: org.my.service.startupservice;uses:="org.osgi.framework"
Notice that we have exported org.my.service.startupservice package. Similarly, we import this package in our consuming bundle.

And to add some notes; The code used for consuming the service is not the best way. You should make the code very simple and easy to understand without involving Exceptions handling,Null pointer checks and ServiceListeners.

On the Reference Architecture for SOA Adoption

2009-01-20T15:31:00.002+02:00

While talking with various people and thinking over the case of S.O.A. (note not SOA) in order to explore in more depth the holistic view needed, i came across Hariharan’s post on Reference Architecture for SOA Adoption. Hariharan states:

Fundamental objective of SOA is aligning your IT capabilities with business goals. SOA is not just IT architecture nor business strategy. It should be perfect combination of IT infrastructure and business strategy. So, when we are planning for SOA adoption, we need a strong roadmap and blueprint. Reference architecture model will be a blueprint of SOA.
Reference architecture is like a drawing of your corporate building. Before and after construction, you need blueprint for verification and reference. Similar to that SOA reference architecture could be a reference model for your enterprise business system. Reference architecture should be defined properly during the SOA roadmap definition phase. Refer my last blog in which I was talking about SOA roadmap.
Fundamentally, while defining reference architecture model for corporate, we should consider the following components as part of architecture.
1. Infrastructure and components services
2. Third party communication and data sharing services
3. Business rules services
4. Business process services
5. Data sharing and transformation services
6. Identity and Security Services
7. Packaged Application access services
8. Integration and Event management services
9. Presentation Services
10. Registry and Repository
11. Messaging and Quality
12. Governance
Currently, if you look at the SOA market, there are many product and service providers comes up with its’ own SOA reference architecture. Many major SOA players like IBM, TIBCO, Web Methods, BEA Oracle and MGL has defined reference architecture based on their product catalog and services. If we refer, all has its own model with explanation which may puzzle the SOA adaptors to choose the right approach. One important point should be remembered that product vendor or service provider’s reference model may not suit for your requirements completely. We should consider that as just one of the reference point to finalize the vendor. We should design the right blueprint model for our corporate and SOA need. This is a crucial step that should be planned as part of the SOA implementation roadmap.

Good starting point. But more or less, again the ‘bird eye’ view needed for S.O.A., is not complete. The reason, is services (and actually web-services) oriented view. As stated here S.O.A. is more than integration. It is integration as well, as is about the following of strategies and the future of the in hand infrastructure. So as concerning the types 1,2,6,8 and 12, the reference architecture should avoid or be irrelevant (as possible) to the ‘marketing effect’ proposed by vendors, as stated here. This last issue is recognized in the above statement in the last paragraph when stating :

“Many major SOA players like IBM, TIBCO, Web Methods, BEA Oracle and MGL has defined reference architecture based on their product catalog and services. If we refer, all has its own model with explanation which may puzzle the SOA adaptors to choose the right approach. One important point should be remembered that product vendor or service provider’s reference model may not suit for your requirements completely”

but not enough explored. In any way, i am not against any vendor, or trying to construct a polemic view against them when referring to ‘marketing effect’. They are trying to do business and make money, extent their ROIs and so on. Thats ok. My point is that we must have in mind the inner or hidden points on every co-operator when making our business or working.

So keeping in hand the 12 above points, but we have to be more holistic and therefore more concrete and safe in our reference. This reference is going to be the ground to explore our enterprise niches and evolve through market changes. A nice starting point in what a reference architecture should look like, is proposed from OASIS, among others proposals and standards, in SOA Reference Architecture. Compliant to this reference is the Conceptual Integration Technical Reference Architecture from the Washington State Department of Information Services. In the conceptual Integration Technical R.A. there is this diagram:

The conceptual reference architecture depicted in the diagram above (and defined in this document) adopts the Service-Oriented Architecture Reference Model (SOA-RM) developed by the Organization for the Advancement of Structured Information Standards (OASIS).

The SOA-RM defines its purpose as follows:

“A reference model is an abstract framework for understanding significant relationships among the entities of some environment. It enables the development of specific architectures using consistent standards or specifications supporting that environment. A reference model consists of a minimal set of unifying concepts, axioms and relationships within a particular problem domain, and is independent of specific standards, technologies, implementations, or other concrete details.” ([SOA-RM], p. 4).

“The goal of this reference model is to define the essence of service oriented architecture, and emerge with a vocabulary and a common understanding of SOA. It provides a normative reference that remains relevant for SOA as an abstract and powerful model, irrespective of the various and inevitable technology evolutions that will impact SOA.” ([SOA-RM], p. 4).

As you can see, the Reference is quite abstract. While the SOA-RM is a powerful model that provides the first vendor-neutral, open-standard definition of the service-oriented approach to integration, its abstract nature means that it is not capable of providing the architectural guidance needed for the actual design of integration solutions (or integrated software systems). That guidance comes from the definition of a reference architecture that rests on the foundation of the reference model’s concepts. The reference architecture builds on those concepts by specifying additional relationships, further defining and specifying some of the concepts, and adding key (high-level) software components necessary for integration solutions.

In the diagram above, SOA-RM concepts are shaded yellow. Concepts and components particular to the conceptual reference architecture defined by this document are shaded cyan. Relationships between concepts (indicated by arrows) are defined in the SOA-RM if the arrows connect concepts shaded yellow. Relationships between cyan-shaded concepts or between cyan-shaded and yellow-shaded concepts are particular to the reference architecture.

In order to conclude with Hariharan’s 12 points, which they try to describe and define in a next detailed plan the web services characteristics, we need to have some preliminaries which should work as the base for the next step Hariharan’s analysis:

Firstly, in the context of the organisation in hand (for whom we will define a more solid S.O.A. Reference Architecture), there should be an analysis of the models that exist and the models that we will going to need in the near future (obtained from the strategic decisions in short and long term of the organisation). So analysis of Models of the various entities that play crucial role in the infrastructure. In general these entities lying into 2 main categories:

Data Modelling
Generic Entity Modelling

What are the nature and model of the data need to be interchanged inside the infrastructure? what formats, categories and relations (dependencies) they follow? This analysis some times maybe needed in a system’s view note. From the point of view that the main system A used in the organisation is handling this data inside it (e.g. time/date/currency/locationGIS/Post code etc…).

What are the main compound entities that they exist and should be interchanged inside the organisation’s infrastructure? How they depend on the predefined data models above? E.g. Customer, Order, Products etc which they will be analysed in the context of main data models and their dependencies and more over, they will incorporate any missing data (used) elements missing from the Data Modelling analysis, depicting relations hierarchies etc. (Usually any missing attributes for the Data Modelling, should be declared and bound on the Entities relations and modelling).

So, now, we can describe our flows (existing) and found the business and IT rules that are needed in order to be able to have some services on the fly or from compounding existing ones. These rules and analysis, will show as if there is a need for the ‘canonical modelling’ that we are using. Depends of coarse on the systems used, to be used and therefore from the strategic decisions taken by the organisation’s next steps (e.g. in Telco's double/triple play which demand an extension to the product models) and of coarse from the vendors compliant modelling schemes of the existing or new systems to be used. Therefore, Hariharan’s list is extending to:

A1.1. Existing Data Modelling
A1.2. Existing Entities Modelling
A1.3. To-Be Data Modelling
A1.4. To-Be Entities Modelling
1. Infrastructure and components services
2. Third party communication and data sharing services
3. Business rules services
4. Business process services
5. Data sharing and transformation services
6. Identity and Security Services
7. Packaged Application access services
8. Integration and Event management services
9. Presentation Services
10. Registry and Repository
11. Messaging and Quality
12. Governance

Now we have the flows parts. Again, having the A1.1-A1.4, we can construct the existing data and entities flows, meaning a more generic Business flows that although they correspond to business analysis, they include the IT –systems steps for complying with the Modelling restrictions by organisation’s infrastructure. These are not only services, although if we generalise the term service, we are in. There might be p2p systems interconnections, hidden to the business flow for demanding, retrieving and consolidating data and so on. \

So we should obtain also an

A2.1 Existing Flows
A2.2 New-proposed Flows

from this point we should be able, having

a) our system’s capabilities
b) our modelling rules

The candidate services, which comply with A2.1 and A2.2 and create the web services (or the RPCs, XMLs, SQLs and so on) for obtaining the repository of available actions. So the list becomes:

A1.1. Existing Data Modelling
A1.2. Existing Entities Modelling
A1.3. To-Be Data Modelling
A1.4. To-Be Entities Modelling
1. Infrastructure and components services
2. Third party communication and data sharing services
A2.1 Existing Flows
A2.2 New-proposed Flows
3. Business rules services
4. Business process services
5. Data sharing and transformation services
6. Identity and Security Services
A6.1. Services Decomposition Strategy
7. Packaged Application access services
8. Integration and Event management services
9. Presentation Services
10. Registry and Repository
11. Messaging and Quality
12. Governance and Orchestration

With these in mind we should be able to develop an Decomposition Strategy (A6.1 in the list) by which we should be able to deconstruct and reconstruct meaningful information from existing services, categorise all the possible services to be used in the organisation, be able to have loosely coupled services and explore with the best possible way under the organisation’s context the services orchestration abilities and of coarse re-usability. In all the above we should have in mind Mike Kavis post concerning the Top 10 Reasons Why People Make SOA Fail.

[

They fail to explain SOA's business value.
They underestimate the impact of organizational change.
They fail to secure strong executive sponsorship.
They attempt to do SOA "on the cheap."
They lack the required skills to deliver SOA.
They have poor project management.
They think of SOA as a project instead of an architecture.
They underestimate the complexity of SOA.
They fail to implement and adhere to SOA governance.
They let the vendors drive the architecture.

]

If you have any additions or need help constructing a more concrete and specific Reference model for your instance, give a drop. All comments are welcomed.

S.O.A.s ‘nature’. On the integration issue – Web Services: soapbuilders reloaded and the prisoner’s dilemma

2009-01-07T16:10:00.001+02:00

As stated in S.O.A.s ‘nature’ and integration, S.O.A. is more than integration. But one of the most practical and real-world aspects and promises of the SOAs is integration. Especially integration from the interoperability point of view. In this aspect, the issue is behind the Web Services Interoperability abilities, that they could (after appropriate analysis and models consolidation) can provide to the proposed solution.

Interoperability is one of the design principles that distinguish Web Services as service middleware from its forefathers: RPC and distributed object computing and from its uncle: message-oriented middleware.

Over the last couple of years there has been a trend toward vendors offering complete SOA suites, focusing on strong cohesion (real or imagined) at the suite level. In this model, there's less motivation for cross vendor interoperability and open integration. This trend exposes the dangers of going with a SOA suite: vendor lock-in; isolated, non-integrated components; and exposure to the risk of needing ongoing consulting services for bespoke integration. As vendor consolidation slows this year, but continues, infrastructure suites will continue to dominate and selecting products to populate a heterogeneous services infrastructure will remain difficult. Actually, this is the ‘marketing’ case of the SOA adoption, instead of S.O.A., as discussed in SOA marketing and expenses. On the misleading promise on SOAs and what is dying?.

Under this assumptions, and any activity that tries to provide a solid ground for neutral, or open interoperability should be welcomed. So I’m glad for this announcement of the Web Services Test Forum (WSTF), a vendor and end-user coalition set up to test Web Services interoperability scenarios. The approach is described in this essential post by Oracle’s Gilbert Pilz and another by Doug Davis from IBM.

December 8, 2008 – The Web Services Test Forum (WSTF) launched today, providing an open community to improve the quality of the Web services standards, with initial membership from Active Endpoints, AIAG, Axway, CISCO, eviware, FORD Motor Co., Fujitsu, Hitachi, IBM, Oracle, Red Hat, Software AG, Teamlog, TIBCO Software Inc, . Using customer-based scenarios, interoperability is validated in a multi-vendor testing environment. Customers and vendors alike, independent of their geographic location, can dynamically test their applications against available implementations to ensure interoperability is preserved. As an open community, WSTF has made it easy to introduce new interoperability scenarios and approve work through simple majority governance.

So it seems, that the fore mentioned ‘marketing’ SOA issues, were in vendors attention, and some ‘fix-ing’ actions is there to take place. In a long term of coarse, but is something.

This forum is an evolution in the interoperability story begun by the soapbuilders group. This group of SOAP stack vendors and interested 3^rd parties was created in 2001 – kicked off by this seemingly innocent email from Tony Hong. Credit goes to IBM and Microsoft for nurturing the idea with support. The work of soapbuilders was carried on by Web Services Protocol Workshops that Jorgen Thelin ran at Microsoft.

And the ‘fix-ing’ action proof:

We need ask ourselves, why is this group coming together now? Why has the prophecy not come true? Vendors are being beaten up (not enough, IMO) by end-users for their interop failings – so in one way it’s a “look, we are doing something” measure.

Also, how does this relate to the work of the Web Service Interoperability Organization (WS-I)? Is this an implicit sigh of desperation at the lack of progress at the WS-I? For example, it’s taken a long time to get the crucial Reliable Secure Profile out of that hopper.

On the WSTF announcement call, Burton Group’s Anne Thomas Manes asked why this effort is not driven through the WS-I. Steve Harris from Oracle explained that while WS-I is a consensus driven effort, WFTF brings a different approach to interoperability. Harris highlighted vendor collaboration and lowering the barriers to entry as differentiators. IBM's Karla Norsworthy added that WSTF is complementary to WS-I, a more lightweight approach, giving the example that the forum could easily bring a few vendors together to test a scenario. Many of the WSTF’s 15 members are also members of WS-I.

Will WSFT make a real contribution to interoperability?

I’m impressed and made hopeful by a number of things. Firstly, that membership obliges a vendor to support live endpoints for each scenario. This means that debate and negotiation will not be the deliverables of the forum; live test results, published on the internet will be. We can view testing between the implementations as a fully connected network, i.e., every endpoint is tested with every other, so each additional implementation adds considerably (as c=(n^2 – n) / 2 for you topology geeks) to the level of interoperability assurance.

Secondly, having end-users like Ford and AIAG in the forum is an important step. Sure, there are a small number of end-users right now, but the usefulness of this forum grows dramatically with the number of end-user enterprises participating. The forum process encourages end-users to submit requirements for the tested scenarios which usefully turns attention onto things other than wire formats.

Actually, the real question is whether the participants, especially the vendors, recognise this opportunity as a prisoner’s dilemma scenario? If everybody co-operates, everybody benefits.

A number of issues may limit the WSTF’s impact. Clearly, not having Microsoft as a member is a problem. While industry manoeuvres might keep analysts and vendors themselves amused, end-users couldn’t care less; they just want software to work. So having Microsoft involved is a core credibility issue for the WSTF. According to WSTF spokespeople, other members will be hosting Microsoft endpoints. That’s not quite the same thing. Microsoft are committed to WS-I and also to Apache Stonehenge which has similar aims to WSTF.

I think other web services approaches need to be included. To launch an initiative purely directed at Web Services in December 2008 looks antiquated, although the IBM and Oracle spokespeople did claim other approaches, like RESTful web services are not excluded in theory.

As with any community based activity, the only real measure of its success is its interactivity. Let’s give WSTF time and then count the live interoperable implementations and end-user organizations active in the forum.

SOA marketing and expenses. On the misleading promise on SOAs and what is dying?

2009-01-07T15:43:00.001+02:00

Continuing thinking over the “SOA and the real Service Integration issue”, I fall on a tone of new-years posts concerning the ‘2009s SOA death’. Once again, I believe that the whole thing is about conceiving SOAs. I post a draft thought concerning the way that SOA need to be conceptualise both from Architects, Developers and Business owners perspectives. In most cases, the ‘philosophy’ of analysing requirements, services and environments for SOA, didn’t had the appropriate ‘holistic’ nature. So, lots of problems raised, both in financial aspects but also in designing principles. All the above, and more were stated by Mike Kavis in his post Top 10 Reasons Why People Make SOA Fail.

[

They fail to explain SOA's business value.
They underestimate the impact of organizational change.
They fail to secure strong executive sponsorship.
They attempt to do SOA "on the cheap."
They lack the required skills to deliver SOA.
They have poor project management.
They think of SOA as a project instead of an architecture.
They underestimate the complexity of SOA.
They fail to implement and adhere to SOA governance.
They let the vendors drive the architecture.

]

Dave Linthicum summed it up in his post when he stated:

[

First, there are not enough qualified architects to go around, and you'll find that most of the core mistakes were made by people calling themselves "architects," who lack the key skills for moving an enterprise towards SOA. They did not engage consultants or get the training they needed, and ran around in circles for a few years until somebody pulled their budgets.

Second, the big consulting firms drove many SOA projects into the ground by focusing more on tactics and billable hours than results and short- and long-term value.

Third, the vendors focused too much on selling and not enough on the solution. They put forth the notion that SOA is something they have to sell, not something you do.
Finally, the hype was just too much for those charged with SOA to resist. Projects selected the technology first, then the approach and architecture. That's completely backwards.

]

Anne Thomas Manes blogged the SOA is Dead;Long Live Services, and stated that “SOA is survived by its offspring: mashups, BPM, SaaS, Cloud Computing, and all other architectural approaches that depend on “services”.” The logic that follows her thoughts is more or less :“Successful SOA (i.e., application re-architecture) requires disruption to the status quo. SOA is not simply a matter of deploying new technology and building service interfaces to existing applications; it requires redesign of the application portfolio. And it requires a massive shift in the way IT operates. The small select group of organizations that has seen spectacular gains from SOA did so by treating it as an agent of transformation. In each of these success stories, SOA was just one aspect of the transformation effort. And here’s the secret to success: SOA needs to be part of something bigger.”

And I fully agree with this last statement. Actually, don’t forget that SOA = Service Oriented Architecture.

The illusions of the above SOAs concepts, become when considered the SOA as option, tool, software platform or some product under the SOA – name umbrella that will magically solve all problems. More or less, was a marketing oriented illusion from some (actually the biggest) vendors on the street. And all this illusions, combined with the financial aspects (more and more and expensive new software, service from vendors and external co-operators, consulting etc ) gave a avoiding aroma in the real S.O.A. approach. Ok, if you mean SOA (the above marketing term) and S.O.A. (the Service Oriented Architecture) i agree, that if its not dead it soon will.

As stated in this blog in a numerous posts, SOA actually is not expensive. Both the open source community and some vendors, have release tools and frameworks that can do the job, or parts of the job. The only consideration is the correct definition of the job, aka the A on S.O.A.s . Actually, all posts under the SOA and ESB labels are toward this concept.

Anne Tomas Manes, blogged on ‘SOA doesn’t need to be expensive’ the main in-expensiveness that S.O.A. could have (and not SOA):

[

It's a common misconception that SOA is expensive. Many organizations believe that they need to acquire a boat-load of new products and technologies to get started with SOA. First on the list of product acquisitions is an ESB, followed by registries, repositories, and security appliances. In these belt-tightening times, many SOA initiatives will be challenged to raise the funding required to acquire these products. So what's a team to do? Pack up and wait for better times? Or make do with what you have?

In truth (and much to the vendors' dismay), you don't need a bunch of new products to do SOA. SOA is about the way you design your solutions -- not about the technology you use to build them. An ESB is a "nice to have", but it's not a prerequisite. Pretty much all programming environments (Java, .NET, Ruby, Groovy, PHP, JavaScript, COBOL, CICS, etc) now include native support for building services -- both method- or document-oriented services built with SOAP and WSDL and RESTful services built with HTTP. You can also build document-oriented services using your favorite message-oriented middleware product (although MOM protocols will limit your reach and interoperability options). (I described the differences among the three types of services here. And see here and here for some great references on RESTful services.)

The only new product that an organization should really budget for is a management solution -- one that supports administration, monitoring, security, and mediation of service interactions (e.g., AmberPoint, Progress Actional, SOA Software Service Manager. Many platform vendors also provide management solutions--sometimes reselling AmberPoint [Oracle and Tibco]).

For organizations that need an ESB, consider open source solutions. Mike Kavis posted a nice summary of open source SOA stacks here. David Linthicum has also been extolling the benefits of an open source SOA product strategy here. Both Mike and Dave point out that the open source solutions tend to be easier to use and more cohesively integrated than the big vendor alternatives. Unfortunately, none of the open source solutions provides a comprehensive management solution yet.

]

the reason that i am using this part of Ms Manes post, is that i feel that describes the best my overall thesis on this subject and hope yours to.

If you need more applied, real world arguments on this, or how can be applied into your case, please drop me a notice.

Message Auditing in SOA

2008-12-29T13:55:00.001+02:00

Any serious service-oriented system implementation can be comprised of a number of services assembled into compositions. Actually, this is a strategy for overcoming, or partially solve the problem of web services “sea” as posted before. When you have a maze of services being called from many different clients, it can easily result in a formidable amount of information being passed around in form of messages. If principles of service-oriented design are followed, these messages would be in a standardized format. leading to a variety of possibilities as to how mechanisms can be positioned to filter and persist the messages and to also extract business intelligence from them in an arbitrary manner. The impact of standards-based messaging on auditing is as deep and as sweeping as its impact on the field of integration in general.

A Case for Auditing Messages in Service-Oriented Systems

Not much attention has been given to auditing messages in service-oriented eco-systems. Most modern infrastructure and service bus platforms provide simple message auditing mechanisms, for example, by writing messages to a log file or in a proprietary database. Typically, these mechanisms are not optimized for performance and are recommended for diagnostic purposes only.

This traditional ignorance towards auditing is most likely due to the fact that auditing is usually not seen as primary business need. Also, the requirements for auditing can be so daunting and unpredictable that it can be very difficult to comprehend and achieve in a consistent manner, especially across heterogeneous environments.

With the advent of service-orientation, service design principles, standardized message structures and commercial infrastructures that directly enable service-oriented computing, it is becoming significantly easier to build a system that is generic enough to plug into a variety of platforms and also specific enough to capture only certain messages on an "as need" basis. As a result, we can now build message auditing systems without having to creep inside of and customize the service logic.

This effectively enables us to:

respond to arbitrary queries pertaining to compliance requirements
gather business intelligence from arbitrary perspectives by running queries against persisted messages
set up observation systems to raise alarms when certain events happen
extract diagnostic information about systems to better optimize their resources
create a dashboard to observe the overall state of systems in real-time

Let's now explore these requirements in more detail and then establish a high-level design and implementation.

Typical Requirements

Any reasonable service-oriented implementation can have multiple services (and even multiple versions of services) along with multiple XML schemas. The services may be implemented on more than one platform and may be called by diverse clients. The volume and size of messages are not predictable – because well-designed services are typically interoperable, reusable and compassable, we cannot predict how and when new usages of existing services will emerge.

This leads to the following requirements that a message auditing system may need to address:

id	Requirement	Description
1	Flexibility to Handle Messages that may have Different Structures	The auditing system cannot assume any specific structure or schema for the messages. A request or response message at one service may look very different from the request or response at another service. Even different versions of same service may have different message structures
2	Ability to Specify Criteria to Filter Messages so that only Specific Messages are Audited	The message auditing system must support message filtering to specify which messages really need to be audited. It is not incomprehensible to visualize that not all messages need to be audited. An ability to filter the messages reduces the load on the auditing system and creates a meaningful message warehouse, which can be easily managed and processed to gather on-going business intelligence. The challenge in this requirement is that since the messages can be of any structure, the criteria for filtering can also vary.
3	Ability to Map messages to the Service Instances from which they Emanated	The message auditing must be able to support the mapping of an audited message to the instance of the service from which it originated, so that, in case more details have to be found, the log files on the server may be referenced. In the absence of this type of mapping logic, it may be difficult to gain the needed amount of insight into how message were previously processed.
4	Ability to Support Multiple Destinations where Message need to be Stored	The message auditing system must be able to support multiple destinations for the audited messages. Typical examples of where the message may be sent are: the database, e-mail, files and folders, queues and other services. The database is commonly considered the single most important destination, since it is permanent and can be used to generate business and reporting intelligence. However, it is important to support other destinations for real-time reporting, events, and for help with diagnosing problems
5	Ability to Scale to Accommodate Increasing Load	As mentioned previously, because it is impossible to predict how a service eco-system will grow and evolve, it is also not possible to accurately estimate future loads placed upon the message auditing system. Therefore, the system needs to be inherently and seamlessly scalable so that any required infrastructure upgrades can be added without affecting existing services
6	Must be Not be Intrusive to Service Logic	An auditing system should not impose significant change upon existing services. Services must be designed so that their autonomy is maximized while their coupling to the infrastructure is minimized. When incorporating a an auditing system (be it accessed directly by service logic or made available via separate utility services), care needs to be taken so that it should not introduce negative service coupling requirements.
7	Must be Autonomous	Message auditing systems should not be dependent on services. They may depend on the standardized schemas being used to define service contract types (following the service design principle of Standardized Service Contract), but they should not have any direct dependencies on business services, their message structures, or service logic. This enables auditing services (or systems) to exist as independent parts of the enterprise with their own life cycles.
8	Must Act as an Optional Feature	Not all services need auditing and those that do may not need auditing all the time. Message auditing logic must therefore be designed to be as "pluggable" as possible, so that it can be easily enabled and disabled without affecting services and their supporting infrastructure.
9	Must be Flexible and Extensible	The auditing requirements can be so different from one organization to another or even from one service inventory to another in the same company, that the auditing system should not be built to address a predefined set of auditing requirements. Instead, the system must act as a platform which can be customized or extended to accommodate evolving requirements.
10	Ability to Define Arbitrary Queries for Reporting or Event Generation Purposes	Ad-hoc reporting is often ignored in traditional auditing systems. In order to support this requirement, messages must be stored in a manner that makes it is possible to query them to generate both reports and events. Keeping in mind that each service operation has its own message structure, it must be possible to specify the query in an arbitrary manner. This is also due to the fact that the requirements for reports and events cannot be predicted since the auditing on a system may lead to the need for ad-hoc queries.
11	Ability to Generate Automated Reports	It has become increasingly common for systems to comply to regulation-based auditing requirements. With the advent of Sarbanes-Oxley Act in particular, companies are required to provide high-quality reporting capabilities in order to respond to auditing-based compliance queries.

Solution

There are several SOA infrastructure platforms that simply have insufficient auditing capabilities. Building your solutions on such a foundation may force you to later adopt a third-party auditing solution. However, auditing requirements can become so varied that it simply may not be feasible to build or use one system that can address them. Therefore, you need to place reasonable assumptions and constraints on what a given auditing system can and cannot do.

For example:

Service instances must be identifiable by some unique ID that can be stored as metadata along with the messages.
The auditing system should not be responsible for decrypting messages.
The auditing system may be allowed to add its own information as headers to the messages.
It must be possible to identify the version of service, operation and schemas being used from the message.

To limit the scope of a message auditing system, the following points must also be considered:

Auditing logic should not be used for routing purposes. The message processing chain should be part of service-logic.
Sometimes, the word "auditing" is used to describe a business requirement for applications. For example, in a customer management application, the administrator must be able to see the history of how customer information was modified and by whom. This should not be confused with "message auditing", which is logic that is considered to be part of an agnostic system (a system that is not bound to a specific business context).
Auditing systems may alter messages by adding their own header information. This should not be considered as a "breakage" in the message integrity because these headers are generally used for auditing purposes only.
The message store should not be used for non-repudiation. This is again is due to the fact that headers may be added to the message for auditing.

Architecture and Design

There can be many different ways to build an auditing system. Here, we will explore one possible approach based on the context and requirements discussed so far.

Overall the solution can be broken down into the following parts:

Service Instance Manager
Message Interceptor
Filter Manager
Filters-Service Instance Mapper
Destination Manager
Filter-Destination Manager
Report Manager
Report Scheduler

1	Service Instance Manager	The message auditing must be done in a way that it should be possible to track from which instance of what service the message was captured. So, it is important to have a mechanism in place that consistently and uniquely defines each service instance. It may also be a good idea to implement a self-registration process so that when an instance of a service is started, it can register itself in a database.
2	Message Interceptor	The capturing mechanism must be pluggable so that it is easily enabled or disabled. It has to be non-intrusive to service logic and to enable high levels of scalability, it should be designed to simply capture and send messages to a queue. This module is also responsible for gathering metadata about the message (e.g. the service instance identifier to map the messages to the service log files). All of this metadata must be either added to the messages via headers or it may be captured as separate fields. The module must be able to capture request and response messages together and send them as pairs for auditing. This ensures that request and response message information will be bundled for future processing and reporting purposes. This module should not apply filters to the messages since that may introduce time-consuming logic that could adversely affect the runtime service performance. Typically, a message capturing mechanism can be developed by implementing message handlers or interceptors.
3	Filter Manager	A filter is a simple function that would take request and response messages and associated metadata and determine whether a given set of messages needs to be audited or not. Typically a filter would look into the message content and apply a rule to find out if certain conditions resolve to true or not. Some common types of filters that can be implemented and readily used are • look for a certain string in the message to be present or absent • apply a given XPath query to the message that results in a Boolean response • apply a given XQuery query to the message that results in a Boolean response • apply interfaces to define arbitrary criteria based on the message content While defining the filter, it must be indicated whether to apply it on the request or the response. The definitions of these filters may be created and stored in a database or configuration file
4	Filter-Service Instance Mapper	A mechanism to map the filters with service instances is needed to determine which filters to apply for a given set of messages and their metadata. This can also be done in a database or a configuration file. Note, however, that a database may be more suitable do to the need to sometimes establish many-to-many relationships.
5	Destination Manager	The destination for filtered messages represents the location they need to be sent to for storage. Some possible destinations can be the database, a queue, e-mail, files and folders, URLs, other services, or any implementation of a simple interface that can accept the request, response, message metadata and filter and is able to send them to whatever destination they need to go. All destinations may be stored in configuration file or a database.
6	Filter-Destinations Mapper	It needs to be specified for each filter where the messages must be sent. This can be achieved by specifying a list of destinations for each filter register.
7	Report Manager	The database is a mandatory destination if it is important to run reports on audited messages. Defining reports at a basic level would require setting up query that needs to be executed to select only relevant messages and further provide some way to render these messages. A simple yet powerful solution for this is storing the messages in XML type columns and specifying queries in terms of XPath or XQuery queries. As mentioned previously, the messages may be based on an arbitrary structure and it may therefore be easy to specify rendering formats with technologies like XSLT
8	Report Scheduler	This is a regular scheduling mechanism that can run reports based on the provided schedule and report definition.

Putting It All Together

Figure 1 identifies some of the interfaces that will enable the creation of the previously listed solution parts. For the sake of brevity, only important interfaces are listed.

Figure 1

Once all these components are in place, we can study how the system will work at runtime and how the original requirements can be met. At service start-up, the service instance must register itself with a central registry as displayed in Figure 2.

Figure 2: At service startup, the service instance object is created and registered

The following steps demonstrate how the auditing system would work at runtime:

A service client sends a message to a service instance.
The auditing interceptor captures the service instance identity, request, and its metadata, and stores this information locally.
The service logic processes the request and constructs the response.
The auditing interceptor captures the response.
The auditing interceptor combines the request, response, metadata, and service instance identity, and sends this combined packet to a queue.
The response is returned to the client.

These steps are illustrated in Figure 3.

Figure 3: This figure shows how audit interceptor captures the request and response and simply submits to a queue. This makes the auditing system less intrusive and more scalable. Note that filters are not applied at this point to reduce the additional processing.

It evident that the auditing interceptor acts in a non-intrusive and scalable manner. It does not depend on message structures and individual service instances.

Let's now see how the messages can be processed by the queue listeners:

1. The queue listener picks the message packet that has request, response, service instance identifier, and other message metadata.

2. The listener finds the mapped filters for given service instance.

3. The filters are applied to the messages and, as a result, each filter indicates whether the message was a match or not. It is possible that many filters are provided for a given instance of the service and more than one may indicate a match.

4. If no filter is specified or no filter indicates that the message must be audited, the message is simply discarded.

5. For every filter that indicates that the given set of messages must be audited, it is determined what the destinations for that filter are.

6. The message packet is then sent to every destination mapped for the given filter.

These steps are further illustrated in Figure 4.

Figure 4: This diagram shows how the messages are actually audited by queue listener. First, for given service instance, the registered filters are looked up, then the filters are applied to messages to check if they need to be audited. If yes, then destinations are looked up for given filter and the message are sent to all destinations associated with each filter.

The sequence of processing steps shown in Figure 4 represents a flexible processing model that allows for customizable behaviour for each service instance and filter. It is extensible since any new filter may be added to the service instance and any new destination may be provided for each filter.

Once the reports have been defined using XPath or XQuery, the queries can be run against the messages. Since the message structure can be arbitrary, it is important to save them in XML type columns.

Let's now consider some sample cases and see how the auditing system can support these requirements:

1. You want to track all activities of a user

It is a common situation where you may want to track a user's activity (e.g. for the purpose of diagnosing a problem). Assuming that each message would have a user ID embedded in it, you can create an XPath filter and add that to all service instances. If you want to quickly diagnose the problem, you could specify your e-mail address as the destination. Once this filter is enabled, the messages will start coming to your inbox. After you have collected relevant messages, you can disable the filter.

2. You want to find how a lookup service is being used.

You might have a service that lets users lookup some service providers in a given area. You may want to find out how this is being used to improve user interface. You can capture all messages for this service and query using XQuery or XPath to count how many times zip, state or city were provided, what mile radius is usually selected by users, and so on. Based on this, the user-interface may pre-select some values.

Care must be taken to ensure that interaction with the queue is efficient (for example, it may be executed in a separate thread). If the messages hold sensitive data, the message database must be designed and managed so that the data in not compromised. Since the message auditing will simply store the messages without association with entitlements, it may not be possible to create reports based on some access control. Messages with attachments pose another challenge since the attachments themselves may also need to be stored and queried.

Conclusion

With the advent of XML-based technologies, the evolution of service design principles, service design patterns, and industry standards, message auditing has become viable commodity within the modern SOA eco-system. Adding auditing logic to your systems opens up a flood gate of opportunities for extracting business intelligence from messages.

The beauty of message auditing with XML is that it does not bind you to a specific data structure and therefore allows for the extraction of information in an arbitrary manner. This can be the best defence against the random nature of service-oriented system auditing, which can span from performance data to compliance requirements, from user behaviour to drawing business intelligence, and many more usages.

BPEL Environments and Complexity management

2008-12-29T12:42:00.001+02:00

Service-oriented architecture (SOA) has become mainstream technology for integrating disparate systems and applications. For building composite applications, Business Process Execution Language (BPEL) has emerged as the standard for business process flow orchestration and application integration within organizations. Many IT organizations are deploying composite applications that use BPEL to automate critical business processes. If your application needs to be available 24/7, you need a reliable BPEL infrastructure. In this article, we first present a typical BPEL engine architecture and then outline its manageable entities, and the typical management functions you need to worry about as an administrator. Finally, we conclude with approaches and best practices for managing your complete BPEL infrastructure.

To effectively manage BPEL infrastructure, administrators need to manage all the components that make up the BPEL ecosystem. Before diving into management functions you need to concern yourself with as an administrator, let's look briefly at the BPEL ecosystem.

The BPEL Ecosystem

By BPEL ecosystem, we mean the system components and resources used for executing the BPEL process and those that ensure guaranteed availability of the BPEL process. Although some of those systems may not be directly under your control, you need to take some action to monitor their availability. Unequivocally, if one or more of these components go down, it will put the business process execution at risk. Figure 1 depicts the architecture of a typical BPEL engine.

A BPEL process typically depends on one or more partner links. A partner link might be a service owned by the organization, such as an EJB running an application server or a database stored procedure over which you have immediate control. It might also be a Web service provided by a partner or vendor over which you do not have any control.

The BPEL processes are run in a runtime environment called a BPEL server or engine. The availability and service level associated with BPEL processes are completely dependent on the BPEL engine. BPEL engines can run in an application server environment. You can have a single instance or a clustered application server environment if your composite application requires high availability. A BPEL server may depend on some of the resources, such as data sources or messaging services, made available by the application server environment.

BPEL processes are long running, and hence they need a persistent store to persist state. Typically the persistence store is a relational database management system. This persistence store is called a dehydration store.

All of these software components run on several nodes, or hosts, that constitute the critical piece of your BPEL ecosystem.

To summarize, a BPEL ecosystem comprises the following:

BPEL processes and partner links
BPEL engine
Dehydration store
Gateway to BPEL engine
The application server on which the BPEL server is running
Hosts on which the BPEL server, the dehydration store, and the gateway are running

The health of a BPEL process that you manage is a function of any of the entities that constitute the BPEL ecosystem, and you need to concern yourself with managing and monitoring all of these entities.

Let's dive down into the management functions for each component and which approaches you should follow to manage them, starting with the management aspects of BPEL processes.

BPEL Runtime Governance and Suitcase Deployment

As an administrator, you are responsible for deploying BPEL processes to a single server or multiple servers. Think about transitioning a BPEL process from test to production or deploying the process to additional BPEL engines to add scalability and availability to your system.

Typically BPEL processes are packaged in an archive known as a BPEL suitcase, which contains a .bpel file, WSDL for partner links, and classes and schemas required by the BPEL process. If you are using an SCA-compliant composite application, it may contain a BPEL module. As an administrator, you may be deploying the BPEL suitcase to more than one BPEL engine. You may want to deploy BPEL processes during off-peak times, so you want to automate the deployment procedure.

To automate deployment of BPEL processes, you can build scripts by using tools provided by your BPEL server vendors. If you're using a management tool, it may provide support for uploading the BPEL suitcases to the software library and then enable you to deploy them as needed. Figure 2 outlines the deployment of BPEL suitcases from a software library.

If your BPEL processes depend on resources such as adapters, it makes sense for you to automate the deployment and configuration of them if you want to avoid any manual errors.

Manage Versioning of BPEL Processes

In a dynamic environment, BPEL processes may change, depending on business needs. You need to implement a method of storing different versions of BPEL that have previously been deployed to the BPEL engine. This will help you track changes between different versions and back track any changes, should you run into issues. You may need to compare different versions of processes within one BPEL server or across BPEL servers. Management tools may provide the ability to store/retrieve and compare versions of a process and dependent artifacts such as WSDL and XML Schema that helps in the quick resolution of issues. The software library will allow you to roll back to a previous version of the BPEL process, should the need arise.

Monitor BPEL Process and Partner Links

Monitoring the health of BPEL processes is critical to meeting service-level agreements for your processes. You can probably ensure availability and meet service-level requirements by ensuring performance and functioning of the partner links your BPEL processes depend on and identifying problems in a proactive manner. You can develop some SOAP tests to verify the availability and performance of partner links. Use either a SOAP testing tool provided by a vendor or an open source Web service testing tool such as soapUI. The correct approach is to automate tests to be performed at regular intervals. You want to be alerted when a particular partner link has an unscheduled service shutdown or a longer-than-expected response time. If you're using a management tool to manage your BPEL infrastructure, check whether it provides such a capability.

SOAP Test Example

The Elevation Query Web Service returns the elevation in feet or meters for a specific latitude/longitude (WGS 1984) point from the USGS Seamless Elevation datasets hosted at http://eros.usgs.gov.

WSDL

Operation
getElevation
Input Parameter X: 45.890610
Input Parameter Y: 7.157390
other parameters: left empty
Output Parameter: elevation of latitude/longitude provided.

You can monitor this Web Service by executing a SOAP test at regular intervals through automated tests. These SOAP tests monitor the availability and perceived performance of the Web service. Figure 3 depicts automating testing of Web services.

Monitoring BPEL Processes from the End-User Perspective

Your BPEL process may be accessed from different locations, and you want to monitor the availability and performance from those locations. Say you're part of a global organization and users in North America, Europe, and Asia access your BPEL processes. Although it's challenging to monitor the performance of BPEL processes from the end-user perspective, you can create an agent at the customer locations to test the performance from their end. However, manually performing these types of tasks from a BPEL process that is accessed from multiple customer sites could be challenging. Hence, you can use the help of management tools to monitor the performance of your BPEL applications from the end-user perspective as depicted in Figure 3.

Error/Fault Monitoring

A BPEL process may raise errors and faults for several reasons, including data exceptions or non-availability of partner links, although you need to limit the number of faults in BPEL processes by proactively monitoring the processes, as we described earlier. You need to drill down into the processes that have raised faults, and you should be able to diagnose the cause of the faults and make sure that this does not happen again. You may want to monitor the processes that failed and help diagnose issues by using tools provided by your BPEL vendor and take corrective action as appropriate. It will also help if you implement some methodology to monitor SOAP messages sent to and received by the BPEL process and the partner links.

Gateway to BPEL Engines

BPEL process instances are created when invoked by a client and the traffic is routed through Web servers and load balancers. You want to be alerted when these Web servers or load balancers are not available or not meeting the expected performance goals.

Managing the BPEL Engine

The BPEL engine is by far the most important part of the BPEL ecosystem. You have to monitor the availability and performance of BPEL engines and need to be alerted when a BPEL engine is not performing as expected. You may have multiple instances of clustered BPEL engines. Hence, you need to know if any one of the engines is not performing or whether the load balancer is working properly. More important, you want to find out proactively when all the engines are not performing because it will lead to total disruption of service.

Configuration Management and Versioning

You may change the configuration of your BPEL engines from time to time. You may want to track any interim changes and to backtrack changes to a previous version. If you follow ITIL practices, this should sound familiar. You can store the configuration of a BPEL engine in a configuration management database (CMDB) or version control system. It can make your life easier to have your management tool provide integration with the CMDB in which you store your configuration.

Keeping a gold image of your BPEL ecosystem configuration and using it to compare with your current configuration will help in identifying issues attributable to configuration changes. And keeping track of components added to and/or removed from your BPEL system will also help in analyzing performance issues - for example, performance was bad during the first week of January but improved afterward, because there was one extra BPEL engine in the cluster after the first week of January. By comparing the current configuration with the gold image, you can monitor changes in some parameters that can have a big impact on BPEL instance throughput, such as dispatcher thread min/max parameters.

BPEL Engine Monitoring

The health of the BPEL engine is critical for your applications. Besides the status of the BPEL engine, there are some statistics you should focus on. They include system statistics such as memory; CPU consumed; and business metrics such as open and closed instances, synchronous and asynchronous process latency, and load factors. In the event of abnormal behavior such as high process latency or load factors, you have to proactively resolve the issues before they affect BPEL processes deployed in the engine.

As an administrator for your BPEL engine, you have to perform some of the following operations:

Archive information about completed BPEL processes
Remove from the database all XML messages that have been successfully delivered and resolved
Purge stale instances
Re-execute failed processes

Ideally, these mundane tasks should be automated in a production environment. You can build some automated scripts to perform these operations and schedule them to be performed at regular intervals. Many management tools help you with automation by providing a graphical user interface.

Dehydration Store

The BPEL engine stores process data in the dehydration store. This is a critical piece of the puzzle. As we discussed earlier, the dehydration store is typically a relational database. If you want high availability for your dehydration store, you will probably use a clustered database. Your database administrators should make sure that the database is available and performing properly.

Monitoring Adapters

Your BPEL processes may depend on adapters that access resources such as databases, messaging services, and EIS that may reside locally or remotely. The status and performance of adapters may have a heavy impact on the response time of your BPEL engine and processes.

Managing the Application Server

A typical BPEL engine runs in an application server environment. For example, Oracle BPEL Process Manager can be deployed on a J2EE-compliant application server such as Oracle Application Server, Oracle WebLogic, IBM WebSphere, or JBoss Application Server. The BPEL engine may depend on several resources and services provided by the application server, such as JDBC DataSource, JMS providers, JCA connectors, and shared libraries. The health of the application server, along with the performance of these resources, may directly influence the performance of your BPEL engine. Each application server provides several health indicator metrics. You should automate a mechanism that would proactively issue an alert before anything goes wrong with your application server.

Managing the Hosts/Nodes

Your BPEL infrastructure may be running on several nodes or machines. These server nodes may run into several resource issues during runtime. For example, they may run out of disk space due to excessive logging or run out of memory or CPU due to a spinning process. You need to proactively monitor these metrics and fix any issues before they become a problem for your infrastructure.

Bringing It All Together

It's now clear that the BPEL infrastructure includes a lot of entities. Monitoring these as independent entities could be a challenge.

A typical SOA environment will have hundreds of artifacts like Web services, BPEL processes, EJBs, adapters, and other resources. Monitoring a flat list of these artifacts is not productive. The dependency between these components and the impact of any changes you make is critical to determine. Distributed ownership of these components also requires that service-level agreements be established between providers and consumers of these components. These issues are addressed with a runtime governance offering. So it's important to employ a management tool that provides:

Translation of design-time discovery into production to help manage all the components and their dependencies in context
SLA capabilities to capture and measure SL compliance of your SOA application and SOA components

Think about a complicated BPEL process that uses several heterogeneous systems and applications. Say you're using Oracle's BPEL engine on an Oracle WebLogic server running on a Linux box. Your business processes involve partner links that depend on adapters that connect to IBM MQSeries applications and applications from Oracle's PeopleSoft Human Resources product family that run IBM WebSphere. Your BPEL engine uses an Oracle Real Application Clusters Database as the dehydration store. You want to monitor all these products together from a single management console. There are several management products on the market that may meet your needs. Figure 4 shows how you can manage the Oracle BPEL engine running on an Oracle WebLogic server from Oracle Enterprise Manager Grid Control 10g

An integrated management solution lets IT administrators manage complexity within a BPEL and SOA environment. Without this management solution, IT will see an increase in incidents, lower service levels, and increased end-user frustration. Moreover, as IT scales out with a new BPEL infrastructure, new personnel will be required to manage this complex distributed architecture. In addition, a move from legacy to SOA architecture can increase costs without a management strategy and toolset. You should consider investing in the right management product that makes sense for your infrastructure.

Another compelling reason why you should consider a management tool is for alignment between IT and your business. As an IT administrator, you need to monitor and report issues in terms of business processes. You should have monitoring statistics aligned with business processes. To accomplish this you need to have a view of business process flows that should be synched up with actual flows in the BPEL server. BPEL management tools provide you with a view of business processes and reports monitoring statistics for each business process.

Best Practices

Here are some of the best practices you can follow to ensure the availability of your BPEL infrastructure:

Establish service-level objectives for your BPEL processes and partner links.
Keep a library of BPEL suitcases in your software library. It will help in rebuilding a system in case of server failure.
Automate routine operations such as purging old process instances.
Monitor the performance of partner links.
Monitor the whole BPEL ecosystem, not just the BPEL engine.
Keep track of BPEL ecosystem membership/topology changes.
Keep a gold image of your configuration when everything is stable and keep updating it after every configuration change. This will help you find the cause of any possible problem due to configuration changes.
Monitor BPEL server-specific J2EE artifacts such as the JMS queues and data sources used by the BPEL server in addition to J2EE constructs used by BPEL processes.
Make sure that you select a management solution that can maximize your productivity and help you deliver maximum service through automation.

Conclusion

With SOA, managing the complete infrastructure - involving lots of heterogeneous components and products - has become challenging. For IT administrators, managing the service-level agreements for composite applications can be a monumental task. Using the right tools and methodology to make sure your BPEL infrastructure is highly available can make your job much easier.

SOAs ‘nature’ and integration.

2008-12-24T12:42:00.001+02:00

An interesting post from Joe McKendrick contains the views of some consultants, as concerning SOA and integration. The issue, deals with a Shakespearian quest ‘to be or not to be’ with ‘be’ as integration.

“SOA is integration. SOA is not integration. Got that?

There’s a renewed debate raging about the relationship between the practices of service oriented architecture and integration.

SOA is more than EAI 2009, but where do we start? […]”

As Gartner analyst Yefim Natis states, few organizations have one comprehensive SOA — most organizations have multiple islands of SOA that will eventually need to be brought together. Ann Thomas Manes, pointed out that “Many organizations mistakenly perceive SOA as an integration strategy. But it is not. SOA is about architecture. To achieve SOA, you must re-architect your systems.” But Loraine takes more of a pro-SOA-is-integration stance, noting that “most companies aren’t getting into SOA for a complete rebuild. Most companies deploy SOA because it’s so darn helpful with simplifying integration:”

“…companies don’t want Extreme Makeover. They’re looking for a slight update, something that ties the room together, as interior designers like to say. … And here’s another hard truth: Although David Linthicum and others believe that agility is the ROI for SOA, many companies are realizing SOA ROI through integration.”

Ann Thomas Manes urges practitioners to look beyond integration when planning SOA efforts. “It’s fine to use service oriented middleware to implement integration projects, but then you need to readjust your expectations. Most organizations that I speak with say that the goals of their SOA initiative are to reduce costs and increase agility. Unfortunately, these organizations aren’t likely to achieve these goals if their projects only focus on integration….”. SOA should be about more than simply linking app A to app B, and so on. SOA is about innovation. Every SOA-related proposal out there should include the word “innovation” somewhere in the text. And, let’s face it, SOA would be downright boring (make that Boring with a capital B) if it was just another means of app or systems integration, and nothing more.

But, nevertheless, integration is still an essential phase in the evolution from soloed, proprietary systems to full-functioning service oriented architecture. In many cases, it can help provide the initial justification for commencing with an SOA approach. And for many SOA proponents, especially those facing organizational resistance, the key is to just start.

And, remember, SOA may involve integration, but it truly is about innovation.

Actually, the truth, or the need (with the Aristotelian notion) behind doing SOA is that the main goals of SOA initiatives are to reduce costs and increase agility. The getting out from silos and proprietary systems is just the good reason of taking SOA approach in projects and have the acceptance of CEOs. How many times the “monolithic-systems” overcome has been stated in such meetings and proposals? And of coarse, at the end, how many times did a true SOA solution broke that “monolithity”. Some will say always, in the context of the project. But the true SOA should guide towards “the standardization of the core architecture elements (…) and furthermore to establish an optimized integration approach for existing assets of the environment.” as stated in the SOA and the real Service Integration issue post. The problems of real life environment (both IT, infrastructure and financials) force towards an analytical solution, meaning solving one problem at a time. This is the main headache for considering SOAs in every extend you may wish to use it.

The most useful approach to SOA, is the change of the adopted philosophy. The ideal SOA, considered somewhere in the Platonian world of Ideas, as a successful concept, has to do with a non-analytical approach, but mostly with a holistic one. The web services and the integration (imagine behind that word how many systems/applications/business approaches/files/tools… might be hiding) the business needs and the time constrains are giving the first bird-eye view, or the context of the environment to be SOA-ised. But this is not enough. If it was, then correctly, we should be speaking about re-engineering and therefore reinventing the wheel. A solution of this type, is something like.. “drop everything out and put the new, SOA”. SOA should deal with ALL of these. According to the project at hand, an holistic view should be taken in every aspect of the analysis. We should thing with the duality in mind (this project, the whole enterprise environment) and in a time agnostic attitude (as-is and to-be in one context). Ok, the magicians are out of job nowadays, but none said that doing SOA should be easy. In the beginning. After having a good and approved idea of the total-holistic SOA environment in the context enterprise, then the compliance rules are straight forward and therefore the base of the SOA for the project(s) at hand to start with. SOA is about innovation, integration and re-architecture. But more of all, is about the way considering the very same topics, issues, environments and enterprises with a new eye.

more to follow, stay tuned and if you would like help establishing a SOA strategy for your specific case, give me a shout!

SOA and the real Service Integration issue

2008-12-23T13:23:00.001+02:00

A real problem considering a SOA project, either in its beginning or in extending an all ready existing one, is the Web Services (ws) hell, that is hiding in the corner. Beginning the SOA approach, especially when having limited time to deliver and / or pure business analysis results, defining and architecting a solution that is compliant with your client (both in the kick-off architecture and in the way of working – aka good adaptation of the new environment) is possibly to provide a sea of ws. From the other hand, extending or taking a project in an already existing SOA environment that was developed by someone else, you have a small daemon hiding in the corner that always play with you in order to make you build more and more new ws in order to build your components.

In any case, in the SOAs you are always face the problem of exponential increase of the ws pool. In order to avoid such a problem, you have to give effort in focusing on the standardization of the core architecture elements (imagine that in an existing case, developed by someone else, you have to understand and comply with the already taken way) and furthermore to establish an optimized integration approach for existing assets of the environment.

Having these in mind, you have either to use some existing ground of work to extent by, or to … re-invent the wheel.

Re-inventing the wheel, means that you are prepared to give much more effort in business analysis and re-engineering and furthermore to develop all the necessary tools and bridges, both for the interoperability elements of the infrastructure, as well as for the semantics and conceptual transformations that will take place in order to implement the needed physical (system) bridges.

Or using some existing assets…

Java Business Integration (JBI) is an effort focused on standardizing the core architecture elements of integration architectures. It is a specification developed under the Java Community Process (JCP) for an approach to implementing a Service Oriented Architecture (SOA). The JCP reference is JSR-208. JBI extends Java EE and Java SE with business integration service provider interfaces (SPIs). It enables the creation of a Java business integration environment for the creation of composite applications. It defines a standard runtime architecture for assembling integration components to enable a SOA in an enterprise information system.

Following the JBI road, you might be able to use another ally for the SOA stacks you gonna need.

Open ESB is an open source integration platform based on JBI technology. It implements an Enterprise Service Bus (ESB) using JBI as the foundation. This allows easy integration of Web Services to create loosely coupled, enterprise-level integration solution.

Open ESB Architecture

Because Open ESB is built on top of the JBI specification, it makes sense that, before diving into the architecture of Open ESB, we take a look at the JBI architecture, which is illustrated in Figure 1.

As the figure shows, JBI adopts a pluggable architecture. At the heart of a JBI runtime is a messaging infrastructure called Normalized Message Router (NMR) that is connected to a bunch of JBI components. JBI components are architectural building blocks of a JBI instance and are plugged into the NMR to interact with each other. The interaction among components carries out the functional logics of the JBI instance.

Normalized Message Router

The primary function of the NMR is to route normalized messages from one component to another. It enables and mediates the inter-component communication. When a component needs to interact with another component, the component does so by generating a normalized message and sending it to the NMR. The NMR will route the message to the destined component based on some routing rules. After the destined component gets and processes the message, it will generate a response message if required, and will send the response message to the NMR. It is the NMR's responsibility to deliver the response message back to the original component.

The NMR uses a WSDL-based messaging model to mediate the message exchanges between components. From the NMR's point-of-view, all JBI components are service providers and/or service consumers. The WSDL-based model defines operations as a message exchange between a service provider and a service consumer. A component can be a service provider, a service consumer, or both. Service consumers identify needed services by a WSDL service name rather than end-point address. This provides the necessary level of abstraction and decouples the consumer from the provider, allowing the NMR to select the appropriate service provider transparently to the consumer.

An instance of an end-to-end interaction between a service consumer and a service provider is referred to as a service invocation. JBI mandates four types of interactions: One-Way, Reliable One-Way, Request-Response, and Request Optional-Response. These interactions map to the message exchange patterns (MEPs) defined by WSDL 2.0 Predefined Extensions.

The NMR also supports various levels of quality of service for message delivery depending on application needs and the nature of the messages being delivered.Component - Service Engine and Binding Component
A JBI component is a collection of software artifacts that provide or consume Web Services. As mentioned previously, JBI components are plugged into the NMR to interact with other components. JBI defines two types of components, Service Engines (SEs) and Binding Components (BCs).

A Service Engine is a component that provides or consumes services locally within the JBI environment. Service Engines are business logic drivers of the JBI system. An XSLT Service Engine, for example, can provide data transformation services, while a BPEL Service Engine can execute a BPEL process to orchestrate services, or enable execution of long-lived business processes. A Service Engine can be a service provider, a service consumer, or both.

A Binding Component is used to send and receive messages via particular protocols and transports to systems that are external to the JBI environment. They serve to isolate the JBI environment from the particular protocol by providing normalization and de-normalization from and to the protocol-specific format, allowing the JBI environment to deal only with normalized messages.

Distinguishing between these two types of components is more functional. In fact, JBI uses only a flag to distinguish these two. The programming model and APIs of these two types are otherwise identical. However, by convention, Service Engines and Binding Components implement different functionality in JBI.

Service Unit and Service Assembly

JBI runtime hosts components and so acts as a container for components. Components in turn act as containers for Service Units (SUs). A Service Unit is a collection of component-specific configuration artifacts to be installed on SEs or BCs. One can also think of a Service Unit as a single deployment package destined for a single component. The content of a Service Unit are completely opaque to JBI, but transparent to the component it is deployed to. An SU contains a single JBI-defined descriptor file that defines the static services provided and consumed by the Service Unit.

Service Units are often grouped into an aggregated deployment file called a Service Assembly (SA). A Service Assembly includes a composite service deployment descriptor, detailing to which component each SU contained in the SA is to be deployed. A service assembly represents a composite service.

Lifecycle Management

JBI also defines a JMX-based infrastructure for lifecycle management, environmental inspection, administration, and reconfiguration to ensure a predictable environment for reliable operations.

Components interfere with JBI via two mechanisms: service provider interfaces (SPIs) and application program interfaces (APIs). SPIs are interfaces implemented by the binding or engine; APIs are interfaces exposed to bindings or engines by the framework. The contracts between framework and component define the obligations of both framework and JBI component to achieve particular functional goals within the JBI environment.

Unless you're doing component development, it's unlikely that you need to work with these SPIs and APIs directly.

Open ESB

Once we've understood the architecture of JBI, Open ESB becomes really simple. Open ESB is an implementation of the JBI specification. It extends the JBI specification by creating an ESB from multiple JBI instances. The instances are linked by a proxy-binding based on Java Message Service (JMS). This lets components in separate JBI instances interoperate in the same fashion as local ones (see Figure 2).

ESB administration is done by the Centralized Administration Server (CAS), a bus member that lets the administrator control the system directly.

Open ESB includes a variety of JBI components, such as the HTTP SOAP Binding Component, the Java EE Service Engine, and the BPEL Service Engine

A Sample Service Integration Scenario

In this section, we'll examine a simple use case and illustrate a possible service integration solution using Open ESB.

Problem Description

ABC Movie Theatres is a fast-growing movie theatre chain. To better serve its customers, the management has decided to put in place a new ticket booking service, the Booking Service, which will be responsible for handling most of the ticket purchasing requests generated from various systems such as Web-based applications, ticket vending machines and points of sale (POS) at box offices.

The business logic of handling a booking request is somewhat complicate. But in a nutshell, it involves the following steps:

When a booking request is received, the Booking Service will first try to process the ticket information. This could include checking for ticket availability; holding the tickets for the customer if tickets are indeed available or provide alternatives to the customer otherwise; applying any applicable promotions and calculating the total dollar amount
Then the Booking Service will charge the customer using the payment information included in the booking request
Finally the Booking Service will send a confirmation to the customer using the contact information included in the request.

High-Level Solution Description

Considering the fact that a significant amount of the logic needed by the new booking service has been implemented in different applications over time, and that these applications have been proved stable over time, it makes a lot sense to leverage these existing IT assets for cost efficiency. So the architect team at ABC comes up with the solution of service-enabling and -consolidating existing logics, bringing the services into the ESB and exposing them as new composite services that are made available via different protocols.

To do this, they have to identify candidates for service-enabling. Currently the company has a billing system developed and used by the finance department and it has been working perfectly over years. This system runs over the HTTP protocol. Another system is the notification system developed by the customer relationship department. This system listens to a JMS message queue and, when it gets a new message, sends out notifications to the customer by the means specified in the message, e.g., an e-mail or a voice mail message. These two systems become the ideal candidates for billing customers and sending confirmations.

The company also has a system for processing ticket orders. This system, however, is severely old and impossible to scale up to handle the ever-increasing transaction volume due to recent acquisitions. The architect team has decided it's time to write a replacement application; it's also decided that the replacement application, the Ticket System, will be built using EJB technologies due to the transactional nature of the application.

The architect team also decided that the new Booking Service will be made available via the SOAP-over-HTTP protocol as well as the File protocol to support different client systems.

Solution Details

When it comes to Open ESB development, it's all about creating JBI Service Units and packing them in a Composite Application (or Service Assembly, in JBI terms). Figure 3 illustrates the Service Units for this solution and the interactions among those units. As mentioned earlier, Service Units are deployed to their corresponding JBI components. For simplicity's sake, we'll use the term Service Unit and Component interchangeably provided it doesn't cause any confusion in the particular context.

At a high level, two BCs, the Booking Service SOAP BC and the Booking Service File BC, are created to enable the Booking Service and expose it to the outside world via the SOAP and File protocol respectively, allowing different kinds of clients to consume the service. A client application can invoke the Booking Service via either of these protocols. Invocation requests generated by client applications are then routed to the Booking Process, a BPEL Service Engine. The Booking Process orchestrates the Ticket Service, the Billing Service, and the Notification Service to fulfil the request.

File and SOAP BC - The Booking Service

In Open ESB, a File BC is a JBI binding component that binds file systems to WS-I Web Services. A File BC scans a pre-configured file location for new files. If a new file is found, the component generates a Web Service call using the content in the file as the payload of the Web Service input message. Response messages will also be written to the file system by the component. Many of the properties of a File BC, such as the file location, file name, and time interval for the component to scan the specified location for new file, can be configured.

A SOAP BC works the same way as the File BC, only instead of scanning a file system directory, a SOAP BC accepts WS-I SOAP messages over the HTTP protocol.

A File BC and a SOAP BC are created in this solution to enable the Booking Service and expose it to external systems via these protocols. External systems that wish to consume the service do so by either sending a WS-I-compliant message, or dropping the message in the specific file location.

External requests received by the previous two BCs will be routed, by the NMR, to the Booking Process, a BPEL Service Engine that executes a BPEL business process to orchestrate services.

BPEL Service Engine - The Booking Process

A BPEL Service Engine is a JBI runtime component that provides services for executing WS-BPEL-compliant business processes. The contract between a business process and partner services is described in WSDL.

A BPEL SE can save business process data to a persistent store if configured to do so. This is required for recovering from system failure and running long-lived processes. A BPEL SE can be deployed to a clustered environment to achieve high scalability. The service engine's clustering algorithm automatically distributes processing across multiple engines. When the business process is configured for clustering, the BPEL Service Engine's failover capabilities ensure throughput of running business process instances. When business process instances encounter an engine failure, any suspended instances are picked up by all available BPEL Service Engines.

The Booking Process in this solution is a BPEL SE and is at the heart of this solution. It does some simple message transformation and, most importantly, invokes the Ticket Service, Billing Service and the Notification Service. Figure 4 shows a simplified version of the Booking Service process.

Java EE Service Engine - The Ticket Service

A Java EE Service Engine brings Java EE components into the Open ESB runtime as Web Services. A Java EE Service Engine acts as a bridge between a Java EE application server and a JBI environment for Web Service providers and Web Service consumers deployed in the application server. Java EE Web components or EJB components that are packaged and deployed as Web Services on a Java EE container can be transparently exposed as service providers in JBI environment.

In this solution, the Ticket System is implemented using EJB Session Beans and wrapped as a JAX-WS Web Service. The service is then brought into the Open ESB runtime by the Ticket Service SE.

HTTP and JMS BC - The Billing Service & the Notification Service

Let's face it - all BCs work the same way. This is the beauty of Open ESB architecture. We've looked at two BCs, the File and the SOAP BC. Similarly an HTTP BC binds the HTTP protocol to the Web Service, and a JMS BC binds the JMS protocol.

In this solution, the HTTP BC and the JMS BC are used to bring the Billing Service and the Notification Service into the Open ESB runtime respectively - as stated previously, the Billing Service runs over the HTTP protocol and the Notification Service over the JMS.

It's important to be aware that, although these components are shown connected directly in the figure, they never communicate directly to each other. Instead, components send massages to and receive messages from the NMR. The NMR is responsible for transforming messages and routing messages to the appropriate destinations.

GlassFish & NetBeans - Development & Deployment

Open ESB runs on any OSGi R4-compliant runtime. GlassFish has a built-in JBI runtime and is bundled with NetBeans for easy development. NetBeans provides a comprehensive GUI development environment. The java.net community is working on a new project, GlassFish ESB aimed at creating a community-driven ESB for the Glassfish Enterprise Server platform.

Developing the process in NetBeans involves creating the needed JBI modules and including them in a composite application. Figure 5 shows a screenshot of creating the composite application in the NetBeans IDE.

Conclusion
Open ESB provides a robust and flexible platform for building service-oriented integration solutions. Its component-based architecture allows maximum extensibility and interoperability. It's based on industry standards and is easy to use. It seamlessly integrates with other Java enterprise technologies.

References

JSR 208: Java Business Integration (JBI)
Open ESB Project
The Community Wiki for JBI, JBI Component Development and Open ESB.
JDJ for useful resources and the scenario

Open JBI Components

The overall goal of Project Open JBI Components is to foster community-based development of JBI components that conform to the Java Business Integration specification (JSR208). You can join this project as a JBI component developer or as part of an existing JBI component development team.

About JBI Components

The JSR208 specification provides for three installable JBI components: Service Engines, Bindings, and Shared Libraries. JBI components operate within a JBI container, which is defined by the JSR208 specification. Two popular implementations of JBI containers are Project Open ESB and ServiceMix, an alternative approach which has been mentioned in previous posts.

SOA agility continued: Leveraging Data Services and provide agility in enterprise

2008-12-16T14:03:00.001+02:00

The promise of SOA is flexibility and agility. SOA people give the definition of agility as the ability to adapt to change at the speed of business. In today’s global economy which has been fuelled by collaboration and Internet technologies, businesses change at a much faster rate than ever before. So how does SOA help companies become agile? In previous post i have consider my case of agility concerning the developing and architecting of SOA enabled applications, after facing some real world bottlenecks and difficulties.

Continuing with the notion of developing and expanding it to the agility of the enterprise (yours or mainly yours customer as an asset to provide to), the main issue should be to abstract the necessary data services that the other layers of the architecture use. The trick here is the ‘magic’ logical (or conceptual) representation of an entity (in this example and in many real world cases, the customer), not the physical implementation. From the point that in software we can deal with symbolic transformations, the option of abstracting the semantics into simpler (but yet transformable back) entities, is the magic. (Although, here i don’t refer to semantics, keep in mind, that in some legacy SOA platforms of biggest vendors, the semantics is on main focus as a next step for tools to by – they mention it as the heart of orchestration, organizing and identifying the correct services in the SOA-ws sea – which is the next step of my consideration here).

Firstly, let’s look at a logical view of a typical SOA.

You can see in this diagram how each layer of the architecture has been abstracted and is mutually exclusive or “loosely coupled” from the other layers of the architecture. Why is this important? The answer is simple…..ease of change! Here are some advantages of this approach.

Share services, components, rules, etc. across the enterprise (reuse)

Isolate changes and reduce dependencies (speed to market)

Minimize impact of business changes (speed to market)

Easier to maintain (maintainability)

Finalizing here, to give a clear applicable explanation of the above, let me give an example of how this approach helps companies become agile.

Use Case: New customer data from a new client

Let’s say your company provides a service for customers in the retail industry. You have a website that offers online services for consumers and your white label solution is tailored to look like it is hosted by the individual retailers. The problem is that each retailer has their own customer database. In the past you would have to write a ton of code for each retailer that you signed up to use your services. With SOA, now it is a simple data mapping exercise. By abstracting data services, the other layers of the architecture use the logical representation of customer, not the physical implementation. Behind the scenes, the data services layer is translating the request for customer data from logical view to physical implementation. So when you bring on new clients, you simply use a tool in the data services layer to map the new clients customer data to the standard logical definition as defined by the architecture.

You can see from this example that all three retailers have an entirely different implementation of their customer database including different naming conventions and even different attributes. In the data services layer, you can map all of these physical implementations to one standard customer definition. You can also see how the business processes all use the logical view of customer. This allows us to add and change customer definitions on the back end without changing code on the front end. If two more retailers were to sign up tomorrow, we can map their definitions to the logical customer view and be done. No Code!!! That is agile!

If you would like help establishing a data services strategy, give me a shout!

The above schematics and context (reason) for writing this post are from Mike Kavis . Also I would like to thank him for the definition of the Use Case above and the schemes usage.

SOA Open Source stacks

2008-12-16T13:29:00.001+02:00

Dave Linthicum wrote a post called Open Source SOA provides some major advantages. In his post Dave stated:

When it comes to SOA, I think open source provides two major advantages:

First, it’s typically much less expensive than the tools and the technology that are proprietary.

Second, they are typically much more simplistic and easier to understand and use.

To the second point, simplicity. The open source SOA vendors seem to take a much more rudimentary approach to SOA, and their tools seem to be much easier to understand and, in some cases, use. While some people want complex, powerful tools, the reality is that most SOAs don’t need them. If you’re honest with the requirements of the project, you’ll see that good enough is, well, good enough.

Great points. I would also add another clear advantage which I learned the hard way. On a previous enterprise wide SOA initiative, I drank the cool-aid that the vendor stack was an integrated stack and was simpler to deploy and manage over a stack of a mix of vendors. What I found out is that the mega vendors (IBM, Oracle, etc.) have bought so many pure play tools (rules engines, BPMs tools, data services and MDM tools, governance tools, etc.) that the smooth integration ends when the Power Point decks are closed. In reality, the mega vendor stacks are a hodge podge of rushed acquisition and integration efforts.The important thing, is that the underlying architecture of each tool within the stack are completely different and there are very few people (if any) within the organization who understands the complete stack. In fact, we were dealing with two very different organizations when dealing with support and they were not in sync. Eventually the entire company was consumed by another mega vendor (you can probably guess which acquisition this was) and the whole product roadmap was turned upside down.

Now let’s look at some of the well established open source stack vendors like WSO2, MuleSource, and RedHat. These vendors do not suffer from acquisition madness and chaos. If fact, they are all built on a consistent architecture and do offer smooth integration between the various layers of the stack. Do they have all of the features of the commercial products? No. Do they have enough features for most SOA initiatives. Definitely. MikeKavis wrote a post on CIO.com called Tight Budgets? Try open source SOA. Here is a quick summary of the advantages he discussed:

Try before you buy

Lower cost of entry

Cost effective support

Core competency

For the people by the people

So what open source options do I have, you might ask? The following picture shows the open source tools that some people prefer for their new SOA initiative. There are using a combination of WSO2, Intalio, Drools, Liferay, and PushToTest.

This is just one example of many. You can mix and match tools from different open source communities or you could standardize on one community. Here is an example of Red Hat’s jBoss SOA stack.

And MuleSource has a well known suite of tools as well.

Many organizations are still not very comfortable with open source for mission critical initiatives. There is a debunk for many of the open source myths in the past (here, here, and here).

If there ever was a time to embrace open source, the time is now in this harsh economy. As commercial SOA vendors continue to get gobbled up by the mega vendors, it is time to seriously consider alternatives.

The above schematics and the first place to step on (context) for writing this post are from Mike Kavis . Also I would like to thank him for using his comments and graphs (although their are not his - Please refer to Mule, JBoss and WSO2 sites for these and extended info for these tools).

Adapting with the SOA – Mashing up, the case of Enterprise Mashups

2008-12-16T12:39:00.001+02:00

The Dilemma

When defining a SOA scheme, or developing web services (of coarse in a SOA compliant way or SOA enabled) you try to sell or develop a reusable asset both to you (for easier future development) and your clients (in order to deliver them a dynamic, adaptable and easy to extent asset). Nowadays, lots are mentioned around enterprise mashups. Actually, when working in web development projects 4-5 years ago, the idea of mashups, was very helpful and many times the focus of my work. Thing of the Microsoft’s Sharepoint logic, whereas you just by dropping in web-parts, you were able to develop in just a ‘lego’ -way some potral, an intranet and what so ever web presence. Knowing the tool’s abilities and the business/project requirement needs the whole job was straight forward. The same for example, holds for the Plone CMS system, which just by re-using existing parts of it (that were tested and user friendly) you were able to deliver a very user friendly CMS system, making your clients very happy that they could manage some web aspects and deploys from a desk and by paying a web developer.

In the world of the SOA era, this “lego” approach is hiding behind the re-usage promise. Companies and developers, software architects and business, are now familiar with the bottlenecks of the above notion. The managers and so, of the SOA ‘environments’, came above the notion of SaaS. In general, Software as a Service (SaaS, typically pronounced 'sass') is a model of software deployment where an application is hosted as a service provided to customers across the Internet. By eliminating the need to install and run the application on the customer's own computer, SaaS alleviates the customer's burden of software maintenance, ongoing operation, and support. Conversely, customers relinquish control over software versions or changing requirements; moreover, costs to use the service become a continuous expense, rather than a single expense at time of purchase. Using SaaS also can conceivably reduce that up-front expense of software purchases, through less costly, on-demand pricing. From the software vendor's standpoint, SaaS has the attraction of providing stronger protection of its intellectual property and establishing an ongoing revenue stream. The SaaS software vendor may host the application on its own web server, or this function may be handled by a third-party application service provider (ASP). This way, end users may reduce their investment on server hardware too.

the…Philosophy

As a term, SaaS is generally associated with business software and is typically thought of as a low-cost way for businesses to obtain the same benefits of commercially licensed, internally operated software without the associated complexity and high initial cost. Consumer-oriented web-native software is generally known as Web 2.0 and not as SaaS. Many types of software are well suited to the SaaS model, where customers may have little interest or capability in software deployment, but do have substantial computing needs. Application areas such as Customer relationship management (CRM), video conferencing, human resources, IT service management, accounting, IT security, web analytics, web content management and e-mail are some of the initial markets showing SaaS success. The distinction between SaaS and earlier applications delivered over the Internet is that SaaS solutions were developed specifically to leverage web technologies such as the browser, thereby making them web-native. The data design and architecture of SaaS applications are specifically built with a 'multi-tenant' backend, thus enabling multiple customers or users to access a shared data model. This further differentiates SaaS from client/server or 'ASP' (Application Service Provider) solutions in that SaaS providers are leveraging enormous economies of scale in the deployment, management, support and through the Software Development Lifecycle.

Key characteristics

The key characteristics of SaaS software, in general, include:

network-based access to, and management of, commercially available software

activities that are managed from central locations rather than at each customer's site, enabling customers to access applications remotely via the Web

application delivery that typically is closer to a one-to-many model (single instance, multi-tenant architecture) than to a one-to-one model, including architecture, pricing, partnering, and management characteristics

centralized feature updating, which obviates the need for downloadable patches and upgrades.

SaaS is often used in a larger network of communicating software - either as part of a mashup or as a plugin to a platform as a service. Service oriented architecture is naturally more complex than traditional models of software deployment.

SaaS applications are generally priced on a per-user basis, sometimes with a relatively small minimum number of users and often with additional fees for extra bandwidth and storage. SaaS revenue streams to the vendor are therefore lower initially than traditional software license fees, but are also recurring, and therefore viewed as more predictable, much like maintenance fees for licensed software.

Ok. This is how the thing is defined to be like. This is the encyclopaedias bird-eye view. Now lets apply these. And we have a real world case. From the above Key characteristics, the 3rd and the 5th are of great importance. The business analysis should give some hinds, but there is not only that. The new startup that I am exposing here as example, is building a SaaS solution that will be consumed by various types of customers and partners. These customers and partners may want to consume our data services as a RSS feed, gadget, SMS message, web page, within a portal or portlet, or a number of different ways. I do not want to spend the rest of my life developing new output mediums for our services. Instead, I would rather spend my time adding new business services to enhance our product and service offerings hence contributing to the bottom line.

Enterprise Mashups to the rescue

Enterprise mashups will allow me to offer my partners and customers the ultimate flexibility to access our products and services in ways that are convenient for them without having to wait on my IT shop to decide if (a) we think the request is important enough in our priority list, (b) if we have the time and resources to work on it, and (c) how much we will charge them. On the IT side of the house, with an enterprise mashup strategy in place we can be assured that whatever mashups our customers and partners create, they will be subject to the same security and governance as the services we have developed. The diagram below shows a logical view of how our SOA will be designed.

As you can see, we have clearly abstracted the various layers within the architecture and they all inherit our overall security policies. SOA governance is applied to this architectural approach to enforce our standards and design principles. Overall IT governance provides oversight over the entire enterprise which includes legacy systems (we don’t mention any legacy yet), third party software, etc.

Now let’s add the enterprise mashup layer. We want to hide the complexity of our architecture from the end user and expose data services to them to consume. At the same time we want these mashups to be equally secure as the services we write and adhere to the same governing principles. Enterprise mashup products provide tools to make managing this layer easy and efficient. The diagram below shows the enterprise mashup layer inserted into the architecture as a layer on top of SOA.

Enterprise Mashups in simple terms

Deepak Alur (JackBe’s VP of Engineering) discussed how enterprises have been focusing more on infrastructure and technology and not on the consumers of data. As he coined it, many shops are “developing horizontally and not addressing the needs of the users”. He talked about how users were doing their own brute force mashups by cutting and pasting data from various places into Excel. This creates various issues within the enterprise due to lack of data integrity, security, and governance. It is ironic how corporations spend huge amounts of money on accounting software and ERP systems, yet they still run the business out of user created Excel spreadsheets! The concept of enterprise mashups addresses this by shifting the focus back to the user consumption of data. Here are some of the requirements for mashups that Deepak pointed out:

User driven & user focused

Both visual & non-visual

Client & server side (although most are server)

Plug-n-Play

Dynamic, Adhoc, Situational

Secure & Governed

Sharable & Customizable

Near zero cost to the consumer

Jackbe’s enterprise mashup tool is called Presto. Presto is an Enterprise Mashup Platform that allows consumers to create “mashlets” or virtual services. IT’s role is to provide the security and governance for each data service that will be exposed for consumer use.

Presto Wires is a user friendly tool to allow users to create their own mashups by joining, filtering, and merging various data services (as shown in the picture below).

In this example the user is combing multiple data points from many different organizations in an automated fashion. They could then present this data to multiple different user interfaces and devices. All without waiting on IT.

How this solves my Dilemma

Back to my dilemma. By leveraging a tool like Jackbe’s Presto or WSO2’s Mashup Server, I can now present various data services in a secured and governed fashion to my customers and partners without being concerned on how they want to consume it. Whether they want the mashup on their own intranet, as a desktop gadget, as an application on Facebook, or what ever they dream of, all I need to be concerned with is the SLA of my data services. This also makes my product offering more competitive than my competitors who have proprietary user interfaces that do not provide the flexibility and customization that the customers desire.

As mentioned in the title, this is the Adaptation with your SOA cake. For those organizations who are disciplined enough to implement SOA and follow the best practices of design and governance, the reward can be an simple addition of an Enterprise Mashup Platform on top of your SOA stack. This is the ultimate flexibility and agility that SOA promises.