Showing posts with label Java. Show all posts
Showing posts with label Java. Show all posts

Tuesday, November 19, 2019

Better integration tests with WireMock

No matter if you follow the classical test pyramid or one of the newer approaches like the Testing Honeycomb you should start writing integration tests at some point during development.
There are different types of integration tests you can write. Starting with persistence tests, you can check the interaction between your components or you can simulate calling external services. This article will be about the latter case.
Let us start with a motivating example before talking about WireMock.

The ChuckNorrisFact service

The complete example can be found on GitHub.
You might have seen me using the Chuck Norris fact API in a previous blog post. The API will serve us as an example for another service that our implementation depends on.
We have a simple ChuckNorrisFactController as the API for manual testing. Next to the “business” classes there is the ChuckNorrisService that does the call to the external API. It uses Spring’s RestTemplate. Nothing special.
What I have seen many times are tests that mock the RestTemplate and return some pre-canned answer. The implementation could look like this:
@Service
public class ChuckNorrisService{
...
  public ChuckNorrisFact retrieveFact() {
    ResponseEntity<ChuckNorrisFactResponse> response = restTemplate.getForEntity(url, ChuckNorrisFactResponse.class);
    return Optional.ofNullable(response.getBody()).map(ChuckNorrisFactResponse::getFact).orElse(BACKUP_FACT);
  }
 ...
 }
Next to the usual unit tests checking for the success cases there would be at least one test covering the error case, i.e. a 4xx or 5xx status code:
  @Test
  public void shouldReturnBackupFactInCaseOfError() {
    String url = "http://localhost:8080";
    RestTemplate mockTemplate = mock(RestTemplate.class);
    ResponseEntity<ChuckNorrisFactResponse> responseEntity = new ResponseEntity<>(HttpStatus.SERVICE_UNAVAILABLE);
    when(mockTemplate.getForEntity(url, ChuckNorrisFactResponse.class)).thenReturn(responseEntity);
    var service = new ChuckNorrisService(mockTemplate, url);

    ChuckNorrisFact retrieved = service.retrieveFact();

    assertThat(retrieved).isEqualTo(ChuckNorrisService.BACKUP_FACT);
  }
Doesn’t look bad, right? The response entity returns a 503 error code and our service will not crash. All tests are green and we can deploy our application.
Unfortunately, Spring’s RestTemplate does not work like this. The method signature of getForEntity gives us a very small hint. It states throws RestClientException. And this is where the mocked RestTemplate differs from the actual implementation. We will never receive a ResponseEntity with a 4xx or 5xx status code. The RestTemplate will throw a subclass of RestClientException. Looking at the class hierarchy we can get a good impression of what could be thrown:
Therefore, lets see how we can make this test better.

WireMock to the rescue

WireMock simulates web services by starting a mock server and returning answers you configured it to return. It is easy to integrate into your tests and mocking requests is also simple thanks to a nice DSL.
For JUnit 4 there is a WireMockRule that helps with starting an stopping the server. For JUnit 5 you will have to do it yourself. When you check the example project you can find the ChuckNorrisServiceIntegrationTest. It is a SpringBoot test based on JUnit 4. Let’s take a look at it.
The most important part is the ClassRule:
  @ClassRule
  public static WireMockRule wireMockRule = new WireMockRule();
As mentioned before, this will start and stop the WireMock server. You could also use the rule as normal Rule to start and stop the server for each test. For our test this isn’t necessary.
Next, you can see several configureWireMockFor... methods. These contain the instructions for WireMock when to return what answer. Splitting the WireMock configuration into several methods and calling them from the tests is my approach to using WireMock. Of course you could set up all possbile requests in an @Before method. For the success case we do:
  public void configureWireMockForOkResponse(ChuckNorrisFact fact) throws JsonProcessingException {
    ChuckNorrisFactResponse chuckNorrisFactResponse = new ChuckNorrisFactResponse("success", fact);
    stubFor(get(urlEqualTo("/jokes/random"))
        .willReturn(okJson(OBJECT_MAPPER.writeValueAsString(chuckNorrisFactResponse))));
  }
All methods are imported statically from com.github.tomakehurst.wiremock.client.WireMock. As you can see, we stub an HTTP GET to a path /jokes/random and return a JSON object. The okJson() method is just shorthand for a 200 response with JSON content. For the error case the code is even more simple:
  private void configureWireMockForErrorResponse() {
    stubFor(get(urlEqualTo("/jokes/random"))
        .willReturn(serverError()));
  }
As you can see, the DSL makes it easy to read the instructions.
Having WireMock in place we can see that our previous implementation does not work since the RestTemplate throws an exception. Therefore, we gotta adjust our code:
  public ChuckNorrisFact retrieveFact() {
    try {
      ResponseEntity<ChuckNorrisFactResponse> response = restTemplate.getForEntity(url, ChuckNorrisFactResponse.class);
      return Optional.ofNullable(response.getBody()).map(ChuckNorrisFactResponse::getFact).orElse(BACKUP_FACT);
    } catch (HttpStatusCodeException e){
      return BACKUP_FACT;
    }
  }
This already covers WireMock’s basic use-cases. Configure an answer for a request, execute the test, check the results. It’s as simple as that.
Still, there is one problem you will usually encounter when you run your tests in a cloud environment. Let’s see what we can do.

WireMock on a dynamic port

You might have noticed that the integration test in the project contains an ApplicationContextInitializer class and that its @TestPropertySource annotation overwrites the URL of the actual API. That is because I wanted to start WireMock on a random port. Of course you can configure a fixed port for WireMock and use this one as hard-coded value in your tests. But if your tests are running on some cloud providers infrastructure you cannot be sure that the port is free. Therefore, I think a random port is better.
Still, when using properties in a Spring application we have to pass the random port somehow to our service. Or, as you can see in the example, overwrite the URL. That is why we use the ApplicationContextInitializer. We add the dynamically assigned port to the application context and then we can refer to it by using the property ${wiremock.port}. The only disadvantage here is that we now have to use a ClassRule. Else, we couldn’t access the port before the Spring application is being initialized.
Having solved this problem, let’s take a look at one common problem when it comes to HTTP calls.

Timeouts

WireMock offers many more possibilities for responses than just simple answers to GET requests. Another test case that is often forgotten is testing timeouts. Developers tend to forget to set timeouts on the RestTemplate or even on URLConnections. Without timeouts both will wait for an infinite amount of time for responses. In the best case you will not notice, in the worst case all your threads wait for a response that will never arrive.
Therefore, we should add a test that simulates a timeout. Of course, we can also create a delay with e.g. a Mockito mock, but in that case we would guess again how the RestTemplate behaves. Simulating a delay with WireMock is pretty easy:
  private void configureWireMockForSlowResponse() throws JsonProcessingException {
    ChuckNorrisFactResponse chuckNorrisFactResponse = new ChuckNorrisFactResponse("success", new ChuckNorrisFact(1L, ""));
    stubFor(get(urlEqualTo("/jokes/random"))
        .willReturn(
            okJson(OBJECT_MAPPER.writeValueAsString(chuckNorrisFactResponse))
                .withFixedDelay((int) Duration.ofSeconds(10L).toMillis())));
  }
withFixedDelay() expects an int value representing milliseconds. I prefer using Duration or at least a constant that indicates that the parameter represents milliseconds without having to read the JavaDoc every time.
After setting a timeout on our RestTemplate and adding the test for the slow response we can see that the RestTemplate throws a ResourceAccessException. So we can either adjust the catch block to catch this exception and the HttpStatusCodeException or just catch the superclass of both:
  public ChuckNorrisFact retrieveFact() {
    try {
      ResponseEntity<ChuckNorrisFactResponse> response = restTemplate.getForEntity(url, ChuckNorrisFactResponse.class);
      return Optional.ofNullable(response.getBody()).map(ChuckNorrisFactResponse::getFact).orElse(BACKUP_FACT);
    } catch (RestClientException e){
      return BACKUP_FACT;
    }
  }
Now we have nicely covered the most common cases when doing HTTP requests and we can be sure that we are testing close to real world conditions.

Why not Hoverfly?

Another choice for HTTP integration tests is Hoverfly. It works similar to WireMock but I have come to prefer the latter. The reason is that WireMock is also quite useful when running end-to-end tests that include a browser. Hoverfly (at least the Java library) is limited by using JVM proxies. This might make it faster than WireMock but when e.g. some JavaScript code comes into play it does not work at all. The fact that WireMock starts a webserver is very useful when your browser code also calls some other services directly. You can then mock those with WireMock, too, and write e.g. your Selenium tests.

Conclusion

I hope this article could show you two things:
  1. the importance of integration tests
  2. that WireMock is pretty nice
Of course, both topics could fill many more articles. Still, I wanted to give you a feeling of how to use WireMock and what it is capable of. Feel free to check their documentation and try many more things. As an example, testing authentication with WireMock is also possible.

Wednesday, June 22, 2016

camunda BPM platform OSGi 2.0.0 released

It has been a while since we had the last release of camunda BPM platform OSGi that included some new features and I am glad to be able to announce the new major version today.

The new version includes one new feature,some dependency adjustments and a restructuring of the whole project.

The new feature is the OSGi Event Bridge, which I already explained here. So now you'll be able to receive camunda process events in an OSGi way.

The most notable change in the dependencies is the change from OSGi 4.2 to version 4.3. This version enables e.g. the usage of generics and of the Require-Capabilityand Provide-Capability headers (one example how you could use them is explained in another blog post).

Finally, the whole project is now more modularized. Using one of the 1.x.x versions, many features were included in the camunda-bpm-osgi module, which you always needed. That ways, you would always have the classes for file install, process application or Blueprint present, if you used them or not. With the new structure you can better choose, which features you want to use and which not to.

Configadmin, Fileinstall and Processapplication are now separate bundles and no longer contained in camunda-bpm-osgi. What is left in the "main" module are the capabilities to find process definitions in your bundles, EL resolving, locating scripting engines and utility classes, e.g. for classloading. Also, all integration tests (except for the Karaf ones), are now located in a central itest module.

I hope all those changes ease the use for you to combine the powers of OSGi and camunda BPM. If you have any feedback or would like to make a wish for a new feature, feel free to leave a comment, open an issue on GitHub or open a pull request.

Wednesday, February 24, 2016

Extension/Service/Plugin mechanisms in Java

Since I started to deep dive into OSGi I was wondering more and more how frameworks that have some way of extension mechanism, e.g. Apache Camel where you can define your own endpoint or the Eclipse IDE with its plugins, handle finding and instantiating extensions. I remember very well a presentation from the JAX 2013, it was by Kai Tödter, where he showed the combination of Vaadin and OSGi. While the web app was running he could add and remove menu entries, just by starting and stopping the bundles.
For a while now I have taken a look at several approaches on how to create an extensible application and you can find resources for every single method. I want to give a medium sized (not short ;)) overview here of the different ways I know to make a Java application extensible. Also, I will add a list of advantages and disadvantages, from my point of view, to each method. For every method I try to give a simple example.
To avoid confusion, when I write about the advantages and disadvantages, I will write from the point of view, as if you want to provide this extension mechanism in your framework, not from the API consumer point of view.

Passing the object

This is the most obvious method. The framework defines a method which takes the SPI interface and you simply pass the object. Camel, next to other methods, makes use of this (example taken from the Camel FAQ):
CamelContext context = new DefaultCamelContext();
context.addComponent("foo", new FooComponent(context));
Internally, Camel doesn't do much magic (code taken from Camel on GitHub).
public void addComponent(String componentName, final Component component) {
    ObjectHelper.notNull(component, "component");
    synchronized (components) {
        if (components.containsKey(componentName)) {
            throw new IllegalArgumentException("Cannot add component as its already previously added: " + componentName);
        }
        component.setCamelContext(this);
        components.put(componentName, component);
        for (LifecycleStrategy strategy : lifecycleStrategies) {
            strategy.onComponentAdd(componentName, component);
        }

        // keep reference to properties component up to date
        if (component instanceof PropertiesComponent && "properties".equals(componentName)) {
            propertiesComponent = (PropertiesComponent) component;
        }
    }
}
Every component has to have an unique name and is somehow bound to a lifecycle. Removal of a component is also possible, but has to be made somewhere from the user code.

Advantages

  • Easy and straightforward
  • No need for an additional framework
  • Compiler checks for the correct interface

Disadvantages

  • Access to central class (the plugin/service/component holder) is necessary
  • Allowing changes during the runtime is possible but complicated, since it has to be assured the component is removed everywhere
  • Your framework has to take care of the whole component lifecycle and any additional requirements it enforces

Interface and Reflection

This method is used quite often (basically it is also how the ServiceLoader works, see next section) and you can find it with small variances. The differences are where and how exactly interface and implementation name reach the application. Placing them somewhere inside a properties file or passing them to the framework during startup are most common. The implementation is then instantiated using reflection. Creating a context with an InitialContextFactory works like this e.g.:
  Properties env = new Properties();
  env.put(Context.INITIAL_CONTEXT_FACTORY,
          "org.jboss.naming.remote.client.InitialContextFactory");

Advantages

  • Easy and straightforward
  • No need for an additional framework
  • No need to provide central class (in properties file approach)

Disadvantages

  • No type safety (if text based)
  • Your framework has to take care of the whole lifecycle and any additional requirements it enforces
  • Check for correct wiring only during runtime (if text based, check either at startup or when the code is being called, where the former is better than the latter)

java.util.ServiceLoader

Frameworks using the java.util.ServiceLoader can also be found quite often. What the ServiceLoader does is, it uses during runtime a ClassLoader and checks the META-INF/services directory for a text file, whose name equals the passed interface (SPI) name and then reads the class name inside that file. Then it instantiates the class via Reflection. All the magic happens in the LazyIterator inside the ServiceLoader class (see OpenJDK). Basically, it's just reading a file and instantiating the object. E.g. Camel and HiveMQ use this method.

Advantages

  • Easy and straightforward
  • ServiceLoader is part of JDK
  • No need for an additional framework

Disadvantages

  • No lifecycle
  • Class has to provide standard constructor
  • Support for runtime changes must be implemented (as mentioned here)
  • Check for correct wiring only during runtime (the filename or the string inside the file could be wrong)

(Eclipse) Extension Points

Picture under BSD license, see here
As far as I know the concept of Extension Points never got popular outside Eclipse, although it is possible to include them in every application. To achieve loose coupling the definition of places where you can add your plugin and the plugins themselves is extracted into XML files.
To define an extension point you need something like this:
<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
   <extension-point
     id="de.blogspot.wrongtracks.FooService"
     name="FooService"
     schema="schema/de.blogspot.wrongtracks.FooService.exsd"/>
The extension provider then has to define an appropriate extension for that point:
<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
   <extension
         point="de.blogspot.wrongtracks.FooService">
      <implementation
            class="com.example.impl.FooServiceImpl"
            id="com.example.impl.FooServiceImpl"
            name="FooServiceImpl">
      </implementation>
   </extension>
</plugin>
I got to admit, that I am not completely sure how exactly you can integrate the extension points, but I guess you will need quite a lot from the basic Eclipse runtime. There is a blog post, which explains how you can use extension points without depending on OSGi.

Advantages

  • Extensions can be added during runtime
  • Good tool support inside Eclipse
  • Wrong wiring only affects single extension
  • Loose coupling (more or less, since the extensions depend on the extension point id)

Disadvantages

  • Dependencies to Eclipse
  • Overhead from the Eclipse platform (I actually cannot prove this point but I assume there must be a considerate overhead involved in comparison to the previous methods)
  • Check for correct wiring only during runtime

Spring XML

The Spring framework tried to find a way for loosely coupled components long before CDI, as we know it today, appeared. Their solution was an XML file in which the different classes are being wired together (I am well aware of the fact that nowadays there are also other ways, but since they are also based on annotations they don't differ enough from CDI as that I'll give them an own paragraph). In the basic XML file you define all your beans and Spring will take care of the instantiation. It is also possible to distribute the configuration among several XML files. A very simple example (taken and modified from the Spring documentation) looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="accountDao"
        class="org.springframework.samples.jpetstore.dao.jpa.JpaAccountDao">
    </bean>

    <bean id="petStore" class="org.springframework.samples.jpetstore.services.PetStoreServiceImpl">
        <property name="accountDao" ref="accountDao"/>
    </bean>
</beans>
If you want to provide your users a way to add their services/plugins to the framework, you'll have to provide a setter method where the users can add their object. E.g. like this (taken from camunda documentation):
<bean id="processEngineConfiguration" class="org.camunda.bpm.engine.spring.SpringProcessEngineConfiguration">
  ...
  <property name="processEnginePlugins">
    <list>
      <bean id="spinPlugin" class="org.camunda.spin.plugin.impl.SpinProcessEnginePlugin" />
    </list>
  </property>
</bean>

Advantages

  • Spring is lightweigth
  • Lifecycle support from Spring

Disadvantages

  • XML needs to be maintained
  • No auto detection, users have to write the XML when they want to add something
  • The Spring IoC container is needed
  • Correct wiring is only checked at startup

OSGi Services

OSGi was created embracing runtime changes and bundles dynamically providing and removing their services. With this in mind OSGi strongly supports applications being extended by services, provided by different bundles. The simplest approach is to implement a ServiceListener or a ServiceTracker. Both should be created on bundle start and they will react when a new implementation of the service appears. A ServiceListener can be as simple as this (taken from the Knoplerfish tutorial):
 ServiceListener sl = new ServiceListener() {
   public void serviceChanged(ServiceEvent ev) {
      ServiceReference sr = ev.getServiceReference();
      switch(ev.getType()) {
        case ServiceEvent.REGISTERED:
          {
             HttpService http = (HttpService)bc.getService(sr);
             http.registerServlet(...);
          }
          break;
        default:
          break;
      }
   }
 };

 String filter = "(objectclass=" + HttpService.class.getName() + ")";
 bc.addServiceListener(sl, filter);
Where bc is a BundleContext object. And a ServiceTracker can be used like this:
ServiceTracker<HttpService,HttpService> serviceTracker = new ServiceTracker<HttpService, HttpService>(bc, HttpService.class, null);
serviceTracker.open();
There are more elegant ways to get hold of an OSGi service using Blueprint, Declarative Services or the Apache Felix Dependency Manager but the ServiceListener is the basic way.

Advantages

  • OSGi lifecycle support
  • Changes during runtime "encouraged" ;)
  • Compiler checks wiring (not for the ServiceListener but for the rest)
  • Problems with services are restricted to single bundle

Disadvantages

  • You have to buy the whole OSGi package: imports, exports, bundles and everything
  • Having the full OSGi lifecycle makes the world more complicated since every service can disappear at every moment

Note about PojoSR/OSGi Light

Since the biggest disadvantage of OSGi is that you have to get the whole package, I want to mention here another approach, which is called PojoSR or OSGi Light. The goal of it is to give you the OSGi service concept without the rest that comes with OSGi. Unfortunately, I could not find much documentation about it and the activity around this project seems to be very low at the moment. There is an article here and the PojoSR framework itself. Also, it looks like PojoSR is now a part of Apache Felix called "Connect", but its version is 0.1.0. So if anyone of you knows more about it, please let me know.

CDI

Contexts and Dependency injection was a big step for Java EE, allowing developers to write more loosely coupled code. The CDI container takes care of automagically wiring the different parts together. The developer only has to use the correct annotations. Depending on which CDI beans are present at runtime, concrete implementations can be changed without changing the code that uses them. When trying to use a class the basic injection looks like this:
@Inject
private MyServiceInterface service;
If there is need to get all of the implementations (which we actually want here), then the class Instance must be used:
@Inject @Any
private Instance<MyServiceInterface> services;
Since Instance is an Iterable a simple for-each loop can be used to access all the objects. Alternatively the select() method can be used to further specify requirements.

Advantages

  • Compiler checks for correct type
  • CDI container checks correct wiring at startup
  • Part of JEE standard but can also be used without application serve (use a JSR-330 implementation like Guice or HK2)r
  • CDI lifecycle support

Disadvantages

  • A CDI container is needed
  • Changes during runtime are not possible
  • Annotatiomania (at least if you don't watch out)

Summary

As you can see many different frameworks/methods evolved in the Java ecosystem. Every single one with its specific advantages and disadvantages. I think we can summarize the different extension mechanisms as three types (with their members):
  1. String and well-known location ("Interface and Reflection", "ServiceLoader", "(Eclipse) Extension Points", "Spring XML")
  2. Programmatic wiring ("Passing the object", "Interface and Reflection", "OSGi Services")
  3. Classpath scanning ("CDI")
Of course the three types are not exclusive. You may provide your users more than one way and let them choose. Also CDI is not exactly the only framework that uses classpath scanning. Spring with its two other ways for configuring the IoC container relies on that method, too.

I hope this article provides an good and sufficient overview of the different methods on how to create an extensible framework. Choosing the right one will make your users surely happy. If you know another method, which I forgot, please let me know, I will gladly add it here.

Please note that the lists of advantages and disadvantages are based on my reasoning. I tried to be objective but like every programmer I have my favorites and my experiences with the frameworks that may make me a little bit biased.

Saturday, February 13, 2016

camunda BPM OSGi - Event Bridge

I have implemented the eventing feature already some months ago but I haven't managed to advertise it a little bit more until now. So, let's praise my work ;)

I'll start with some background information, which you can skip if you're familiar with camunda BPM and the OSGi EventAdmin. Then, some information about the what and how follows.

Let's start with OSGi eventing.

OSGi Event Admin

The Event Admin is a part of the OSGi Compendium Specification. It is a way to communicate between bundles in a decoupled way by sending events. The communication follows a publish/subcribe scheme.

One bundle obtains the EventAdmin service, creates an Event object and sends it. Every event is created with a certain topic and can contain arbitrary String properties in a key-value way. Topics are hierarchical separated by a "/" and wildcards are allowed. E.g. org/osgi/framework/BundleEvent/STARTED is a topic used by the OSGi framework.

Events can be sent in a synchronous or asynchronous way and additional LDAP filters can be used based on the properties.

You can find a good example on the Apache Felix website.

Now that we know a little bit about the EventAdmin let's take a look at camunda BPM.

camunda BPM events

During the execution of a process certain events occur, e.g. a task is being assigned or a process end. To be able to "see" those events the user has to register either an ExecutionListener or a TaskListener (for more details see here and here).

The common way to register the listeners is to directly add them to the process definition, i.e. the .bpmn file. But there are certainly cases where we do not own the process file but would like to receive events (e.g. for monitoring).

Let's see how to achieve this in an OSGi environment.

camunda BPM OSGi - Event Bridge

I gotta admit the idea of an event bridge is not my own, because the CDI extension for camunda BPM already has an CDI event bridge. Anyways, for OSGi this feature was missing. I'll explain to you what happens internally and how you can use it.

What happens?

The OSGi event bridge implementation exports a service that is a BpmnParseListener. Whenever the engine parses a process definition this listener will become active and attach TaskListener and ExecutionListener wherever possible. But these listeners aren't full implementations. They are dynamic proxies with a special InvocationHandler.

When the InvocationHandler is being invoked it checks if the OSGi event bridge is still active and if the EventAdmin is present. If yes, it instantiates a new OSGiEventDistributor, which creates a new event and fills the properties.

I've tried to use all properties the camunda events provide and put them into the event properties. You can see a full list in this class.

This is basically what is happening. So, what can you do with the event bridge?

How to use it?

Before you can make use of the OSGi event bridge you have to add the OSGiEventBridgeActivator as a BpmnParseListener to your ProcessEngineConfiguration. You do this with the method setCustomPreBPMNParseListeners(). Unfortunately, there is no way to add the listener to an already created engine. After adding the listener events are being published. The event topics are:
  • org/camunda/bpm/extension/osgi/eventing/TaskEvent
  • org/camunda/bpm/extension/osgi/eventing/Execution
Of course you can use an asterisk after ../eventing/ to match both.

Wherever you want to listen to events, you can create your own EventHandler and subscribe to the topic you need/want. A simple example would be:

EventHandler eventHandler = new EventHandler() {
  @Override
  public void handleEvent(Event event) {
    Logger.getLogger("Event occured: " + event.getTopic());
  }
};
Dictionary props = new Hashtable();
props.put(org.osgi.service.event.EventConstants.EVENT_TOPIC, org.camunda.bpm.extension.osgi.eventing.api.Topics.ALL_EVENTING_EVENTS_TOPIC);
bundleContext.registerService(EventHandler.class.getName(), eventHandler, props);

Since many information is inside the event properties you can also use a more sophisticated LDAP filter expression based on that information. E.g. if you only want to receive events for a certain process you can do this:

EventHandler eventHandler = new EventHandler() {
...
};
Dictionary<String, String> props = new Hashtable<String, String>();props.put(EventConstants.EVENT_TOPIC, Topics.ALL_EVENTING_EVENTS_TOPIC);
props.put(EventConstants.EVENT_FILTER, "(processDefinitionId=invoice");
bundleContext.registerService(EventHandler.class.getName(), eventHandler, props);

And that's it. At the moment there is no way to limit the applications that are allowed to receive events, so everybody can see all the events if he subscribes to them. If you have an idea how to do this in a nice way, please let me know.

I hope you can make good use of the OSGi event bridge. My plan is to release camunda BPM OSGi 2.0.0 (which includes the event bridge) shortly after camunda BPM 7.5.0 is being released.

Thursday, January 29, 2015

Cluster your service with the ConfigurationAdmin and Apache Karaf Cellar using the camunda BPM engine as example

Introduction

Initially, this was supposed to be a short introduction about the topic in the title and an opportunity for me to get to know Apache Karaf Cellar. Unfortunately, I couldn't finish the topic until today because I had some unexpected problems. So basically this is going to be a post about the problems I encountered. At the end you'll find a TL;DR; if you just want to get started.

Short introduction into the Configuration Admin Service

From the OSGi wiki: "Configuration Admin is a service which allows configuration information to be passed into components in order to initialise them, without having a dependency on where or how that configuration information is stored."(http://wiki.osgi.org/wiki/Configuration_Admin)

Basically you write a key-value property and a service which can use it. All the "magic" is done by the ConfigurationAdminService, which is part of the OSGi Compendium Specification. A good introduction can be found here. Also the Admin will store it somewhere for you.

Short introduction into Apache Karaf Cellar

Taken from the Cellar website: "Cellar is a clustering solution for Apache Karaf powered by Hazelcast. Cellar allows you to manage a cluster of Karaf instances, providing synchronisation between instances."(http://karaf.apache.org/index/subprojects/cellar.html)

I liked the idea to provide a service on one Karaf instance and see it appear on every instance in the cluster. Especially the combination with a MangedServiceFactory seems like a great idea.

To read more about Cellar see here.

Set up your Apache Karaf

For my example I want to use my MangedProcessEngineFactory from the 1.1.0-SNAPSHOT version of camunda BPM OSGi. You can just clone the repository on GitHub and built it with mvn install.

Because I am quite lazy I started two Karaf instances on my laptop. If you want to do that, too, you'll have to change some port numbers for the second Karaf instance. First, the ports in the etc/org.apache.karaf.management.cfg:

rmiRegistryPort
rmiServerPort

Second the SSH port in the etc/org.apache.karaf.shell.cfg (forgetting this caused me a some trouble). Next we gotta install Cellar on each Karaf instance. Because we want to use the current version, we'll use version 3.0.1 of Cellar. You can find the general installation guide here for instructions about installation and start. Basically you just have to call from the Karaf console

feature:repo-add mvn:org.apache.karaf.cellar/apache-karaf-cellar/3.0.1/xml/features
feature:install cellar

If you somehow plan to build Cellar yourself, I'll recommend to comment out the "samples" module in the root POM. All your Karaf instances should discover each other automatically. Now we got to install and share the camunda-feature (or whichever you want to use) into the cluster.

Install and share a feature

To do this task we have two choices. One would be to activate the listeners in every Karaf instance and use the "basic" commands. Therefore you'll have to set the bundle listener value in the org.apache.karaf.cellar.node.cfg to true (we won't need the other ones in this example):

bundle.listener = true
config.listener = false
feature.listener = false

The other choice would be to use the cluster:* commands. Both will (should) produce the same result so choose whichever you prefer.

As I mentioned, if you prefer the first option (listeners), you can just install everything as usual because the cluster synchronizes every change:

feature:repo-add mvn:org.camunda.bpm.extension.osgi/camunda-bpm-karaf-feature/1.1.0-SNAPSHOT/xml/features
feature:install camunda-bpm-karaf-feature-minimal

(Please note that you'll need my example project installed locally to use it)

(Also please note that there is currently a bug in the camunda feature.xml. You'll have to change the version of camunda-connect-core to 1.0.0-alpha3 to make it work)

If you want to use the "cluster-versions" of those commands, you have to type:

cluster:feature-repo-add default mvn:org.camunda.bpm.extension.osgi/camunda-bpm-karaf-feature/1.1.0-SNAPSHOT/xml/features
cluster:feature-install default camunda-bpm-karaf-feature-minimal

Those commands work like the basic ones but you always have to provide a group.

You should see that the feature got installed on both Karaf instances (check e.g. with features:list | grep -i camunda). Now we need a database.

Setting up the database

I gotta admit, this is were my first problems occurred. Starting from funny and ending at a being a little bit annoyed. My first problem was that I tried to use the in-memory version of H2. This won't work because, logically, every Karaf instance runs in its own JVM. So, because of multiple applications, I started h2 in server mode (see here for more information).

java -cp h2*.jar org.h2.tools.Server jdbc:h2:tcp://localhost/~/test

The next problem was that because of some exceptions the ProcessEngines started and stopped in seemingly random orders. Having the databaseSchemaUpdate property set to create-drop caused problems with tables not being present because of random dropping/creating. I recommend to create the tables yourself (here are the sqls).

This didn't solve all of my database problems. I suspected H2 of not being capable of handling the same user logging in twice (which it is capable of as far as I know now). After that I switched to MySQL.

Setting up MySQL in Karaf

MySQL is a little bit more complicated to set up than H2 because we have to create a proper datasource. First, we need to install Apache Karaf DataSources:

feature:install jdbc

Next, create the datasource

jdbc:create -u sa -p sa -url jdbc:mysql://localhost:3306/test -t MySQL test

The datasource create command has to be executed on both Karafs because the datasource-*.xml that'll be created in the deploy directory won't be copied. For the ProcessEngine to be able to find the MySQL datasource it needs a JNDI name. To give a datasource a JNDI name we need Apache Karaf Naming.

feature:install jndi

Now the datasource will automatically get a JNDI name (check with jndi:names). If you don't see the jndi:* commands you'll have to install the feature manually on the second Karaf.

Finally we need the MySQL connector jar. We can find it here. Simply drop the jar into the deploy directory.

The MySQL database works fine for me so far. Let's take a look at the configuration file.

The configuration file

When I started with this "experiment" I thought that making the use of the etc/ directory in Karaf would be a good idea but now I gotta say: Please, don't try to do this file based. I tried a lot of combinations and it didn't work out. The closest I got was the configuration arriving on both Karafs but only one engine being created. Jean-Baptiste and Achim were really trying to help me on the mailing list. Nevertheless, I couldn't get it running. You are free to try.

Karaf watches the etc/ directory for configuration files. To deploy one for the ManagedProcessEngineFactroy you'll have to name it org.camunda.bpm.extension.osgi.configadmin.ManagedProcessEngineFactory-1.cfg.

I switched to a bundle which contains the configuration.

The configuration bundle

As mentioned before, for a ManagedServiceFactory to create a service it needs one or more configurations. We'll use a simple version of the configuration:

databaseSchemaUpdate=false
jobExecutorActivate=true
processEngineName=TestEngine
databaseType=mysql
dataSourceJndiName=osgi:service/jdbc/test

If you want to try H2, the configuration would look like this:

databaseSchemaUpdate=false
jdbcUrl=jdbc:h2:tcp://localhost/~/test
jobExecutorActivate=true
processEngineName=TestEngine
jdbcUsername=sa
jdbcPassword=sa

To make it simple the bundle just uses a BundleActivator, gets hold of the Configuration Admin and provides the property, like this:

public class Activator implements BundleActivator {

    public void start(BundleContext context) throws Exception {
        ServiceReference ref = context.getServiceReference(ConfigurationAdmin.class.getName());
        ConfigurationAdmin admin = (ConfigurationAdmin) context.getService(ref);
        String pid = "org.camunda.bpm.extension.osgi.configadmin.ManagedProcessEngineFactory";
        Configuration configuration = admin.createFactoryConfiguration(pid, null);
        Hashtable properties = new Hashtable();
        properties.put("databaseSchemaUpdate","false");
        properties.put("jobExecutorActivate","true");
        properties.put("processEngineName","TestEngine");
        properties.put("databaseType","mysql");
        properties.put("dataSourceJndiName", "osgi:service/jdbc/test");
        configuration.update(properties);
    }

The activated bundle listener should provide the bundle to all Karafs. Just drop the bundle into the deploy directory.

You should see that the configuration got shared, too. To check just run this command: config:list "(service.pid=org.camunda.bpm.extension.osgi.configadmin.ManagedProcessEngineFactory*)"

TL;DR;

  1. change port numbers in etc/org.apache.karaf.management.cfg and etc/org.apache.karaf.shell.cfg if you run two instances on one machine
  2. feature:repo-add mvn:org.apache.karaf.cellar/apache-karaf-cellar/3.0.1/xml/features
  3. feature:install cellar
    1. Decide if you want to activate the listener or use the cluster:commands for the following things
  4. git clone https://github.com/camunda/camunda-bpm-platform-osgi.git
  5. mvn install the project
  6. feature:repo-add mvn:org.camunda.bpm.extension.osgi/camunda-bpm-karaf-feature/1.1.0-SNAPSHOT/xml/features
  7. feature:install camunda-bpm-karaf-feature-minimal
  8. set up MySQL databse
  9. feature:install jdbc
  10. drop MySQL connector jar into deploy directory
  11. jdbc:create -u sa -p sa -url jdbc:mysql://localhost:3306/test -t MySQL test
  12. feature:install jndi
  13. create configuration bundle and drop it into deploy directory. Configuration:
    databaseSchemaUpdate=false
    jobExecutorActivate=true
    processEngineName=TestEngine
    databaseType=mysql
    dataSourceJndiName=osgi:service/jdbc/test

And you're good to go.

So, this was my trip into the Karaf Cellar world. I hope I could prove the feasibility to you. I'll leave the practical consequences as an exercise to the reader ;-)

Tuesday, November 18, 2014

camunda BPM engine: use custom VariableType to resist the urge to flush

Introduction

I hope all of you are aware of the fact that you can provide a ProcessEnginewith your own VariableTypes. If not, I'll give you a short introduction. Please note that my descriptions are based on camunda-engine 7.1.0. There will be some changes in versoin 7.2.0 and I am not sure if my observations will still be true.

VariableType

 

VariableTypes help the ProcessEngine store your process variables in the table ACT_RU_VARIABLE. I would call them a mediator between the possible variables and the database schema. There are VariableType implementations for
  • Boolean
  • Serizable
  • Date
  • Double
  • Integer
  • JPA Entities
  • Long
  • Null
  • Short
  • String
  • and CustomObjects (about which I'll talk later)

If you try to add an object as process variable, which doesn't belong to one of those types, you'll see this exception:

org.camunda.bpm.engine.ProcessEngineException: couldn't find a variable type that is able to serialize \<object\>
    at org.camunda.bpm.engine.impl.variable.DefaultVariableTypes.findVariableType(DefaultVariableTypes.java:62)
    at org.camunda.bpm.engine.impl.persistence.entity.VariableScopeImpl.getNewVariableType(VariableScopeImpl.java:315)
    at org.camunda.bpm.engine.impl.persistence.entity.VariableScopeImpl.createVariableInstance(VariableScopeImpl.java:395)
    at org.camunda.bpm.engine.impl.persistence.entity.VariableScopeImpl.createVariableLocal(VariableScopeImpl.java:332)
    at org.camunda.bpm.engine.impl.persistence.entity.VariableScopeImpl.setVariable(VariableScopeImpl.java:259)
    at org.camunda.bpm.engine.impl.persistence.entity.VariableScopeImpl.setVariable(VariableScopeImpl.java:242)
    at de.blogspot.wrongtracks.StoreDataDelegate.execute(StoreDataDelegate.java:9)
    at org.camunda.bpm.engine.impl.delegate.JavaDelegateInvocation.invoke(JavaDelegateInvocation.java:34)
    at org.camunda.bpm.engine.impl.delegate.DelegateInvocation.proceed(DelegateInvocation.java:39)
    at org.camunda.bpm.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:42)
    at org.camunda.bpm.engine.impl.bpmn.behavior.ServiceTaskJavaDelegateActivityBehavior.execute(ServiceTaskJavaDelegateActivityBehavior.java:49)


Provide your variable type

 

Every ProcessEngineConfiguration should have the methods setCustomPostVariableTypes(List<VariableType>) and setCustomPreVariableTypes(List<VariableType>) so you can add your variable types when configuring the engine.
But wait, why are there two methods, pre and post?
When searching which VariableType can handle the object you want to store as process variable the engine iterates over the list of VariableTypes and the first one, which can handle the object, wins. Maybe you want your own types to have precedence over the default types.

Flushing

 

Now that you know about VariableTypes I want to present to you my use case.

The case

 

Imagine a process that's supposed to run synchronously (i.e. without a wait state) within a JTA transaction and every task needs a result from the preceding one. Additionally, the results are JPA Entities. By default the JPAEntityVariableType would take care of the entity.
The implementation shows that every time setValue() is called the JPAEntityVariableType calls flush() on the EntityManager. Since the process runs synchronously within a transaction the flush results in unnecessary queries on my database during process execution.

The solution

 

Here comes the CustomObjectType class. The CustomObjectType only needs a name and a class to work. The class is used to determine if it can handle a certain object. The CustomObjectType stores all objects in the cache of the ValueField. To get rid of the flush I instantiated a CustomObjectType with the class of my result and passed it to the configuration. Now, every time I put an entity inside the process variables the CustomObjectType places them inside the cache and no flush is called.

 

 The downside

 

Well, nothing comes without a price: If I should ever need a wait state my solution won't work and I'll have to find another solution or live with the flush.

 

Alternatives

 

I am not sure if my solution is the best way to solve my problem. If anyone knows a better way please let me know.

 

Small example

 

I also created a small example to show the use of the CustomObjectType here on GitHub

Wednesday, January 22, 2014

Activiti/camunda BPM: custom behavior and BPMN extension elements using Blueprint

Introduction 

 

In the last few weeks I have been working on a problem regarding OSGi-Blueprint and Activiti. Because it wasn't as easy as I would have hoped I want to share my solution with you. I will start by explaining my environment, show you the problem and then I will explain my first attempt. After that I will present my solution. Finally I will show some ideas how to make it better and things that I did not test.

Just a short hint about the writing: when I reference a class or some XML it's written in italic, e.g. Object. When you see "process engine", I am talking about the whole thing, but when you see ProcessEngine it's the actual class.

My environment

 

I use the Activiti-framework in version 5.12.1, Apache Aries in version 1.0.0 with a little modification and my own ProSt bundles. ProSt can be found here. The README.md explains why and what I changed in Aries.

I haven't tried, yet, but I am pretty sure that camunda BPM suffers the same problem because
both share the same MailActivityBehavior and BlueprintELResolver classes and use <extension-elements> for injection. So if you prefer camunda and see "activiti" somewhere you just have to replace it with "camunda" in your head ;-)

The problem 

 

In general, I just wanted to send an e-mail during my process-execution. Sounds pretty easy, right?

My SendMailWithAttachmentBehaviour class extends the previously mentioned MailActivitiBehavior class. The process definition contains all the necessary information to send the e-mail, e.g. from, to and subject. Only the attachment is missing, which I get from the execution environment.

Because I use Blueprint I cannot use the activit:class or type="mail" attributes in the process definition. I have to declare the class this way:
activiti:deleExpression="${sendMailWithAttachmentBehavior}"

A little hint: the name in the braces has to match the one used as bean id in the blueprint.xml.

The other ways do not work with OSGi because of class visibility etc.

The easy part was to extend the BlueprintELResolver class (ProStBlueprintELResolver) and add a way to add custom behavior classes at the moment.

So, what happens when the process engine tries to resolve the expression?
When the bundle is loaded Blueprint creates a dynamic proxy and registers it at the ProStBlueprintELManager.
After the process reaches the ServiceTask which delegates to the ${sendMailWithAttachmentBehaviour} the process-engine asks its ExpressionLanguageResolvers if they know something with the name "sendMailWithAttachmentBehaviour". Logically the proxy is found.
After that the process engine tries to set the extension-elements at the class.
First it tries to find setter methods and if it cannot find setters it tries field injection. (see ClassDelegate.applyFieldDeclaration())
Both ways do not work.
But why?
Of course a proxy does not have any fields. But why is it not possible to just add the setters to the SendMailWithAttachmendBehaviour class?
The call is proxy.getClass().getMethods() and according to the documentation this will return all the methods of the interfaces that the proxy was created with. ActivityBehavior does not declare the setSubject() etc. methods because they are only needed for e-mails.

First attempt

 

At first I thought the solution was quite obvious. I would just export a second interface containing the setters like this:

<bean id="sendMailWithAttachment" class="de.blogspot.wrongtracks.prost.example.behaviour.SendMailWithAttachmentBehaviour" />
  
<service ref="sendMailWithAttachment">
  <interfaces>
<value>org.activiti.engine.impl.pvm.delegate.ActivityBehavior</value>
<value>de.blogspot.wrongtracks.prost.example.behavior.ExtensionElementsMailSetter</value>
  </interfaces>
</services>
But wait, if you take a look at the (old) context.xml you can see that my reference listener just listens for ActivityBehavior and not the other interface. That's why the created proxy won't contain the methods from the other interface. Too bad...

The solution

 

I found the solution accidentally while reading the Apache Aries Blueprint documentation. This chapter points out that you can also listen for service references.
I changed the methods to accept a ServiceReference instead of a ActivitiyBehavior and when the expression should be resolved I use the BundleContext to get the service. At that point it is not a proxy, it is the implementation.

Then everything works just fine. I don't even need the setter interface anymore.
You can see the solution when you look at the new context.xml and the previously mentioned ProStBlueprintELResolver. (the previously showed link pointed to an old version so nothing would be spoiled ;-) )

That's it, that is my solution to add custom behavior to the process engine and use extension elements in the BPMN XML.

How could we improve the whole thing?

 

Strangely, I have no idea how the whole thing could be improved. I would like to hear your ideas. Also, I would like to know if you think that the way presented here is good or bad or something in between.

What didn't I try?

 

You should note that I have not tried to find out how JavaDelegates behave in the same situation. I just did not have time and I wanted to show you my solution as soon as I finished it.

Sunday, July 15, 2012

Sorry for the delay

I know I promised you some information about Ivy and I am really, really late.
But studies and watching the European Soccer Championship consumed lots of time.
And right now, I am back at learning some Python and I gotta admit (again) that

a = []

looks much more elegant than

List<Object> a = new ArrayList<Object>();

Of course the compact list syntax is a feature and Groovy e.g. has the same ability.

And now to the promised Ivy part (it won't be long, because I don't want to duplicate the tutorial, which can be found here):
For me, Ivy is quite straightforward with its ivy.xml and the dependencies in there and the ivy.xml looks like the dependencies in a Maven POM.
Also adding <ivy:retrieve/> to your build.xml shouldn't be too complicated ;)
Having different repositories like Maven is also possible.
And, what I recently found out is, that Gradle and SBT work with Ivy, too.

So, give it a try.

Saturday, June 16, 2012

Getting started with Apache Ivy

Well, no big knowlegde sharing today, I just wanna let you guys know what I am up to.
2 days ago I discovered, that there is something called Apache Ivy. I was quite surprised that I never recognised it before.
Ivy is a dependency manager and it seems to me like a good way to keep your Ant build.xmls and to have the (very comfortable) dependency resolution from Maven.

So, my task for the weekend is to find more out about Ivy. I hope I'll be able to write a little bit more on Sunday.

Tuesday, May 15, 2012

@Nonbinding

I experiment recently a lot with CDI and one point I stumbled upon were the return values of my annotation functions.
Consider an annotation like this:

@Qualifier
@Retention(...)
@Target(...)
public @interface MyInterface{
 String value();
}

I tried to combine this annotation with a producer method and to use the value in the method (by asking the InjectionPoint for its value). Every time my Eclipse tried to deploy the .war it showed me an error.
After a while I found out that the only way to evade this, is to use the annotation @Nonbindung.
The altered annotation looks like this:

@Qualifier
@Retention(...)
@Target(...)
public @interface MyInterface{
 @Nonbinding
 String value();
}


Maybe I didn't read the spec good enough, but it took me a while to find that out.
So I hope you guys won't have the same problem like me ;)

Thursday, May 3, 2012

new vs. valueOf() in Java primitive wrappers

Did you ever take a close look at the Javadoc which is written at, for example, Integer.valueOf()?

No? Ok, I'll tell you ;)

It says you should prefer using the valueOf() method instead of using Integer's constructor. You'll find the same annotation at all the other wrapper classes for the primitive types in Java. (in Double, the Javadoc says the same, but double doesn't have a cache, weird...)

The reason is that Java will cache some values, so the VM won't create a lot of new objects.
I cannot imagine a certain case, when I explicitly need a new Integer or something similar (maybe in a JUnit-Test). A friend of mine mentioned when it's the key of a WeakHashMap.

So the next time you want to type new Integer(...), consider using Integer.valueOf(...) ;)

 

Copyright @ 2013 Wrong tracks of a developer.

Designed by Templateiy