Assemble your custom Apache Karaf with the karaf-maven-plugin

I was quite happy to find out there is a Maven Plugin with which you can assembly a full Apache Karaf and include your own features/bundles.
From time to time I like to test my bundles in a real environment. Because of that the plugin is a great way to save the steps of unzipping a new Karaf, adding my feature and installing it.
So the plugin basically serves my laziness ;-)
But before the lazy part starts (for me and you) we have to do some work to get the plugin running.

I will start to describe the things I figured out. Then I will show you my final configuration and at the end I will talk about the problems I encountered.
Of course you can take a look at the documentation (here and here), too.

Karaf-assembly

 

To start your assembly project you just need an empty maven project with the packaging "karaf-assembly" and the plugin, of course.

To configure the features for the plugin (so the features will end up in the Karaf) there are three options:

1. startupFeature
2. bootFeature
3. installedFeature

Here is an example:

<configuration>
  <bootFeatures>
    <feature>standard</feature>
    <feature>management</feature>
    <feature>camunda-bpm-karaf-feature-minimal</feature> 
  </bootFeatures>
</configuration>

All three types result in a different configuration. Since I don't want to copy the documentation I'll give a very brief explanation.

startupFeatures

All the bundles from your feature will appear in the startup.properties, copied to system/ and started with the Karaf.

bootFeatures

All the bundles from your feature will be copied to system/. The features you listed will appear in org.apache.karaf.features.cfg and installed when starting Karaf. The path to your feature.xml will be added to org.apache.karaf.features.cfg as feature repository.

installFeatures

All the bundles from your feature will be copied to system/. The path to your feature.xml will be added to org.apache.karaf.features.cfg as feature repository.

You can see that every kind of *Features gets a little bit less serious than the one before. Please note that "compile" dependencies in your POM will be treated like a startupFeature.

All the dependencies you want to include have to be ether of type "kar" or have to have the classifier "feature" and type "xml", e.g:

<dependency>
  <groupId>org.apache.karaf.features</groupId>
  <artifactId>standard</artifactId>
  <version>3.0.2-SNAPSHOT</version>
  <classifier>features</classifier>
  <type>xml</type>
  <scope>runtime</scope>
</dependency>

Other dependencies will be ignored.

That was all I could figure out about the configuration of the plugin. Now let's have a look at my project.

My project

 

As mentioned before my project contains no classes or anything under src/resources. It just has the pom.xml that looks like this (Google Drive link).
I added a small shell script because the karaf start file wasn't executable and because I didn't want to move to target/assembly/... every time. Also I had a small problem with Java (see following heading).
The script looks like this:

export JAVA_HOME=$(/usr/libexec/java_home -v 1.6)
chmod 777 ./target/assembly/bin/karaf
./target/assembly/bin/karaf start

Nothing fancy ;-) So, that's already all about my project. Finally, I want to tell you about the problems I faced.

Issues

Plugin version

I had the problem that when a feature contained nested features the nested ones wouldn't be resolved. It took me a while and some remote debugging to find the problem. After I asked in the mailing list I was told that the problem existed in my version (3.0.1) and is fixed in the next one.
So you should definitely use the 3.0.2-SNAPSHOT version despite the fact that it's a snapshot. Jean-Baptiste did some great improvements in that version. The logging is way better and you can have nested features.

Ordering of dependencies

After upgrading my version I could see that all of my bundles were successfully installed into the system/ directory. But after starting my Karaf they weren't deployed. The "mvn:" URL for my feature was missing in the org.apache.karaf.features.cfg "featuresRepositories" property.
I found out that the problem was in the order of my dependencies.

My feature was the first dependency and then followed the Apache Karaf dependencies. Like this: 
<dependencies>
  <dependency>
    <groupId>org.camunda.bpm.extension.osgi</groupId>
    <artifactId>camunda-bpm-karaf-feature</artifactId>
    <version>1.1.0-SNAPSHOT</version>
    <classifier>features</classifier>
    <type>xml</type>
    <scope>runtime</scope>
  </dependency>
  <dependency>
    <groupId>org.apache.karaf.features</groupId>
    <artifactId>framework</artifactId>
    <version>3.0.2-SNAPSHOT</version>
    <type>kar</type>
  </dependency>
  <dependency>
    <groupId>org.apache.karaf.features</groupId>
    <artifactId>standard</artifactId>
    <version>3.0.2-SNAPSHOT</version>
    <classifier>features</classifier>
    <type>xml</type>
    <scope>runtime</scope>
  </dependency>
</dependencies>

The problem is that the framework Kar contains all the configuration files. So when the plugin tries to update the config-file with my feature it is not present. So be careful that the framework kar is your first dependency.

Java 8

 

Edit: As Jean-Baptiste told me (thank you again) the Java 8 problem is only related to version 3.0.1 which I can hereby confirm. So if you have followed my advice and use 3.0.2 you can skip this part.

Being the young and hip person I am ;-) my MacBook was already running Java 8. When I assembled and started a Karaf it would start without a problem (at least it seemed so). But hitting tab only showed this small amount of commands:

Every command, even help, would answer with a NullPointerException. The NPE itself looked like this:

2014-10-07 16:55:55,232 | ERROR | Local user karaf | ShellUtil                        | 37 - org.apache.karaf.shell.console - 3.0.1 | Exception caught while executing command
java.lang.NullPointerException
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:61)[37:org.apache.karaf.shell.console:3.0.1]
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)[37:org.apache.karaf.shell.console:3.0.1]
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:477)[37:org.apache.karaf.shell.console:3.0.1]
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:403)[37:org.apache.karaf.shell.console:3.0.1]
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)[37:org.apache.karaf.shell.console:3.0.1]

At first I thought something was missing. But checking the logs again, looking at what happened during startup, revealed some IllegalArgumentExceptions:

2014-10-07 16:53:23,402 | INFO  | FelixStartLevel  | ServiceRecipe                    | 19 - org.apache.aries.blueprint.core - 1.4.0 | Unable to create a proxy object for the service .component-1 defined in bundle org.apache.karaf.deployer.features at version 3.0.1 with id 25. Returning the original object instead.
java.lang.IllegalArgumentException
at org.objectweb.asm.ClassReader

I found out (thank you internet) that this is a Java 8 related problem. The command

export JAVA_HOME=$(/usr/libexec/java_home -v 1.6)

solved my problem. To always start my Karaf with Java 6 I added this line to my start script (see previous heading).

That was all about my karaf-maven-plugin experience. I am sure there are some more hidden things I couldn't figure out. I hope my experience will be useful for someone else.
Have fun with your own custom Karaf!

Read More

Create a ProcessEngine with the ConfigurationAdminService

There is a new feature in the camunda BPM OSGi extension and I would like to introduce it to you. So, let's start with the news.

What's new?

 

The OSGi extension now exports a ManagedServiceFactory to provide another way to configure and automatically share a ProcessEngine. The factory will be automatically exported when the OSGi compendium classes are present. You can then provide your configuration and the engine will be created and exported.

If you've never heard of the ConfigurationAdminService I would like to give you a short introduction.

What is the ConfigurationAdminService?

 

The ConfigurationAdminService is supposed to make the provision and change of configuration during easier. When you provide a configuration object (a dictionary) the service will find the according ManagedService or ManagedServiceFactory based on a pid (persistent id) and pass the configuration to it.

There are (way ;-) ) better descriptions in the OSGi Alliance blog and the Apache Felix documentation if you want to learn a little bit more about it. Let's see how we can use the service.

How to use it?

 

As I mentioned before the configuration is just a dictionary. The keys have to corresspondent to the fields of a ProcessEngineConfiguration object. Simply create a HashTable and put everything in it you need to run your engine:
    Hashtable<String, Object> props = new Hashtable<String, Object>();
    props.put("databaseSchemaUpdate", ProcessEngineConfiguration.DB_SCHEMA_UPDATE_CREATE_DROP);
    props.put("jdbcUrl", "jdbc:h2:mem:camunda;DB_CLOSE_DELAY=-1");
    props.put("jobExecutorActivate", true);
    props.put("processEngineName", "TestEngine");

Next you gotta get the ConfigurationAdminService and call createFactoryConfiguration() with the following PId:
org.camunda.bpm.extension.osgi.configadmin.ManagedProcessEngineFactory

There is also a constant in the ManagedProcessEngineFactory interface. After that pass your dictionary to the Configuration object by calling the update() method. And that's it. Your ProcessEngine will be created and exported.

Now that you know how to use the service I would like to tell you what makes it special.

Why use the ConfigurationAdminService?

 

I remember when I first read about the ConfigurationAdminService my thought was: "That's a really great idea!". By using the service you have several ways of providing configuration for your ProcessEngine. The easiest thing to image is that you store your configuration files in separate bundles. Every time something changes you update that bundle.

Depending on your environment there are more ways. In Apache Karaf you could place a file named
org.camunda.bpm.extension.osgi.configadmin.ManagedProcessEngineFactory.cfg
in the etc directory. Karaf would find the factory and pass the configuration to it.
Apache Felix and Equinox also provide ways to read and use configuration files.

Also, the ConfigurationAdminServices helps you to provide different configurations for different environments. At least text files are to change and provide than .class files.

Finally I want to tell you some details about the implementation.

How is it implemented?

 

I gotta admit that the implementation is not that special. The factory uses Commons BeanUtils to find the setters for the properties. Because the setters of ProcessEngineConfiguration provide a fluent way I couldn't use the classes BeanUtils or PropertyUtils. That's why I "combine" the setter-name on my own and invoke the method with MethodUtils.

Every time the configuration of a ProcessEngine changes I stop that engine, unregister it and create a new one and register that one. That is the only way to "change" the configuration of a ProcessEngine. Maybe a ProcessEngine/Configuration needs an update() method.

I would appreciate any hints or recommendations on how to improve the factory. Since it's my first try implementing a ManagedServiceFactory.

So, enjoy the new service!

Read More

camunda BPM platform OSGi presents: integration with Process Application API

I am happy to announce that there is a new way to configure a ProcessEngine and deploy processes.
You can now use the Process Application API.
Luckily, using this API in your project is quite easy.
There are three things you have to do:

1.  provide a processes.xml file
2.  make a subclass of org.camunda.bpm.extension.osgi.application.OSGiProcessApplication
3. export it as OSGi service

After that the process will be deployed and the engine will be started and exported.
To show you how easy it can be I created an example project.

Please note that the feature is right now only usable when using Blueprint.
Also you'll have to build camunda-bpm-platform and camunda-bpm-platform-osgi yourself. But the next releases should be right around the corner ;-)

Unfortunately, I wasn't able to activate the process application local scan for process definitions (see here). I couldn't figure out a way to find resources inside an embedded jar.
Neil Bartlett mentioned the BundleWiring class. Seems like I have to wait until we upgrade the project to OSGi 4.3.
If anyone knows a way please let me know.

So, enjoy the OSGiProcessApplication and give me some feedback if you want to!

Read More

camunda BPM Platform OSGi 1.0.0 released

Today I am happy to announce the version 1.0.0 release of the camunda BPM Platform OSGi project.
Especially because I am the maintainer of the project ;-)

The project started on 17th of November when we moved the "old" OSGi module out of the core platform and made it a community project.
So let's start with a review.

What did we do?

 

First of all we now have a lot more test coverage. At the beginning there were zero tests and now we should have roughly 80% test coverage across all modules. Next to the tests there was a lot of refactoring to have smaller classes with clear responsibilities (that's what refactoring is all about, right? ;-))

Secondly, we have a Apache Karaf feature.xml and a Blueprint example project.

The first contribution from "outside" was the Apache Karaf commands module, which was developed by Elek from DCP Consulting. Thank you, again!

Finally, there is the new OSGiELResolver, which was included a few weeks ago. The new ELResolver gives us some independence from Blueprint.

As you can see, we did quite a few things, but there are still some tasks left.

What's left to do?

 

The ToDo-list states the following:

1. adapt Process Application API for OSGi
2. camunda webapp WAB (cockpit, tasklist, admin)
3. create example for configuring engine using PAX-CDI

Number one Daniel, Roman and I tried to solve in May. All the results are in the platfrom-api-hack branch. They still need some review.
Number two and three are still open.

That's what's left on the ToDo-list, but what else is there to do? 

The future

Of course it would be great to get some feedback from "real world" users and I hope more people will use the OSGi module in the future.
Then the open ToDos got to get done and I guess we'll find some ideas for the future (maybe Apache ServiceMix with camunda BPM).

After we've taken a look at the past, the present and the future there is only one thing left:

Finally

 

A big "thank you" to Daniel and Roman for guidance, support and having time for a hackathon with me! It is a pleasure working with you!

Read More

Consuming arbitrary remote services with the OSGiELResolver (camunda BPM OSGi)

In my last blog post I promised to give a slightly more advanced example about how to use the new OSGiELResolver. And as I promised, here it is ;-)

Prerequisites

The setup is quite simple. We have three bundles:

1. API
2. Service Provider
3. Service Consumer

You can find all the sources here. (feel free to suggest improvements, possible bugs, etc.)
As runtime I used two Apache Karaf instances on my computer (version 2.3.5; I had some problems with 3.0.1).
For remoting we'll use Apache CXF 1.4 (single bundle release).
And of course we'll need camunda BPM platform OSGi, which you'll have to build yourself.
Before I tell you more about the three bundles I'd like to point the book "Enterprise OSGi in Action" out. Without that great book I couldn't have provided this example. It's definitely worth reading.

So, enough advertisement, let's take a look at the bundles.

API bundle

 

The API bundle is really simple. It only contains one interface with a method. We'll need the bundle in both runtimes.

Provider bundle

 

Now we're getting a little bit more serious. The provider bundle contains the service implementation we want to use.
The context.xml contains the important parts for remoting:
<entry key="service.exported.interfaces" 
 value="de.blogspot.wrongtracks.osgielresolver.api.SomethingService"/><entry key="service.exported.configs"
       value="org.apache.cxf.ws"
<entry key="org.apache.cxf.ws.address"
       value="http://localhost:9001/somethingservice"/>

"service.exported.interfaces" should be obvious.
"service.exported.configs" tells Distributed OSGi to look for implementation specific properties.
Lastly "org.apache.cxf.ws.address" lets us define an alternative address. It is quite helpful if you don't want to type the fully qualified name of the class in your browser or other config files.

Consumer bundle

 

Let's take a look at the consumer. This bundle needs a little bit more information to work properly. To be able to consume remote services we need the OSGI-INF/remote-service/remote-services.xml. It doesn't have to be that name or that directory. You can specify the path inside the bundle with the "Remote-Service" header, which I set in the POM to:
      <Remote-Service>OSGI-INF/remote-service/*.xml</Remote-Service>
I won't walk you through the remote-services.xml. I'm sure you'll find better explanations somewhere else. (e.g. in Enterprise OSGi in Action ;-) )

After we configured this we can use the reference tag in the context.xml to find the service.
To make the service work with the OSGiELResolver we have to add two things. In the remote-services.xml the property "processExpression" has to be set and in the context.xml we have to use a filter.
As you may know the ELResolver uses the filter to search for classes. Searching only worked when both, attribute and filter, were set.

The provider Karaf

 

Like I said, I used Karaf as runtime. The "provider" Karaf needs three bundles:

1. API
2. Provider
3. Apache CXF

Just drop them into the deploy directory. It worked best for me when I started them in the order API, CXF and provider. Then everything should work as expected.

The consumer Karaf

 

The "consumer" Karaf needs a little bit more bundles (and if you run it on the same machine you'll have to change three ports). You have to add:

1. API
2. Consumer
3. Apache CXF
4. camunda BPM platform OSGi and dependencies

Drop API, consumer and CXF jars into deploy (again, starting API, CXF and then consumer works best). Adding camunda BPM platform OSGi isn't very difficult because there is a feature.xml (assumed it is installed in your local Maven repository).
To install it type:
features:addurl mvn:org.camunda.bpm.extension.osgi/camunda-bpm-karaf-feature/1.0.0-SNAPSHOT/xml/features

and then:
features:install camunda-bpm-karaf-feature-minimal

This should resolve all you bundles. Now, If you start the consumer bundle you should see the log saying "Started process". Strangely the logger of the service implementation was quiet. But if you uncomment the exception you can see that the service was called.

So, as you can see, the new OSGiELResolver makes it possible to consume arbitrary remote services, which is quite an improvement. I hope my example is understandable and helps to see the possibilities.

Hint

 

When you encounter this exception:
java.lang.IllegalStateException: Invalid BundleContext
just start the CXF bundle again, then it should work.

Read More

camunda BPM OSGi: the new OSGiELResolver

Introduction

Some of you may know that I am the maintainer of the camunda BPM OSGi project.
Several weeks ago I started to implement a new ELResolver (EL = expression language) and because it's finished now I want to do some shameless self-advertising for my work ;-)

The problem

The "old" ELResolver had some limitations: It could only work with one kind of classes (those who implement the JavaDelegate interface) and you had to register the ELResolver as service listener.
Also, the implementation depends on Blueprint because it used the registered component id to find the classes.

The new OSGiELResolver

The new OSGiELResolver doesn't have those limitations. You can use it theoretically with every class and it doesn't depend on Blueprint. If you want to know more, please have a look at the updated README. I would be happy if you could give me some feedback or ideas for improvement.

So far for now. I'll try to put together a more advanced example, soon.

Please note: this change breaks the API because I moved some classes, so this version would be a new major version number, if it weren't for the snapshot ;-)

Read More

First steps with Apache ACE

Introduction

"Apache ACE is a software distribution framework that allows you to centrally manage and distribute software components, configuration data and other artifacts to target systems." (from https://ace.apache.org/)
Well, that sounds good enough to try it out, at least for me.
I like the idea to centrally configure deployments with different version and have a way to automatically distribute those.

Starting ACE

Setting up Apache ACE was pretty easy. The Getting started guide contains all the necessary steps.
My MacBook was the ACE server and my Raspberry Pi a target.

Using the Web GUI is easy and straightforward (nice one guys ;-) ).
But that's for little children. I wanna find a way to automate everything with scripts.

The Client Shell API

Basically there are two ways to talk to the server remotely. One is the Client Shell API and the other way is via REST API. For now I'll stick with the Shell API

1st step: connecting to the server as shell client

Before we can write a script we have to connect to the server.
With some help from the iQSpot people (see here) I figured it out.
They suggest starting the client like this:

java -Dagent.discovery.serverurls="http://server:port"
     -Dorg.apache.ace.server="server:port"
     -Dorg.apache.ace.obr="server:port"
     -Dorg.osgi.service.http.port=-1
     -jar client.jar

Unfortunately, that didn't work for me (even after adding some missing backslashes).
The default is that you should be in the directory of client.jar. "-jar client.jar" wasn't the problem.
The startup searches for the client/conf directory, so when you see this exception:

java.lang.IllegalArgumentException: Bad arguments; either not an existing directory or an invalid interval.
    at org.apache.ace.configurator.Configurator.<init>(Configurator.java:89)
    at org.apache.ace.configurator.Activator.init(Activator.java:33)
    at org.apache.felix.dm.DependencyActivatorBase.start(DependencyActivatorBase.java:76)
    at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:645)
    at org.apache.felix.framework.Felix.activateBundle(Felix.java:2146)
    at org.apache.felix.framework.Felix.startBundle(Felix.java:2064)
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1291)
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)
    at java.lang.Thread.run(Thread.java:722)

You're probably starting the client from a different directory.
To get rid of that exception we have to set a property:
-Dorg.apache.ace.configurator.CONFIG_DIR=
All in all the command to start the client looks like this:

java -Dagent.discovery.serverurls="http://server:port"\
     -Dorg.apache.ace.server="server:port"\
     -Dorg.apache.ace.obr="server:port"\
     -Dorg.osgi.service.http.port=-1\
     -Dorg.apache.ace.configurator.CONFIG_DIR="apache-ace-2.0.1-bin/client/conf"\
     -jar apache-ace-2.0.1-bin/client/client.jar

Now we can start the client.
But to pass a script to the shell we need two more arguments. Thanks again to the iQSpot people. They already pointed out those arguments:

• -Dgosh.args="–-args"
• -Dace.gogo.script.delay=delay
• -Dace.gogo.script=/path/to/script.gogo

What do those three do?
Everything you'll pass as "gosh.args" will be executed immidiately. If pass "--help" for example and start the client you'll see the help output.
The delay is helpful when you want to give your client some time to synchronize with the server.
"ace.gogo.script" should be obvious ;-)
We end up with the following command:

java -Dagent.discovery.serverurls="http://server:port"\
     -Dorg.apache.ace.server="server:port"\
     -Dorg.apache.ace.obr="server:port"\
     -Dorg.osgi.service.http.port=-1\
     -Dorg.apache.ace.configurator.CONFIG_DIR="apache-ace-2.0.1-bin/client/conf"\
     -Dace.gogo.script.delay="3000"\
     -Dace.gogo.script="script.foo"\
     -jar apache-ace-2.0.1-bin/client/client.jar

Now we have to find out, what we should put into "script.foo".
 

Shell commands

Every (basic) command is described here.
The steps are quite simple: cw, ca, cf, ca2f, cd, cf2d
If you don't like or get the abbreviations (it took me a while) there is also a nice picture in the REST API documentation:

What the picture is missing is cw or "create workspace". When using the Shell API you need a workspace, which you can commit later.

The script

 

So, what should skript.foo do? Let's assume we have to upload some generated artifacts from our CI server 
The steps are

1. create workspace
2. add the new Jars as artifacts from certain directory
3. create a new feature
4. add artifacts to feature
5. create a new distribution
6. add new feature and existing ones to distribution
7. add feature to existing target 

I have to admit that it took me quite a while to figure everything out because I'm not very experienced with Apache Felix GoGo.
Creating the workspace is easy: w = (cw)
Now we can call the workspace with $w. Adding the Jars was more difficult. Let's assume the directory is ./toAdd. Then the command looks like this: 

each ([(ls toAdd)]) {$w ca (($it toURL) toString) false}

You "toAdd" can be changed to any path and you could use some wildcards, like ls toAdd/*.jar
I guess if you're used to GoGo the command won't be a surprise. If you're not used to it, I would like to explain the different parts to you:
each takes a list and a function. ls toAdd returns a File array. That's why we need the brackets. They convert the array into a list. After that comes the function, indicated by the braces.
$w ca is the method to create an arfifact. $it is the iterator over the list that is provided by each.
Then we call the methods toURL and toString because reflection makes it possible ;-)

Third step: add all artifacts to feature

each ($w la "(Bundle-SymbolicName=org.camunda.*)") {symbolicName=($it getAttribute "Bundle-SymbolicName"); $w ca2f "(Bundle-SymbolicName="$symbolicName")" "(name=test-feature)"}

Again, we use a for-each-loop.
$w la lists all the bundles that match the passed pattern. (Here, I want to add all camunda bundles, no advertisement ;-))
Then I save the symbolic name in a variable, so it's easier for me later to reference it.
org.apache.ace.client.repository.RepositoryObject has a getAttribute method, which we use here.
Also, please note the semicolon.
We use the symbolic name as part of the first argument for ca2f (create artifact2feature).
The String contains three parts

• "(Bundle-SymbolicName="
•  $symbolicName
• ")"

I don't know why, but we don't ne a "+" for string concatenation. The second argument is the name of the feature. I just assume it to stay the same: "test-feature"
Creating a distribution and a feature2distribution are nothing special.

All in all we end up with the following:

w = (ace:cw)
$w cf test-feature
$w cd test-distro

each ([(ls toAdd)]) {$w ca (($it toURL) toString) false}

each ($w la "(Bundle-SymbolicName=org.camunda.*)") {symbolicName=($it getAttribute "Bundle-SymbolicName"); $w ca2f "(Bundle-SymbolicName="$symbolicName")" "(name=test-feature)"}

$w cf2d "(name=test-feature)" "(name=test-distro)"

$w commit

That should do the trick so far.
Stay tuned for my next steps with ACE ;-)

Read More