28 December 2010

Java EE 6 Integration Testing Easier Than Ever

With jeeunit 0.6.0, you can write and run JUnit integration tests for Java EE 6 components as easy as this:

@RunWith(JeeunitRunner.class)
class MySessionBeanTest {
    
    @Inject
    private MySessionBean mySessionBean;

    @Test
    public void aSimpleTest() {
        boolean result = mySessionBean.doSomething();
        assertTrue(result);
    }
}

The JeeunitRunner builds a WAR on the fly, launches an embedded container, deploys the WAR and delegates all test methods to a proxy runner within the container.

With this approach, you can run your test class (or even a package with multiple test classes) from Eclipse via the context menu. The container is only launched once for all classes.

jeeunit currently supports Glassfish 3.1-b33 or higher. It uses the new Embedded Glassfish API introduced in b33.

Compared to earlier jeeunit versions, it is no longer required to build a test WAR manually or to provide a test suite for running multiple test classes.

Thanks to the excellent service provided by Sonatype OSS Repository Hosting, jeeunit is now available from Maven Central.

18 December 2010

Glassfish Logging with slf4j (Part 2)

My previous post on the same subject is one of the most popular articles in this blog, and since Glassfish still does not officially support slf4j, this update may be helpful to anyone hitting my blog via Google trying to improve their logging experience with Glassfish.

To redirect (almost) all log messages from Glassfish and your applications to the same log file and/or console, follow this recipe:
  • Copy these JARs to [install-root]/lib/endorsed:
    • jul-to-slf4j.jar
    • slf4j-api.jar
    • logback-classic.jar
    • logback-core.jar
  • Build a JAR containing your logback.xml configuration at root level and put it in the same place. [Update 26 Jul 2011: Actually, this is no longer required: Simply create a logback configuration file somewhere in your local file system and add the system property -Dlogback.configurationFile=file:/path/to/my/logback.xml to your Glassfish configuration.]
  • Define a new system property in the jvm-options section of your domain.xml:
    -Djava.util.logging.config.file=${com.sun.aas.instanceRoot}/config/my_logging.properties
  • Create the corresponding file my_logging.properties with the following contents:
  • handlers = org.slf4j.bridge.SLF4JBridgeHandler
    com.sun.enterprise.server.logging.GFFileHandler.flushFrequency=1
    com.sun.enterprise.server.logging.GFFileHandler.file=${com.sun.aas.instanceRoot}/logs/server.log
    com.sun.enterprise.server.logging.GFFileHandler.rotationTimelimitInMinutes=0
    com.sun.enterprise.server.logging.GFFileHandler.logtoConsole=false
    com.sun.enterprise.server.logging.GFFileHandler.rotationLimitInBytes=2000000
    com.sun.enterprise.server.logging.GFFileHandler.alarms=false
    com.sun.enterprise.server.logging.GFFileHandler.formatter=com.sun.enterprise.server.logging.UniformLogFormatter
    com.sun.enterprise.server.logging.GFFileHandler.retainErrorsStasticsForHours=0
    
When you restart Glassfish, you should only see two log messages in the old java.util.logging format, all the rest goes via slf4j and logback to the appenders configured in your logback.xml. The two remaining log messages are issued before Glassfish reads the domain.xml configuration with the adapted configuration, so there is no easy way of avoiding them.

I have been using this approach on various Glassfish versions from 3.0.1 to 3.1-b33. It is rather ugly, but nonetheless effective.

All this may sound rather mysterious, so I'd better add some explanations:

jul-to-slf4j.jar is a bridge for redirecting java.util.logging via the slf4j API to any logging backend of your choice (logback in this case). To make the redirection work, the bridge has to be visible to the classloader evaluating the logging configuration. Since Glassfish starts logging very early in its bootstrapping phase, the lib/endorsed folder is the only place where the bridge has the desired effect.

Unfortunately, class file folders seem to be unsupported for endorsed libraries. Of course, it would be easier to simply put the logback.xml file in lib/endorsed/classes where you can directly edit it. But for a quick change to your logging configuration, you can always use an editor which supports editing ZIP/JAR file contents in place.

I created the contents of my_logging.properties by trial and error, starting with a one-liner containing only the handler. This caused a couple of exceptions as Glassfish seemed to be missing the GFFileHandler settings, so I copied them over from the original logging.properties.

The same procedure should also work for other slf4j backends, so you could replace the logback JARs by a log4j.jar.

An Update on Memory Issues in Glassfish and Weld

Using the most recent Glassfish milestone build 3.1-b33, I took another look at the memory issues that had plagued my project on Glassfish 3.0.1. The last build I tried was b30, which did not even let me deploy my application.

Just in time for Christmas, there are Good Tidings - Weld has improved a lot, and I can no longer see any suspicious memory usage related to Weld. I took my Wicket+EJB+OpenJPA+PostgreSQL application packaged in a single WAR and deployed it to several different Glassfish versions from the command line. Then I used the Eclipse Memory Analyzer to a create a heap dump of the running Glassfish instance.

The Total Heap column in the following table is the total size of the heap as reported by the Memory Analyzer. The Shallow, Retained, and Percentage Columns refer to the amount of heap memory consumed by instances of org.jboss.weld.introspector.weld.jlr.WeldClassImpl

Glassfish Version Total Heap Shallow Retained Percentage
3.0.1 412.1 MB 84.680 378.215.056 87.52 %
3.1-b26 335.3 MB 85.144 205.527.048 58.38 %
3.1-b33 149.6 MB 84.216 7.041.704 4.49 %

The retained heap of an object is the set of objects that will be garbage collected when the given parent object is garbage collected.

So in b33, Weld still holds 5 % of the total heap - whether or not this is still too much or just reasonable is surely debatable. But I'm really happy to see that Weld has improved a lot, so I would no longer consider it as a no-go.

The version of my application I used for these measurements is the last one that actually used Weld, before I decided in August to stop using CDI to get rid of most memory issues.

I took some more measurements with the successor version in which CDI was eliminated essentially just by replacing @Inject by @EJB throughout the sources. This is really the only difference between the two versions, and here are the total heap sizes:

Glassfish Version with @Inject with @EJB
3.0.1 412.1 MB 44.4 MB
3.1-b33 149.6 MB 138.3 MB

The illustrates, as claimed in my previous post, that I was able to save 90 % of heap memory by not using CDI on Glassfish 3.0.1. Doing the same on Glassfish 3.1-b33, I can still save about 9 %.

But I was really surprised to see the heap usage of my application increase by a factor of 3 between Glassfish 3.0.1 and 3.1-b33.

The Memory Analyzer reveals the culprit to be org.glassfish.hk2.classmodel.reflect.impl.TypesCtr with a retained heap of about 80 MB.

I have no clue what this class is doing, but it smells like another case of an oversized reflective model of the application, which was just the problem with earlier Weld versions.

GLASSFISH-15266 is the related issue in Glassfish JIRA.

10 December 2010

The Interface Antipattern

Keeping interfaces and implementations separate is a useful design pattern for building modular systems. If your clients only depend on a service interface, you can switch service implementations without your clients even noticing.

Now there can be too much of a good thing, and the quality of a software system is certainly not measured by the ratio of interfaces to classes.

When you are planning to implement a FrobnicatorService, think twice before creating an IFrobnicatorService interface and a FrobnicatorServiceImpl. How many different Frobnicator implementations are there going to be? If you do need at least two implementations with distinct behaviour, then go ahead and create the interface. If there's only one implementation, then don't bother with the interface.

Resist the temptation of speculative generalization: "Oh, there might be a performance bottleneck, and in that case an alternative CacheingFrobnicatorServiceImpl might help, so I really should start with a service interface now." Think twice: you ain't gonna need it.

And if you do need the interface, you can still pull it out when you need it. Most IDEs have an automatic refactoring Extract Interface. (Unfortunately, none of the major Java IDEs seems to have the opposite refactoring Merge Interface and Implementation.)

Even in Java EE, the times of interface inflation are gone with EJB 3.1, thanks to the no-interface local view. For a stateless session bean, it is enough to implement

@Stateless
public class FrobnicatorService {
    
    public void frobnicate() {
        ...
    }
}

Every public method of this class will be part of the implicit local business interface.

In OSGi, a service does not need to have a separate interface, you can register any class as a service. Even when using Declarative Services, don't get fooled by the XML syntax of the Service Component Descriptor:

<service>
  <provide interface="com.example.FrobnicatorService"/>
</service>


The provide element has an interface attribute, but the attribute value can be any old class.

The same is true in Spring: There is no rule forcing you to inject Spring beans only via their interface. Any old class will do, even when working with automatic transaction proxies (just make sure that CGLIB is on your classpath).

So remember the KISS principle and kill some of the interfaces you don't really need!

01 December 2010

Glassfish and Weld: New builds, new bugs

My article about memory issues with CDI/Weld seems to have caused rather a wave. Someone (not me) posted a link on DZone, which triggered an unprecedented amount of traffic on this blog, there were discussions both in the Glassfish and Weld communities, and I was very happy to see that Weld appears to have made significant progress in terms of resource usage.

So the good news is: First, things are moving in the right direction with Weld, and second, blogging helps - at least in this case, it seems to have been a lot more effective than my previous bug reports and forum messages.

Unfortunately, there is also some bad news. Several users have already reported regressions with Glassfish 3.1-b29 and b30 related to Weld. My application fails to deploy on b30 with a stack trace which looks like a case of GLASSFISH-13131. So I'd better wait and try again with Glassfish 3.1 milestone 8.

29 November 2010

Java EE and Spring: Why I couldn't care less

 Java EE or Spring - for an independent software architect, the question is just as relevant as Catholic or Protestant to an atheist. The question has recently been discussed in a number of articles mostly taking one side or the other:

In this post, I'll try to point out why I'm not fully happy with either of the two.

If there's one thing that matters to me, it's modularity. To put my cards on the table, I'm a strong OSGi supporter, but I'm not even religious about that - in fact, Living without OSGi (which is what I've been doing for nine months now) would be a nice headline for one of my next posts.

Seen from outside, Spring and Java EE share a number of shortcomings:

Preaching to the Converted

When did you last visit the official homepages of Java EE or Spring? Take a moment to click on the links and imagine you've never heard about either. After visiting the homepages, you are none the wiser, all you get is the usual marketing blurb and a couple of newsbites completely meaningless to the uninitiated.

Lack of focus

Java EE and Spring both are animals of the rare species Sus ovipara lactifera velluta unknown in English speaking countries which we call eierlegende Wollmilchsau in German. In plain English, they both try to be everything to everyone and end up doing nothing in particular.

In contrast, OSGi has a clear mission statement The Dynamic Module System for Java and provides a link to its key benefits right on its homepage.

Bloat

Ever heard about lean development? Let's have a look at the official Javadocs:
  • Java EE 6: 1597 classes
  • Spring 3.0.5: 2312 classes

Legacy

Backward compatibility is a good thing for veteran users of a framework. Users don't want to change all of their application code just for upgrading to a new framework version.

On the other hand, backward compatibility can be extremely confusing to new users: There's two or three solutions for the same kind of problem, the legacy one is not marked as legacy or deprecated clearly enough, tutorials and example code still use the old style, and you can only resort to your favourite search engine or to trial and error to make things work consistently.

Love Thy User

Don't just list what you can do for your users, ask them what they want you to do for them. Looking at the Java EE and Spring reference documentation, I can see that both frameworks solve lots of problems I've never had in my life.

On the other hand, looking for a solution for some specific standard kind of problem, it is often surprisingly hard to find a comprehensive code example.

User documentation should be structured by user stories, not by technology.

Enough Rope

Both Spring and Java EE give you enough rope to hang yourself. It's easy to write messy and ugly code on top of both frameworks, and I've seen enough of that.

You can write loosely coupled and elegant applications on top of either, but it takes a good architect to set up guidelines and restrict the number of choices.

Development Process Integration

As an application developer, you don't just write code to run on a given framework. You want to run integration tests on your applications, you want to deploy and debug them directly in your favourite IDE, and you want your batch build and continuous integration to be able to deploy your applications.

When choosing a framework, always consider the entire process chain - some lack of tooling for your favourite framework may be reason enough to pick the second best instead.

Modularity

My ideal framework/container/app server - pick any name you like - would allow me to write truly modular applications, enforcing modularity at design time, at compile time, and at run time. The framework itself would be modular, allowing me to pick out those and only those modules actually required by my application.

It would actively support OSGi as the most mature module system for Java, and maybe a few others on top.

I expect some further degree of convergence between Java EE, Spring and OSGi within the next two years or so. There has been a lot slideware and a number of proof-of-concept or early adopter projects, but my feeling is that none of this has reached production quality yet.

To sum up, frameworks in general are alive and kicking, but the age of monolithic all-in-one frameworks is definitely over.

28 November 2010

Java EE 6 Integration Testing with Spring

Half a year ago, I created and published the jeeunit project to fill in some gaps for running integration tests on Java EE 6 components in an embedded EJB container, mainly targeted at Embedded Glassfish 3.x. One of the alternatives I considered and dismissed before writing my own solution was the Spring Test Context.

At the time, I was left with the impression you had to pollute your Java EE components with Spring annotations like @Transactional to make declarative transactions work in the Spring container, and I did not (and still do not) like the idea of writing XML just to create a couple of Java beans and wire them up.

After taking a closer look at Spring 3.0.x these days, I realized a number of facts which are not quite obvious from the Spring reference manual:
  • You can do most of your Spring configuration (but sadly not all of it!) in Java instead of XML, which makes your application context type-safe and easier to read, and gives you refactoring support from your IDE.
  • Spring supports some (but not all) of the Java EE 6 annotations as alternatives to its own flavours.
  • You can run Stateless Session Beans and JPA entities in a Spring container with declarative transactions without introducing dependencies on Spring APIs or annotations.
jeeunit now includes an example project jeeunit-example-test-spring demonstrating this approach. The details are documented in the jeeunit Wiki page Testing on Spring.

This solution is not intended to replace the original in-container-testing approach of jeeunit, because transactional Spring beans and EJBs have different run-time semantics. However, for many integration tests, these differences are irrelevant.

Using the Spring Test Context instead of Embedded Glassfish and jeeunit has two main advantages:
  • Spring starts noticeably faster than Embedded Glassfish.
  • Launching a single test from your suite is much easier than with jeeunit. E.g. in Eclipse, you can simply use Run as JUnit Test just as with plain old unit tests.

04 November 2010

CDI - A Major Risk Factor in Java EE 6

Update 18 Dec 2010: This post mainly relates to Glassfish 3.0.1. As of 3.1-b33, the situation has improved a lot.

Evaluating platforms and architectures for a new enterprise web application earlier this year, around March/April, scheduled to be released in 2011, I was bold enough to settle on Java EE 6.

In retrospective, I do not regret this decision, but I had to spend a significant amount of time analyzing and debugging platform problems. Based on this experience with Java EE 6, which may not be representative of course, the only advice I can give is: When Java EE 7 is released, wait a year or two before using it for production, unless you have the time and resources to act as beta tester.

The Java EE 6 specifications were released in December 2009, together with Glassfish v3, the first "production-quality release" of a Java EE 6 server. "Marketing release" would have been more appropriate, since going by the number of bugs in Glassfish 3.0, the main driving factor for the release date was quite obviously not a quality benchmark, but simply the need for Sun as the Java EE champion to release a working server simultaneously with the specifications.

The Problem


Java EE 6 was advertised as the lightest Java Enterprise Edition ever, with Contexts and Dependency Injection (CDI, JSR-299) as one of the most prominent innovations. However, for my project, CDI in the guise of Weld, its reference implementation, turned out to be the largest single source of problems. In addition to a number of functional bugs where injection would not work in certain scenarios, there are at least two severe memory issues:
  1. Weld causes memory leaks on application redeployment.
  2. Even in the first deployment of an application, Weld uses enormous amounts of memory.
The first issue had plagued us for months, and my team had grown accustomed to kill all Glassfish and Eclipse processes after three or four redeployments and to re-initialize their entire workspace, which was rather a pain.
Investigating the memory leaks using the Eclipse Memory Analyzer, I realized that Weld claimed almost 90 % of the heap memory of my application. Most of this was due to instances of org.jboss.weld.introspector.weld.jlr.WeldClassImpl, which appears to be a reflective model of a class inspected by the CDI BeanManager. Some instances of this class were as large as 1 MB.

I should add that our application uses Wicket for the web front end, injecting EJBs into custom Wicket components. Custom components inherit from Wicket base classes with a fairly deep inheritance hierarchy and a large number of methods. This accounts for the size of the reflective models.

Our usage of CDI was fairly basic, so it was just an hour's work to replace all @Inject annotations by Java EE 5 style @EJB annotations. After that, the heap usage of our application was reduced by 90 %, we no longer had any memory leaks, and we finally had a hassle-free edit-save-debug cycle in our development environment.

The figure of 90 % heap usage by Weld was observed on Glassfish 3.0.1, running on a 32-bit JVM on Windows XP.

This week, I retested the same application with Glassfish 3.1-b26 (using a newer version of Weld), running on a 64-bit JVM on Ubuntu 10.04. The result was not as drastic as before, but still about 50 % of my heap memory was consumed by Weld.

Alternatives


This article is mainly about problems with the CDI reference implementation Weld, not about CDI as such. But these implementation details have a ripple effect:
  • Glassfish 3.x still is the only certified Open Source Java EE 6 server. It uses a newer version of Weld in the current 3.1 promoted builds, but I can see no indication of the Weld issues being solved in time for the 3.1 release.
  • JBoss AS 6 is likely to be the next in line for full Java EE 6 compliance. However, JBoss also includes Weld, so you're bound to run into the same kind of problems on JBoss.
  • There are alternative CDI implementations, CanDI from Caucho and OpenWebBeans from Apache. Unfortunately, unlike JPA, there is no pluggable CDI provider interface that would allow you to simply replace Weld by some other CDI implementation on Glassfish or any other server.
  • I tried running my application on Resin 4.0.8 to see if CanDI was any better than Weld. However, Resin had bugs in the JPA and EJB areas and I was unable to deploy my application, let alone run and monitor it. (Resin only aims at supporting the Java EE 6 Web Profile, but that would have been sufficient for my case.)

Conclusion


All in all, if you want to work with Java EE 6, there is currently no way around Weld. And until Weld reaches production quality, the only way to build production quality applications on Java EE 6 is to avoid using CDI.

Free and Open Source Software is all about choice - so I would really love to see a pluggable CDI provider interface that would allow users to work with Glassfish and some CDI implementation other than Weld.

The second best solution might be to join forces from different CDI project to make Weld stable enough for production usage.

The third option would be another fully Java EE 6 compliant Open Source server (maybe Geronimo?) not including Weld.

No idea if any of these options is realistic, but this is on my wishlist for 2011.

19 September 2010

OpenJPA and OSGi

In my previous review of OSGi support in JPA 2.0 persistence providers, I promised to explain my setup for using OpenJPA in an OSGi environment.

The solution I'm going to outline is by no means final, and in the meantime, my first shot has been greatly simplified using some valuable input from the Apache Aries User mailing list.

In the past, brute force solutions like DynamicImport-Package were more or less the only way to let a JPA provider bundle access all the required classes for a persistence unit from application bundles. Since Release 4.2, the OSGi Enterprise Specification standardizes a whole lot of OSGi service wrappers for well-known Java EE APIs, including JPA, JDBC, JTA and JNDI, so this should be the preferred way of integrating  JPA providers into an OSGi environment.

The Apache Aries project provides (partial) implementations for some of the OSGi enterprise services and a whole lot of other things beyond the scope of this article. Fortunately, all their service implementations are nicely modular, so it is quite easy to use just the JPA and JNDI Services from Aries to make OpenJPA work on OSGi, following the OSGi Enterprise spec rather closely (though not fully).

Here is an overview of the required steps:
  • Get OSGi bundles for OpenJPA and all its dependencies, plus the required Aries bundles.
  • Publish the datasource to be used by your persistence unit as an OSGi service.
  • Make sure that your persistence.xml satisfies a number of restrictions.
  • Run the OpenJPA enhancer and package all classes and metadata of your persistence unit into a bundle with a special manifest header.
  • Start your OSGi framework. The JPA Service will publish an EntityManagerFactory service for each persistence unit.
  • In your client bundles, import your persistence bundle, OpenJPA and javax.persistence, get the appropriate EntityManagerFactory from the service registry and use it just like in an unmanaged Java SE environment.
Each step is explained in more detail in the following sections. I'm using Java 1.6.0, Maven 2.2.1 for collecting the dependencies, Eclipse 3.5.2 for doing most of the work, and the corresponding Equinox version as OSGi runtime.

Get the dependencies


The main JAR in the OpenJPA binary distribution is an OSGi bundle, but most of the included dependencies are plain old JARs, so you need to go shopping for osgified versions of all required libraries. All the required bundles for this example can be found in the SpringSource Enterprise Bundle Repository.

The following is the best part of a Maven POM for copying all the dependencies to a local directory target/platform/plugins.

You need to add the OSGi framework, javax.persistence (I copied both of them from Eclipse 3.5.2) and a JDBC driver bundle for your database. Then copy the whole lot to a new directory to be used as target platform for your Eclipse workspace.


<project>
  <properties>
    <aries.version>0.2-incubating</aries.version>
    <output.dir>${project.build.directory}/platform/plugins</output.dir>
  </properties>
  <dependencies>
    <dependency>  
      <groupId>javax.transaction</groupId>  
      <artifactId>com.springsource.javax.transaction</artifactId>  
      <version>1.1.0</version> 
    </dependency>    
    <dependency>
      <groupId>net.sourceforge.serp</groupId>
      <artifactId>com.springsource.serp</artifactId>
      <version>1.13.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.aries</groupId>
      <artifactId>org.apache.aries.util</artifactId>
      <version>${aries.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.aries.jndi</groupId>
      <artifactId>org.apache.aries.jndi.api</artifactId>
      <version>${aries.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.aries.jndi</groupId>
      <artifactId>org.apache.aries.jndi.core</artifactId>
      <version>${aries.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.aries.jndi</groupId>
      <artifactId>org.apache.aries.jndi.url</artifactId>
      <version>${aries.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.aries.jpa</groupId>
      <artifactId>org.apache.aries.jpa.api</artifactId>
      <version>${aries.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.aries.jpa</groupId>
      <artifactId>org.apache.aries.jpa.container</artifactId>
      <version>${aries.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>com.springsource.org.apache.commons.collections</artifactId>
      <version>3.2.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>com.springsource.org.apache.commons.lang</artifactId>
      <version>2.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>com.springsource.org.apache.commons.pool</artifactId>
      <version>1.5.3</version>
    </dependency>
   <dependency>
      <groupId>org.apache.openjpa</groupId>
      <artifactId>openjpa-all</artifactId>
      <version>2.0.1</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>com.springsource.slf4j.api</artifactId>
      <version>1.5.0</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>com.springsource.slf4j.log4j</artifactId>
      <version>1.5.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.log4j</groupId>
      <artifactId>com.springsource.org.apache.log4j</artifactId>
      <version>1.2.15</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-dependency-plugin</artifactId>
        <executions>
          <execution>
            <id>copy-bundles</id>
            <phase>package</phase>
            <goals>
              <goal>copy-dependencies</goal>
            </goals>
            <configuration>
              <outputDirectory>${output.dir}</outputDirectory>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  <repositories>
    <repository>
      <id>com.springsource.repository.bundles.external</id>
      <name>SpringSource Enterprise Bundle Repository - External Bundle Releases</name>
      <url>http://repository.springsource.com/maven/bundles/external</url>
    </repository>
  </repositories>

</project>



Register a data source service


A persistence unit requires a data source, either from a javax.sql.DataSource or from a JDBC driver URL. This may be a matter of taste - in this article, I'm focusing on the DataSource.

Create a DataSource for your database using any suitable driver and register it as an OSGi service with the javax.sql.DataSource interface and optionally some additional identifying properties of your choice, e.g. name=myds, in case you have more than one DataSource in your system.

It is useful to do this from a separate bundle, so that this bundle will be the only one in your system with a direct dependency on the JDBC driver.


Adapt your persistence.xml


If you have a persistence.xml which works on Java SE oder Java EE, you will probably have to modify it  to make it work with OSGi and Aries. Here is an example:

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
  <persistence-unit name="myapp" transaction-type="RESOURCE_LOCAL">
  <non-jta-data-source>osgi:service/javax.sql.DataSource</non-jta-data-source>   
    <mapping-file>META-INF/orm.xml</mapping-file>
    <exclude-unlisted-classes>true</exclude-unlisted-classes>
  </persistence-unit>
</persistence>


First, you need to specify the RESOURCE_LOCAL transaction type explicitly, or else OpenJPA will complain about a missing transaction manager. This is because Aries creates a container EntityManagerFactory and not a stand-alone one.

Second, your data source has to be a non-jta-data-source. Note the special syntax for accessing your data source as an OSGi service via JNDI. To make this work, you need to include the Aries JNDI Service bundles. To select a specific data source by properties, you can append an LDAP filter to the JNDI URL, e.g. osgi:service/javax.sql.DataSource/(name=myds).

Third, you need to prevent the persistence provider from scanning your persistence unit for entity classes, since this may lead to classloader problems. To do so, either list all your classes explicitly if you use JPA annotations, or include all mapping files if you use XML metadata and set exclude-unlisted-classes to true to prevent the provider from scanning your bundle.

If you get funny exceptions like

org.apache.openjpa.persistence.PersistenceException: Error extracting class information from "bundleresource://23.fwk1421571929/".
Caused by: java.io.FileNotFoundException: /myapp/plugins/test.openjpa/bin (Is a directory)

then check if you have set this option. (This problem may be specific to Eclipse, but I'm not sure about that.)

Package your persistence unit bundle


To turn your persistence unit into a persistence bundle, make sure you have all mapping data resources and all entity classes included. In addition to the usual OSGi headers, insert the following one into your manifest:

Meta-Persistence: META-INF/persistence.xml

This is the hook for Aries to discover your persistence bundle using the extender pattern. If all goes well, an EntityManagerFactory will be registered as an OSGi service.

In theory, your persistence bundle should only depend on javax.persistence and not on org.apache.openjpa. In practice, you currently need to include this unwanted import, or else you will get exceptions of the following kind:

java.lang.NoClassDefFoundError: org/apache/openjpa/enhance/PersistenceCapable

Apparently, OpenJPA tries to load one of its own classes via your persistence bundle class loader, which is incorrect.

Use the EntityManagerFactory service in your clients


Finally, to work with your persistence unit from client bundles, just get hold of the appropriate EntityManagerFactory service any way you like, using the filter (osgi.unit.name=myapp).


Update 26 Sep 2010: I have submitted some sample code for inclusion into the OpenJPA test suite. The sample is attached to OPENJPA-1815.

15 September 2010

OSGi Support in JPA 2.0 Persistence Providers

About three years ago, I started working with Hibernate in an OSGi environment, and at that time, I had to roll my own solution for resolving all sorts of classloader issues.

Since then, with OSGi 4.2 and JPA 2.0, there have been major specification releases, followed by implementations; OSGi is growing ever more popular, and the current release of the JPA standard has integrated a lot of functionality which used to be available in proprietary API extensions only.

The OSGi 4.2 Enterprise Specification includes JPA, JTA and JDBC services, so in theory, using JPA in an OSGi environment should stop being an issue, as soon as this specification is supported by all OSGi and JPA implementations.

In practice, the specifications are not yet fully covered by the available implementations, and the degree of OSGi support varies a lot.

In the past few months, I have worked with the current versions of the three certified JPA 2.0 persistence providers, and in my opinion, the ranking for OSGi support is quite clear:
  1. Eclipselink
  2. OpenJPA
  3. Hibernate
Some more details on each provider:

Eclipselink


Eclipselink has a fully osgified distribution containing OSGi bundles of Eclipselink itself and all dependencies. To make your entity classes visible to Eclipselink, all you need to do is add a JPA-PersistenceUnits manifest header to each bundle containing one or more persistence units. This is enough for Eclipselink to load all your XML metadata and entity classes.

No ugly hacks, no buddy policies or fragments, it just works out of the box.

So for a quick start with JPA under OSGi, Eclipslink is definitely the best choice, even though the OSGi 4.2 JPA Service Specification is not yet fully supported (for instance, the Meta-Persistence header does not work), but this will be of little or no concern to most users.

Leaving OSGi aside, I have been bitten by a number of bugs in Eclipselink, discussed in earlier articles. If you do not use any of these features, you may be very happy with Eclipselink. For me, this set of bugs is currently a no-go, so I have migrated my OSGi applications from Eclipselink to OpenJPA.

OpenJPA


The silver medal goes to OpenJPA: OSGi does not appear to be on the top of the agenda of the OpenJPA core developers, but since OpenJPA is used under OSGi by Aries, another Apache project, there is at least a certain level of OSGi support.

In the binary distribution, the OpenJPA aggregate JAR is an OSGi bundle, but most of the dependencies are plain old JARs, so you need to find osgified versions of the dependencies on your own. The same goes for the Maven artifacts.

Once you have downloaded the appropriate OSGi bundles, you still have to do some extra work to ensure that OpenJPA finds your persistence units and your entity classes, and there are a couple of minor issues with the enhancer and with user-defined value handlers.

I will explain the details of my setup and some of the problems that should be addressed in OpenJPA in my next post.

Hibernate


Sadly, regarding OSGi support, Hibernate 3.5.x is not much different from the 3.2.x release I discussed in my earlier articles. The distribution contains plain old JARs, no OSGi bundles. The latest osgified version available from the SpringSource Enterprise Bundle Repository is 3.4.0, and the SpringSource versions never used to work for me, so I expect that the steps described in my 2008 article are still required to make Hibernate work on OSGi.

Disclaimer: I did not try to make Hibernate 3.5 work on OSGi, due to its lack of support of a number of JPA 2.0 features used heavily by my applications. For this reason, I have stopped using Hibernate, both on OSGi and on Java EE.

12 September 2010

JPA 2.0: Ordered Collections

In earlier posts, I wrote about problems with persistent maps, a new feature introduced in JPA 2.0. Ordered collections with explicit order columns are another JPA 2.0 feature which, again, is not yet fully robust in all JPA 2.0 compliant and TCK tested implementations.

I'm currently working with a simple JPA model for OpenStreetMap (OSM). There are ways and nodes, and each way has a sequence of nodes. Of course the order of the nodes is important, and a node may be used more than once for a given way.

Thus, the nodes collection has to be a List, not a Set, and we need an explicit order column specifying the sequence number of the nodes along the way. Sorting the nodes by ID would not make any sense, obviously.

Here is a snippet from the model:

@Entity
@Table(name = "ways")
public class OsmWay
{
    @Id
    private long id;
    
    @ManyToMany(cascade = CascadeType.ALL)
    @OrderColumn(name = "sequence_id")
    @JoinTable(name = "way_nodes", 
               joinColumns = @JoinColumn(name = "id"), 
               inverseJoinColumns = @JoinColumn(name = "node_id"))
    private List<OsmNode> nodes = new ArrayList<OsmNode>();
}    

This entity is represented by two tables:

CREATE TABLE ways
(
  id bigint NOT NULL
);

CREATE TABLE way_nodes
(
  id bigint NOT NULL,
  node_id bigint NOT NULL,
  sequence_id integer,
);

I've tested this scenario on Hibernate 3.5.3, Eclipselink 2.1.1 and OpenJPA 2.0.1: Hibernate and OpenJPA pass, Eclipselink fails with 2 issues.

First, the generated DDL is incorrect: There is a PRIMARY KEY (id, node_id) for way_nodes. This should be (id, sequence_id).

Second, the order of the list items is not maintained in all contexts. It is ok when the collection is lazily loaded, e.g.

OsmWay way = em.find(OsmWay.class, wayId);
List<OsmNode> nodes = way.getNodes();

but it is broken when using a fetch join:

String jpql = "select distinct w from OsmWay w join fetch w.nodes";
TypedQuery<OsmWay> query = em.createQuery(jpql, OsmWay.class);
List<OsmWay> ways = query.getResultList();
OsmWay way = ways.get(0);
List<OsmNode> nodes = way.getNodes();

The funny thing is, the nodes are neither ordered by sequence_id nor by node_id. In my test case, the way has 5 nodes, one of which appears twice in different positions.

So here is some more evidence for my claim that the JPA 2.0 TCK is insufficient.

08 August 2010

Ubuntu VPN Client Setup

Not a Java topic this time: Setting up an IPSec VPN client on my Linux box was a complete nightmare. The final solution is rather easy, but I wasted hours and hours trying different clients with different configurations which simply did not work...


This is the environment:
  • Linux client machine running Kubuntu 10.04.
  • Network interface eth0 on client configured via DHCP from a Netgear Router.
  • The router acts as DNS server.
  • The home network is 192.168.0.0/24.
  • The company network has a FortiGate firewall with an IPSec VPN.
  • The IPSec tunnel uses a pre-shared key and XAUTH username/password authentication.
  • The company network is 172.16.0.0/16, VPN clients get a virtual IP adress from 172.19.0.0/24 via DHCP-over-IPSec.
The problem:
  • There is no official Linux VPN client for FortiGate. (This is rather ironic, given that Fortinet products include GPLed Linux code, which became public after a GPL violation injunction in 2005.)
  • Ubuntu has just too many too buggy VPN clients and too little readable documentation on VPN setup. (I tried KNetworkManager, kvpnc, ike and openswan, mostly getting only error messages I could not make sense of. ike did work ocasionally, but most of the time, the incoming packets from the tunnel would not reach my client port.)
My solution:
  • strongswan for the IPSec tunnel
  • dnsmasq for merging the public and private DNS zones
  • eth0 now uses manual configuration instead of DHCP
Now for the details: The XAUTH credentials and the pre-shared key are stored in /etc/ipsec.secrets:

: XAUTH myusername "MyPassword"
: PSK "MyPreSharedKey"

The general and connection specific settings for IPSec are in /etc/ipsec.conf:

config setup
     nat_traversal=yes
                                                             
conn office
     left=%defaultroute
     leftsourceip=172.19.0.99
     right=firewall.example.com
     rightsubnet=172.16.0.0/16
     auto=add
     authby=xauthpsk


The only flaw is the explicit virtual IP address: I tried using leftsourceip=%config to let the client use an address assigned by the server, but this did not work as FortiGate does not support the ModeCfg protocol (at least not in the firmware version we are currently using), and I could not find out how to set up DHCP-over-IPSec with strongswan, so I ended up using an explicit address not part of the DHCP pool.

After editing these files and starting the tunnel via

ipsec restart
ipsec up office

I could successfully ping addresses from the office network.

In contrast both to the Windows FortiClient and the Shrew VPN Client (ikea), strongswan does not require a virtual network device and does not change the routing table. You can see the IPSec tunnel and the corresponding policies using the commands

ip xfrm state
ip xfrm policy

The log output from the strongswan IKEv1 daemon named pluto can be found in /etc/auth.log.

Now for DNS: Of course I don't want all my DNS requests for things like schweinebildchen.com to hit the office DNS server while the tunnel is up, so I'm using dnsmasq as a simple DNS proxy which redirects all DNS requests from my local machine either to my home router or to the office DNS server via the tunnel, depending on the domain.

The dnsmasq configuration in /etc/dnsmasq.conf is as simple as this:

server=/example.com/172.16.0.2
server=192.168.0.1


Finally, I changed the configuration of my network interface eth0 from auto to manual, using the IP address 192.168.0.5 and set this as nameserver in /etc/resolv.conf.

This means that all DNS requests are first handled by dnsmasq on my local machine, requests for example.com get forwarded to the office DNS server (via the IPSec tunnel), and everything else gets routed to my Netgear router (and from there to my ISP).

27 July 2010

A Fish in the Clouds

How long does it take to find a hosting company and get an enterprise web application up and running from scratch? Could be weeks if there are enough lawyers and pointy-haired bosses involved. And even if you can directly talk to the providers, it usually takes a couple of days to request quotes and compare them.

That's the way we started for our current project, but now we've decided to have a go at cloud computing. (So at last this adds another buzzword to my CV...)

I vaguely remembered Arun Gupta's blog about Glassfish on Amazon EC2, and using that together with the Ubuntu EC2 Starter's Guide, it just took me about half an hour to set up a virtual machine in the Amazon Cloud with Ubuntu Server 10.04, JDK 1.6.0_20 and Glassfish 3.0.1, start the Glassfish domain and access the admin console from my local web browser.

And downloading the 78 M Glassfish zip file on the EC2 instance from Oracle's server took less than 2 seconds...

Not bad for a start. I would have expected to spend about a day to get as far as that.

I was also using Ubuntu 10.04 on my local machine, working with the command line interface from the
ec2-api-tools Ubuntu package most of the time.

Of course it will take some more time to set up the database server, deploy our web app and to secure the system. And it's too early to tell if we really require all the elasticity of EC2, which is likely to be more expensive in the long run than a conventional dedicated server cluster.

At any rate, installing a site in virtually no time and running it for less than 10 cents per hour is rather impressive. Thumbs up for Amazon Web Services!

26 July 2010

Building a Java EE 6 Web Application with Eclipse Helios, Maven and Glassfish

This is a short tutorial showing how to use Eclipse 3.6 (Helios) with Glassfish and Maven for building and running a Java EE 6 web application from a multi-module source tree.

We are going to import and run the Wicket gmap2-examples, demonstrating the use of Google Maps via Apache Wicket, but this is purely incidental - even if you prefer mainstream JSF or another web framework to Wicket, you may find this tutorial useful. There are absolutely no Wicket specifics involved; gmap2 was just a handy small but non-trivial example.

If you haven't worked with Maven before, you will get an idea how Maven can save you a lot of work managing the third-party dependencies of your project automatically.

This is what you'll see in the end:


There's quite a few things on your shopping list before you can start. If you are reading this, you probably already have the following on your disk:
You'll need to install these features into Eclipse:
  • Maven Integration for Eclipse (also known as m2eclipse)
  • Maven Integration for Eclipse (Extras)
  • Glassfish Java EE Application Server Plugin for Eclipse
Eclipse 3.6 has a new way of installing plugins via Help | Eclipse Marketplace. This is an integrated web client which lets you search for plugins by free text, so you no longer have to copy and paste update site URLs into the Update Manager - but you can still do so if you prefer.

The m2eclipse book has step-by-step instructions and screenshots explaining the installation process: start here with the m2eclipse installation. When you get to the Extras, the only ones required for the rest of this tutorial are Maven Integration for WTP and Maven SCM Integration. The WTP integration will let Eclipse recognize your Maven projects with war packaging as Dynamic Web Projects.

The Maven SCM Integration lets you fetch Maven projects directly from source code repositories like Subversion, Mercurial or others. Note that this integration simply invokes the corresponding command line clients like /usr/bin/svn.

Having installed m2clipse via the Eclipse Marketplace, use the same procedure to install the Glassfish Plugin.

After restarting Eclipse, set your Glassfish preferences via Window | Preferences | Glassfish Preferences. For this tutorial, you should check Start the Glassfish Enterprise Server in verbose mode and uncheck the other items.

Next, define a server runtime environment via Window | Preferences | Server | Runtime Environment | Add... Select the server type Glassfish | Glassfish Server Open Source Edition 3 (Java EE 6), check Create a new local server and click Next. Select your Glassfish installation directory - this is required to be the parent of the modules and domains directories, e.g. /home/hwellmann/glassfish-3.0.1/glassfishv3/glassfish. Click Next and fill in the domain name and the administrator credentials. If you did not change the Glassfish defaults, you can probably just click on Finish.

The Servers view has opened automatically. You can select your server and click on the Run button to  launch Glassfish from Eclipse. The Eclipse plugin launches Glassfish indirectly via an asadmin subprocess.

You will see some Glassfish log messages in the console. To stop Glassfish, do not hit the stop button in the Console view. This will just kill the asadmin process, but not Glassfish itself, and Eclipse and Glassfish will get terribly out of sync. Make sure to select the Servers view and hit the stop button there to avoid trouble.

Now for the interesting part: Let us import and build the gmap2 example project.

This project has a parent project with two subprojects, or modules in Maven terms. Maven requires you to store the module subprojects of a parent project in subdirectories of the parent directory. Eclipse normally cannot handle overlapping or nested directory structures for projects in the same workspace. Fortunately, m2eclipse works some magic to flatten the project structure, making Eclipse happy.

To import the example sources directly from the Subversion repository, select File | Import... | Maven | Check out Maven Projects from SCM and click Next. Fill in the SCM URL

https://wicket-stuff.svn.sourceforge.net/svnroot/wicket-stuff/tags/wicketstuff-core-1.4.9.2/gmap2-parent

and select the SCM provider svn, then click Next and Finish.

After a while, you will see three projects in your workspace:
  • gmap2
  • gmap2-examples
  • gmap2-parent

The parent project gmap2-parent has two subfolders gmap2 and gmap2-examples which are also represented as top-level Eclipse projects - this is a special feature of the m2eclipse integration.

The gmap2-examples subproject is a web application, as indicated by the Maven packaging type war. Using this and other information specified in the Maven POM, m2eclipse was able to turn this Eclipse project into a Dynamic Web Modules project - have a look at the Project Facets in the project properties to convince yourself.

Opening the project, you will notice two classpath containers Maven Dependencies and Web App Libraries. The latter contains just one item gmap2 with an open folder icon, indicating a reference to another subproject in our workspace that will go into WEB-INF/lib. The Maven Dependencies container displays all JARs required by our project, e.g. wicket-1.4.9.jar and slf4j-api-1.5.8.jar. All these were downloaded automatically by Maven into your local Maven repository on your hard drive, and m2eclipse makes Eclipse reference them from that location.

We are now ready to launch the web app: Select the gmap2-examples project folder and then Run As... | Run on Server from the context menu. Select the existing Glassfish server you created before and click Finish. After a while, you see the gmap2-examples welcome page in a web browser window.

Click on one of the links, e.g. marker listener, to see Google Maps in Firefox in Eclipse via Wicket on Glassfish on Java on Linux. (OK, maybe it's not Firefox or Linux on your machine...)

Note that the Java compilation and the WAR assembly and deployment were done by Eclipse, not by Maven.

However, you can also run a Maven build from Eclipse. To build our entire project hierarchy, select gmap2-parent and open Run As | Maven build... from the context menu. Enter the goals clean install and click Run. You will see the Maven messages in the console window. When Maven has finished, click Refresh (F5) on gmap2-parent, and open gmap2-examples/target to see the WAR file compiled by Maven.

20 July 2010

JPA 2.0: Querying a Map

Welcome back to more merriment with Maps in JPA 2.0!

After watching 3 out of 4 persistence providers choke on a model with a map in the previous post, let us now continue our experiments and see how our guinea pigs can handle JPQL queries for maps.

Recall that the JPQL query language has three special operators for building map queries: KEY(), VALUE() and ENTRY().

Now let us try and run the following query on a slightly modified model, compared to the previous post.

select m.text from MultilingualString s join s.map m where KEY(m) = 'de'

The corresponding model is:

@Embeddable
public class LocalizedString {

    private String language;

    private String text;

} 
 
@Entity
@Table(schema = "jpa", name = "multilingual_string")
public class MultilingualString {

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE)
    @Column(name = "string_id")
    private long id;

    @ElementCollection(fetch=FetchType.EAGER)
    @MapKeyColumn(name = "language_key")
    @CollectionTable(schema = "jpa", name = "multilingual_string_map", 
                     joinColumns = @JoinColumn(name = "string_id"))
    private Map<String, LocalizedString> map = new HashMap<String, LocalizedString>();
}

This time I've changed the model so that the map key is stored in its own column, which gives Hibernate and Eclipselink at least a chance to digest the model and proceed to the query. OpenJPA is fine with either version of the model.

DataNucleus is out of the game by now. I even tried replacing the @Embeddable by an @Entity and a few other things to cheat it into accepting my model, but in the end I gave up.

Now, Ladies and Gentleman, the winner and sole survivor is: OpenJPA again!

Both Hibernate and Eclipselink fail, merrily throwing exceptions. Hibernate only seems to have stubbed out the KEY() and VALUE() operators in their parser code (see HHH-5396 for the gory details and elaborate stack traces).

And Eclipselink's famous last words are:

Error compiling the query [select m.text from MultilingualString s join s.map m where KEY(m) = 'de'], 
line 1, column 9: unknown state or association field [text] of class [LocalizedString]. 
 

Not sure what the poor soul is trying to tell me.

To sum up: Should you ever consider working with persistent maps à la JPA 2.0, beware! Here be dragons...

17 July 2010

JPA 2.0: Mapping a Map

JPA 2.0 has added support for persistent maps where keys and values may be any combination of basic types, embeddables or entities.

Let's start with a use case:

The Use Case


In an internationalized application, working with plain old Strings is not enough, sometimes you also need to know the language of a string, and given a string in English, you may need to find an equivalent string in German.

So you come up with a LocalizedString, which is nothing but a plain old String together with a language code, and then you build a MultilingualString as a map of language codes to LocalizedStrings. Since you want to reuse LocalizedStrings in other contexts, and you don't need to address them individually, you model them as an embeddable class, not as an entity.

The special thing about this map is that the keys are part of the value. The map contents look like

'de' -> ('de', 'Hallo')
'en' -> ('en', 'Hello')

The Model


This is the resulting model:


[Update 20 July 2010: There is a slight misconception in my model as pointed out by Mike Keith in his first comment on this post. Editing the post in-place would turn the comments meaningless, so I think I'd better leave the original text unchanged and insert a few Editor's Notes. The @MapKey annotation below should be replaced by @MapKeyColumn(name = "language", insertable = false, updatable = false) to make the model JPA 2.0 compliant.]

@Embeddable
public class LocalizedString {

    private String language;

    private String text;

    public LocalizedString() {}

    public LocalizedString(String language, String text) {
        this.language = language;
        this.text = text;
    }
    
    // autogenerated getters and setters, hashCode(), equals()
} 
 
@Entity
@Table(schema = "jpa", name = "multilingual_string")
public class MultilingualString {

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE)
    @Column(name = "string_id")
    private long id;

    @ElementCollection(fetch=FetchType.EAGER)
    @MapKey(name = "language")
    @CollectionTable(schema = "jpa", name = "multilingual_string_map", 
                     joinColumns = @JoinColumn(name = "string_id"))
    private Map<String, LocalizedString> map = new HashMap<String, LocalizedString>();

    public MultilingualString() {}
    
    public MultilingualString(String lang, String text) {
        addText(lang, text);
    }
    
    public void addText(String lang, String text) {
        map.put(lang, new LocalizedString(lang, text));
    }

    public String getText(String lang) {
        if (map.containsKey(lang)) {
            return map.get(lang).getText();
        }
        return null;
    }
    
    // autogenerated getters and setters, hashCode(), equals()
}



The SQL statements for creating the corresponding tables:

CREATE TABLE jpa.multilingual_string
(
  string_id bigint NOT NULL,
  CONSTRAINT multilingual_string_pkey PRIMARY KEY (string_id)
)

CREATE TABLE jpa.multilingual_string_map
(
  string_id bigint,
  language character varying(255) NOT NULL,
  text character varying(255)
)

The Specification


The most important and most difficult annotation in this example is @MapKey. According to JSR-317, section 2.1.7 Map Keys:

If the map key type is a basic type, the MapKeyColumn annotation can be used to specify the column mapping for the map key. [...]
The MapKey annotation is used to specify the special case where the map key is itself the primary key or a persistent field or property of the entity that is the value of the map.

Unfortunately, in our case it is not quite clear whether we should use @MapKey or @MapKeyColumn to define the table column for our map key. Our map key is a basic type and our map value is not an entity, so this seems to imply we should use @MapKeyColumn.

On the other hand, our key is a persistent field of the map value, and I think the whole point of the @MapKey annotation is to indicate the fact that we simply reuse a property of the map value as the map key, so we do not need to provide an extra table column, as the given property is already mapped to a column.

The way I see it, replacing @MapKey by @MapKeyColumn(name = "language_key") - note the _key suffix! - is also legal, but then we get a different table model and different semantics: The table jpa.multilingual_string_map would have a fourth column language_key, this language_key would not necessarily have to be equal to the language of the map value.

Another open question: Is it legal to write @MapKeyColumn(name = "language")? If so, this should indicate that the language column is to be used as the map key, so this would be equivalent to the @MapKey annotation. On the other hand, you might say that this annotation indicates that the application is free to use map keys that are independent of the map values, so this contract would be violated if the column name indicated by the annotation is already mapped.

The Persistence Providers


I've tried implementing this example with the current versions of Hibernate, Eclipselink, OpenJPA and DataNucleus. I did not succeed with any of them. Only OpenJPA provided a workable solution using @MapKeyColumn, but as I said, I'm not sure if this usage is really intended by the specification.

[Update 20 July 2010: With the corrected model, the updated verdict is: Only OpenJPA passes the test, the other three bail out for various reasons.]

Let's look at the contestants in turn:

Hibernate


Using the mapping defined above, Hibernate 3.5.3-Final complains:

org.hibernate.AnnotationException: Associated class not found: LocalizedString

Apparently Hibernate is expecting the map value to be an entity not an embeddable.

Using @MapKeyColumn(name = "language"), the exception is

org.hibernate.MappingException: Repeated column in mapping for collection: MultilingualString.map column: language

Finally, with @MapKeyColumn(name = "language_key"), Hibernate no longer complains about duplicate columns, but I end up with a redundant table column in my database which I was trying to avoid.

Another problem with Hibernate is different behaviour when working with XML mapping data instead of annotations (which is what I prefer for various reasons, but that's a topic for another post).

Using XML metadata for this example, Hibernate happily ignores the table names from the metadata and simply uses the default names. I filed a bug report in April 2010 (HHH-5136), with no reaction ever since.


Eclipselink


Using Eclipselink 2.1.0, I simply get a rather cryptic exception

java.lang.NullPointerException
 at org.eclipse.persistence.internal.queries.MapContainerPolicy.compareKeys(MapContainerPolicy.java:234)

With @MapKeyColumn=(name = "language"), Eclipselink also complains about a duplicate column, and changing the name to language_key, my test finally passes, at the expense of a redundant column, as with Hibernate.

OpenJPA


With OpenJPA 2.0.0, the message is

org.apache.openjpa.persistence.ArgumentException: Map field "MultilingualString.map" is attempting to use a map table, 
but its key is mapped by another field.  Use an inverse key or join table mapping.

which I can't make sense of. Switching to @MapKeyColumn=(name = "language"), the new message is

org.apache.openjpa.persistence.ArgumentException: 
"LocalizedString.text" declares a column that is not compatible with the expected type "varchar".  

Its seems OpenJPA is confused by the column name text which sounds like a column data type. After adding @Column(name = "_text") to LocalizedString.text, my test case works and my database table only has three columns.

DataNucleus


DataNucleus 2.1.1 complains

javax.persistence.PersistenceException: Persistent class "LocalizedString" has no table in the database, 
but the operation requires it. Please check the specification of the MetaData for this class.

I'm getting the same message with all three variants of the annotation, so it appears that DataNucleus simply cannot handle embeddable map value and expects them to be entities.

Conclusion


Mapping maps with JPA is much harder than you would think, both for the user and for the implementor. Hibernate, Eclipselink and OpenJPA have all passed the JPA TCK. DataNucleus would have liked to do so, but they have not yet been granted access to the TCK.

All four implementors failed this simple map example to various degrees, which implies that there are features in the JPA 2.0 specification which are not sufficiently covered by the TCK.

An Open Source TCK for JPA would help in detecting and eliminating such gaps instead of leaving that to the initiative of individuals.

16 July 2010

JPA 2.0 Frustration

JPA 2.0 is one of the areas where Java EE 6 can make your life a lot easier compared to Java EE 5 - at least if your life is somehow connected to Java software development.

So much for the marketing blurb. In practice, working with JPA 2.0 means
  • trying to understand a specification (JSR 317) which may be quite challenging to read for the average developer and occasionally somewhat too vague even for experts
  • making sense of obscure stack traces from the JPA provider of your choice
  • discovering numerous bugs and omissions in implementations claiming to be JPA 2.0 compliant.
If you think this sounds fun, then read on...

Actually, this post is just an introductory note to a series of articles on specific use cases that seem to be particularly hard to get right.

I'm not unhappy about the spec as such - Object Relational Mapping (ORM) is a challenging topic and not exactly the stuff you expect first year computer science students to understand. JPA 2.0 narrows the gap between the JPA standard and vendor-specific extensions or native ORM features.

Still, there are some areas not covered by the standard where you have to fall back to vendor extensions or even write your own code: for instance, I would really like to see more flexible support for enum types, a standard for user-defined column types and an addition for spatial objects and queries, based on OGC standards.

The main source of frustration with JPA 2.0 is simply the lack of specification compliance of the available implementations. Implementing any but the most trivial persistence mappings and queries at application level can require hours of trial and error to get the expected results. Yes, most of the time the problem may be in your application code. But with JPA 2.0, chances are really high that your persistence provider has a bug.

The situation is not helped by the policy of Sun/Oracle/the JCP (I'm not really sure who's in charge of that) to keep the JPA TCK (Technology Compliance Kit) under a non-disclosure agreement - see this blog post from DataNucleus, which was linked in a related thread in the Glassfish forums recently. (DataNucleus is a JPA implementor not beloging to the inner circle of the JCP.)

So far, I've looked at
  • Hibernate
  • Eclipselink
  • OpenJPA
  • DataNucleus,
and sadly, I've had problems with all of them. I'm going to provide specific examples in the following posts.

26 June 2010

Early Adopter

My mum keeps telling me I used to be gifted child. Of course I don't remember everything I did at the age of three, but I seem to have signed up for the java.net forums at that time. Quite impressive!

14 May 2010

Java XML Binding with Property Change Support

In an earlier post, I was talking about generating Java classes with xjc with property change support which is not enabled by default. This post explains the details, an in fact the Maven build explained below is a lot a easier than my original approch with Ant which required some downloads to set things up.

The general idea is same as described by Kohsuke Kawaguchi in a post of 2007, but his POM is broken with the current stage of the Maven repositories. In addition, there is a tweak with the PropertyListener customization.

So here is a little example schema:

<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema" xmlns:tns="http://www.example.org/person/" targetNamespace="http://www.example.org/person/">
    <complexType name="Person">
      <sequence>
        <element name="firstName" type="string"></element>
        <element name="lastName" type="string"></element>
        <element name="address" type="tns:Address"></element>
      </sequence>
    </complexType>

    <complexType name="Address">
      <sequence>
        <element name="street" type="string"></element>
        <element name="houseNumber" type="string"></element>
        <element name="city" type="string"></element>
        <element name="postalCode" type="string"></element>
        <element name="country" type="string"></element>
      </sequence>
    </complexType>
</schema>

We use the following file bindings.xjb to customize the Java classes generated by xjc. We make all classes serializable and include support for PropertyChangeListeners.

<?xml version="1.0" encoding ="UTF-8"?>
<jaxb:bindings
    schemaLocation="person.xsd"
    version="2.1"
    xmlns:li="http://jaxb.dev.java.net/plugin/listener-injector"
    xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
    xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <jaxb:bindings node="/xs:schema">
    <jaxb:globalBindings>
      <jaxb:serializable uid="1" />
    </jaxb:globalBindings>
    <li:listener>java.beans.PropertyChangeListener</li:listener>    
  </jaxb:bindings>
</jaxb:bindings>


The following POM takes care of the code generation:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>jaxb-properties</groupId>
  <artifactId>jaxb-properties</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>JAXB PropertyListener Demo</name>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.3</version>
        <configuration>
          <source>1.6</source>
          <target>1.6</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.jvnet.jaxb2.maven2</groupId>
        <artifactId>maven-jaxb2-plugin</artifactId>
        <version>0.7.3</version>
        <executions>
          <execution>
            <goals>
              <goal>generate</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <extension>true</extension>
          <generatePackage>com.example.jaxb</generatePackage>
          <schemaIncludes>
            <schemaInclude>person.xsd</schemaInclude>
          </schemaIncludes>
          <bindingIncludes>
            <bindingInclude>bindings.xjb</bindingInclude>
          </bindingIncludes>
          <args>
            <arg>-Xinject-listener-code</arg>
          </args>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>com.sun.xml.bind</groupId>
            <artifactId>jaxb-xjc</artifactId>
            <version>2.1.13</version>
          </dependency>
          <dependency>
            <groupId>org.jvnet.jaxb2-commons</groupId>
            <artifactId>property-listener-injector</artifactId>
            <version>1.1-SNAPSHOT</version>
            <exclusions>
              <exclusion>
                <groupId>com.sun.xml.bind</groupId>
                <artifactId>jaxb-xjc</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
        </dependencies>
      </plugin>

    </plugins>
  </build>
</project>


You do need to use the 1.1-SNAPSHOT version of the property-listener-injector, as the 1.0 release only supports VetoableChangeListeners and not the PropertyChangeListeners we want to use. The <li:listener> element in the bindings file is used to define the listener class.

It is important to exclude the jaxb-xjc dependency from the property-listener-injector, or else it will try to download an non-existing snapshot version. Looks like something is broken in the java.net repository, but then again, working with snapshot releases is always dangerous....

13 May 2010

Glassfish Logging with slf4j

Update 18 Dec 2010: There is an extended version of this article.

It can be annoying to have each third-party library or framework in your application logging to a different logfile. slf4j with its various adapters enables you to collect logging events from various APIs (org.slf4j.api, java.util.logging, org.apache.commons.logging) and redirect them to a logging backend of your choice.

I've always preferred log4j or, more recently, logback over java.util.logging, so after working with Glassfish v3 for a while, I tried to tweak it to use logback over slf4j and the jul-to-slf4j bridge.

To redirect java.util.logging used by Glassfish v3, you put the following libs on your classpath:
  • jul-to-slf4j.jar
  • slf4j-api.jar
  • logback-classic.jar
  • logback-core.jar
(You can replace the logback libs by any other supported backend like log4j, see the slf4j docs for more details.)
The problem with Glassfish is:
  • You need to take care of its class loader hierarchy and make sure that the logging jars get picked up early enough.
  • Glassfish does some all-too-clever logger manipulation in its LogManagerService which will get in your way if you don't like the Glassfish defaults: It redirects all System.out messages to a logger.
I did not manage to fully supersede Glassfish logging by slf4j and logback, but at least I can copy all Glassfish log events to my logback application log like this:
  • Put the logging libs in [install-root]/lib/endorsed.
  • Build a JAR containing your logback.xml configuration and put it in the same place.
  • Edit an entry in [instance-root]/config/logging.properties, setting
    handlers = org.slf4j.bridge.SLF4JBridgeHandler
Now if you omit a ConsoleAppender from your logging configuration, the Glassfish server.log will no longer contain any log output from your application. It will still contain the log messages from Glassfish itself, but they will also be included in your own logback logfile.

09 May 2010

jeeunit: In-Container Integration Testing for Java EE 6

Recently, I've started working with Java EE 6 in general and Glassfish v3 in particular. Some software engineering best practices do not really depend on the framework or even the language you work with, so I was looking for a convenient way of doing automatic integration tests.

I've been in the habit of using JUnit not just for unit tests, but also for integration or system tests, so this was a natural starting point.

Traditionally, Java EE containers were heavy-weight machinery so that people preferred writing their tests to run out-of-container, paying the price of emulating or mocking some of the container functionality.

Maybe I've been looking in the wrong places, but most of the out-of-container testing approaches like ejb3unit seem to carry more baggage than the container itself, at least when it comes to working with Glassfish v3.

Anyway, as I could not find a ready-to-go solution, I wrote a little JUnit extension called jeeunit together with an example project showing how to set things up with Glassfish v3 and Maven.

The jeeunit project is available from Google Code under an Apache License.

03 March 2010

The Ultimate Software Metric

28 February 2010

Misconceptions about Java Internationalization

Let me start with a joke:
What do you call someone who speaks three languages?
Trilingual.
What do you call someone who speaks two languages?
Bilingual.
What do you call someone who speaks one language?
American.
To be fair on Americans, even most of us multilingual Europeans tend to be biased when it comes to internationalization, tacitly assuming that text is written left-to-right and can be sorted from A to Z.

Most Java developers are familiar with resource bundles backed by properties files. The basics can be found in the Internationalization Trail of the Java Tutorial. Multilingual Java applications often come with a set of properties files, e.g.
  • MyApp_de_AT.properties
  • MyApp_de.properties
  • MyApp_es.properties
  • MyApp.properties
where MyApp.properties contains the "default" message resources in English, MyApp_de.properties and MyApp_es.properties contain the German and Spanish resources, respectively, and MyApp_de_AT.properties contains some country specific variants for the Austrian flavour of German. Usually, files for country specific variants are sparsely populated, containing only those properties that actually differ from the mainstream language version, like Jänner (de_AT) vs. Januar (de) vs. January (en).

However, you may be surprised in this case to end up with a German string even when you requested a resource for an English locale.

Assume nothing is a sound principle for robust software development, and you should not assume that English is the default or fallback language. In fact, the fallback for resources from an unsupported locale is the system default locale, which is based on the host environment.

See the documentation for ResourceBundle.getBundle() and Locale.getDefault() for more details.

So when the default locale of your system is de_DE and you request a resource for locale en_US, the lookup order for the properties files is
  1. MyApp_en_US.properties
  2. MyApp_en.properties
  3. MyApp_de_DE.properties
  4. MyApp_de.properties
  5. MyApp.properties
Hence, ResourceBundle.getString() will return a German string from MyApp_de.properties, since the first three files do not exist and the English resources are preceded by the German ones in this sequence.

There are two solutions:
  1. As a user, set your default locale to en when launching the application.
  2. As a developer, make sure to provide a properties file for locale en (which may be empty).
The method for changing the default locale depends on your Java VM and your operating system. Setting the system property user.language may work on some platforms, but not with the Sun JDK 1.6.0 under Linux. Instead, you need to set the environment variable LANG before launching the Java VM.

The preferred solution is the second one, of course. Even when MyApp_en.properties is empty, it will be picked up as entry point for resource lookup. If a given key cannot be found in this file, the parent file MyApp.properties will be used as fallback, which is just the desired behaviour.