23 October 2011

JBoss AS 7: Catching up with Java EE 6



In my Java EE 6 server comparison of June 2011, JBoss AS 6.0.0 was not exactly the shining star, both in terms of performance and usability.

The next major release JBoss AS 7, claiming to be lightning fast, was published in July 2011, followed by two maintenance updates in August and September.

Time to take another look at JBoss and check if it now compares more favourably to other servers.

This post is based on JBoss AS 7.0.2 (Full Profile) and the same real-world application which I had ported from Spring to Java EE 6 for the GlassFish/Resin/JBoss AS 6 comparison.

With some minor modifications, I was able to deploy the same WAR on JBoss AS 7 and to repeat the measurements I did for JBoss AS 6.

Compared to JBoss AS 6 which did not even offer a Getting Started Guide, the documentation of JBoss AS 7 has improved a lot, but there are still quite a few gaps to be closed. At any rate I found sufficient input for my experiments in the following two documents:


Preparing the server

My application contains a JPA 2.0 persistence unit developed on Hibernate 3.6.x and MySQL 5.1, so first of all I had to configure the data source and the JDBC driver.

To do so, I added this section to the main configuration file of JBoss AS 7 in standalone/configuration/standalone.xml:

<datasource jndi-name="jdbc/myapp" pool-name="myapp" enabled="true" jta="true" use-java-context="true" use-ccm="true">
    <connection-url>
        jdbc:mysql://localhost/myapp
    </connection-url>
    <driver-class>
        com.mysql.jdbc.Driver
    </driver-class>
    <driver>
        mysql-connector-java-5.1.13.jar
    </driver>
    <transaction-isolation>
        TRANSACTION_READ_COMMITTED
    </transaction-isolation>
    <pool>
        <min-pool-size>
            4
        </min-pool-size>
        <max-pool-size>
            30
        </max-pool-size>
        <prefill>
            true
        </prefill>
        <use-strict-min>
            false
        </use-strict-min>
        <flush-strategy>
            FailingConnectionOnly
        </flush-strategy>
    </pool>
    <security>
        <user-name>
            MYUSER
        </user-name>
        <password>
            SECRET
        </password>
    </security>
</datasource>
<drivers>
    <driver name="mysql-connector-java-5.1.13.jar" module="com.mysql"/>
</drivers>

Then I copied both the JDBC driver mysql-connector-java-5.1.13.jar and my application myapp.war to standalone/deployments/ and started the server via

bin/standalone.sh

Troubleshooting


Of course, this did not work out of the box. The first exception I had was

java.lang.ClassCastException: org.dom4j.DocumentFactory cannot be cast to org.dom4j.DocumentFactory

This looked like a classloader conflict for dom4j. My WAR has a dom4j.jar in its WEB-INF/lib, but dom4j is also provided by JBoss as an transitive dependency of Hibernate. To resolve the conflict, I rebuilt my WAR without dom4j.

The next exception was

java.lang.ClassNotFoundException: com.sun.xml.bind.Locatable 

My original application included jaxb-impl-2.2.2.jar, which I'd had to remove for JBoss AS 6, so I re-added this dependency to my WAR and redeployed it.

JBoss AS 7 marks any failed deployment with a foo.war.failed file in the deployment directory and will not attempt to redeploy the application until the marker file is deleted.

After restarting the server, the next problem was a

org.jboss.weld.exceptions.DeploymentException: WELD-001408 Unsatisfied dependencies for type [MyType] with qualifiers [@Default] 
at injection point [[field] @Inject private MyClient.myType]

This was confusing. I had beans.xml files in all my application modules, and the same WAR did not cause any CDI exceptions on other servers. So I enabled DEBUG logging for Weld to find out what was going on by adding

<logger category="org.jboss.weld">
    <level name="DEBUG"/>
</logger>

to the logging section of standalone.xml. This did not have the desired effect. I also had to change the

<level name="INFO"/>

child elements of the console-handler and the periodic-rotating-file-handler from INFO to ALL. (The Logging Configuration documentation does not mention this, but some additional information is hidden in the Getting Started Guide, section Configure Logging in JBoss Application Server 7.)

Now the DEBUG output from Weld revealed the cause of the problem:

17:33:39,799 DEBUG [org.jboss.weld.ClassLoading] (MSC service thread 1-8) WELD-000119 Not generating any bean definitions 
from MyType because of underlying class loading error
Caused by: java.lang.NoClassDefFoundError: Lorg/dom4j/Document;

Strange, dom4j had caused a classloader conflict at first, and now it was no longer visible at all? I had a closer look at Class Loading in AS7 to understand the background of this.

In short, JBoss AS 7 has a module structure, and server modules are not visible to applications by default, unless they are imported or exported explicitly.

Both dom4j and jaxb-impl are JBoss modules, so I re-deleted jaxb-impl and jaxb-api from my application and made the JBoss modules globally visible by adding the following to standalone.xml:

<subsystem xmlns="urn:jboss:domain:ee:1.0">
    <global-modules>
        <module name="org.dom4j" slot="main"/>
        <module name="com.sun.xml.bind" slot="main"/>
    </global-modules>
</subsystem>

(Global visibility is not the best solution, of course, but for this experiment I would rather modify the server configuration than add JBoss-specific manifest headers or deployment descriptors to my application WAR.)

Finally, I had another ClassNotFoundException for an internal Xerces class from the JDK required by my application. (Yes, such a dependency is bad practice, but as I said above, I'm using a real-world application for these tests...)

I made this class visible to my application by adding

<path name="com/sun/org/apache/xerces/internal/impl/io"/>

to modules/sun/jdk/main/module.xml - no idea if you are supposed to do it this way, but it works for me...

Now at last I was able to deploy and run my application and take some measurements.


Performance measurements


The following table compares some performance indicators for JBoss AS 7.0.2, JBoss AS 6.0.0 and GlassFish 3.1.1. For JBoss AS 6, I did not repeat the measurements but simply copied the results of June 2011. For GlassFish, I had been working with a pre-release build at that time, so I've now re-run my tests with the official GlassFish 3.1.1 release.

JBoss AS 7.0.2 JBoss AS 6.0.0 GlassFish 3.1.1
Empty server startup time 1.9 s 12 s 3.2 s
Empty server heap memory 10.5 MB 100 MB 26.5 MB
Empty server PermGen memory 36.3 MB 70 MB 28.4 MB
MyApp deployment time 5.8 s 47 s 13 s
Server + MyApp restart time 8 s 30 s 14.5 s
Server + MyApp heap memory 52.8 MB 236 MB 55.3 MB
Server + MyApp PermGen memory 80.9 MB 175 MB 84.5 MB
MyApp redeployment time 3.5 s 30 s 7 s

All in all, there is an amazing improvement of performance and memory footprint in JBoss AS 7, compared to JBoss AS 6. JBoss AS 7 is now at a competitive level with Resin and Glassfish and actually outperforms Glassfish in almost all of these tests.

I don't know if this indeed qualifies as lightning fast, but it is now safe to say that JBoss AS 6 was blatantly fat and slow, as confirmed by its own successor.

Redeployment


I tested redeploying my application by simply touching the WAR file in the deployments directory and watching the memory usage via jvisualvm.

I could see the PermGen usage grow at a constant rate, and after a couple of redeployments, there was an OutOfMemoryError, so JBoss AS 7 appears to have a new classloader leak. (The same test did pass successfully on JBoss AS 6.)

Eclipse Integration


In contrast to the drastic improvements of the server itself, I could find no significant progress with the Eclipse integration. I installed the JBoss AS Tools from JBoss Tools 3.3.0.M3 into a fresh copy of Eclipse Indigo 3.7.1 with m2e 1.0 and m2e-wtp 0.14.0.

I was able to deploy my application directly from the workspace, but editing and saving a source file did not redeploy the application. An Incremental Publish did not help either. Only a Full Publish was sufficient to redeploy the application.

This is a major productivity issue for Eclipse users. Redeploy-on-save works smoothly with the built-in Tomcat integration of Eclipse WTP, so something must be broken in JBoss AS Tools.

JPA Providers


JBoss AS 7.0.2 comes bundled with Hibernate 4.0.0.CR2. This is a bad smell - an official product release should never contain any release candidates of upstream components, especially not for components originating from the same company.

The good news is that my application which was developed on Hibernate 3.6.0 works without problems on this Hibernate 4.x release candidate.

But the real bad news is that JBoss AS 7 does not currently support other persistence providers like Eclipselink, OpenJPA or DataNucleus (according to the JPA Reference Guide). With GlassFish and Resin, you can simply drop the JARs of your preferred provider and its dependencies in a designated folder of your server installation and edit your persistence.xml to override the default provider of the server.

JBoss AS 7 appears to require an adapter per persistence provider, which to me looks like an unfortunate and unnecessary design decision.

At any rate, this lack of flexibility is a severe restriction. In most of my projects, I had to replace Hibernate by OpenJPA due to a number of bugs related to new JPA 2.0 features.

Conclusion


Given the drastic performance improvements in JBoss AS 7, which was released only six months and a half after JBoss AS 6, I can only wonder why RedHat and the JBoss community ever spent any resources on an AS 6 release which clearly was not competitive, instead of focusing on AS 7.

Anybody looking for a production-quality Java EE 6 server in the first half of 2011 was better off with GlassFish 3.1.1 or one of its pre-releases.

With AS 7, JBoss is now back in the game, it is surprisingly lean and fast, and it even has the potential to take over the lead from GlassFish.

But performance is not the only factor that counts. The documentation continues to be sketchy and far below the standard of JBoss AS 5.

Different projects have different needs, but for me, the top-level performance of JBoss AS 7 is more than outweighed by classloader leaks, productivity issues of the Eclipse integration and lack of support for JPA providers other than Hibernate.

Each of these is currently a blocker for using JBoss AS 7 in production. GlassFish 3.x has taken more than a year to reach production quality. Let's see if JBoss AS 7.x can do the same lightning fast.

9 comments:

Alexis MP said...

Aren't you comparing a web profile (JBoss 7.0.2) and a full Java EE 6 platform (GlassFish 3.1.1)?

Harald Wellmann said...

Not exactly - I made sure to use the "Everything" distribution of JBoss. Yes, it's not certified and I don't know if it really covers the full Java EE 6 profile.

Anyway, it shouldn't make much of a difference, as both GlassFish and JBoss lazily activate their components, and my test application requires the Web profile only.

Jaikiran said...

Harald,

Do you have more details about the PermGen issue on redeployment? Maybe a sample application or any other details? I _think_ we had a test somewhere where we used to redeploy around 6k times (via Arquillian) the same deployment to check for these leaks. I think that had passed 7.0.0. Not sure if something broke after that. If you have any more details, do let us know. In the meantime, I'll if I can find that testcase.

Jaikiran said...

JBoss AS 7 marks any failed deployment with a foo.war.failed file in the deployment directory and will not attempt to redeploy the application until the marker file is deleted.

I thought we fixed that https://issues.jboss.org/browse/AS7-1237 in 7.0.0.Final itself. We even have a testcase for that in our testsuite. Do you you redeploy the application? Is it an archive or an exploded deployment?

Harald Wellmann said...

@Jaikiran: Regarding the *.failed marker, my application is an archive.

I've just double checked: I commented out the global-modules in my standalone.xml to provoke a deployment failure. Then I stopped the server, reverted the configuration and restarted the server.

The application does not get redeployed until I delete the *.failed marker files.

The PermGen issue is more difficult, of course. I'll try to look at a heap dump with the Eclipse Memory Analyzer, to get a clue what the root cause might be.

This will take some time - if I can narrow down the problem, I'll get back to you or post a message in the JBoss forum.

Best regards,
Harald

Jaikiran said...

Hi Harald,


I've just double checked: I commented out the global-modules in my standalone.xml to provoke a deployment failure. Then I stopped the server, reverted the configuration and restarted the server.

The application does not get redeployed until I delete the *.failed marker files.


Thanks for those details. I think I see what's going on. It appears to be specific to server restart. If the archive timestamp gets changed without a server restart, it seems to work fine by identifying the changed archive and redeploying the archive (without requiring a manual step to delete the .failed marker). There appears to be a issue when a server restart is involved. I'll take a look at this tomorrow.


The PermGen issue is more difficult, of course. I'll try to look at a heap dump with the Eclipse Memory Analyzer, to get a clue what the root cause might be.

This will take some time - if I can narrow down the problem, I'll get back to you or post a message in the JBoss forum.


That would be great. Thanks! By the way, I just tested that 6k redeployment scenario that I mentioned earlier, against 7.0.2 and it didn't show any PermGen failures. So I guess it specific to the app you are using. So any details (whenever you get the time) will help in narrowing down the isssue.

Thanks!

Harald Wellmann said...

I've been poking around in a heap dump, and there's a BeanMetaDataCache holding references to multiple copies of my application, which looks rather suspicious.

See http://community.jboss.org/thread/174014 for details.

Jaikiran said...

Harald,

Thanks for creating that forum thread about the PermGen issue, with the details.

About the .failed marker deployment issue, I've just opened a AS7 dev list discussion here http://lists.jboss.org/pipermail/jboss-as7-dev/2011-October/004304.html

Arron Ferguson said...

In order to redeploy on JBoss AS 7, one needs to set the exploded view (i.e. extracted WAR) to redeploy within the admin console.

That said, I had huge problems attempting to deploy anything other than the out-of-the-box JBoss projects (created from the JBoss Tools plugin for Eclipse) and like you said, don't even think about using anything other than janky Hibernate, it simply won't work.

I also could not get Primefaces to work with it either. The whole JBoss motto appears to be: our way or the highway and don't ask question.

Could have something to do with the fact that one can find Tomcat and Glassfish hosting but very rarely any JBoss hosting. It's simply too much pain and effort to get working.