07 June 2011

Java EE 6 Server Comparison: Introduction

A year and a half after the Java EE 6 specification release, there are now three Open Source servers certified for the Java EE 6 Web Profile:
  • Glassfish 3.x
  • JBoss 6.x
  • Resin 4.0.x
Time to take a look at these three servers in direct comparison and do some experiments, not with a toy or benchmark application developed for this purpose, but with a real-life enterprise application.


Taking an application developed on server X and porting it to servers Y and Z might cause some bias towards server X, so my scenario is the following:
  • MyApp is an enterprise application originally developed on Tomcat 6 and Spring 3, using JPA 2.0 (Hibernate on MySQL), declarative transactions, dependency injection, and Spring MVC in the web layer. In the persistence and service layers, this application has no compile time dependencies on Spring, thanks to the fact that Java EE 6 annotations like @Inject or @TransactionAttribute are also supported by Spring.
  • MyApp is built with Maven, my development environment is Eclipse 3.6.2 with m2eclipse.
The goals of my experiments:
  • Make MyApp run on a Java EE 6 server (using JPA, JTA, EJB, CDI, Servlet, JSP) and restrict the run-time use of Spring to the web layer, using a CDI-to-Spring bridge.
  • Build a WAR that can be deployed to each of Glassfish, JBoss and Resin.
  • Work with the Eclipse plugin for each server, compare the edit-save-deploy cycles and developer productivity in general.
  • Work with the stand-alone servers, and compare startup and deployment times and memory usage.
Disclaimer: I started working with Glassfish in March 2010 and I haven't really used either JBoss or Resin before these experiments, so I may be biased towards Glassfish. I'm not in any way affiliated with Oracle, Red Hat or Caucho, and the views expressed in this blog are my own and not my employer's.

A note on Java EE 6 Profiles and certification: Glassfish is of course certified both for the Full Profile and for the Web Profile. JBoss AS 6.0.0.Final was announced as a fully certified implementation of the Java EE 6 Web Profile, but for some reason it is not yet listed on the official Java EE compatibility page. JBoss also claims to support the Full Profile, but has not taken the pains of undergoing certification for it. Resin is explicitly focused on the Web Profile and has recently been certified. Resin also supports JMS, which is not part of the Web Profile.

MyApp only requires the Web Profile. For my experiments, I've used the Full Profile installation of Glassfish, the default server configuration of JBoss, and the Open Source Resin installation (not Resin Pro).

Performance Measurement Scenarios


For each server, I ran the following tests:

  • Empty server startup time. ("Empty" means without any applications deployed by the user, running the server out of the box, which may include some management applications.)
  • Empty server total heap and PermGen size. (After starting the server and forcing a garbage collection from jvisualvm.)
  • MyApp deployment time (copying it to the autodeploy directory of the running server).
  • Server + MyApp restart time (restarting the server with my deployed application).
  • Server + MyApp total heap and PermGen size. (After starting the server with the deployed applicaton and forcing a garbage collection from jvisualvm).
  • MyApp redeployment time (after touching myapp.war in the autodeploy directory).
  • Server + MyApp startup time from Eclipse. (Using Run on Server on the top-level web project).
  • Server + MyApp total heap and PermGen size, running from Eclipse (after forcing GC as above).
  • MyApp redeploy time after changing a source file in Eclipse.
  • Checking for memory leaks after redeploying.

Read the whole story


7 comments:

Reza Rahman said...

Harald,

Thanks so much for your kind words about Resin and the Resin team. We take your comments very seriously and will make sure to address them before the upcoming Resin 4.1 release. Please let us know if you have any more comments for us. This is the ideal time for that since we are currently focused on stability, bugs and performance before we have the 4.1 release.

Cheers,
Reza

struberg said...

Hi Harald!

Nice comparison, but I miss the most important number in the performance tests: How fast is the web application itself?

I found factor 15 differences when I first tested a big webapp on different EE servers and on tomcat + OpenWebBeans.
And it really makes a difference if a page renders in 300ms or in 5 seconds...

LieGrue,
strub

Harald Wellmann said...

@strub: You're right, I left out runtime performance, but that was a deliberate decision.

It takes some kind of test harness, which I didn't have at hand, and most of the work in my application happens in the persistence level, so I don't know if my results would have been representative for apps with a lot of web traffic.

Reza Rahman said...

We'd actually be very interested to know that since runtime performance, especially vis-a-vis plain Servlet containers, has always been a focus for us. As far as I know, the only recent well-known benchmark in this area has been GlassFish (Grizzly) vs. Tomcat (in which Grizzly seemed to fare quite well).

You have a good point that pure web performance concerns do get outweighed by persistence tier performance for most realistic web applications.

Opperbolt said...

Pitty this does not mention websphere, which is Java EE6 full profile certified. I thought this was a strict superset of the web profile.

Harald Wellmann said...

@Jan: I published this series of articles before WebSphere AS 8.0 was released.

Gurkan Erdogdu said...

Maybe you can also look at http://siwpas.mechsoft.com.tr for Siwpas.