JBoss AS 7: Catching up with Java EE 6

In my Java EE 6 server comparison of June 2011, JBoss AS 6.0.0 was not exactly the shining star, both in terms of performance and usability.

The next major release JBoss AS 7, claiming to be lightning fast, was published in July 2011, followed by two maintenance updates in August and September.

Time to take another look at JBoss and check if it now compares more favourably to other servers.

Read more »

Java EE 6 Server Comparison: Glassfish

In April 2010, I started a Java EE 6 project, and Glassfish 3.0 was the obvious server choice, or rather, the only option at that time. For the next couple of months, I spent a considerable amount of time not working on my application, but analyzing, reporting and working around bugs in the server, mainly in the CDI and JPA areas. This appeared to be the price you had pay for being an early adopter of the new Java Enterprise standard.

Read more »

OSGi Support in JPA 2.0 Persistence Providers

About three years ago, I started working with Hibernate in an OSGi environment, and at that time, I had to roll my own solution for resolving all sorts of classloader issues.

Since then, with OSGi 4.2 and JPA 2.0, there have been major specification releases, followed by implementations; OSGi is growing ever more popular, and the current release of the JPA standard has integrated a lot of functionality which used to be available in proprietary API extensions only.

The OSGi 4.2 Enterprise Specification includes JPA, JTA and JDBC services, so in theory, using JPA in an OSGi environment should stop being an issue, as soon as this specification is supported by all OSGi and JPA implementations.

In practice, the specifications are not yet fully covered by the available implementations, and the degree of OSGi support varies a lot.

In the past few months, I have worked with the current versions of the three certified JPA 2.0 persistence providers, and in my opinion, the ranking for OSGi support is quite clear:

1. Eclipselink
2. OpenJPA
3. Hibernate

Some more details on each provider:

Eclipselink

Eclipselink has a fully osgified distribution containing OSGi bundles of Eclipselink itself and all dependencies. To make your entity classes visible to Eclipselink, all you need to do is add a JPA-PersistenceUnits manifest header to each bundle containing one or more persistence units. This is enough for Eclipselink to load all your XML metadata and entity classes.

No ugly hacks, no buddy policies or fragments, it just works out of the box.

So for a quick start with JPA under OSGi, Eclipslink is definitely the best choice, even though the OSGi 4.2 JPA Service Specification is not yet fully supported (for instance, the Meta-Persistence header does not work), but this will be of little or no concern to most users.

Leaving OSGi aside, I have been bitten by a number of bugs in Eclipselink, discussed in earlier articles. If you do not use any of these features, you may be very happy with Eclipselink. For me, this set of bugs is currently a no-go, so I have migrated my OSGi applications from Eclipselink to OpenJPA.

OpenJPA

The silver medal goes to OpenJPA: OSGi does not appear to be on the top of the agenda of the OpenJPA core developers, but since OpenJPA is used under OSGi by Aries, another Apache project, there is at least a certain level of OSGi support.

In the binary distribution, the OpenJPA aggregate JAR is an OSGi bundle, but most of the dependencies are plain old JARs, so you need to find osgified versions of the dependencies on your own. The same goes for the Maven artifacts.

Once you have downloaded the appropriate OSGi bundles, you still have to do some extra work to ensure that OpenJPA finds your persistence units and your entity classes, and there are a couple of minor issues with the enhancer and with user-defined value handlers.

I will explain the details of my setup and some of the problems that should be addressed in OpenJPA in my next post.

Hibernate

Sadly, regarding OSGi support, Hibernate 3.5.x is not much different from the 3.2.x release I discussed in my earlier articles. The distribution contains plain old JARs, no OSGi bundles. The latest osgified version available from the SpringSource Enterprise Bundle Repository is 3.4.0, and the SpringSource versions never used to work for me, so I expect that the steps described in my 2008 article are still required to make Hibernate work on OSGi.

Disclaimer: I did not try to make Hibernate 3.5 work on OSGi, due to its lack of support of a number of JPA 2.0 features used heavily by my applications. For this reason, I have stopped using Hibernate, both on OSGi and on Java EE.

JPA 2.0: Ordered Collections

In earlier posts, I wrote about problems with persistent maps, a new feature introduced in JPA 2.0. Ordered collections with explicit order columns are another JPA 2.0 feature which, again, is not yet fully robust in all JPA 2.0 compliant and TCK tested implementations.

I'm currently working with a simple JPA model for OpenStreetMap (OSM). There are ways and nodes, and each way has a sequence of nodes. Of course the order of the nodes is important, and a node may be used more than once for a given way.

Thus, the nodes collection has to be a List, not a Set, and we need an explicit order column specifying the sequence number of the nodes along the way. Sorting the nodes by ID would not make any sense, obviously.

Here is a snippet from the model:

@Entity
@Table(name = "ways")
public class OsmWay
{
    @Id
    private long id;
    
    @ManyToMany(cascade = CascadeType.ALL)
    @OrderColumn(name = "sequence_id")
    @JoinTable(name = "way_nodes", 
               joinColumns = @JoinColumn(name = "id"), 
               inverseJoinColumns = @JoinColumn(name = "node_id"))
    private List<OsmNode> nodes = new ArrayList<OsmNode>();
}    

This entity is represented by two tables:

CREATE TABLE ways
(
  id bigint NOT NULL
);

CREATE TABLE way_nodes
(
  id bigint NOT NULL,
  node_id bigint NOT NULL,
  sequence_id integer,
);

I've tested this scenario on Hibernate 3.5.3, Eclipselink 2.1.1 and OpenJPA 2.0.1: Hibernate and OpenJPA pass, Eclipselink fails with 2 issues.

First, the generated DDL is incorrect: There is a PRIMARY KEY (id, node_id) for way_nodes. This should be (id, sequence_id).

Second, the order of the list items is not maintained in all contexts. It is ok when the collection is lazily loaded, e.g.

OsmWay way = em.find(OsmWay.class, wayId);
List<OsmNode> nodes = way.getNodes();

but it is broken when using a fetch join:

String jpql = "select distinct w from OsmWay w join fetch w.nodes";
TypedQuery<OsmWay> query = em.createQuery(jpql, OsmWay.class);
List<OsmWay> ways = query.getResultList();
OsmWay way = ways.get(0);
List<OsmNode> nodes = way.getNodes();

The funny thing is, the nodes are neither ordered by sequence_id nor by node_id. In my test case, the way has 5 nodes, one of which appears twice in different positions.

So here is some more evidence for my claim that the JPA 2.0 TCK is insufficient.

JPA 2.0: Querying a Map

Welcome back to more merriment with Maps in JPA 2.0!

After watching 3 out of 4 persistence providers choke on a model with a map in the previous post, let us now continue our experiments and see how our guinea pigs can handle JPQL queries for maps.

Recall that the JPQL query language has three special operators for building map queries: KEY(), VALUE() and ENTRY().

Now let us try and run the following query on a slightly modified model, compared to the previous post.

select m.text from MultilingualString s join s.map m where KEY(m) = 'de'

The corresponding model is:

@Embeddable
public class LocalizedString {

    private String language;

    private String text;

} 

@Entity
@Table(schema = "jpa", name = "multilingual_string")
public class MultilingualString {

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE)
    @Column(name = "string_id")
    private long id;

    @ElementCollection(fetch=FetchType.EAGER)
    @MapKeyColumn(name = "language_key")
    @CollectionTable(schema = "jpa", name = "multilingual_string_map", 
                     joinColumns = @JoinColumn(name = "string_id"))
    private Map<String, LocalizedString> map = new HashMap<String, LocalizedString>();
}

This time I've changed the model so that the map key is stored in its own column, which gives Hibernate and Eclipselink at least a chance to digest the model and proceed to the query. OpenJPA is fine with either version of the model.

DataNucleus is out of the game by now. I even tried replacing the @Embeddable by an @Entity and a few other things to cheat it into accepting my model, but in the end I gave up.

Now, Ladies and Gentleman, the winner and sole survivor is: OpenJPA again!

Both Hibernate and Eclipselink fail, merrily throwing exceptions. Hibernate only seems to have stubbed out the KEY() and VALUE() operators in their parser code (see HHH-5396 for the gory details and elaborate stack traces).

And Eclipselink's famous last words are:

Error compiling the query [select m.text from MultilingualString s join s.map m where KEY(m) = 'de'], 
line 1, column 9: unknown state or association field [text] of class [LocalizedString]. 

Not sure what the poor soul is trying to tell me.

To sum up: Should you ever consider working with persistent maps à la JPA 2.0, beware! Here be dragons...

JPA 2.0: Mapping a Map

JPA 2.0 has added support for persistent maps where keys and values may be any combination of basic types, embeddables or entities.

Let's start with a use case:

The Use Case

In an internationalized application, working with plain old Strings is not enough, sometimes you also need to know the language of a string, and given a string in English, you may need to find an equivalent string in German.

So you come up with a LocalizedString, which is nothing but a plain old String together with a language code, and then you build a MultilingualString as a map of language codes to LocalizedStrings. Since you want to reuse LocalizedStrings in other contexts, and you don't need to address them individually, you model them as an embeddable class, not as an entity.

The special thing about this map is that the keys are part of the value. The map contents look like

'de' -> ('de', 'Hallo')
'en' -> ('en', 'Hello')

The Model

This is the resulting model:


[Update 20 July 2010: There is a slight misconception in my model as pointed out by Mike Keith in his first comment on this post. Editing the post in-place would turn the comments meaningless, so I think I'd better leave the original text unchanged and insert a few Editor's Notes. The @MapKey annotation below should be replaced by @MapKeyColumn(name = "language", insertable = false, updatable = false) to make the model JPA 2.0 compliant.]

@Embeddable
public class LocalizedString {

    private String language;

    private String text;

    public LocalizedString() {}

    public LocalizedString(String language, String text) {
        this.language = language;
        this.text = text;
    }
    
    // autogenerated getters and setters, hashCode(), equals()
} 

@Entity
@Table(schema = "jpa", name = "multilingual_string")
public class MultilingualString {

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE)
    @Column(name = "string_id")
    private long id;

    @ElementCollection(fetch=FetchType.EAGER)
    @MapKey(name = "language")
    @CollectionTable(schema = "jpa", name = "multilingual_string_map", 
                     joinColumns = @JoinColumn(name = "string_id"))
    private Map<String, LocalizedString> map = new HashMap<String, LocalizedString>();

    public MultilingualString() {}
    
    public MultilingualString(String lang, String text) {
        addText(lang, text);
    }
    
    public void addText(String lang, String text) {
        map.put(lang, new LocalizedString(lang, text));
    }

    public String getText(String lang) {
        if (map.containsKey(lang)) {
            return map.get(lang).getText();
        }
        return null;
    }
    
    // autogenerated getters and setters, hashCode(), equals()
}

The SQL statements for creating the corresponding tables:

CREATE TABLE jpa.multilingual_string
(
  string_id bigint NOT NULL,
  CONSTRAINT multilingual_string_pkey PRIMARY KEY (string_id)
)

CREATE TABLE jpa.multilingual_string_map
(
  string_id bigint,
  language character varying(255) NOT NULL,
  text character varying(255)
)

The Specification

The most important and most difficult annotation in this example is @MapKey. According to JSR-317, section 2.1.7 Map Keys:

If the map key type is a basic type, the MapKeyColumn annotation can be used to specify the column mapping for the map key. [...]
The MapKey annotation is used to specify the special case where the map key is itself the primary key or a persistent field or property of the entity that is the value of the map.

Unfortunately, in our case it is not quite clear whether we should use @MapKey or @MapKeyColumn to define the table column for our map key. Our map key is a basic type and our map value is not an entity, so this seems to imply we should use @MapKeyColumn.

On the other hand, our key is a persistent field of the map value, and I think the whole point of the @MapKey annotation is to indicate the fact that we simply reuse a property of the map value as the map key, so we do not need to provide an extra table column, as the given property is already mapped to a column.

The way I see it, replacing @MapKey by @MapKeyColumn(name = "language_key") - note the _key suffix! - is also legal, but then we get a different table model and different semantics: The table jpa.multilingual_string_map would have a fourth column language_key, this language_key would not necessarily have to be equal to the language of the map value.

Another open question: Is it legal to write @MapKeyColumn(name = "language")? If so, this should indicate that the language column is to be used as the map key, so this would be equivalent to the @MapKey annotation. On the other hand, you might say that this annotation indicates that the application is free to use map keys that are independent of the map values, so this contract would be violated if the column name indicated by the annotation is already mapped.

The Persistence Providers

I've tried implementing this example with the current versions of Hibernate, Eclipselink, OpenJPA and DataNucleus. I did not succeed with any of them. Only OpenJPA provided a workable solution using @MapKeyColumn, but as I said, I'm not sure if this usage is really intended by the specification.

[Update 20 July 2010: With the corrected model, the updated verdict is: Only OpenJPA passes the test, the other three bail out for various reasons.]

Let's look at the contestants in turn:

Hibernate

Using the mapping defined above, Hibernate 3.5.3-Final complains:

org.hibernate.AnnotationException: Associated class not found: LocalizedString

Apparently Hibernate is expecting the map value to be an entity not an embeddable.

Using @MapKeyColumn(name = "language"), the exception is

org.hibernate.MappingException: Repeated column in mapping for collection: MultilingualString.map column: language

Finally, with @MapKeyColumn(name = "language_key"), Hibernate no longer complains about duplicate columns, but I end up with a redundant table column in my database which I was trying to avoid.

Another problem with Hibernate is different behaviour when working with XML mapping data instead of annotations (which is what I prefer for various reasons, but that's a topic for another post).

Using XML metadata for this example, Hibernate happily ignores the table names from the metadata and simply uses the default names. I filed a bug report in April 2010 (HHH-5136), with no reaction ever since.

Eclipselink

Using Eclipselink 2.1.0, I simply get a rather cryptic exception

java.lang.NullPointerException
 at org.eclipse.persistence.internal.queries.MapContainerPolicy.compareKeys(MapContainerPolicy.java:234)

With @MapKeyColumn=(name = "language"), Eclipselink also complains about a duplicate column, and changing the name to language_key, my test finally passes, at the expense of a redundant column, as with Hibernate.

OpenJPA

With OpenJPA 2.0.0, the message is

org.apache.openjpa.persistence.ArgumentException: Map field "MultilingualString.map" is attempting to use a map table, 
but its key is mapped by another field.  Use an inverse key or join table mapping.

which I can't make sense of. Switching to @MapKeyColumn=(name = "language"), the new message is

org.apache.openjpa.persistence.ArgumentException: 
"LocalizedString.text" declares a column that is not compatible with the expected type "varchar".  

Its seems OpenJPA is confused by the column name text which sounds like a column data type. After adding @Column(name = "_text") to LocalizedString.text, my test case works and my database table only has three columns.

DataNucleus

DataNucleus 2.1.1 complains

javax.persistence.PersistenceException: Persistent class "LocalizedString" has no table in the database, 
but the operation requires it. Please check the specification of the MetaData for this class.

I'm getting the same message with all three variants of the annotation, so it appears that DataNucleus simply cannot handle embeddable map value and expects them to be entities.

Conclusion

Mapping maps with JPA is much harder than you would think, both for the user and for the implementor. Hibernate, Eclipselink and OpenJPA have all passed the JPA TCK. DataNucleus would have liked to do so, but they have not yet been granted access to the TCK.

All four implementors failed this simple map example to various degrees, which implies that there are features in the JPA 2.0 specification which are not sufficiently covered by the TCK.

An Open Source TCK for JPA would help in detecting and eliminating such gaps instead of leaving that to the initiative of individuals.

Hibernate and OSGi: An elaborate solution

A much cleaner and flexible way of osgifying a plain old JAR with lots of dependencies is adding the required OSGi headers to each JAR, so you will replace the megabundle with a bunch of small bundles, one per JAR, which you can reuse in other contexts.

In an ideal OSGi world, every JAR would be an OSGi bundle, so you would have nothing to worry about. Unfortunately, most Java libraries still come as plain old JARs and you have to add the headers on your own.

If you are lucky, someone has done the job for you already. There are a number of third-party repositories offering osgified versions of popular Java libraries, e.g. the OSGi bundle repository or the SpringSource Enterprise Bundle Repository.

SpringSource even has an OSGi bundle of Hibernate itself. However, their version does not work, at any rate not in my setup, so I had to build my own Hibernate bundle.

At least, I was able to use the SpringSource OSGi version for each dependency of Hibernate.

These are the problems I had with the SpringSource version of Hibernate (3.2.6.ga):

• The javax.transaction package does not resolve. It is imported with a specific version range. However, in Java 1.6.0, this package is contained in the JRE and comes from there without a version. I had to drop the version directive to make Hibernate use the javax.transaction package from the JRE.
• I got an exception on running a HQL query, since ANTLR could not load the Hibernate token class.
• I had a very mysterious ClassNotFoundException when accessing some of my model classes. As it turned out, the reason was my usage of lazy loading, where Hibernate injects CGLIB proxy classes into my model classes. The exception was due to the fact that my model bundle did not have access to the CGLIB classes. Rather than declaring a dependency on CGLIB for each of my model bundles, I added the following header to my Hibernate manifest:

Require-Bundle: com.springsource.net.sf.cglib;visibility:=reexport

This means that every bundle with a dependency on Hibernate automatically inherits the dependency on CGLIB. Even St. Peter the Evangelist who usually preaches about using Import-Package instead of Require-Bundle admits this to be one of the rare cases where the latter has its merits.

So far, with this approach, we have not used any Equinox buddy policies, but we still need to deal with the application model classes and resources.

For a while, I thought a fragment bundle would be the definitive solution, which would work not only on Equinox.

I created a bundle com.acme.myapp.hibernate.fragment, containing no Java classes, but only a manifest and some resources, i.e. the Hibernate configuration file and all my Hibernate mapping files. I added all model packages and all relevant JDBC driver packages to the Import-Package header (using resolution:= optional for the JDBC packages.) The host for this fragment is the org.hibernate.osgi bundle, of course.

This worked perfectly when launching my application from within the Eclipse IDE. However, I had an unpleasant surprise when running our batch builds which are based on the Eclipse PDE Ant runner.

PDE kept complaining about cyclic dependencies in my bundles. And not even in my own bundles, also between some of the third-party libraries used by Hibernate, in particular between jaxen and dom4j.

(I had a look at the sources of these two libraries to understand what was going on here: each of them contains some helper classes for the other one, and each must have been compiled against an older version of its friend - really scary...)

I had already spent half a day in working around this problem by repackaging each cycle of libraries in one bundle, and I was going to file an Eclipse bug report. When searching for similar issues, I found bug 208011 which not only describes the problem, but also offers a partial solution:

allowBinaryCycles = true

is a property you can set in your top-level build.properties file as of Eclipse 3.4. (Not a word about this in the Eclipse Online Help, and not even in the comments of the batch build template files!)

According to Chris Aniszczyk, this option will be accessible from the IDE UI in Eclipse 3.5M3.

allowBinaryCycles did suppress the error message regarding the dom4j-jaxen cycle.

But alas, now I had another error complaining about a cycle between Hibernate, my hibernate fragment and my model bundles. Apparently, Eclipse regards the fragment as part of its host (which is ok), and since the fragment depends on an application bundle, and the application bundle depends on Hibernate, so the PDE batch build now complains about a cycle between Hibernate and my application bundle.

So for now, I left all the resources and the JDBC dependencies in the Hibernate fragment, but reverted to using buddy declarations in Hibernate and my model bundles to break the dependency cycle.

I also created a fragment com.acme.myapp.antlr.fragment which imports org.hibernate.hql.ast.HqlToken and thus make the custom token class of Hibernate visible to ANTLR. (See the SpringSource bug report.)

All in all, this looks 95 % clean to me, so I think I'm going to leave it at that for a while...

Hibernate and OSGi: A pragmatic solution

The simplest thing you can do to turn a plain old JAR with a number of external dependencies into an OSGi bundle is wrapping this JAR and all its dependencies in another JAR and adding an OSGi manifest with all dependencies on the bundle classpath.

For example:

Bundle-SymbolicName: org.hibernate.osgi
Bundle-ClassPath: hibernate3.jar,
lib/antlr.jar,
lib/cglib.jar,
lib/commons-logging.jar,
...
Export-Package: org.hibernate,
org.hibernate.configuration,
...

With this megabundle, you can trivially solve all classpath or visibility issues between Hibernate and its dependencies.

You still have to do something to make the model classes of your application and the mapping files accessible to Hibernate.

Eclipse Equinox has an extension of the OSGi standard called buddy policies, which enables you to define some kind of classloader callback.

Your application depends on Hibernate, but you do not want Hibernate to depend on your application. Even if you wrap your own Hibernate bundle, you want to be able to use it in more than one of your applications.

There are several flavours of buddy policies, I will just mention one of them: Adding the header

Eclipse-BuddyPolicy: registered

to your Hibernate bundle manifest will tell Hibernate to ask all its buddies for classes it cannot load using its own classloader.

In each bundle of your application containing some Hibernate model classes, you then add a dependency on Hibernate and a header telling Hibernate that your bundle is a buddy:

Eclipse-RegisterBuddy: org.hibernate.osgi
Require-Bundle: org.hibernate.osgi

(The buddy thing does not work unless there is an actual dependency by Require-Bundle or Import-Package.) This will enable Hibernate to load classes and resources from your bundle, so it will also be able to locate your Hibernate mapping files if you put them into your bundle next to the model classes. Make sure to define your mappings in terms of resources, not files.

You can use the same approach for your JDBC drivers. More likely than not, your JDBC JAR does not come with an OSGi manifest, so you have to wrap it in a bundle anyway. Add a dependency on Hibernate and make your JDBC bundle a buddy of Hibernate. Logically, this is upside-down, but it has the advantage that you can exchange your JDBC drivers without changing the Hibernate bundle.

Alternatively, you can add optional dependencies to the Hibernate manifest on any JDBC driver you are planning to use, e.g.

Require-Bundle: com.microsoft.sqljdbc;resolution:=optional,
org.postgresql;resolution:=optional

Of course, this solution is a nightmare to any OSGi purist, but it does work and it is easy to set up.

There are the following major drawbacks:

• You are bound to Equinox. Other OSGi implementations do not have a buddy policy equivalent. To my best knowledge, you would have to resort to DynamicImport-Package which opens the gates much wider.
• The megabundle approach does not scale. It will be okay if Hibernate is the only component in your system you treat this way. If you have three or four of such heavyweight components you will end up wrapping general purpose JARs like commons-logging.jar over and over again.
• And you will be in real trouble if some of the wrapped third-party dependencies occur in more than one megabundle and are used on the API of your components. With multiple copies of the same JAR in different bundles, and each bundle having its own classloader, you will end up with class cast or assignment exceptions, because classes loaded via different class loaders are always incompatible, even if they have the same fully qualified name.

Hibernate and OSGi: The problem

Making Hibernate work in an OSGi environment is not trivial. Over the last couple of months, I have been experimenting with various approaches, and I'm not yet fully satisfied with the solutions I have found.

The funny thing is, there must be a whole bunch of people facing the same problem, and you do find a couple of postings or example code with the help of your favourite search engine, however, none of the articles or examples seem to match my environment or my requirements.

Now here is an outline of my approaches, which may or may not work for you. Take a look and give me some feedback.

Before talking about the solutions, we all know the answer is 42, but we haven't really worked out the question yet, so let me first list the issues you have to deal with:

• Hibernate has a lot of third-party dependencies.
• When you use an XML configuration file for your session factory, Hibernate must be able to load ist.
• Hibernate needs to load your JDBC driver classes.
  
• When using Hibernate mapping files for your entity classes, Hibernate needs to be able to load them as resources.
• While processing the mapping files, Hibernate reads the names of your entity classes and loads them using Class.forName().
• Hibernate internally uses ANTLR to parse HQL queries, and it passes a custom token class to ANTLR. Unfortunately, ANTLR requires you to specify this class by name and uses Class.forName() load the custom token class.
• When using lazy loading with the CGLIB proxy generator, Hibernate injects a dependency on CGLIB into your model classes. Thus, any bundle using you model classes may have to load CGLIB classes.

There may be more issues, depending on the way you work with Hibernate. In my project environment, some issues mentioned by other people simply do not occur, while others lead to new problems.

• We do not use Hibernate annotations.
• Our OSGi configuration is static. All bundles are installed and started during the framework startup phase, registering all services. We do not have to worry about adding new classes to the session factory when a bundle gets installed at run-time. (For this reason, the extender pattern approach is a bit too heavyweight for my taste.)
• We do not use a web container, or Spring, or anything else, just naked OSGi - well, Eclipse Equinox, in fact.
• Our IDE is Eclipse and we want to be able to launch our OSGi application including Hibernate from Eclipse.
• We also use Eclipse PDE batch builds.
  

Having described to problem in some detail, I will explain a pragmatic and a more elaborate solution in separate articles.