Tip for my future self: Java 7 and Maven on OS X

I am finally upgrading to Java 7 on my Mac box and I've been hit by MNG-4226. Maven was still using the Java 6 installation, as described in this Stack Overflow post.

Solution, create file /etc/mavenrc with JAVA_HOME=`/usr/libexec/java_home -v 1.7`.

Google Cloud Storage support in Derquinse Bocas

Starting with the just released 1.1.0 version of Derquinse Bocas, a new experimental back-end based on Google Cloud Storage is included. In this first version, the XML API 2.0 is used and only service account authentication is supported.

Besides, version 1.1.0 also includes a revision of the API, incorporating bucket support (in order to group entries) and simplifying entry creation methods. This will be described in a future post.

Announcing Conquiris

After a few months from the first release,  I'd like to announce the succesor to lucis: Conquiris. Conquiris is a set of libraries to help developing applications based on Apache Lucene. It is not a search server, as Solr fills that role really well. It is based on Java SE 6 and Lucene 3.6.1

Why the delay? Well, I was waiting to have some documentation in place to make the announcement, but it was taking so long that I have just gone for it.

So, what are the available resources to get in touch with Conquiris?:

• The project Home Page, where the documentation will be hosted.
• The Google Code Project Page, where you can get the source code and post issues.
• The Conquiris Google+ Page, for announcements.
• The API documentation, hosted on GitHub Pages.
• The Google Group for general discussion.

As with other projects, continuous integration is kindly provided by CloudBees DEV@cloud. The Jenkins instance is at https://derquinse.ci.cloudbees.com.

"Big Memory" support and new Resources for Derquinse Bocas

Java brought automatic memory management to the mainstream. However, managing the heap is a non-trival task that can require a great amount of effort from the garbage collector. This effort grows with the heap size and may lead to long collection pauses that hurt your system's throughput and response times.

In Bocas, when using a memory-backed repository or a cache, most of the memory consumption comes from the stored values, which are opaque byte streams. If we could get those values out of the Java heap we could use that precious managed space for more value-adding objects.

Enter "big memory" support: starting with version 1.0.4, Bocas supports the use of direct buffers as the backing storage of both memory-backed repositories and caches, using off-heap memory to store the repository values. As the memory-backed repositories are used mainly for testing purposes, this feature is most useful in caches.

This support allows for better utilization of the Java heap, reducing the load for the Java garbage collector. However, take your off-heap caches into account when sizing the heap and planning the memory distribution of your machines.

I would also like to announce new resources for the project:

• A new Google+ Page, used for announcements.
• More documentation is being added to the Project Web Site.
• The General Discussion Google Group.

As a side note, version 1.0.5 has just been released and is already in Maven Central.

New Resources for Derquinse Commons

One of the goals of open sourcing some project is getting other people to use it. However, code alone is  not enough for that, and some other resources have to be provided so that other people start looking at your code. So, I'm going to try to devote a little more time to providing documentation for the existing codebase than including new features. So for the Derquinse Commons Project, we have:

• A new Project Web Site, still work in progress, that will host the project (non-javadoc) documentation.
• Project announcements will be done through the new Google+ Page for the project.
• A new General Discussion Google group.

As before, the API documentation will continue to be hosted at GitHub Pages and the source code on Google Code.

Announcing Derquinse Bocas

I'd like announce a little project I've been working on lately. It called Bocas and it is a Java and REST API to access key-value repositories with the content-addressed storage constraint, that is, the key must be a hash of the value. In the current implementation, SHA-256 is used as the hash function.

A few releases have been already deployed in Maven Central, and documentation is being built at the project web site. As this site is still short of content, you may find the API documentation useful, above all the main API interface.

The project is hosted on Google Code and has the Apache 2 license.

Stay tuned for more news about this project.

New home for Derquinse Commons API documentation

One of the main issues in Derquinse Commons is the lack of documentation,  including the lack of a stable home for the Javadoc-generated documentation.

From now on, and thanks to the wonderful GitHub pages service, you can find the API docs at http://apis.derquinse.net/derquinse-common.

Non-API docs are still missing, and I hope to be able to write some soon.

Derquinse Commons now in Maven Central

One if the best features in Maven is dependency management. Even with its known problems it is a great step forward and many other build systems reuse Maven's repository structure.

One of the essential parts of this vision is the Maven Central Repository, your one stop shop for most of the dependencies in your projects.

Publishing to Maven Central has always had its requirement set (you can find an updated list here), one of them being that the repository must be self-contained. That is, no artifact may depend on another artifact that is not part of the repository.

Back in 2008, when I started both derquinsej and lucis, this requirement was fulfilled with no problem and synchronization with Central was provided thanks to Sourceforge's SSH access.

However, as time passed, some projects such as Hibernate starting publishing its artifacts in external repositories and many other interesting projects were hosted in repositories different of Central, such as Java.net. So, for future projects Central Publishing was not an option and I prepared an external repository for my own projects.

Fortunately, the situation has improved a lot:

• Now many interesting external repositories, such as JBoss and Java.net, are synchronized back in Central.
• Central navigation and search has been enhanced.
• Sonatype provides a repository for open source projects with the option of being published in Central. The process is very well documented and the response is fast and great (kudos to Juven Xu). 

So, from now on, you can find Derquinse Commons in Maven Central.
  

Follow up to Maven Release Plugin version problems

In a previous post, I warned about some bugs that caused the Maven Release Plugin to generate wrong tag versions. While those bugs are still opened today, the real culprit was MRELEASE-697 which is fixed in version 2.2.1, which can be used safely.

A Lucis update

After a too long period of inactivity at last we have a significant update to Lucis. The migration to Lucene 3.x has been put on hold to work on some more important changes (and I don't know if it will happen before the move to Lucis' successor project). So, to avoid confusion, version numbering has changed so that the first two numbers match those of the Lucene version used.

I have just released version 2.4.31 (should be in central soon), the main changes being:

• Reindexing indexes can now hold a checkpoint.
• Reindexing indexes can know decide to skip a run.
• Cancellation policy for index services has been defined, using interruptions, providing graceful shutdowns. As always for all your java concurrency-related questions, go to the source.
• It is know possible to use external executors.
• Many more tests.

Also, as of this version, the javadoc is no longer uploaded to the repository, as it can be built from source, but the relevant parts are online for your reading convenience.

UPDATE: Already in Central.

Stick to version 2.0 of Maven Release Plugin

As reflected on bugs MRELEASE-694 (related to MRELEASE-689 and MRELEASE-691), there are some problems generating the release version numbers (like maintaining SNAPSHOT). This may bite without notice if use non-interactive mode. These bugs are still open as of version 2.2.1.

It's a nice coincidence that because of another bug, the Maven Eclipse Plugin warns you if you use a Maven Release Plugin version other than 2.0 :)

Revisiting Lucis

The lucis project is a key component of an increasing number of web sites. However, it is starting to show its age. Rushed into production in the summer of 2008 it included some "incorrect" decisions that were to be mended in its successor project, which was also going to include all the important updates. So lucis algo received small evolutions.

However, as the first release of conquiris is delayed again and again, and the number fo projects using lucis increases far more than expected, it is clear that a new impulse was needed.

So today, I've released version 0.0.31 (which should be in Maven Central in a few hours). The only (but fundamental) change is the upgrade from Lucene 2.4.1 to 3.1.0. As the new Lucene version already breaks backwards compatibility, I've gone the whole way and removed use of (now) deprecated parts of Lucene. So this version of lucis is totally incompatible with the previous stable one (0.0.29).

I hope to post about the changes real soon now.

As I said, the "only" change is the Lucene version, so this release won't be integrated in Port@l and GeNews until more improvements are added.

Port@l 3.0 en la LSWC

Sé que voy con un poco de retraso, pero más vale tarde que nunca. Sólo comentar que el pasado 27 de octubre participé en mesa redonda "Soluciones Web Avanzadas" dentro de la Libre Software World Conference, con una breve presentación sobre Port@l 3.0. Podéis encontrar la presentación aquí embebida y en el siguiente enlace.

Port@l 3.0 en la LSWC

View more presentations from andresrc.

Permgen problems and Running Eclipse on Java 1.6 update 21

See this post. Reposted here for future reference.

New Home...

... for this blog (sort of) and my common Java libraries.

Some months ago I bought the domain derquinse.net and started a new incarnation of my common Java libraries, now called derquinse-commons, hosted on Google Code, and based on Google's guava-libraries.

But is hasn't been until recently that I have configured this blog to use the new domain, and performed the first releases (in plural, as the different subprojects are now released independently), which may be found in the Maven Repository (that, as of now, will not be synchronized with Central as there are dependencies that are not part of it). Right now, I've released:

• derquinse-commons-base 1.0.0: Base libraries.
• derquinse-commons-orm 1.0.1: Base object/relational libraries.
• derquinse-commons-orm-hib3 1.0.1: Hibernate 3.5 implementation of the ORM module.

Besides some common POMs. The collections module will be next, with Hierarchy and friends.

Stay tuned.

Lucene 2.9.0

A few weeks ago Lucene 2.9.0 was released. The list of changes is impressive, and it's certainly an interim release paving the way to release 3.0, where I suppose many of the now deprecated methods will start to disappear.

One of the most interesting changes for me is LUCENE-1382, which will greatly simplify checkpoint management in lucis.

Looking forward to 3.0.

Counting and Grouping Queries in Lucene

When using a Lucene index to look up some information you have access to some querying facilities not found in other kind of repositories. However, in a classical trade-off, you lose some features such as the aggregate queries easily performed in relational databases.

Anyway, if you need to perform this kind of operations, they may be easily implemented using hit collectors. So, I've included in lucis two simple operations, counting and grouping results:

The LucisQuery object is used to decouple index control policy (when to open and close it, etc) from the queries themselves.

The counting query just needs the Lucene query to perform and the (optional) filter to apply. The result holds the number of documents found and the time the query needed.

For the grouping query you must provide the list of field names you want to group by (in order) and the query result is the same that the counting query plus the root group (the one corresponding to the first field name), where a group is something like (partial API showed, see the source):

So, for each collected value of the provided field you get a child group which itself contains the groups representing the nested fields. The number of hits in a group may not be equal to the sum of the hits in the children groups if any of the fields is multivalued.

Pointcut expressions for stereotypes

Some of the new additions to Spring in version 2.5 are annotation-driven configuration, classpath scanning for managed components and @AspectJ support for AOP. These features can gretly reduce the amount of XML needed to configure your application context.

Together with classpath scanning came stereotypes, which, as the reference documentation states, make ideal targets for pointcuts. In putting this idea (stereotype as pointcut target) you may find two different candidate expressions in the documentation (the Service stereotype is used as an example): @target(org.springframework.stereotype.Service) and @within(org.springframework.stereotype.Service) with no clear differentiation between them.

However, some of the differences started to show up as Spring auto-proxying mechanism started to try creating proxies for classes that had no stereotype and were not advised in any other way. After a non-immediate web-searching session, the real problem stood up: if you use @target every class is proxied just in case a new subclass with the annotation is introduced. Even though the issue is marked as resolved, I have suffered it in 2.5.6SEC01.

So, in the meantime, just use @within.

'This' type and Covariant builders

When designing your own APIs, either by using interfaces or class hierarchies, you usually find the need to reference the type of the actual implementacion of the interface or class you are writing. This is an actual problem, and there's even a proposal in Project Coin (small language changes for Java 7) to deal with it.

One of this cases, and the one the rest of this post is focused on, is using the 'This' type as the return type of some methods. This is particularly useful for method chaining in fluent interfaces.

Up until Java 5, and the introduction of covariant return types, it was impossible to deal with this issue, which led to frequent casting. But even with Java 5+, the solution is based in manually overriding every method in the subclass (or subinterface) changing the return type to the desired subtype, calling super and returning this.

So, I have checked-in a partial solution to this issue. The type signature of the Type class This<T extends This<T>> (equivalent to that of the Enum class) allows the implementation type to be referenced inside the class, and its value obtained by calling thisValue. But not only the final implementation type, but also some siblings in the hierarchy (as showed in the tests), as the Java type system cannot guarantee this.

That's the reason for the validation in the constructor of This, the reason it is an abstract class instead of an interface, and one of the many reasons it is only a partial soluction.

One application of the This type is for builders (see the builder pattern and item 2 in Effective Java 2nd Ed.). If your builder is part of a hierarchy you are faced with the problem described in this post, so I've also included a base class for them, the covariant builder.

Spring and REST

Here's a post on REST Support in Spring 3.0 by Arjen Poustma. I have mixed feelings about this. On the one side I'd prefer better integration with JAX-RS in general and Jersey in particular. On the other side, I appreciate choice and the closer integration.