Jarc: Javadoc generation

During the preparation of the libreload-java package, I discovered that the javadoc tool - the JDK tool used to generate an HTML documentation of a Java API - does not generate the javadoc based on Java dependency. It can either generate the javadoc for whole packages, or for a subset of Java files (e.g. File*.java). But we cannot ask it to generate the documentation for one specific class and all the classes depending on this class.

I guess that my way of doing things is different: I do not have one source tree per project, but one source tree that contains the code for all of them (and much, much more code that is not released). A particular Debian package contains only a subset of the code available in my source tree - the code needed for this particular package. One way to see this is that a package is a view on my source repository. Ant or javadoc cannot easily extract only the sources files needed for a particular view (unless I list all the files individually, which would be unmaintainable), so the jarc tool relies on Java dependency.

When I tried to build the javadoc for the libreload-java package, I noticed that the test units files (used by Junit) were part of the javadoc, although they clearly are not part of the API. This is because the javadoc tool had to process the whole directory. I cannot move these two files in another directory, because they need to have access to constructors and methods that are not public.

So what I did is that I integrated the javadoc tool inside jarc. Because jarc knows the list of dependent Java files during a jar file build, I just have to feed it to the javadoc tool to have the exact API documentation corresponding to the jar file built. The new version of jarc (0.2.22) supports a new parameter (-javadoc ) that can is used to request the generation of the javadoc at the same time the package is build and unit tested.

One can notice that the Debian source packages that are distributed do not contain the whole tree, but only the subset of the files needed to rebuild this package. To do this (I obviously did not want to release my whole source tree), I had to rely on a trick. My hard disk is mounted with the strictatime options, which means that each time a file is accessed, the data/time of this access is kept. When I build a Debian package I start to build it from my whole source repository:

$ touch run-stamp
$ SRC=../src dpkg-buildpackage -D -b -us -uc

Then I run this command to copy only the files that where used for this build into a new directory:

$ find ../src -anewer run-stamp -type f -exec cp --parents -t src {} \;

I can then build again the packages, but this time with a source package that contains only these files (and that can be used to eventually rebuild the binary packages):

$ SRC=src dpkg-buildpackage -I

Note the difference in the $SRC variable content: ../src is my complete source repository and src is just the subset for this package.

RELOAD: Access control policy script tester

There was some interest at the last IETF meeting in my presentation of Access Control Policy script, so I spent some time working in improving this. The first step was to release a new version of the I-D. To encourage adoption of this draft, I also released a library permitting to develop and test new access control scripts without having a complete implementation of RELOAD. This was released as a Debian package named libreload-java in my Debian/Ubuntu repository. This package is also a reference implementation for the I-D but, as it contains a subset of the RELOAD library I am current developing, it can also be considered as an early release of this library. As always, this is released under an Affero GPL 3 license, and the code source is available.

To explain how to develop a new access control script, I will now go through the example which is also in the libreload-java package:

import static org.implementers.net.reload.AccessControlPolicyTester.kind;
import static org.implementers.net.reload.DataModel.Standard.*;

These two lines permit to simplify the code by importing the kind() static method and the available Data Models.

private static final long KIND_ID = 4026531841l;

Here we declare a constant defining a new Kind-Id in the private range.

AccessControlPolicyTester tester = new AccessControlPolicyTester("myusername",
  kind(KIND_ID, SINGLE,
    "String.prototype['bytes'] = function() {\n" +
    "  var bytes = [];\n" +
    "    for (var i = 0; i < this.length; i++) {\n" +
    "      bytes[i] = this.charCodeAt(i);\n" +
    "    }\n" +
    "    return bytes;\n" +
    "};\n" +
    "return resource.equalsHash(signature.user_name.bytes());\n"));

This code creates a new instance of the tester by passing a user name (that will be used to generate an X509 certificate) and a new kind. The new kind uses the Kind-ID declared previously, uses the SINGLE Data Model and will use an Access Control Policy as described in the ECMAScript code. Note that more calls to the kind() method can be added to the constructor of the tester.

Map<Long, DataModel<?>> kinds = new HashMap<Long, DataModel<?>>();

This code declares a variable that will contains the data items meant to be stored. The key of the map is a long that contains a Kind-Id. The value is either an instance of the DataModel.Single class, the DataModel.Array class or the DataModel.Dictionnary if the Data Model is respectively SINGLE, ARRAY or DICTIONARY. In our case, we use the SINGLE Data Model, meaning that only one item can be stored at a specific Resource-ID for this specific Kind-ID.

kinds.put(KIND_ID, new DataModel.Single(KIND_ID, 0l, new Value.NotExists(System.currentTimeMillis(), 60, tester.signer(), tester.privateKey())));

This piece of code does multiple things, so let's decompose:

First it retrieves the signer (an Identity instance that consist of a X509 Certificate, a user name and a Node-ID) and the private key that was used for the certificate of the signer. These two objects are automatically created by the constructor of the tester object.

Then it creates a new Value instance with a store_time equal to the current time, a lifetime of one minute, an exists value of FALSE and the identity and private key just retrieved. This is the object that we want to store (it is more or less equivalent to StoredData).

Then a new instance of DataModel.Single is created with our new Kind-ID, a generation_counter of 0 and the value to store (it is more or less equivalent to StoreKindData).

Finally this object is stored in the kinds map, with the same Kind-ID as key.

tester.store(tester.signer().userName().getBytes(UTF_8), 0, kinds)

In this code, we finally try to store the items that we just created. The first parameter is the Resource-Name, which in our case is the user name (converted to a byte array). The second parameter is the replica number, 0 for the original replica. The third parameter is the set of items. The method will return false if one items fails the access control policy verification, and true if all the items pass the verification.

The DataModel.Array and DataModel.Dictionary works in the same way, excepted that DataModel.Array is a sparse array implementing java.util.List and DataModel.Dictionary is implementing java.util.Map.

The program can be compiled and executed with the following commands:

$ javac -classpath /usr/lib/reload/reload.jar Main.java
$ java -classpath usr/lib/reload/reload.jar:. Main


In addition to the jar file and the example, the package contains the Javadoc for all the classes and methods accessible to the tester, and a copy of the I-D.

Update 2011/05/14:

The command lines above do not work. I uploaded a new package (0.1.1) so the following command lines should work now:

$ javac -classpath /usr/lib/reload/reload.jar AcpTest.java
$ java -classpath usr/lib/reload/reload.jar:. AcpTest


Hardcover warning sticker

Yesterday I received a brand new book that I plan to use as reference for my work for a long time. This is a 1000+ pages hardcover, and after unwrapping it from its plastic protection, I immediately open it. At the exact time I let the two sides of the book fall on the table, I new that I will break the spine, but it was too late. Here's what the book looks like now:



Now I knew the right way to open an hardcover book the first time, but as I buy perhaps one or two each year, it's difficult to remember it in the excitation of having new stuff to play with.

What is done is done, and it will be a good exercise to rebuild the book one day (I am slowly investing in bookbinding tools), but seen the cost of of this kind of books, here's a suggestion for publisher: The book was wrapped in plastic, so why not inserting inside an explanation similar to the link above or at least putting a warning sticker on the wrapping? Or perhaps Amazon can just just add this kind of sticker automatically.

Jarc: Now running Java 1.7

At last, installable packages for OpenJDK 7 are available in the experimental Debian repository. So I released a new version of jarc that is now directly supporting Java 7 code but, because the openjdk-7 packages are in an experimental repository, I uploaded jarc 0.2.21 in an experimental repository so people would not be hit with dependency issues. I updated my Ubuntu/Debian repository page to explain how to add my experimental repository to the repository configuration file.

The reason to release this experimental version of jarc is that I need two the new features of the new JVM for my RELOAD implementation. The first one is the support of TLS 1.2, aka RFC 5246, which is used by RELOAD to authenticate and encrypt all the TCP connections. The only thing missing to completely implement RELOAD will be an implementation of DTLS 1.0 (which is to datagrams what TLS is to streams). I guess I will have to bite the bullet and write this one myself.

The second feature that I needed is an implementation of the SCTP protocol. TCP is great for transferring files but it has one major flaw when used with multiplexed client/server transactions, which is called the Head-Of-Line problem - a message lost will block all the subsequent messages, even if they belong to a different transaction, until the lost message is retransmitted and received. UDP does not have this issue but comes with its own set of problems. SCTP is somewhere between UDP and TCP, and this is why, in my opinion, it is a good transport protocol for something like RELOAD (well, between RELOAD nodes with a public IP address, as most NATs - one exception been the one I supervised the development at 8x8 - do not NAT SCTP).

So the plan will be to add SCTP to RELOAD, using either TLS (RFC 3436) or DTLS (RFC 6083) depending on the DTLS implementation described above, and to write an I-D, perhaps in time for IETF 81 in Quebec City.