Into The Labyrinth – using the JavaCompiler API from Jython

The Plumber

After having neglected Java for years I began to re-examine it this month together with Jython and my initial reaction was a culture shock.

Java is infamous for being a “plumbing language” i.e. you have to subclass some classes or implement a few interfaces and then plug them into a framework. Alternatively you have to call framework methods that expect a bunch of interrelated objects as parameters. None of those class implementations is particularly complicated but you have to figure out how all the objects are related to each other and essentially deal with configurations and dependencies on object level. It is easy to mess things up with every additional point of failure. There is also a tendency for abstraction inversion: you have to undertake many concrete steps within a complex machinery to create a simple building block. Abstraction inversion is an indication of a system being overdesigned.

The topic of this article is Javas compiler API and it is a nice show case to highlight some differences which are not really situated on language level but in the way problems are approached. Take Pythons compile function for example. Pythons compile function has the following signature

compile: (code_str, file_name, eval_mode) -> code_object

and is dead simple to use:

>>> compile("import EasyExtend.langlets", "<input>", "exec")
<code object <module> at 00EB0578, file "<input>", line 1>

If you need to read the source from a file you do just this

>>> compile(open("module.py").read(), "input", "exec")
<code object <module> at 00EB0578, file "<input>", line 1>

If you want to store the resulting bytecode in an appropriate file you need to do a little more work

import os, struct, marshal, imp
 
def write_code(filename, code):
    f = open(filename, "wb")
    try:
        mtime = os.path.getmtime(filename)
        mtime = struct.pack('<i', mtime)
        MAGIC = imp.get_magic()
        f.write(MAGIC + mtime)
        marshal.dump(code, f)
    finally:
        f.close()

This creates a Python bytecode file ( usually a ‘ pyc’ file ) and serializes the code object. Something like this could be implemented on the method level of the code object itself and a single call code.write(filename) would suffice to store the code.

How to compile Java dynamically?

Java is different. The Java Compiler API is specified in JSR 199. It contains 20 classes/interfaces and you have to figure out their interplay in order to compile a source string. Half of them are JavaFile or JavaFileManager classes/interfaces so what’s actually peripheral has moved to the center. The JavaCompiler API is new to Java 6 and it seems dynamic compilation didn’t work out of the box prior to release 6. I suppose one had to call javac from the command line or use a different compiler such as Janino.

There isn’t any easy way to get into the compiler API. The JSR 199 isn’t exactly a design document but a terse javadoc API documentation. Tutorials are rare and mostly superficial in that they cover the most simple use case only. A notable exception is an introduction written by Andrew Davison. It covers the most relevant use cases and more. Alternatively one might skim through the tests of the JDK 6. It contains expression evaluation code written by the JSR 199 implementor Peter von der Ahé. Once again tests are documentation. The Jython 2.5 code I’ll present follows Davisons article and is mostly a transcript of his Java code in the relevant parts.

The JavaCompiler API from Jython

Prerequisites: a JDK for Java 6 has to be installed first. The Java compiler tools are implemented in

<JDK-Path>/lib/tools.jar

and that path has to be added to the Java class path when Jython is invoked. Alternatively you can add the path to thePYTHONPATH inside of your application. I’ll chose the latter approach here:

import sys
sys.path.append(os.path.join(os.environ["JAVA_HOME"],"lib", "tools.jar"))

The environment variable JAVA_HOME has to be set to your JDK path – not to the JRE path. This might have to be changed. In case of my Windows notebook the path is

JAVA_HOME = C:\Programme\Java\jdk1.6.0_13.

Another configuration aspect concerns access to protected members of Java classes. By default this is disabled in Jython. We will need to access protected member functions once we override a Java class loader. For that reason one has to set following disrespectful flag in Jythons registry file

python.security.respectJavaAccessibility = false

For more information see the Jython FAQ.

Now we are ready to import the compiler tools:

from javax.tools import*
from com.sun.tools.javac.api import JavacTool
 
compiler = JavacTool.create()
assert compiler

The CompilationTask

After having created a compiler instance we now care about the CompilationTask which is defined in JSR 199. It is basically a callable, an interface which specifies a parameter-less call() method that returns a value of type Boolean. It is this callmethod that has to be overridden in subclasses and called for compilation. In case of the JavaCompiler framework one doesn’t have to override the CompilationTaskexplicitly but fetches it from the compiler object using the the getTask method:

JavaCompiler.CompilationTask getTask(Writer out,
                             JavaFileManager fileManager,
                             DiagnosticListener&lt;? super JavaFileObject&gt; diagnosticListener,
                             Iterable&lt;String&gt; options,
                             Iterable&lt;String&gt; classes,
                             Iterable&lt;? extends JavaFileObject&gt; compilationUnits)

The parameters need further examination but once we have understood how to create the objects required to fetch the CompilationTask the compilation is performed by

compiler.getTask(...).call()

The meaning of the parameters of the CompilationTask is as follows:

Parameters:

  • out – a Writer for additional output from the compiler; use System.err if null
  • fileManager – a file manager; if null use the compiler’s standard filemanager
  • diagnosticListener – a diagnostic listener; if null use the compiler’s default method for reporting diagnostics
  • options – compiler options, null means no options
  • classes – class names (for annotation processing), null means no class names
  • compilationUnits – the compilation units to compile, null means no compilation units

For now we will ignore the out parameter as well as compiler options and class names passed to the annotation processors.

The compilationUnits holds the source code to be compiled. In case of Jython it will be a Python list containing a single JavaFileObject.

So a call of getTask will have the following shape

compiler.getTask(None,
                 fileManager,
                 diagnosticListener,
                 None,
                 None,
                 [source]).call()

Diagnostics

DiagnosticCollectors are used for error reporting. A DiagnosticCollector implements a DiagnosticListener interface. It is parametrized by some type S and holds a possibly empty list of Diagnostic&lt;S&gt; objects which can be fetched.

We only need to know as much about diagnostics to handle failure cases:

 if not task.call():
     msg = "\n  "+"\n  ".join(str(d) for d in diagnostics.getDiagnostics())
     raise JavaCompilationError(msg)

The JavaCompilationError is a custom Jython exception. The JavaCompiler API doesn’t raise an exception on compilation failure.

JavaFileObject

The JavaFileObject specifies a file abstraction. For our own purposes we need two of them: one that is readable and holds the source code ( a string ) and one that is writable and holds the bytecode ( an array of bytes ). Both derive from SimpleJavaFileObject which provides an implementation of the JavaFileObject interface.

from java.net import URI
from java.io import*
 
class StringJFO(SimpleJavaFileObject):
    '''
    JavaFileObject implemention used to hold the source code.
    '''
    def __init__(self, className, codestr):
        self.codestr = codestr
        super(StringJFO, self).__init__(URI(className),
                                        JavaFileObject.Kind.SOURCE)
 
    def getCharContent(self, errs):
        return self.codestr
 
class ByteArrayJFO(SimpleJavaFileObject):
    '''
    JavaFileObject implementation used to hold the byte code.
    '''
    def __init__(self, className, kind):
        super(ByteArrayJFO, self).__init__(URI(className), kind)
        self.baos = ByteArrayOutputStream()
 
    def openInputStream(self):
        return ByteArrayInputStream(self.getByteArray())
 
    def openOutputStream(self):
        return self.baos
 
    def getByteArray(self):
        return self.baos.toByteArray()

In case of the ByteArrayJFO a writable ByteArrayOutputStream is created that can be fetched by the framework using the openInputStream method.

An instance of StringJFO will become our compilationUnit and an instance of ByteArrayJFO will be returned by a still to be defined FileManager:

class ByteJavaFileManager(ForwardingJavaFileManager):
    def __init__(self, fileManager):
        super(ByteJavaFileManager, self).__init__(fileManager)
        self.code = None
 
    def getJavaFileForOutput(self, location, className, kind, sibling):
        self.code = ByteArrayJFO(className, kind)
        return self.code

A Java compiler in Jython

Now we can put all those things together and define a compileClass function.

def compileClass(className, codeStr, *flags):
    compiler = JavacTool.create()
    assert compiler, "Compiler not found"
    diagnostics = DiagnosticCollector()
    jfm = ByteJavaFileManager(compiler.getStandardFileManager(diagnostics,
                                                              None,
                                                              None))
    task = compiler.getTask(None,
                            jfm,
                            diagnostics,
                            flags,
                            None,
                            [StringJFO(className+".java", codeStr)])
    if not task.call():
        e = "\n  "+"\n  ".join(str(d) for d in diagnostics.getDiagnostics())
        raise JavaCompilationError(e)
    return jfm.code

In order to initialize the StringJFO object we have to pass a file name which corresponds to the the class name we use. The return values of compileClass is an object of type ByteArrayJFO that holds the byte code.

Adding a ClassLoader

We are almost done. To complete our exercise we have to make the byte code executable which means we have to define a class loader and create an actual Java class. Since we operate from within Jython it will be immediately wrapped into a Python type.

from java.lang import ClassLoader
from java.lang import ClassNotFoundException
 
class ByteClassLoader(ClassLoader):
    def __init__(self, code):
        super(ByteClassLoader, self).__init__(ClassLoader.getClassLoader())
        self.code = code
 
    def findClass(self, className):
        code = self.code.getByteArray()
        cl = self.defineClass(className, code, 0, len(code))
        if cl is None:
            raise ClassNotFoundException(className)
        else:
            return cl
 
def createJavaClass(className, codeStr, *compilerflags):
    loader = ByteClassLoader(compileClass(className, codeStr, *compilerflags))
    return loader.loadClass(className)

To a very good end we have hidden the JavaCompiler API behind a single function with a tiny interface. There is not something Python has been needed for actually. The JavaCompiler API is a Swiss Army knife style API that requires a whole object tree to be created to achieve a simple effect.

Java classes from within Jython

Finally we want to demonstrate the value of our implementation showing a few simple examples. The jcompile.py module contains the complete code and can be downloaded here.

The first example is our “Hello Jython”:

codeStr = """
public class Foo {
    public static void main(String args[])
    {
        System.out.println("Hello, "+args[0]);
    }
}
"""
 
Foo = createJavaClass("Foo", codeStr)
print Foo  # &lt;type 'Foo'&gt;
 
Foo.main(["Jython!"])  #  Hello, Jython!

In our next example we import some symbols from a Java library:

codeStr = '''
import java.lang.Math;
public class Prime {
    public static Boolean isPrime(double n)
    {
        for(int d=2;d&lt;=Math.sqrt(n);d++)
        {
            if(n%d == 0)
                return false;
        }
        return true;
    }
}
'''
 
Prime = createJavaClass("Prime", codeStr)
Prime.isPrime(2)     # True
Prime.isPrime(9971)  # False
Prime.isPrime(9973)  # True

————— Update ! We. 2009-24-06 —————–

Right now it is not possible to derive a class from a dynamically compiled Java class. If Foo is a dynamically generated Java class and Baris defined as

class Bar(Foo):
    pass

then a ClassNotFoundException is raised. The only possible workaround I see right now is to store the class to the disk and import it right after this. A possible adaption of the createJavaClass function looks like:

def createJavaClass(className,
                    codeStr,
                    todisk = True,
                    remove = True,
                    *compilerflags):
    '''
    Compiles and loads a new Java class.
    '''
    compiled = compileClass(className, codeStr, *compilerflags)
    if todisk:
        code = compiled.getByteArray()
        clsfile = open(className+".class", "wb")
        try:
            code.tofile(clsfile)
        finally:
            clsfile.close()
        jclass = __import__(className)
        if remove:
            os.remove(className+".class")
        return jclass
    else:
        loader = ByteClassLoader(compileClass(className, codeStr, *compilerflags))
        return loader.loadClass(className)
  1. […] back into Jython. One key element is Java code generation and dynamic compilation using the Java 6 Compilation API. Another one is interface extraction of Jython classes using the rich reflection API provided by […]

Leave a Reply