Stitches of a flea language – defining Java annotations in Jython

Jython annotations – anyone?

The last few days I tried to figure out how to create Jython annotations. A Jython annotation is defined here as a Java annotation lifted from Jython to Java. So one essentially defines a Java annotation in a Jython class. A Jython annotation shall not be confused with a decorator. A Python ( or Jython ) decorator is just a higher order function ( or callable ). It might create attributes in Jython objects but those are not the same as Java annotations reflected by the Java runtime. Without Jython annotations Jython is right now essentially stuck in a pre version 1.5 Javaverse and Jython classes are disconnected from modern Java frameworks and cannot be plugged.

Jython annotations in Jython 2.5 don’t work out of the box. It is not much known yet about how or when Jython annotations will be supported by core Jython. The lead Jython developer Frank Wierzbicki announced something along the lines in his 2008 PyCon conference talk but this is now about 16 months ago. I could temper my impatience if Jython annotations were just around the corner but what can we expect after those 16 months?

In this article I introduce a library that enables lifting of meta-data from Jython to Java and loading Java back into Jython. One key element is Java code generation and dynamic compilation using the Java 6 Compilation API. Another one is interface extraction of Jython classes using the rich reflection API provided by core Jython.

Lifting up Jython classes

For every Jython class JyClass one can generate a Java class JaFromJyClass by means of interface extraction. We assume JyClass to be a subclass of a Java class, e.g. Object, and translate the Jython class

class JyClass(Object):
    def foo(self, *args):
        print "foo", args

into a corresponding Java class

public class JaFromJyClass extends Object{
    PyObject jyobject;
 
    public PyObject foo(PyObject[] args)
    {
        return jyobject.invoke("foo", args);
    }
}

This class is basically a proxy for the jyobject member variable of type PyObject which is a Jython API type. Once we have generated the Java code from Jython we can dynamically compile and load the Java code into Jython:

JaFromJyClass = createJavaClass("JaFromJyClass", source)
jainstance = JaFromJyClass()
jainstance.jyobject = JyClass()
jainstance.foo(9)  # prints 'foo 9'

This was straightforward and hints on our translation strategy. Next we review the Jython to Java translations in more detail.

Jython to Java translations

PyObject Injections

We cannot be glad with the way the jyobject was assigned to the jainstance in the previous example. The particular assignment protocol implies that the Jython script has always control over the instantiation of Jython classes. But once we plug the class into a framework the framework takes over. A better solution is to inject the PyObject using a factory mechanism.

public JaFromJyClass() {
    super();
    jyobject = JyGateway.newInstance("JyClass", this, null);
    jaobject = (Object)jyobject.__tojava__(Object.class);
}

The factory is called JyGateway. The JyGateway is a Java class which defines HashMap called registry

public static HashMap<String, PyDictionary> registry = new HashMap<String, PyDictionary>();

The keys of the Java HashMap are Strings that represent class names. The PyDictionary is a dictionary of Jython functions. Right now two functions are defined: newInstance and callStatic. Both of them correspond to static methods of JyGateway. If JyGateway.newInstance(“JyClass”, this, null) is called the newInstance Jython function is fetched from the registry using “JyClass” as a key. The third argument of JyGateway.newInstance contains an array of Java objects passed as arguments to the newInstance function which returns a new PyObject. If the constructor doesn’t take an argument null is passed as in the example. The particular JyClass will never be exposed to Java code.

Overrdiding superclass methods

Aside from jyobject we have also defined jaobject in the JaFromJyClass constructor which has the type Object. Here Object is just the superclass of both JaFromJyClass and JyClass. The jaobjectis defined for the purpose of overriding superclass methods: we cannot simply change the signature of superclass methods in particular not the return value.

If public void foo(int x) is a method defined in the superclass of JyClass, the Java method generated from JyClass is

public void foo(int arg) { jaobject.foo(arg) }

The method foo called from jaobject is still the method implemented in Jython. The Jython method is just called with Java arguments and returns a Java value ( if any ) that gets converted back to Jython.

Calling static methods

Calling static or classmethods of Jython objects from Java is similar to calling JyGateway.newInstance:

public static PyObject bar(PyObject args[])
{
     return JyGateway.callStatic("JyClass", "bar", args);
}

Defining Jython metadata

There are three kinds of meta-data which can be added to Jython classes which are extracted for dynamic Java generation. Those are called jproperty, annotation and signature. They serve different purposes.

signature

A signature decorator is defined to assign Java argument and the return types to a Jython method. Without the signature decorator a default translation is applied:

def foo(self, arg):
    ...

—–>

public PyObject foo(PyObject[] args) {
    return jyobject.invoke("foo", args);
}

If we decorate foo with the following signature decorator

@signature("public int _(char)")
def foo(self, arg):
    ...

we get the translation

public int foo(char arg0){
    PyObject args[] = new PyObject[1];
    for(int i=0;i<1;i++) {
        args[0] = Py.java2py(arg0);
    }
    return (Integer)jyobject.invoke("foo", args).__tojava__(int.class);
}

The name of the function in the signature declaration string is of no relevance. That’s why we have used a single underscore.

annotation

The annotation decorator applies to methods and classes. We have to wait for Jython 2.6 for proper class decorator syntax but the semantics is the same when we write

cls = annotation(value)(cls)

The value passed to annotation is a string which must conform Java annotation syntax with the leading @ character being stripped.

@annotation("Override")
def foo(self, arg):
    ...

is a valid annotation which corresponds to the Java method

@Override
public PyObject foo(PyObject[] args) {
    return jyobject.invoke("foo", args);
}

Annotations can be stacked and also combined with the signature decorator. So we can define three new decorators

test  = annotation("Test")(signature("public void _()"))
setUp = annotation("Before")(signature("public void _()"))
beforeClass = annotation("BeforeClass")(signature("public static void _()"))

and use them within e.g. JUnit 4

from org.junit import*
from org.junit.Assert import*
 
class TestClass(Object):
    @beforeClass
    def start(self):
        print "Run 'TestClass' tests ..."
 
    @test
    def test_epoweripi(self):
        from math import e, pi
        assertTrue( abs( e**(pi*1j) + 1 ) < 10**-10 )

jproperty

The jproperty object is a descriptor which assigns a Java type and zero or more annotations to a Jython attribute.

class JyClass(Object):
    x = jproperty("private int", "X", "Y")

This is how it translates

pubic class JyClassBase(Object)
{
    @Y @X
    private int x;
}

A JyClass instance reads ( and sets ) jproperty values from the corresponding Java class instances it was assigned to. Remember that the jyobject instance construction looked like this

jyobject = JyGateway.newInstance("JyClass", this, null);

With this an instance of the Java class was passed to the newInstance factory function. Not only holds a javaobject a jyobject but also a jyobject holds a javaobject. Reading / writing jproperty values is the primary reason for this cyclic dependence.

Class import heuristics

Whenever a Java class gets compiled, names have to be imported from classes/packages using the import-statement. Jython applies some heuristics to extract class and package names from annotations/Jython code and creates Java import statements accordingly. An important precondition for a class/package to be found is that it has been imported in Jython code. This isn’t particularly cumbersome. When you define an annotation

annotation('SupportedAnnotationTypes("*")')

the name SupportedAnnotationTypes has to be made public using a normal Jython import:

from javax.annotation.processing import SupportedAnnotationTypes

This is not much different from considering an evaluation of the parameter string using Pythons eval.

The annotation class has a class attribute namespace which is used for Jython class and annotation extraction. If the heuristics fails to extract a class the namespace can be manually updated:

annotation.namespace.update(locals())

Jynx

Within the next few days I’ll launch a new project on code.google.com called jynx which will contain tools particularly suited for Jython utilization of modern Java frameworks and APIs. The relevant source code for this article can be found here and you can examine and play with it.

  1. We’re intending to provide this kind of functionality via the separate Clamp project: http://github.com/groves/clamp/tree/master. It’s a bit similar to what you’re proposing but with a slicker API, here’s what it looks like:

    http://github.com/groves/clamp/blob/c771c7d356fe354d279046352dab39c60628a7a4/src/python/clamp/test/test_making_interface.py

    Clamp isn’t finished, e.g. it doesn’t do annotations yet, but that’s on the roadmap. I wonder if we could make Jython’s Java integration layer recognize annotations and automagically make them act like decorators. i.e. allowing

    from javax.inject import Inject

    class Foo:
        @Inject
        @javamethod(Void.TYPE, BarService)
        def setBarService(service):

    Clamp is super important and could use some development help if you’re at all interested, let us know via the jython-dev mailing list or irc://irc.freenode.net/#jython

  2. kay says:

    Hi Philip, the reason why I refused to provide a slicker API is that Java syntax permits the use of arbitrary Java expressions within annotations. I don’t want to exclude the general case for a more convenient programming experience.

    I’d rather suggest to define per project annotations as I did for JUnit 4.X. All relevant annotations like @After, @BeforeClass etc. can be expressed together with the function signature of the required framework function within a single expression which creates another decorator:

    test = annotation("Test")(signature("public void _()"))

    So one can just use

    @test
    def test_foo(self):
        ....

    instead of the clunky annotation expression. It manages the same thing.

    I also don’t like the idea of building a DSL for expressing arguments and return types when there is already a normative annotation syntax specified for those cases in Python 3.0. I don’t see a particular reason why Jython cannot adopt this already in version 2.6.

    There are a few other things I dislike about the Clamp example. Foo instances need the Reflector class to be instantiated. How shall this work when a class is passed into a framework that controls instantiation? Once again JUnit 4 is a simple framework practising this.

  3. You can embed expressions inside of Python decorators too. So ideally our annotations would support decoration via @Inject and @Inject(arg=’val’ * 2). That’s a bit tricky but doable I think.

    We want to utilize type annotations — we don’t simply because they’re not implemented yet. It’d be another blocking dependency on clamp when clamp should have happened yesterday. So until we have them we’re settling for the simpler decorator syntax.

    We wouldn’t mind including function annotations in Jython 2.x but somebody needs to do the work (someone attempted it last year but obviously didn’t make it too far). It’s mostly just parser work, but we really have only one ANTLR guru on the dev team

    AFAIK Reflector is just a test

  4. Jim Baker says:

    Jython 2.5 actually includes three features from Python 2.6: named tuples, the ast module (the higher-level version of what’s available in _ast), and class decorators. One reason we made certain class decorators were available is that we wanted to have syntactic support for Jython class annotations, as you defined them.

    Thanks for working on this problem, it will certainly help converge us on a good solution!

  5. kay says:

    Jim, it is really hard to figure out anything about Jython.

    The project looks like total crap on the first sight with 5 years old releases linked on the Jython homepage and scattered and aged information across Wiki pages, source forge hosting, no specs/change requests and so on. Right now I try to figure out how to adapt the Java classpath at runtime for dynamic compilation and I have no idea what other programmers with other environments can rely upon. It always comes at a surprise that the project is not comatose again and left unmaintained.

    Once I started to deal with Jython seriously I was surprised about it running not only smooth but providing really excellent Java integration and reflection capabilities – with the notable exception of annotations.

    Thanks for working on this problem, it will certainly help converge us on a good solution!

    I hope dynamic Java compilation is taken seriously. With dynamic compilation one moves into the Jython script in order to reflect the Java environment. This is BTW more in the spirit of the Javaverse as an immersive platform independent environment than dealing with hundreds of command line tools in the OS shell.

  6. Looks like I did not edit my comment well, I’ll try again:

    I would definitely consider putting Python annotations into 2.6. Thanks for looking into the problem of annotations and publishing your findings, and I’d love it if you would take a look at clamp and comment. I also really like the idea of extending our compile() builtin so that it can compile Java sources, though probably not before 2.6.

    We are painfully aware that Jython’s documentation is scattered and weak. A bunch of us are working on a Jython book that will be published by Apress and also made available on the web with an open source license. When that work is done I expect it will become a solid user guide for Jython 2.5.

  7. kay says:

    Hi Frank, I commented on Clamp in the articles comment section already. I cannot say much more about Clamp because it is yet another mostly undocumented project. All I found was a TODO list and I’m not sure what to make of it. I also didn’t find Clamp when I googled for “Jython”+”annotations” so I simply overlooked it.

    I suspect the implementation in jannotations.py might serve as a better starting point given that it has at least some some running use cases ( JUnit 4 integration ).

    Note that I wouldn’t just extend the compile() builtin but provide a completely different function for Java compilation. The Java Compiler API lets one define annotation processors and pass command line arguments. One also has to specify the class name separately. This has been considered in the jcompiler.py module and maybe it suffices to clean up the module and put it into the stdlib.

    Right now it is not possible to inherit from a Java class created dynamically without dumping Java byte code to disk and re-load it but this is just a bug in Jython and I already created a ticket in the bug-tracker.

    About documentation. It’s certainly worthwhile to write a book but books just reflect past efforts. What about adopting the PEP process? As Jython has matured under your project leadership it would be just good to give Jython a less experimental appeal.

  8. Jim Baker says:

    I do hope that our documentation efforts will pay off, you certainly should see some vast improvements on that front in the next 3-6 months, including a revamped web site with documentation tracking C(ore) Python’s doc set. That is, something like http://docs.python.org/library/, but specifically addressing any points of difference. And there will be the book that Frank, I, and others are working on.

    A PEP process for Jython may make sense, but I’m not certain it’s necessary at this time. We already have the PEP process of Python itself, so for us it’s much more narrow in terms of Java integration issues. I rather see more discussion over blogs, IRC, wikis, and mailing lists, especially around good ideas. And most importantly we rather see useful, working code. When we find that process too chaotic, then we may need to rely on PEPs.

  9. kay says:

    The PEP process might seem a bit heavyweight but it specifies the Python language. The only gripe I have with it is that it started late in Pythons evolution and that much of the core implementation has no known rationale. Jython deviates from Python in many fine points and those are not all bugs or maybe they are but who knows?

    Example: yesterday I struggled for hours with compiling a Python string and dump it to a file. Jython allows compiling Python to a code object but the co_code attribute doesn’t show up in dir(code_obj) and when I accessed it, it was empty. There is no load_compiled() API function in imp as it is documented in the Python docs – is_builtin() and is_frozen() are missing too. I opened a bug-ticket because their omission leads to failures in ihook library. The test_importhooks.py test simply fails due to missing modules. There are two implementations of the imp module, one in org.python.core and one in org.python.core.modules. Why not three? If one wants to compile a string using their API functions one has to create an InputStream object first. I finally read the source code of the Jython implementation to understand how to use the API because it is undocumented ( against the advices on the Jython homepage ). This was yesterday. Today I attempted to load a Python script that contained the string “München”. I got SyntaxErrors with UTF encodings – iso-8859 is missing for unknown reasons.

    All of this looks like random complexity and writing a spec is a process of clarification. It’s not necessarily meant to lead to endless bike shedding discussions as we have seen them on python-dev. I also don’t want to search IRC archives and mailing lists from 2002 to get a hint.

    To say something positive, I succeeded in porting EasyExtend 4 to Jython. It is still incomplete and slow but some of the crucial tests are running.

  1. There are no trackbacks for this post yet.

Leave a Reply