Jython annotations – anyone?
The last few days I tried to figure out how to create Jython annotations. A Jython annotation is defined here as a Java annotation lifted from Jython to Java. So one essentially defines a Java annotation in a Jython class. A Jython annotation shall not be confused with a decorator. A Python ( or Jython ) decorator is just a higher order function ( or callable ). It might create attributes in Jython objects but those are not the same as Java annotations reflected by the Java runtime. Without Jython annotations Jython is right now essentially stuck in a pre version 1.5 Javaverse and Jython classes are disconnected from modern Java frameworks and cannot be plugged.
Jython annotations in Jython 2.5 don’t work out of the box. It is not much known yet about how or when Jython annotations will be supported by core Jython. The lead Jython developer Frank Wierzbicki announced something along the lines in his 2008 PyCon conference talk but this is now about 16 months ago. I could temper my impatience if Jython annotations were just around the corner but what can we expect after those 16 months?
In this article I introduce a library that enables lifting of meta-data from Jython to Java and loading Java back into Jython. One key element is Java code generation and dynamic compilation using the Java 6 Compilation API. Another one is interface extraction of Jython classes using the rich reflection API provided by core Jython.
Lifting up Jython classes
For every Jython class `JyClass` one can generate a Java class `JaFromJyClass` by means of interface extraction. We assume `JyClass` to be a subclass of a Java class, e.g. `Object`, and translate the Jython class
class JyClass(Object): def foo(self, *args): print "foo", args |
into a corresponding Java class
public class JaFromJyClass extends Object{ PyObject jyobject; public PyObject foo(PyObject[] args) { return jyobject.invoke("foo", args); } } |
This class is basically a proxy for the `jyobject` member variable of type PyObject which is a Jython API type. Once we have generated the Java code from Jython we can dynamically compile and load the Java code into Jython:
JaFromJyClass = createJavaClass("JaFromJyClass", source) jainstance = JaFromJyClass() jainstance.jyobject = JyClass() jainstance.foo(9) # prints 'foo 9' |
This was straightforward and hints on our translation strategy. Next we review the Jython to Java translations in more detail.
Jython to Java translations
PyObject Injections
We cannot be glad with the way the `jyobject` was assigned to the `jainstance` in the previous example. The particular assignment protocol implies that the Jython script has always control over the instantiation of Jython classes. But once we plug the class into a framework the framework takes over. A better solution is to inject the PyObject using a factory mechanism.
public JaFromJyClass() { super(); jyobject = JyGateway.newInstance("JyClass", this, null); jaobject = (Object)jyobject.__tojava__(Object.class); } |
The factory is called `JyGateway`. The `JyGateway` is a Java class which defines HashMap called registry
public static HashMap<String, PyDictionary> registry = new HashMap<String, PyDictionary>(); |
The keys of the Java HashMap are Strings that represent class names. The `PyDictionary` is a dictionary of Jython functions. Right now two functions are defined: `newInstance` and `callStatic`. Both of them correspond to static methods of `JyGateway`. If `JyGateway.newInstance(“JyClass”, this, null)` is called the `newInstance` Jython function is fetched from the registry using “JyClass” as a key. The third argument of `JyGateway.newInstance` contains an array of Java objects passed as arguments to the `newInstance` function which returns a new PyObject. If the constructor doesn’t take an argument `null` is passed as in the example. The particular `JyClass` will never be exposed to Java code.
Overrdiding superclass methods
Aside from `jyobject` we have also defined `jaobject` in the `JaFromJyClass` constructor which has the type `Object`. Here `Object` is just the superclass of both `JaFromJyClass` and `JyClass`. The `jaobject`is defined for the purpose of overriding superclass methods: we cannot simply change the signature of superclass methods in particular not the return value.
If `public void foo(int x)` is a method defined in the superclass of `JyClass`, the Java method generated from `JyClass` is
public void foo(int arg) { jaobject.foo(arg) } |
The method `foo` called from `jaobject` is still the method implemented in Jython. The Jython method is just called with Java arguments and returns a Java value ( if any ) that gets converted back to Jython.
Calling static methods
Calling static or classmethods of Jython objects from Java is similar to calling `JyGateway.newInstance`:
public static PyObject bar(PyObject args[]) { return JyGateway.callStatic("JyClass", "bar", args); } |
Defining Jython metadata
There are three kinds of meta-data which can be added to Jython classes which are extracted for dynamic Java generation. Those are called `jproperty`, `annotation` and `signature`. They serve different purposes.
signature
A `signature` decorator is defined to assign Java argument and the return types to a Jython method. Without the signature decorator a default translation is applied:
def foo(self, arg): ... |
—–>
public PyObject foo(PyObject[] args) { return jyobject.invoke("foo", args); } |
If we decorate `foo` with the following `signature` decorator
@signature("public int _(char)") def foo(self, arg): ... |
we get the translation
public int foo(char arg0){ PyObject args[] = new PyObject[1]; for(int i=0;i<1;i++) { args[0] = Py.java2py(arg0); } return (Integer)jyobject.invoke("foo", args).__tojava__(int.class); } |
The name of the function in the signature declaration string is of no relevance. That’s why we have used a single underscore.
annotation
The `annotation` decorator applies to methods and classes. We have to wait for Jython 2.6 for proper class decorator syntax but the semantics is the same when we write
cls = annotation(value)(cls) |
The value passed to `annotation` is a string which must conform Java annotation syntax with the leading `@` character being stripped.
@annotation("Override") def foo(self, arg): ... |
is a valid annotation which corresponds to the Java method
@Override public PyObject foo(PyObject[] args) { return jyobject.invoke("foo", args); } |
Annotations can be stacked and also combined with the signature decorator. So we can define three new decorators
test = annotation("Test")(signature("public void _()")) setUp = annotation("Before")(signature("public void _()")) beforeClass = annotation("BeforeClass")(signature("public static void _()")) |
and use them within e.g. JUnit 4
from org.junit import* from org.junit.Assert import* class TestClass(Object): @beforeClass def start(self): print "Run 'TestClass' tests ..." @test def test_epoweripi(self): from math import e, pi assertTrue( abs( e**(pi*1j) + 1 ) < 10**-10 ) |
jproperty
The `jproperty` object is a descriptor which assigns a Java type and zero or more annotations to a Jython attribute.
class JyClass(Object): x = jproperty("private int", "X", "Y") |
This is how it translates
pubic class JyClassBase(Object) { @Y @X private int x; } |
A `JyClass` instance reads ( and sets ) `jproperty` values from the corresponding Java class instances it was assigned to. Remember that the `jyobject` instance construction looked like this
jyobject = JyGateway.newInstance("JyClass", this, null); |
With `this` an instance of the Java class was passed to the `newInstance` factory function. Not only holds a `javaobject` a `jyobject` but also a `jyobject` holds a `javaobject`. Reading / writing `jproperty` values is the primary reason for this cyclic dependence.
Class import heuristics
Whenever a Java class gets compiled, names have to be imported from classes/packages using the import-statement. Jython applies some heuristics to extract class and package names from annotations/Jython code and creates Java import statements accordingly. An important precondition for a class/package to be found is that it has been imported in Jython code. This isn’t particularly cumbersome. When you define an annotation
annotation('SupportedAnnotationTypes("*")') |
the name `SupportedAnnotationTypes` has to be made public using a normal Jython import:
from javax.annotation.processing import SupportedAnnotationTypes |
This is not much different from considering an evaluation of the parameter string using Pythons `eval`.
The `annotation` class has a class attribute `namespace` which is used for Jython class and annotation extraction. If the heuristics fails to extract a class the `namespace` can be manually updated:
annotation.namespace.update(locals()) |
Jynx
Within the next few days I’ll launch a new project on `code.google.com` called jynx which will contain tools particularly suited for Jython utilization of modern Java frameworks and APIs. The relevant source code for this article can be found here and you can examine and play with it.
We’re intending to provide this kind of functionality via the separate Clamp project: http://github.com/groves/clamp/tree/master. It’s a bit similar to what you’re proposing but with a slicker API, here’s what it looks like:
http://github.com/groves/clamp/blob/c771c7d356fe354d279046352dab39c60628a7a4/src/python/clamp/test/test_making_interface.py
Clamp isn’t finished, e.g. it doesn’t do annotations yet, but that’s on the roadmap. I wonder if we could make Jython’s Java integration layer recognize annotations and automagically make them act like decorators. i.e. allowing
from javax.inject import Inject
class Foo: @Inject @javamethod(Void.TYPE, BarService) def setBarService(service):Clamp is super important and could use some development help if you’re at all interested, let us know via the jython-dev mailing list or irc://irc.freenode.net/#jython
Hi Philip, the reason why I refused to provide a slicker API is that Java syntax permits the use of arbitrary Java expressions within annotations. I don’t want to exclude the general case for a more convenient programming experience.
I’d rather suggest to define per project annotations as I did for JUnit 4.X. All relevant annotations like @After, @BeforeClass etc. can be expressed together with the function signature of the required framework function within a single expression which creates another decorator:
test = annotation("Test")(signature("public void _()"))So one can just use
@test def test_foo(self): ....instead of the clunky annotation expression. It manages the same thing.
I also don’t like the idea of building a DSL for expressing arguments and return types when there is already a normative annotation syntax specified for those cases in Python 3.0. I don’t see a particular reason why Jython cannot adopt this already in version 2.6.
There are a few other things I dislike about the Clamp example. Foo instances need the Reflector class to be instantiated. How shall this work when a class is passed into a framework that controls instantiation? Once again JUnit 4 is a simple framework practising this.
You can embed expressions inside of Python decorators too. So ideally our annotations would support decoration via @Inject and @Inject(arg=’val’ * 2). That’s a bit tricky but doable I think.
We want to utilize type annotations — we don’t simply because they’re not implemented yet. It’d be another blocking dependency on clamp when clamp should have happened yesterday. So until we have them we’re settling for the simpler decorator syntax.
We wouldn’t mind including function annotations in Jython 2.x but somebody needs to do the work (someone attempted it last year but obviously didn’t make it too far). It’s mostly just parser work, but we really have only one ANTLR guru on the dev team
AFAIK Reflector is just a test
Jython 2.5 actually includes three features from Python 2.6: named tuples, the ast module (the higher-level version of what’s available in _ast), and class decorators. One reason we made certain class decorators were available is that we wanted to have syntactic support for Jython class annotations, as you defined them.
Thanks for working on this problem, it will certainly help converge us on a good solution!
Jim, it is really hard to figure out anything about Jython.
The project looks like total crap on the first sight with 5 years old releases linked on the Jython homepage and scattered and aged information across Wiki pages, source forge hosting, no specs/change requests and so on. Right now I try to figure out how to adapt the Java classpath at runtime for dynamic compilation and I have no idea what other programmers with other environments can rely upon. It always comes at a surprise that the project is not comatose again and left unmaintained.
Once I started to deal with Jython seriously I was surprised about it running not only smooth but providing really excellent Java integration and reflection capabilities – with the notable exception of annotations.
I hope dynamic Java compilation is taken seriously. With dynamic compilation one moves into the Jython script in order to reflect the Java environment. This is BTW more in the spirit of the Javaverse as an immersive platform independent environment than dealing with hundreds of command line tools in the OS shell.
Looks like I did not edit my comment well, I’ll try again:
I would definitely consider putting Python annotations into 2.6. Thanks for looking into the problem of annotations and publishing your findings, and I’d love it if you would take a look at clamp and comment. I also really like the idea of extending our compile() builtin so that it can compile Java sources, though probably not before 2.6.
We are painfully aware that Jython’s documentation is scattered and weak. A bunch of us are working on a Jython book that will be published by Apress and also made available on the web with an open source license. When that work is done I expect it will become a solid user guide for Jython 2.5.
Hi Frank, I commented on Clamp in the articles comment section already. I cannot say much more about Clamp because it is yet another mostly undocumented project. All I found was a TODO list and I’m not sure what to make of it. I also didn’t find Clamp when I googled for “Jython”+”annotations” so I simply overlooked it.
I suspect the implementation in jannotations.py might serve as a better starting point given that it has at least some some running use cases ( JUnit 4 integration ).
Note that I wouldn’t just extend the compile() builtin but provide a completely different function for Java compilation. The Java Compiler API lets one define annotation processors and pass command line arguments. One also has to specify the class name separately. This has been considered in the jcompiler.py module and maybe it suffices to clean up the module and put it into the stdlib.
Right now it is not possible to inherit from a Java class created dynamically without dumping Java byte code to disk and re-load it but this is just a bug in Jython and I already created a ticket in the bug-tracker.
About documentation. It’s certainly worthwhile to write a book but books just reflect past efforts. What about adopting the PEP process? As Jython has matured under your project leadership it would be just good to give Jython a less experimental appeal.
I do hope that our documentation efforts will pay off, you certainly should see some vast improvements on that front in the next 3-6 months, including a revamped web site with documentation tracking C(ore) Python’s doc set. That is, something like http://docs.python.org/library/, but specifically addressing any points of difference. And there will be the book that Frank, I, and others are working on.
A PEP process for Jython may make sense, but I’m not certain it’s necessary at this time. We already have the PEP process of Python itself, so for us it’s much more narrow in terms of Java integration issues. I rather see more discussion over blogs, IRC, wikis, and mailing lists, especially around good ideas. And most importantly we rather see useful, working code. When we find that process too chaotic, then we may need to rely on PEPs.
The PEP process might seem a bit heavyweight but it specifies the Python language. The only gripe I have with it is that it started late in Pythons evolution and that much of the core implementation has no known rationale. Jython deviates from Python in many fine points and those are not all bugs or maybe they are but who knows?
Example: yesterday I struggled for hours with compiling a Python string and dump it to a file. Jython allows compiling Python to a code object but the co_code attribute doesn’t show up in dir(code_obj) and when I accessed it, it was empty. There is no load_compiled() API function in imp as it is documented in the Python docs – is_builtin() and is_frozen() are missing too. I opened a bug-ticket because their omission leads to failures in ihook library. The test_importhooks.py test simply fails due to missing modules. There are two implementations of the imp module, one in org.python.core and one in org.python.core.modules. Why not three? If one wants to compile a string using their API functions one has to create an InputStream object first. I finally read the source code of the Jython implementation to understand how to use the API because it is undocumented ( against the advices on the Jython homepage ). This was yesterday. Today I attempted to load a Python script that contained the string “München”. I got SyntaxErrors with UTF encodings – iso-8859 is missing for unknown reasons.
All of this looks like random complexity and writing a spec is a process of clarification. It’s not necessarily meant to lead to endless bike shedding discussions as we have seen them on python-dev. I also don’t want to search IRC archives and mailing lists from 2002 to get a hint.
To say something positive, I succeeded in porting EasyExtend 4 to Jython. It is still incomplete and slow but some of the crucial tests are running.