Jython

Python – Hibernate – Jynx

Posted in Hibernate, Jynx, Jython on September 4th, 2009 by kay – 2 Comments

Jynx 0.4 goes Hibernate

In Jynx 0.4 JPA/Hibernate annotations are supported. Although this is still work in progress some of the more complex nested annotations were tested as well as Hibernate extension annotations which cannot be single-name imported along with the corresponding JPA annotations without conflicts.

Jynx 0.4 provides other new features as well. One can now use @signature decorators to express Java method overloading. A simple Java parser is integrated. A Java parser was necessary to improve the Java class detection heuristics used to determine required imports when a Java proxy is created from a Jython class and compiled dynamically. Finally there is a new @bean_property decorator which creates a private attribute foo along with public getters and setters given a bean_property decorated method def foo(_):_. Full documentation of Jynx as well as its changes can be found here.

Using Hibernate from Jython

Starting and closing sessions and managing simple transactions is not difficult in Hibernate. In Jynx two context managers for with-statements are defined which hide open+close and begin+commit/rollback boilerplate from the programmer. Code for Hibernate sessions and transactions lives then in with-statement blocks.

class hn_session(object):
    '''
    Context manager which opens/closes hibernate sessions.
    '''
    def __init__(self, *classes):
        sessionFactory = buildSessionFactory(*classes)
        self.session   = sessionFactory.openSession()
 
    def __enter__(self):
        return self.session
 
    def __exit__(self, *exc_info):
        self.session.close()
 
class hn_transact(object):
    '''
    Context manager which begins and performs commits/rollbacks hibernate transactions.
    '''
    def __init__(self, session):
        self.tx = session.beginTransaction()
 
    def __enter__(self):
        return self.tx
 
    def __exit__(self, type, value, traceback):
        if type is None:
            self.tx.commit()
        else:
            self.tx.rollback()

A simple session using a single Entity Bean may then look like:

from __future__ import with_statement
 
from jynx.lib.hibernate import*
 
@Entity
class Course(Serializable):
    @Id
    @Column(name="COURSE_ID")
    @signature("public int _()")
    def getCourseId(self):
        return self.courseId
 
    @Column(name="COURSE_NAME", nullable = False, length=50)
    @signature("public String _()")
    def getCourseName(self):
        return self.courseName
 
    @signature("public void _(String)")
    def setCourseName(self, value):
        self.courseName = value
 
    @signature("public void _(int)")
    def setCourseId(self, value):
        self.courseId = value
 
with hn_session(Course) as session:
    course  = Course()
    course.setCourseId(121)
    course.setCourseName(str(range(5)))
    with hn_transact(session):
        session.saveOrUpdate(course)

Boilerplate Reduction

The standard class decorator for creating a Java class from a Jython class in Jynx is @JavaClass. In Jynx 0.4 some slightly extended decorators are introduced in particular @Entity and @Embeddable. Not only do they make Jython code more concise because one doesn’t have to stack @Entity and @JavaClass but translating with @Entity turns some automatically generated Java attributes into transient ones i.e. a @Transient annotation is applied which prevents those attributes to be mapped to table columns.

The massive boilerplate needed for defining a valid Entity Bean in the preceding example can be reduced using the @bean_property decorator:

@Entity
class Course(Serializable):
    @Id
    @Column(name="COURSE_ID")
    @bean_property(int)
    def courseId(self): pass
 
    @Column(name="COURSE_NAME", nullable = False, length=50)
    @bean_property(String)
    def courseName(self): pass

Applied to def courseId(self): pass the @bean_property decorator will cause the following Java code translation

    @Id @Column(name="COURSE_ID") private int courseId;
    int getCourseId() { return courseId; }
    int setCourseId(int value) { courseId = value; }

which specifies a complete Java Bean property.

Example

In the following example two Entities are associated using a one-to-one mapping between primary keys.

@Entity
class Heart(Serializable):
    @Id
    @bean_property(int)
    def id(self):pass
 
@Entity
class Body(Serializable):
    @Id
    @bean_property(int)
    def id(self):pass
 
    @OneToOne(cascade = CascadeType.ALL)
    @PrimaryKeyJoinColumn
    @bean_property(Heart)
    def heart(self):pass

Now we can check the behavior:

# session 1
with hn_session(Heart, Body) as session:
    body = Body()
    heart = Heart()
    body.heart = heart
    body.id = 1
    heart.id = body.id
    with hn_transact(session):
        session.saveOrUpdate(body)
        session.saveOrUpdate(heart)
 
# session 2
with hn_session(Heart, Body) as session:
    with hn_transact(session):
        b = session.get(Body, 1)
        assert b
        assert b.heart
        assert b.heart.id == 1

Summary

With Hibernate support in Jython we notice another clear departure from the CPython world and its web frameworks and components. Hibernate is distinctively Java and special techniques are needed to create compile time Java properties in a dynamic language. Jython has long been a second citizen in Python land. I suspect this is going to change with support of Java frameworks which alone have as many users/downloads as Python.

Jynx 0.3 – how to fix custom class loaders for use with Jython

Posted in Jynx, Jython on August 12th, 2009 by kay – Be the first to comment

Broken class loaders

Jynx 0.2 contained an ugly workaround for a bug I couldn’t fix for quite a while. The bug can be described as follows: suppose you defined code of a Java class A and compiled it dynamically:

A = JavaCompiler().createClass("A", A_source)

When you attempt to build a subclass

class B(A): pass
a NoClassDefFoundError exception was raised:

Traceback (most recent call last):
  File "C:\lang\Jython\jcompile.py", line 185, in <module>
    class B(A):pass
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
java.lang.NoClassDefFoundError: org/python/core/PyProxy (wrong name: A)

In that case the Jython runtime failed to create a proxy class for B while locating PyProxy which is a Jython core interface. From the traceback it wasn’t clear how to locate the error and I started to debug into Jython from Netbeans.

This is what happened: Jynx defines a ByteClassLoader class which is custom class loader for dynamic compilation of A. When A is loaded with loadClass a findClass method is called to locate A and this method had to be overwritten. The ByteClassLoader was bound to A automatically and used by Jython to locate interfaces such as org.python.core.PyProxy. This didn’t work and explains the failure. A possible fix is to respond to classes which cannot be dealt with from ByteClassLoader and delegate a findClass call to the parent class loader.

Curiously Jython stopped using ByteClassLoader after I changed the inheritance hierarchy from

class ByteClassLoader(ClassLoader):
    def __init__(self, code):
        super(ByteClassLoader, self).__init__(ClassLoader.getClassLoader())
        ...

to

class ByteClassLoader(URLClassLoader):
    def __init__(self, code):
        super(ByteClassLoader, self).__init__([], ClassLoader.getSystemClassLoader())
        ...

The URLClassLoader provides the opportunity to add URLs at runtime and therefore modifying the CLASSPATH dynamically.

No disk dumps in Jynx 0.3

Prior to Jynx 0.3 a workaround has been dumping A to disk and load the class from there. We discussed the subtle nuances of selecting the right class loader and loading A from disk moved the machinery into a correct state. This wasn’t only cumbersome but a hurdle when a programmer intended to work within a Java sandbox. With Jynx 0.3 I feel prepared to explore Java integration with Jynx on GAE-J.

Jynx 0.2 released

Posted in Java, Jynx, Jython on July 27th, 2009 by kay – Be the first to comment

I’ve released Jynx 0.2. Jynx is a Jython package which utilizes dynamic Java compilation from Jython and improves on Java scripting. With Jynx 0.2 two major new features are implemented now.

Annotation Extraction

In the initial Jynx 0.1 release an annotation object was defined which can be used as a decorator. A Python class such as

@JavaClass
class TestClass(Object):
    @annotation("Test")
    @signature("public void _()")
    def test_report_test_failure(self):
        assertTrue("length of empty list is 0", len([]) != 0)

equipped with the JavaClass decorator is compiled into a Java class on the fly which acts as a proxy for a Python object and provides the correct interface for being used within a Java framework which expects methods of a particular type signature and annotations. The class defined above can be used within JUnit 4.X.

Jynx 0.2 provides a new classmethod extract of the annotation class which can be used to extract Java annotation classes and acts as a factory function for Jython annotation objects.

# import Test annotation in JUnit 4.X
from org.junit import Test      
 
# a Python annotation object
Test = annotation.extract(Test) 
 
# keep a signature object as a parameter and returns a new Jython
# annotation object. The Java code generator will create a method
# with the correct signature and the @Test annotation
Test = Test(signature("public void _()")  
 
@JavaClass
class TestClass(Object):
    @Test
    def test_report_test_failure(self):
       assertTrue("length of empty list is 0", len([]) != 0)

As we see there is no overhead left here. When programming against a Java API / framework, Jython annotations can be defined within a single file and used application wide.

Classpath Manipulation

For reasons which are not completely transparent to me Java doesn’t permit runtime classpath manipulations. The JDK defines an addURL method in a special classloader called URLClassLoader. This method is protected and cannot generally be accessed without reflection. Internally the Sun JVM uses such a loader class ( or a subclass of it ) and when you are willing to accept a hack and programming against an implementation detail you can use the JVMs default class loader and add new paths to a classpath:

from java.lang import ClassLoader
systemLoader = ClassLoader.getSystemClassLoader()
systemLoader.addURL("file:///C|junit-4.6.jar")

Jynx defines a ClassPath class and a new sys module attribute classpath. Adding a file system path P to sys.classpath results in a method call

systemloader.addURL(URL("file:"+pathname2url(pth)))

which converts the file system path into a Java URL object and adds it to the classpath. Additionally the same path is added to the PYTHONPATH via sys.path:

sys.classpath.append(r"C:\junit-4.6.jar")

The advantage is that each Python package can maintain the Java packages it depends upon and no global CLASSPATH environment variable has to be adapted unless a Java or Jython class defines its own class loader.

Jynx

Posted in Jynx, Jython on July 10th, 2009 by kay – Be the first to comment

I have just released the initial version of my new Jython project Jynx on Google Code. Jynx sums up my latest efforts on the dynamic Java compilation front and it heads into the future of Java framework utilization from Python.

Although Jynx is quite tiny right now it has already enough structure, code and documentation for Jython/Jynx developers to be useful. People are invited to check it out, criticize it and contribute.

Have much fun with Python / Jython programming!

Stitches of a flea language – defining Java annotations in Jython

Posted in Java, Jython on June 30th, 2009 by kay – 9 Comments

Jython annotations – anyone?

The last few days I tried to figure out how to create Jython annotations. A Jython annotation is defined here as a Java annotation lifted from Jython to Java. So one essentially defines a Java annotation in a Jython class. A Jython annotation shall not be confused with a decorator. A Python ( or Jython ) decorator is just a higher order function ( or callable ). It might create attributes in Jython objects but those are not the same as Java annotations reflected by the Java runtime. Without Jython annotations Jython is right now essentially stuck in a pre version 1.5 Javaverse and Jython classes are disconnected from modern Java frameworks and cannot be plugged.

Jython annotations in Jython 2.5 don’t work out of the box. It is not much known yet about how or when Jython annotations will be supported by core Jython. The lead Jython developer Frank Wierzbicki announced something along the lines in his 2008 PyCon conference talk but this is now about 16 months ago. I could temper my impatience if Jython annotations were just around the corner but what can we expect after those 16 months?

In this article I introduce a library that enables lifting of meta-data from Jython to Java and loading Java back into Jython. One key element is Java code generation and dynamic compilation using the Java 6 Compilation API. Another one is interface extraction of Jython classes using the rich reflection API provided by core Jython.

Lifting up Jython classes

For every Jython class JyClass one can generate a Java class JaFromJyClass by means of interface extraction. We assume JyClass to be a subclass of a Java class, e.g. Object, and translate the Jython class

class JyClass(Object):
    def foo(self, *args):
        print "foo", args

into a corresponding Java class

public class JaFromJyClass extends Object{
    PyObject jyobject;
 
    public PyObject foo(PyObject[] args)
    {
        return jyobject.invoke("foo", args);
    }
}

This class is basically a proxy for the jyobject member variable of type PyObject which is a Jython API type. Once we have generated the Java code from Jython we can dynamically compile and load the Java code into Jython:

JaFromJyClass = createJavaClass("JaFromJyClass", source)
jainstance = JaFromJyClass()
jainstance.jyobject = JyClass()
jainstance.foo(9)  # prints 'foo 9'

This was straightforward and hints on our translation strategy. Next we review the Jython to Java translations in more detail.

Jython to Java translations

PyObject Injections

We cannot be glad with the way the jyobject was assigned to the jainstance in the previous example. The particular assignment protocol implies that the Jython script has always control over the instantiation of Jython classes. But once we plug the class into a framework the framework takes over. A better solution is to inject the PyObject using a factory mechanism.

public JaFromJyClass() {
    super();
    jyobject = JyGateway.newInstance("JyClass", this, null);
    jaobject = (Object)jyobject.__tojava__(Object.class);
}

The factory is called JyGateway. The JyGateway is a Java class which defines HashMap called registry

public static HashMap<String, PyDictionary> registry = new HashMap<String, PyDictionary>();

The keys of the Java HashMap are Strings that represent class names. The PyDictionary is a dictionary of Jython functions. Right now two functions are defined: newInstance and callStatic. Both of them correspond to static methods of JyGateway. If JyGateway.newInstance(“JyClass”, this, null) is called the newInstance Jython function is fetched from the registry using “JyClass” as a key. The third argument of JyGateway.newInstance contains an array of Java objects passed as arguments to the newInstance function which returns a new PyObject. If the constructor doesn’t take an argument null is passed as in the example. The particular JyClass will never be exposed to Java code.

Overrdiding superclass methods

Aside from jyobject we have also defined jaobject in the JaFromJyClass constructor which has the type Object. Here Object is just the superclass of both JaFromJyClass and JyClass. The jaobjectis defined for the purpose of overriding superclass methods: we cannot simply change the signature of superclass methods in particular not the return value.

If public void foo(int x) is a method defined in the superclass of JyClass, the Java method generated from JyClass is

public void foo(int arg) { jaobject.foo(arg) }

The method foo called from jaobject is still the method implemented in Jython. The Jython method is just called with Java arguments and returns a Java value ( if any ) that gets converted back to Jython.

Calling static methods

Calling static or classmethods of Jython objects from Java is similar to calling JyGateway.newInstance:

public static PyObject bar(PyObject args[])
{
     return JyGateway.callStatic("JyClass", "bar", args);
}

Defining Jython metadata

There are three kinds of meta-data which can be added to Jython classes which are extracted for dynamic Java generation. Those are called jproperty, annotation and signature. They serve different purposes.

signature

A signature decorator is defined to assign Java argument and the return types to a Jython method. Without the signature decorator a default translation is applied:

def foo(self, arg):
    ...

—–>

public PyObject foo(PyObject[] args) {
    return jyobject.invoke("foo", args);
}

If we decorate foo with the following signature decorator

@signature("public int _(char)")
def foo(self, arg):
    ...

we get the translation

public int foo(char arg0){
    PyObject args[] = new PyObject[1];
    for(int i=0;i<1;i++) {
        args[0] = Py.java2py(arg0);
    }
    return (Integer)jyobject.invoke("foo", args).__tojava__(int.class);
}

The name of the function in the signature declaration string is of no relevance. That’s why we have used a single underscore.

annotation

The annotation decorator applies to methods and classes. We have to wait for Jython 2.6 for proper class decorator syntax but the semantics is the same when we write

cls = annotation(value)(cls)

The value passed to annotation is a string which must conform Java annotation syntax with the leading @ character being stripped.

@annotation("Override")
def foo(self, arg):
    ...

is a valid annotation which corresponds to the Java method

@Override
public PyObject foo(PyObject[] args) {
    return jyobject.invoke("foo", args);
}

Annotations can be stacked and also combined with the signature decorator. So we can define three new decorators

test  = annotation("Test")(signature("public void _()"))
setUp = annotation("Before")(signature("public void _()"))
beforeClass = annotation("BeforeClass")(signature("public static void _()"))

and use them within e.g. JUnit 4

from org.junit import*
from org.junit.Assert import*
 
class TestClass(Object):
    @beforeClass
    def start(self):
        print "Run 'TestClass' tests ..."
 
    @test
    def test_epoweripi(self):
        from math import e, pi
        assertTrue( abs( e**(pi*1j) + 1 ) < 10**-10 )

jproperty

The jproperty object is a descriptor which assigns a Java type and zero or more annotations to a Jython attribute.

class JyClass(Object):
    x = jproperty("private int", "X", "Y")

This is how it translates

pubic class JyClassBase(Object)
{
    @Y @X
    private int x;
}

A JyClass instance reads ( and sets ) jproperty values from the corresponding Java class instances it was assigned to. Remember that the jyobject instance construction looked like this

jyobject = JyGateway.newInstance("JyClass", this, null);

With this an instance of the Java class was passed to the newInstance factory function. Not only holds a javaobject a jyobject but also a jyobject holds a javaobject. Reading / writing jproperty values is the primary reason for this cyclic dependence.

Class import heuristics

Whenever a Java class gets compiled, names have to be imported from classes/packages using the import-statement. Jython applies some heuristics to extract class and package names from annotations/Jython code and creates Java import statements accordingly. An important precondition for a class/package to be found is that it has been imported in Jython code. This isn’t particularly cumbersome. When you define an annotation

annotation('SupportedAnnotationTypes("*")')

the name SupportedAnnotationTypes has to be made public using a normal Jython import:

from javax.annotation.processing import SupportedAnnotationTypes

This is not much different from considering an evaluation of the parameter string using Pythons eval.

The annotation class has a class attribute namespace which is used for Jython class and annotation extraction. If the heuristics fails to extract a class the namespace can be manually updated:

annotation.namespace.update(locals())

Jynx

Within the next few days I’ll launch a new project on code.google.com called jynx which will contain tools particularly suited for Jython utilization of modern Java frameworks and APIs. The relevant source code for this article can be found here and you can examine and play with it.