Archive for July, 2009

Jynx 0.2 released

Posted in Java, Jynx, Jython on July 27th, 2009 by kay – Be the first to comment

I’ve released Jynx 0.2. Jynx is a Jython package which utilizes dynamic Java compilation from Jython and improves on Java scripting. With Jynx 0.2 two major new features are implemented now.

Annotation Extraction

In the initial Jynx 0.1 release an annotation object was defined which can be used as a decorator. A Python class such as

@JavaClass
class TestClass(Object):
    @annotation("Test")
    @signature("public void _()")
    def test_report_test_failure(self):
        assertTrue("length of empty list is 0", len([]) != 0)

equipped with the JavaClass decorator is compiled into a Java class on the fly which acts as a proxy for a Python object and provides the correct interface for being used within a Java framework which expects methods of a particular type signature and annotations. The class defined above can be used within JUnit 4.X.

Jynx 0.2 provides a new classmethod extract of the annotation class which can be used to extract Java annotation classes and acts as a factory function for Jython annotation objects.

# import Test annotation in JUnit 4.X
from org.junit import Test      
 
# a Python annotation object
Test = annotation.extract(Test) 
 
# keep a signature object as a parameter and returns a new Jython
# annotation object. The Java code generator will create a method
# with the correct signature and the @Test annotation
Test = Test(signature("public void _()")  
 
@JavaClass
class TestClass(Object):
    @Test
    def test_report_test_failure(self):
       assertTrue("length of empty list is 0", len([]) != 0)

As we see there is no overhead left here. When programming against a Java API / framework, Jython annotations can be defined within a single file and used application wide.

Classpath Manipulation

For reasons which are not completely transparent to me Java doesn’t permit runtime classpath manipulations. The JDK defines an addURL method in a special classloader called URLClassLoader. This method is protected and cannot generally be accessed without reflection. Internally the Sun JVM uses such a loader class ( or a subclass of it ) and when you are willing to accept a hack and programming against an implementation detail you can use the JVMs default class loader and add new paths to a classpath:

from java.lang import ClassLoader
systemLoader = ClassLoader.getSystemClassLoader()
systemLoader.addURL("file:///C|junit-4.6.jar")

Jynx defines a ClassPath class and a new sys module attribute classpath. Adding a file system path P to sys.classpath results in a method call

systemloader.addURL(URL("file:"+pathname2url(pth)))

which converts the file system path into a Java URL object and adds it to the classpath. Additionally the same path is added to the PYTHONPATH via sys.path:

sys.classpath.append(r"C:\junit-4.6.jar")

The advantage is that each Python package can maintain the Java packages it depends upon and no global CLASSPATH environment variable has to be adapted unless a Java or Jython class defines its own class loader.

Four things I’d change in Python – and a little more

Posted in Python on July 25th, 2009 by kay – 4 Comments

1. Import system

Replace the flat module cache by a set of ModuleTree objects rooted in nodes living on the PYTHONPATH. Apply relative path semantics by default and treat absolute paths as special cases. Internal paths which are used in import statements or for traversing ModuleTree objects and external ones ( file-system, zip-files, URLs etc. ) are related through representations of internal paths [1]. Representations shall be user definable. For ModuleTree objects custom import semantics may be defined. This replaces “import hooks” and provides similar functionality in a much safer and object oriented manner. Further effects: no physical module is imported twice for two different import paths; each module can be used as a script no matter how the path is written. No changes to the language syntax.

[1] What I mean here is a representation of a path algebra in systems which can be considered as the “environment” of Python. This sounds more grandiose than it actually is.

2. Decorators everywhere

This basically reflects my interest in improving Jython compliance with Java and lifting Jython classes to Java classes turning Java classes into Jython class proxies – everything at runtime. This doesn’t work without specifying Java interfaces in Jython. Those consist of two parts: type signatures + annotations. For functions and classes this works in Python without much hassle. With Python 3.0 style function annotations one can even remove a decorator for type signatures. It doesn’t work for members though. In Java you can write

public class CusomerId {
    @Id
    @Column(name = "CustId", nullable = false)
    private Integer cust_id;
}

In Python I want to write similarly

class CusomerId:
    @Id
    @Column(name = "CustId", nullable = False)
    cust_id = jproperty("private int")

which translates into

class CusomerId:
    cust_id = Id(Column(name="CustId", nullable=False)(jproperty("private int")))

This requires that assignment statements ( grammatically expr_stmt’s ) may be decorated, not just functions and classes.

3. A new opcode for code monitoring

I know Ned Batchelders coverage tool and I have written one by myself using EasyExtend. EasyExtends is more powerful in that it doesn’t only provide the simplest type of coverage namely statement coverage. However it uses source code weaving which might affect functionality in a highly reflective language. It would be far better to introduce a new opcode which is weaved into Pythons bytecode and acts as a sensor. The weaving can be activated using a command line option. The overall achievement is improved code monitoring. This solution might also be applied to improve debuggers by setting breakpoints within expressions.

4. Function annotation and the nonlocal statement backports

I wish to see function argument annotations and the nonlocal statement in Python 2.x.

Other things

Admittedly I felt a little depression after the huge disappointment which was Python 3. Instead of a bright future it looked much like a minor legacy transformation which essentially missed the point of relevant trends in language design which are marked by concurrency orientation and unification of declarative dataflow and OO in frameworks + languages like WPF/Silverlight, Flex and JavaFX. The best thing which can be said about Python 3 is that it didn’t turn into a running gag and actually shipped code.

However there are lots of positive news and much progress in many other areas. At many fronts Python performance is targeted: PyPy, Unladen Swallow, Psyco 2, Cython, Shedskin. Package distribution and deployment is addressed just like renovation of the standard library. With PyPy, Unladen Swallow, Jython and IronPython Python becomes or is already GIL free and fit for multicore. The one improvement I’m personally most pleased about is that of Jython. Aside from my eternal pets ( Trail + EasyExtend ) I enjoy exploring the Javaverse, which is incredibly rich, from the Jython + scripting angle with some promising first results, new challenges and also some disappointments. I actually expect the next 600 Python web frameworks of interest will not be written in CPython anymore but in Jython and IronPython using Java/.Net components. When will we see a Jython Enterprise Framework on the JVM which will be as powerful as Spring but as lightweight as Pylons?

Redesign of the code.py and codeop.py modules

Posted in Python on July 24th, 2009 by kay – 2 Comments

Brett Cannon asks for modules of the stdlib to be redesigned. I find the idea rather bizarre to initiate a poll for this but maybe that’s just the future of programming where the quality of an implementation is judged by democratic voting. So I immersed into the hive mind and voted for distutils. Seems like Tarek Ziade addresses this already but I’m not entirely sure he goes far enough. Last time I looked at the source code there were still all kinds of compiler modules in the lib which contain config information closely coupled with application code. That’s not so nice and mostly a refactoring bit.

Some other of the stdlib modules I’d rewrite are not mentioned in the voting list. Maybe they are not sexy enough for the majority of web programmers that dominate all the discussions about Python? Among my favorites are code.py and codeop.py. Here is a brief but incomplete list of requirements and refactorings.

  • The heuristics used to determine incomplete Python commands in _maybe_compile is pretty weak.
  • Can you tell the difference between Compile, CommandCompiler and compile_command in codeop.py?
  • Encapsulate the raw_input function in interact within a method that can be overwritten.
  • provide two methods at_start and at_exit in InteractiveConsole to make startup and shutdown customizable.
  • Separate interactive loop from line processing and implement the line processor as a generator. It’s easier to write custom interactive loops for systems that interface with Python. The default interact method becomes
    def interact(self):
        self.at_start()
        try:
            gen_process = self.process_line()
            line = None
            while True:
                try:
                    prompt = gen_process.send(line)
                    line   = self.user.get_input(prompt)
                except StopIteration:
                    break
        finally:
            self.at_exit()
  • Move the the line terminating heuristics from _maybe_compile into process_line and define a try_parse function together with a try_compile function. I’d go a little further even and define a try_tokenize function which isn’t essential though.
  • Provide a subclass for interactive sessions which can be recorded and replayed and command line options accordingly. This is optional though and not part of a redesign strategy.

There are other modules I’d like to rewrite such as tokenizer.py. Having a lexer in the stdlib which handles Python as a special case would be quite a big deal IMO. But it’s delicate and I struggle with writing lexers which can be both extended in a simple way ( without the trouble of running into ordered choice problems of the current regular expression engine ) and have a high performance. So far I only accomplished the first of the goals, at least partially, but not the second one.

A simple Spring challenge

Posted in Java on July 20th, 2009 by kay – 3 Comments

I got some comments on my Biggus Dickus article in its own comments section as well as on programming.reddit. Many people defended Spring on grounds of its usability, whereas others identified the author of this lines as completely clueless. I don’t want to argue against the latter and they are certainly right that Spring is the way enterprise software shall be written to eliminate Java complexity.

SpringSource made just the mistake of supporting dynamic languages but omitted Jython which wasn’t hip for a while and now countless amateurs, Java mavericks and inevitable crackpots feel attracted by Spring and seek their luck. Once you open the door to this folks they want to feel comfortable in their own way which means they want to get rid of XML configuration files and enable self-management for dynamic languages.

The problem description

It is not much effort to use Springs dependency injection (DI) machinery. The Spring user can follow a Bean creation protocol which looks cumbersome on the first sight but one gets used to it very soon.

Spring defines a BeanReaderInterface implemented by the PropertyBeanDefinitionReader and XmlBeanDefinitionReader classes. So what about adding a JythonBeanDefinitionReader along with a JythonBeanFactory replacing the DefaultListableBeanFactory or another factory of this kind which is typically used? The following protocol shows how to tangle both types and how to create new bean instances without letting the application know anything about configuration logics:

JythonBeanFactory factory = new JythonBeanFactory();
JythonBeanDefinitionReader reader = new JythonBeanDefinitionReader(factory);
reader.loadBeanDefinitions(path);
SomeBean obj = (SomeBean)factory.getBean(name);

How can JythonBeanFactory and JythonBeanDefinitionReader possibly work ? In a simple case the Reader uses the Jython API and imports a Python module which defines parameter-less functions like the following:

def source():
    # creates SomeBean object and returns it

Calling factory.getBean(“source”) will invoke the source() function which returns a SomeBean object. Eventually the object has been cached but at this stage I do not want to complicate the design if it can be avoided.

Both classes can be implemented on an elementary level as a simple exercise of embedding Jython in Java and using the Jython API.

The Challenge

Now try to write both classes s.t. they do fit into the Spring framework. As I said above it is a basic exercise to write them but side stepping the Spring interface hierarchy would just mean to create another DI framework which is off topic here. They shall implement Spring interfaces and they shall replace existing bean readers and factories. This is not hackish and Spring itself has foreseen such extensions as use cases of the framework.

Java Spring – or the Biggus Dickus effect

Posted in Java on July 18th, 2009 by kay – 32 Comments

Looking at the API alone Spring feels like reading a parody on Java enterprise software written by Steve Yegge.

AbstractBeanFactoryBasedTargetSourceCreator ContextSingletonBeanFactoryLocator
AspectJAdviceParameterNameDiscoverer UserRoleAuthorizationInterceptor 
TransactionAwarePersistenceManagerFactoryProxy SQLStateSQLExceptionTranslator 
SimpleBeanFactoryAwareAspectInstanceFactory ...
Legend! Nonstop administrative debris as dadaist poetry. Écriture automatique of the programming office manager or his parrot. This goes on and on for miles and miles – I count 1943 Spring 2.5 API classes+interfaces. This way it tops JEE 6 ( Glassfish 3 ) by about 450 – JEE 6 is basically free of this notational nonsense.

Spring causes a Biggus Dickus effect on me in the role of one of Pilatus guarding soldiers. It lives beyond a tolerable threshold of “software engineering best practices” not becoming comical.

Jynx

Posted in Jynx, Jython on July 10th, 2009 by kay – Be the first to comment

I have just released the initial version of my new Jython project Jynx on Google Code. Jynx sums up my latest efforts on the dynamic Java compilation front and it heads into the future of Java framework utilization from Python.

Although Jynx is quite tiny right now it has already enough structure, code and documentation for Jython/Jynx developers to be useful. People are invited to check it out, criticize it and contribute.

Have much fun with Python / Jython programming!