The state of EasyExtend
Maybe an autumnal look on EasyExtend is justified. EE was and is an alien project which never resonated well with the Python community and its possible users. Actually up to now I don’t no anyone who has ever used it ( besides myself, of course ) and I wouldn’t wonder if this isn’t going to change in the future. For a Python programmer there are numerous alternatives now like python4ply, MetaPython and also 2to3 – why not? – which can be used to extend Python. None of them were available when I started with EE in 2006. Some people might also attempt to revive Logix which is among the more famous “dead” projects in the Python community. Logix might be in style and ambition precisely what Python users are looking for. EasyExtend isn’t even tangential.
Whenever I thought EE becomes stable I challenged it with bigger, more difficult problems: simultaneous transformations of multiple langlets, context sensitive languages, quirky real world grammars, online syntax definitions, source directed transformations, more expressible grammar syntax, language agnosticism etc.
Another major issue is performance. In the past I’ve used Psyco and also Cython. They boosted performance quite well and I got 3-5 times speedup for lexer+parser but I have clearly no performance model and I don’t see why those speedups shall be the limit? Python isn’t the right tool for the right job here and I suspect this had been an impediment for the current implementation already, since I overused containers like tuples and dicts in favor for classes and objects and their slow attribute access.
From EasyExtend to Langscape
The most likely path into the future of EasyExtend is to factor out components like the parser generator, the langlet transformer and most of the csttools and rewrite them in C++. I’ll probably start a completely new project which I intend to call “Langscape”. By means of SWIG it shall be possible to use Langscape also from environments like the JVM or the CLR. As a Python front end I’ll use the code I’ve developed for EasyExtend 4 which will probably never go public in the current form. I’ll still consider doing the functional testing in Python and I also want to preserve interactivity. Both the language front-end as well as back-end bindings become separated from Langscape. Langscape only deals with source code, grammars and CSTs.
Hi Kay! I also wonder if anyone has ever used my python4ply code. I’ve never gotten any feedback from users, and haven’t seen real mention of it in anywhere. That’s sadly just the way it goes, it seems.
Cheers! – Andrew
Apparently, we are acting in the wrong niche where people don’t accept a steep learning curve anymore.
For the near future I expect half baked language extensions and overly complicated implementations even in projects with more than one committer. Or people continue ranting about Python being so much more limited than Ruby and blame Guido for this and what not.
The larger part of me wants to be optimistic and say that isn’t so. Perhaps language tweaks aren’t that useful. I consider the stories I’ve heard about Smalltalk and Lisp instances so tweaked that only the author can use it, which makes me think there are advantages to having steep learning curves. But I wrote my long tutorial which went through the different aspects of how to tweak the language precisely to minimize the learning curve, and I complained about people using the term “internal DSL” for glorified API calls when it’s so easy to reach into the realm of real DSLs. Though easy here means for someone who has had CS training…
Perhaps the problem is that what I’ve done is more a technology demonstration with insufficient practical use? Would something like branch coverage or type checking, or a 2-to-3 converter with run-time checking of incompatibilities be more of a draw? OTOH, that’s a lot of work. 😉
Best regards – Andrew
One very cute example of a an external DSL which is otherwise well integrated with Python are shell-script commands in IPython. I’m not 100% sure but this seems to be limited towards interactive shell commands and one can’t write functions mixing shell script and Python code and load them from modules into IPython. This approach to DSLs coming from interactive programming is also one I’d like to emphasize. It is very simple to intercept a single line of code in a shell, parse it and transform it according to some special syntax but once we want fully embedded expressions / statements it will become more difficult.
I mention IPython here because I don’t think we have a solution seeking for a problem.
In case of EasyExtend all extensions can be rectified to Python code and I mean source code. Although it is a fully syntactical transformation it isn’t entirely different from a simple text substitution. In an EE aware shell with the -p option being set the result of an expr/stmt transformation can be displayed before the value accordingly is printed to stdout. It becomes difficult to write obfuscated code no one can understand who is a knowledgeable Python programmer. Same with complete modules. Transformations are transparent and in no way I do think learning barriers are needed to prevent weak programmers from using it.
Web frameworks are socially accepted and no says CS PhD are required to master Rails or Django. Debugging syntax can be a tedium but using language transformation systems isn’t more difficult than writing web apps. I’d rather claim the opposite is true because there are less environmental conditions that influence success or failure.