Thursday, February 12, 2009

Extending JDT Part I

In my last post, I described some limitations of JDT, and how it is Java-centric and does not support other Java-like languages such as AspectJ, Scala, and Groovy that compile to Java byte code. In this post, I will show you how the AJDT project has made JDT more extensible by using AspectJ to weave into the Eclipse framework in a way that is elegant, generic, and Eclipse-friendly.

I will start by describing a particular limitation of JDT: that Java-like languages are not able to plug in their own variants of the ICompilationUnit class. Then I will describe the how AJDT uses AspectJ and some factory methods to produce custom ICompilationUnits for AspectJ files. Finally, I will describe how we packaged up this functionality in an extension point so in a way that is generic and consumable by other development tools.

ICompilationUnit


An ICompilationUnit "represents an entire Java compilation unit", according to its JavaDocs. It is part of the IJavaElement hierarchy. An IJavaElement supplies a "common protocol for all elements provided by the Java model." The last lines of both JavaDocs for these elements are crucial: "@noimplement This interface is not intended to be implemented by clients." ICompilationUnits are starting point for much of JDT's functionality, including refactoring, content assist, indexing, and eager parsing.

This leaves us in a quandry: Java-like languages must plug into the Java model through creating custom ICompilationUnits if they are to be compatible with JDT, but ICompilationUnit is not allowed to be sub-classed.

Well, just because the JavaDoc says @noimplement, doesn't mean that we can't implement. It really just means that we do so at our own risk, and this is something that AJDT has been doing for years in the AJCompilationUnit class. The problem is not the difficulty of implementing ICompilationUnit.

The Real Problem



The real problem is the lack of control over instantiation of ICompilationUnits. ICompilationUnits are instantiated deep within the framework, in ways that are opaque to third party plugins. What we really want is to have some kind of factory that provides the correct kind of ICompilationUnit for each file. For our purpose, it is sufficient that *.aj files correspond to AJCompiltionUnit and *.java files correspond to CompilationUnit.

This sounds like a job for AspectJ! AspectJ allows us to intercept creations of CompilationUnit objects and determine if a different kind of object should be created instead.

The Solution



As a first pass, we wrote this code:


pointcut compilationUnitCreations(PackageFragment parent, String name, WorkingCopyOwner owner) :
call(public CompilationUnit.new(PackageFragment, String, WorkingCopyOwner)) &&
within(org.eclipse.jdt..*) &&
args(parent, name, owner);

CompilationUnit around(PackageFragment parent, String name, WorkingCopyOwner owner) :
compilationUnitCreations(parent, name, owner) {

String extension = findExtension(name);
if (extension.equals(".aj") {
return new AJCompilationUnitProvider().create(parent, name, owner);
}
return proceed(parent, name, owner);
}



Don't worry if you are unfamiliar with AspectJ, what this snippet does is fairly straight forward. The compilationUnitCreations pointcut identifies a set of points in the execution of the program. In this case, compilationUnitCreations identifies all locations where CompilationUnit objects are constructed. Beneath that, is an around advice declaration. It describes what to do at the pointcut instead of creating a CompilationUnit object. If the file extension is *.aj, then an AJCompilationUnit object is created. Otherwise, a standard CompilationUnit is created.

This is very nice. Using this AspectJ code, the creation of CompilationUnits has been delegated to the aspect, which can now inject AspectJ elements into the Java model. This opens up a world of functionality to AJDT that had been closed. For example, with this simple aspect, the renaming, moving, organize imports, and other kinds of code clean up and refactorings just work. No more ugly exceptions when you try to do these kinds of things.

However, there are still a couple of considerations with this implementation:


  1. It is now our responsibility (i.e., AJDT's) to ensure that the pointcut matches through future versions. For example, a later version of Eclipse may add an argument to the CompilationUnit constructor. AJDT needs to stay on top of this. Not a problem. Unit tests are your friends. We have tests for each of our aspects that ensure they continue to provide the expected functionality as Eclipse evolves. This is no different from the rest of our test suite which helps ensure that AJDT's use of internal Eclipse APIs doesn't break in new versions of Eclipse.


  2. This implementation works for AJDT, but what about other Java-like languages that require the same kind of JDT integration? For this, we can utilize one of the Eclipse platform's strengths$mdash;its extensible plug-in architecture.



Our second (and current) implementation of this aspect is as follows:


pointcut compilationUnitCreations(PackageFragment parent, String name, WorkingCopyOwner owner) :
call(public CompilationUnit.new(PackageFragment, String, WorkingCopyOwner)) &&
within(org.eclipse.jdt..*) &&
args(parent, name, owner);

CompilationUnit around(PackageFragment parent, String name, WorkingCopyOwner owner) :
compilationUnitCreations(parent, name, owner) {

if (inWeavableProject(parent)) {
String extension = findExtension(name);
ICompilationUnitProvider provider =
CompilationUnitProviderRegistry.getInstance().getProvider(extension);
if (provider != null) {
try {
return provider.create(parent, name, owner);
} catch (Throwable t) {
JDTWeavingPlugin.logException(t);
}
}
return proceed(parent, name, owner);
}


The AspectJ part of the snippet is the same as before. The difference is in the advice body. Instead of calling the AJCompilationUnitProvider factory method directly, there is a call to the CompilationUnitProviderRegistry, which has a mapping from file extensions to ICompilationUnitProviders. A plugin can register its own ICompilationUnitProvider by using the org.eclipse.contribution.weaving.jdt.cuprovider extension point. It looks like this:




One benefit of this implementation is that it extends JDT behavior in a way that uses common Eclipse mechanisms—the extension point. This allows other plug-ins to extend in a well-defined and structured manner. This is the approach that the Scala development tools has chosen to follow, and other language development tools are on the way.

A second of the benefit of this approach is that the consumer of the org.eclipse.contribution.weaving.jdt.cuprovider extension point does not need to be aware of the underlying aspect-oriented implementation. From the consumer's point of view, this extension point can be extended just like any other extension point. No understanding of AspectJ or AOP is required to actually use this extension point.

This is how it is possible to make JDT extensible to other languages in a simple and structured way. However, experienced AspectJ programmers may be confused at this point. Eclipse is built on an OSGi framework. Compile time weaving is not possible because JDT is already installed on a user's machine by the time AJDT is installed. And load time weaving is not possible because it cannot handle an OSGi environment. In a future post, I will describe the Eclipse Weaving Service, which is built on top of Equinox Aspects, that allows plugins to weave into Eclipse.

7 comments:

  1. Great article !
    How about debug ?
    Could the same technique used to extend jdt debugger ?

    ReplyDelete
  2. Yes. It can. We are currently extending the debugger in one place, to let language development tools plug in their own icons for choosing main classes. Other locations are certainly possible.

    Do you have any specific ideas in mind?

    ReplyDelete
  3. Wouldn't it have been more simple to file a bug against JDT to make ICompilationUnit public?

    ReplyDelete
  4. Aaron,

    In an ideal world, yes. Instead we get:

    https://bugs.eclipse.org/bugs/show_bug.cgi?id=36939

    Ismael

    ReplyDelete
  5. Thanks for beating me to it Ismael.

    AJDT has been pushing to get bug 36939 fixed for years. I have submitted several patches for it, and so have previous AJDT developers. It also has 37 votes! So, a lot of people wanted this done.

    Based on this, we decided to solve the problem ourselves.

    I don't blame the JDT team for not implementing the fix because the problem that the bug addresses is really outside the scope of the JDT project (ie- non-Java support). Although, you can argue that perhaps JDT's scope *should* change now that Java-like languages are becoming more popular on the JVM.

    ReplyDelete
  6. Andrew,

    Thanks for the response.

    > Do you have any specific ideas in mind?

    I think to mixed-debugging for languages who use java services or extend java language :)

    ReplyDelete
  7. A lot of this is already possible without Eclipse extensions. JSR 45 http://jcp.org/en/jsr/detail?id=45 provides a mechanism for debugging support in other languages that compile to Java byte code.

    There is a line mapping file that maps byte code instructions to line numbers. Debuggers (such as Eclipse's and most other modern debuggers) are able to leverage the mapping file to figure out where the debug arrow should be pointing to.

    That's the first step. The second step is to ensure all the variable names are properly mapped.

    This should be enough for the most basic debugging like stepping through code and inspecting variables.

    Eclipse provides standard extension points for breakpoints, which should be recognized in Java like languages.

    But, niceties like using the display view in your own language and getting the icons right will require weaving into JDT.

    ReplyDelete