Thursday, December 24, 2009

Extending Groovy-Eclipse for use with Domain-Specific Languages

One of the great things about Groovy is how easy it is to use the language to create domain specific languages (DSLs). And one of the great things about Eclipse is extensibility through its plugin architecture. Within Groovy-Eclipse, we are using the second to leverage the first. We have created a set of Eclipse extension points that allow Groovy programmers to create tool support for their own custom Groovy DSL. In this post, I will walk you through the extension points we created and show you how we use them to implement Grails support in STS.

We provide three extension points, one for adding new smarts to the inferencing engine: one for adding new content assist proposals, and one for adding new highlighting rules to the editor.

Extending Syntax Highlighting



The simplest way to extend Groovy-Eclipse is through the org.codehaus.groovy.eclipse.ui.syntaxHighlightingExtension extension point.

Here is what STS does to use the extension point:

<extension
point="org.codehaus.groovy.eclipse.ui.syntaxHighlightingExtension">
<highlightingExtender
extender="com.springsource.sts.grails.editor.groovy.GrailsSyntaxHighlighting"
natureID="com.springsource.sts.grails.core.nature">
</highlightingExtender>
</extension>


As you can see, an extender class (com.springsource.sts.grails.editor.groovy.GrailsSyntaxHighlighting) is associated with a project nature (com.springsource.sts.grails.core.nature). And now, whenever a Groovy Editor is opened for a Grails project, all of the syntax highlighting rules from the extender class is added to the Groovy Editor. For Grails, the extension is simple:

public class GrailsSyntaxHighlighting implements IHighlightingExtender {

public List getAdditionalGJDKKeywords() {
return Arrays.asList(
// domain fields
"constraints", "belongsTo", "hasMany", "nullable", "belongsTo", "mapping",
"hasMany", "embedded", "transients", "id", "tablePerHierarchy", "version",
// domain methods
"list", "save", "delete", "get",
// controller fields
"log", "actionName", "actionUri", "controllerName", "controllerUri",
"flash", "log", "params", "request", "response", "session",
"servletContext",
// controller methods
"render", "redirect"
);
}

public List getAdditionalRules() {
return null;
}
public List getAdditionalGroovyKeywords() {
return null;
}

}


New GJDK keywords are added and nothing else. Also, note that GrailsSyntaxHighlighting implements IHighlightingExtender. And here is what the additional syntax highlighting can give you:

syntax_highlighting

Notice that the special Grails domain class fields such as belongsTo and mapping are highlighted, and below in the controller class, keywords like params and render are highlighted.

Extending the inferencing engine



The Groovy-Eclipse inferencing engine is used to infer the types of expressions within a Groovy file. Because of its dynamic nature, determining the types of all Groovy expressions in a file is undecidable. The good news is that most programs are well-behaved and follow a simple set of rules through which we can infer the types of most expressions. Meta-programming in Groovy can add new members to Groovy objects and classes. This is a feature used by most Groovy DSLs.

Groovy-Eclipse allows DSL programmers to specify the meta-programming through the org.eclipse.jdt.groovy.core.typeLookup extension point. Here is what the extension point looks like in the Grails tool support in STS:

<extension
point="org.eclipse.jdt.groovy.core.typeLookup">
<lookup lookup="com.springsource.sts.grails.editor.groovy.types.GrailsTypeLookup">
<appliesTo projectNature="com.springsource.sts.grails.core.nature"/>
</lookup>
</extension>


Here the class com.springsource.sts.grails.editor.groovy.types.GrailsTypeLookup is defined to be a type lookup for projects that have the com.springsource.sts.grails.core.nature (i.e., this lookup is only activated for Grails projects).

Let's take a look at the GrailsTypeLookup class:

public class GrailsTypeLookup extends AbstractSimplifiedTypeLookup implements ITypeLookup {

private IGrailsElement element;
private GrailsProject gp;

public void initialize(GroovyCompilationUnit unit,
VariableScope topLevelScope) {
gp = GrailsCore.get().getGrailsProjectFor(unit);
if (gp != null) {
element = gp.getGrailsElement(unit);
element.initializeTypeLookup(topLevelScope);
}
}

@Override
protected TypeAndDeclaration lookupTypeAndDeclaration(
ClassNode declaringType, String name, VariableScope scope) {
IGrailsElement declaringElt = gp.getGrailsElement(declaringType);
return declaringElt.lookupTypeAndDeclaration(declaringType, name, scope);
}
}


According to the extension point specification, GrailsTypeLookup must extend ITypeLookup and we choose to let it extend AbstractSimplifiedTypeLookup in order to reduce the amount of coding required.

The initialize method is called when type inferencing is starting for a Groovy file. Here, it is possible to stuff things into the top level scope (such as global variables). For Grails, we determine what kind of Grails element we are performing inference on (e.g., a domain class, controller class, taglib, etc) and modify the top level scope appropriately.

More magic happens in the lookupTypeAndDeclaration method. Again, the type lookup delegates to the specific Grails element to determine what the type is.

Now, let's take a look at what this can do for us. Notice that hovering over Grails keywords will bring up a JavaDoc of the inferred type of that keyword:

hovers_in_controller

The Grails response field in controller classes is of type HttpServletResponse. Similarly, this allows us to get HttpServletResponse aware content assist proposals:

content_assist1

Extending Content Assist



The final step is to hook extensible content assist into the DSL. This can be tricky. For example, in Grails, there are certain fields that if defined have special meaning. There is the constraints field where constraints for domain classes are defined, and the mapping field where object-relational mappings are defined. The closure attached to each of these fields have special keywords that they expect.

This is possible to control through the org.codehaus.groovy.eclipse.codeassist.completion.completionProposalProvider extension point. Here is how it is used in STS:
<extension
point="org.codehaus.groovy.eclipse.codeassist.completion.completionProposalProvider">
<proposalProvider
proposalProvider=
"com.springsource.sts.grails.editor.groovy.contentassist.GrailsProposalProvider">
<appliesTo projectNature="com.springsource.sts.grails.core.nature"/>
</proposalProvider>
</extension>


This extension point wires a com.springsource.sts.grails.editor.groovy.contentassist.GrailsProposalProvider to the Grails project nature. And so (as with the other extension points), theis extra content logic will only occur when inside a Grails project.

com.springsource.sts.grails.editor.groovy.contentassist.GrailsProposalProvider implements org.codehaus.groovy.eclipse.codeassist.processors.IProposalProvider. This interface has three methods to implement:


  • getNewFieldProposals: Return a list of fields that can be defined at the content assist location. For example, here is where all special fields available in Grails domain classes are proposed. This method is only called when content assist is invoked when inside a class body (i.e., only where it is appropriate to define new fields).

  • getNewMethodProposals: Return a list of new methods that can be defined at the invocation location. As with the new fields method, this method is only called when content assist is invoked in a location that is possible to define new methods.

  • getStatementAndExpressionProposals: This method returns all possible special content assist proposals when in the context of an expression or statement. For example, here is where special controller class fields like params, request, and response are inserted.



Let's take a look at what this can do. In this screenshot, you can see that when performing content assist on a reference to a Grails domain class, you can access Grails specific methods like count:

extra_content_assist

And when inside the constraints block (and only when within that block) content assist is augmented with possible constraints to add:

content_assist_constraints

Conclusion



We have worked hard to make sure that Groovy-Eclipse is extensible. It is already being used by some DSLs such as EasyB and by BonitaSoft.

These extension points and APIs are still a work in progress if you have any questions, or require some changes to anything, please raise a bug or send a message to the mailing list.

Monday, December 14, 2009

Getting GMaven to play nicely with Groovy-Eclipse

There has been a lot of talk on the Groovy mailing list lately about how to get GMaven working with Groovy-Eclipse and coincidentally I saw my friend Mike last night who works at boats.com and he told me that his office has solved this problem. Here is what he told me:



Okay, the first things you will need in your POM file are the GMaven mojo and the GMaven runtime. The dependencies for the mojo seem to be a big buggered out of the box, so you will have to tweak the GMaven runtime dependencies. Here is how you do it:


<dependencies>
<dependency>
<groupId>org.codehaus.groovy.maven</groupId>
<artifactId>gmaven-mojo</artifactId>
<version>1.0</version>
<exclusions>
<exclusion>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all-minimal</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.groovy.maven.runtime</groupId>
<artifactId>gmaven-runtime-1.5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.codehaus.groovy.maven.runtime</groupId>
<artifactId>gmaven-runtime-1.6</artifactId>
<version>1.0</version>
</dependency>
</dependencies>


Next is your build configuration. Eclipse will need to be able to find your Groovy source files, so you will need to add your Groovy source as an explicit resource directory.

The mvn:eclipse plugin will also need to add the Groovy nature to your project.

You will also need to add directives for the GMaven plugin. This is where you will have to do a manual tweak: when using mvn to build your project, gmaven generates some stubs for your groovy code before the java code compiles, so that any java->groovy dependencies will be fulfilled. The Groovy code is then compiled and will overwrite the .class files from the previous step. This breaks things in eclipse, since if you attempt to execute code in your workbench, eclipse sees the stubs and somehow their .class files are what end up in your binary output directory.

Therefore, if you want to execute a gmaven project from Eclipse, you simply need to comment out the stub generation directive and delete any generated stub classes. This is what it looks like:


<build>
<resources>
<resource>
<directory>src/main/groovy</directory>
</resource>
</resources>
<plugins>
<plugin>
<artifactId>maven-eclipse-plugin</artifactId>
<configuration>
<additionalProjectnatures>
<projectnature>
org.eclipse.jdt.groovy.core.groovyNature
</projectnature>
</additionalProjectnatures>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.groovy.maven</groupId>
<artifactId>gmaven-plugin</artifactId>
<version>1.0</version>
<executions>
<execution>
<goals>
<!--
<goal>generateStubs</goal>
-->
<goal>compile</goal>
<!--
<goal>generateTestStubs</goal>
-->
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

The result is a groovy/java project that can be managed with maven but can be transparently developed (and more importantly, debugged!) in Eclipse.

There is one minor issue that has come up however: GMaven 1.6 stub generation is broken for enums: it generates public constructors, which do not compile. I submitted an issue to codehaus (GMAVEN-51), and apparently the workaround is to use the gmaven 1.7 runtime. I haven't tried it yet, since things are working okay for us thus far.

Mike


So, it does seem like there is a bit of a problem with stubs being recognized by Groovy-Eclipse when they should not be. The solution would be to generate the stubs in a directory that is not seen by Eclipse, rather than in the default output directory. But not knowing much about GMaven (or Maven for that matter) works, I don't know how feasible this is. Can anyone think of a better solution?

Friday, November 27, 2009

New cross-language refactoring support for Groovy-Eclipse plugin

Thanks to the work of Michael Klenk and his students at the Institute for Software in Switzerland, the Groovy-Eclipse plugin now has cross-language refactoring support for renaming. Its use is seamless. Let me give you an example.

Let's start with a simple Java file and a simple Groovy file:

before

You can rename the field in the Java class:

during1

This will perform the standard Java refactoring on all Java classes in your projects. Additionally, the refactoring will be performed on your groovy classes. If there are any ambiguous references to the Java field, the following dialog will open up allowing you to choose which references should be refactored:

choose

And the result is what you would expect:

after

The refactoring can be initiated from the Groovy side as well:

groovyrefactor_menu

As you can see, extract method an inline method are two other refactorings supported in Groovy code.

This functionality is currently available in the latest snapshot build of Groovy-Eclipse for Eclipse 3.5, available from the following update site:
   http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5/


Please send any questions or problems to the mailing list. We're happy to help. Enjoy!

Thursday, November 19, 2009

Getting AspectJ Pointcut matching timer information in AJDT

Andy Clement has just added pointcut match timing information to AspectJ. A number of people have been asking to have this included in AJDT, and so I did a little work and made it available in the latest dev builds of AJDT for 3.4 and 3.5.

Here's how to display timing information:


  1. Select Verbose and Pointcut matching timers from the AspectJ compiler preferences page (this can be either globally from Eclipse preferences or for a specific project from the project properties page):

    compiler prefs

  2. In the AJDT Event Trace View, ensure that you have Compiler / Task list messages selected:

    event trace



And now on every project build (incremental or full), you will see pointcut matching times spit out to the event trace:

output

Enjoy! Questions can be sent to the AJDT developer's mailing list.

Monday, November 16, 2009

How type inferencing for Groovy in Eclipse is leading to all sorts of Groovy goodness

Of all of the recent improvements of Groovy-Eclipse since M1 has been released, the groovy type inferencing engine is the most pervasive and powerful. This inferencing engine is able to take a groovy file or code snippet and infer the type of any or all of its expressions. It now forms the core of several of the most prominent features of Groovy-Eclipse. Everything that I describe here is currently available in the latest snapshot release of Groovy-Eclipse for Eclipse 3.5 and the update site is here:
http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5


In this post, I am going to describe how the inferencing engine is currently used in Groovy-Eclipse and then I will dive a deep and describe how it is implemented. I'll end the post by describing our plans for immediate the future.

What type inferencing is used for



Underlining statically indeterminate references


The Groovy editor now underlines all references that cannot be determined while editing. This happens in a background thread while you type. So, the following code (notice the mispelling of asImmutable):


def aList = []
def otherList = aList.asImmutabl()
otherList.asImmutable()



...will appear in the editor like this...

underline

...and when the typo is fixed, the underlines go away...

nounderline

Code is re-analyzed after every keystroke (well, actually, it is after every keystroke that is followed by 500ms of no keystrokes). The type of every reference in the file is looked up using the inferencing engine. Every reference that cannot be found is considered to be unknown and will be underlined.

Of course, Groovy being a dynamic language, not all references can be statically linked to a type (other than Object). This is especially the case when working with DSLs (such as Grails) that use metaprogramming to add new methods and fields to existing types. Later, I'll describe our plans for extensibility.

Search



Inferencing is used for Java Search as well. Using the search short cut (CTRL-Shift-G or CMD-Shift-G on Mac) or running search from the Java Search dialog, references from and to Groovy code snippets can be located. However, there are some difficulties here. Compared to Java, Groovy is quite a bit more flexible in the ways that fields and methods can be referenced. For example, this field is assigned a closure and referenced as if it were a method:

class Foo {
def x = { print it }
}

x("Hello!")


And this method is referenced in a way that is statically indistinguishable from a field:


class Foo {
def x() { print "Hello!" }
}

x


Furthermore, the spread operator and the use of default parameters makes the Java way of determining which method or field is referenced impossible. So, Groovy search is significantly more liberal than Java search in what it considers a search match. If a reference corresponds to any field, method, or property in the declaring type (regardless of parameter count and whether it is accessed as a field or method), then the match is considered successful. This leads to the following scenarios.

1. Performing search on the field declaration finds field references here:
searchForField

2. Performing search on the method declaration finds exactly the same references:
searchForMethod

And, notice that references attached to the wrong type are not considered a match.
searchNoResults

Content Assist



Most of the internals of Groovy-Eclipse has been rewritten since the previous 1.5.7 release. The most recent rewrite is of content assist. Although the previous version worked (usually), there was always a noticeable lag while waiting for the content assist window to appear. This previous implementation of content assist used a combination of Java classloading and Groovy AST matching in order to find all completion proposals available at a given location, which was both messy and slow. Now, we have hooked into the inferencing engine and have completely rewritten proposal generation with extensibility in mind. And we have made some remarkable improvements in performance.

Using the following code snippet I performed content assist 5 times under the old implementation and the new and took the average of the last 3 of each:


def x
x.hasProperty("foo").





Old implementation2315 milliseconds
New implementation8 milliseconds


That's right...there is a 3 orders of magnitude speed up under the new implementation, making content assist a much more pleasant feature to use. But please note that this is just the time for calculation of the proposals and it does not include the time required to actually pop up the window.

Open declaration


The implementation of open declaration (or code select as it is called in JDT because it is a general mechanism to determine what the current selection refers to and is also used for things like calculating hovers) also uses inferencing. Pressing F3 whenever the selection is any form of "navigateToMe" in the following code, the expected declaration will open:

navigateBefore ---> navigateAfter

Although Groovy-Eclipse M1 used type inferencing for code select, it was using the inferencing from content assist and was subject to the same mistakes, duplicate answers, and slowness that content assist had.

How it works


The inferencing engine has unified the implementation of several of Groovy-Eclipse's most widely used features. With this unification, we have been able to significantly reduce the amount of duplicated code, improve performance, and plan for extensibility.

There are four interacting components of the inferencing engine:


  1. The Abstract Syntax Tree (or AST): this is an abstract representation of the syntax of a groovy file expressed hierarchically. The code to work on can be obtained either from the disk, or it can be the latest in memory. The AST is generated by the Groovy compiler based on the current state of the editor whenever Eclipse is otherwise idle.

  2. The type inferencing visitor: this visitor walks the AST (i.e., uses a visitor pattern to visit every AST node) and delegates to the type lookups and the requestor described below. A new visitor is created for each call to the inferencing engine.

  3. The type lookups: these objects lookup the type information of a particular expression using whatever mechanism they require. They can store and retrieve information about types in a way that all other lookups can use. In this way, each lookup can do what it can and coordinate with other lookups where required. Several default lookups are provided including a simple lookup, an inferencing lookup (that stores information about types via assignment expressions), and a category lookup (that looks up types in Groovy categories). Type lookups are created from a registry on each call to the inferencing engine.

  4. The requestor: this is the object that collects the type information determined by the lookups. The requestor is general and can be used as a way to store the types of all expressions, lookup the type of a particular expression, or look for expressions with unknown types (just to name a few uses). The requestor is created by client code and past to the visitor to start the inferencing process.



The following diagram shows process and how the different pieces work together:

arch


  1. The visitor visits an AST expression node

  2. The visitor sends this expression to the type lookups.

    • each lookup responds with a type and its confidence in that type (ie, exact, potential, inferred...), or null, if it cannot find a type

    • a lookup may additionally store information in a scope object that is passed around to other lookups so that information can be shared between them


  3. The result of the lookup is sent to the requestor

  4. The requestor processes the result and may choose to end the visit or continue.

  5. If required, the visitor will continue to visit the rest of the abstract syntax of the Groovy file



What's next


One of the core design goals of the inferencing engine is extensibility. The inferencing engine must be extensible in two ways. First, it must be usable to help support new features such as quick fixes and refactoring. Also, DSLs, (most notably Grails) must be able to extend inferencing with their own type lookups. It should be apparent through the description of the variety of ways that the engine is currently used that it versatile and can become the core implementation of any number of new Groovy-Eclipse features. As for DSL extensibility, that has not yet been implemented, but the stubs are available. Other DSLs will need to implement their own type lookup, but there is currently no way for these new lookups to be plugged in. Fortunately, it will be possible to use Eclipse's plugin architecture here and this will be the subject of my next blog post (after it gets implemented).

Saturday, October 3, 2009

AJDT 2.0.1 is released!

Overview

In this service release of AJDT, we have included a number of performance enhancements, as well as a few new features. This is the last scheduled release of the AJDT stream targeting 3.4. The next release, in early 2010, will be targeting the 3.5 stream only. And our second 2010 release will target 3.5. At the same time, we will also release an early development version of AJDT targeting Eclipse 3.6 in time for EclipseCon 2010.

AJDT is available from the following update sites for Eclipse 3.5 and 3.4 respectively:
http://download.eclipse.org/tools/ajdt/35/update
http://download.eclipse.org/tools/ajdt/34/update

Export Feature With AspectJ

A new wizard is available to export features with AspectJ support. This wizard can be initiated from the File -> Export... command:



This wizard supports all of the options that the standard export feature supports. The only difference is that the AspectJ compiler is used to compile the projects instead of the Java compiler:


AJDT support for passing aop.xml file

AJDT now provides UI support to pass aop.xml files to the compiler. AspectJ bug 124460 describes how the AspectJ compiler can use aop.xml files for compile time weaving. And now AJDT provides UI support in the following way.

In project properties of your AspectJ project, select aop.xml management. Here you add and remove aop.xml files that will be sent to the compiler. Note that these files are highlighted using a special icon. Also, note that the name aop.xml is optional. Any xml file can be sent to the compiler in this way. See the following screenshot for an example


In order to use aop.xml files for compile-time configuring of the weaver, you must add the -xmlConfigured option under Non-Standard Compiler Options in the project's AspectJ Compiler options. This work is described in bug 287459.

This work is still experimental. We hope to receive some feedback so that compile time XML configuration of the weaver can be improved. Please send all comments to theAspectJ mailing list, or to bugzilla.

AspectJ 1.6.6

AJDT 2.0.1 contains AspectJ 1.6.6 and this version is available for download separately. See the release notes here.

Bug Fixes

Thirty-four bugs have been addressed in this release. See Bugzilla for the full report.

Thanks!

Thanks to those of you who have submitted patches, bug reports, and contributed to the mailing list for this release. We appreciate your help. Specifically, in this release:

Previous Releases

See here for information on the new features in AJDT 2.0.0.

Tuesday, September 29, 2009

Groovy-Eclipse plugin now supports Eclipse 3.5.1

The Groovy-Eclipse plugin can now be installed into Eclipse 3.5.1. As always, the update site is here:

http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5/

You can stop reading here if you don't care how I got this to work, or why I am a little bit surprised by this.

The Groovy-Eclipse plugin uses a feature patch on the JDT core bundle in order to achieve a high level of integration with JDT. Typically, feature patches can only apply to a specific version of a feature (e.g., 3.5.0.v20090527-2000-7r88FEeFJePyvYeA33DjZ_c1 or some ugliness). This means that Groovy-Eclipse, until recently could only install in a specific version of Eclipse, the one that ships the JDT feature version 3.5.0.v20090527-2000-7r88FEeFJePyvYeA33DjZ_c1. However, Andrew Niefer describes how to get around this by editing the content.xml file to widen the range that the patch applies to. Excellent stuff, and lucky for me, because I didn't want to branch the Groovy-Eclipse code base every time a new service release of Eclipse comes out.

Unfortunately, his instructions were not entirely accurate. Andrew mentions that the range attribute in the <patchscope> element needs to be widened in order allow the patch to be installable on multiple versions. I tried exactly what he suggested, but could not get my patch to install in both 3.5.0 and 3.5.1. After a little bit of exploration, I found that I also needed to change the range attribute in the <lifecycle> element.

All this meant was adding a single line to my ant script, to be executed after the p2 repository is generated:

    
<replace file="${updateSiteDir}/content.xml"
summary="yes" token="${orig.jdt.feature.version.range}"
value="${new.jdt.feature.version.range}"/>

And this turned out to be very simple. Thanks for the hint!

Thursday, June 25, 2009

AJDT 2.0.0 is released!

We have just released AJDT 2.0.0 for Eclipse 3.5 and 3.4.

Get it now from the update sites:
For Eclipse 3.5: http://download.eclipse.org/tools/ajdt/35/update
For Eclipse 3.4: http://download.eclipse.org/tools/ajdt/34/update

In addition to a new Push In refactoring and ITD-aware navigation and Javadocs, you can find out about what's included in this release at our New & Noteworthy page.

AJDT 2.0.0 will also be available in the upcoming SpringSource Tool Suite 2.1.0.

This release would not have been possible with out help from the AspectJ and AJDT community. So, thank you for your bug reports, patches, discussions and being warm and welcoming to newcomers.

Wednesday, June 3, 2009

Update on Greclipse 2.0 (Groovy support in Eclipse)

Today, I decided to run EclEmma on the Groovy-Eclipse plugin test suite. Shockingly, I found that JUnit support, content assist, and code navigation are completely test-fress. So, I took a stab at creating a solid and flexible test infrastructure that makes it easy to write plugin tests, especially ones that exercise the Java model.

How did I do this? I did it by hooking into existing JDT functionality, just like I have been doing for the rest of my work on Greclipse. I downloaded the JDT core tests from CVS and used what I needed. Now, I can write extremely concise tests that exercise the code I want it to. For example, I wrote the following test case that tests code selection on a closure in a groovy script (i.e., this is the code that is executed when you perform Open Declaration in an editor):


public void testCodeSelectClosure() throws Exception {
IPath projectPath = createGenericProject();
IPath root = projectPath.append("src");
String contents =
"def x = {\n"+
"t -> print t\n"+
"}\n"+
"x('hello')\n";
env.addGroovyClass(root, "Script", contents);
GroovyCompilationUnit unit = getGroovyCompilationUnit(root, "Script.groovy");
IJavaElement[] elts = unit.codeSelect(contents.lastIndexOf('x'), 1);
assertEquals("Should have found a selection", 1, elts.length);
assertEquals("Should have found local variable 'x'", "x", elts[0].getElementName());
}


I have written many tests for JUnit support (or is that GUnit?), and some for code selection. I still need to work on content assist. So, I can now be confident that as we make changes towards the 2.0 release, we are not sliding back in functionality. There is a lot of good work that has already been put into the plugin (including groovy-aware type inference for content assist) and we need to make sure that this does not regress as we go forward.

Wednesday, May 27, 2009

Push in refactoring for AJDT

I just finished implementing the new Push In refactoring for the AJDT plugin, available in the latest dev build.

Essentially, the refactoring pushes intertype declarations into their target types. So, when a push in refactoring is applied to the following aspect and class:


aspect Foo {
int Bar.x;
declare parents : Bar implements I;
declare @type : Bar : @Baz;
}

class Bar { }


the result is:


@Baz
class Bar implements I {
int x;
}


The Foo aspect is deleted because it is empty.

It is simple to use. Select the ITDs or set of ITDs that you want to push in. You can select an entire project, source folder, or package to push all of its ITDs into the target types:



And then follow the wizard:



...and...




The result is that all ITDs have been successfully pushed in and all aspects that are made empty because of this refactoring have been deleted.



And undo works, too:



It is also possible to execute the refactoring from an editor. Just select all or part of the ITD you want to push in. However, the command itself is in a different sub-menu:




Please try it out and let me know if it works for you. We are working on adding more refactorings to AJDT. Next on the list is the analogous refactoring pull-out.

Sunday, May 17, 2009

Is it Groovy or is it Java? Do we really care?

Well, I care, but the editor doesn't. Following up on the work that Andy Clement has done to modify the JDT compiler so that it can process Groovy as well as Java, I've spent the last few weeks working on editor support for Groovy in Eclipse. Yes, there is already an Eclipse plugin for Groovy, but this particular plugin is tied to a split compiler (JDT for Java files and Groovy compiler for Groovy files). And I wanted to see if we could use the joint compiler to drive Groovy tool support.

So far, with minimal changes it seems that it is possible. I've been able to implement much of the standard Java editing support for Groovy files (e.g., open declaration, search, syntax highlighting, content assist, etc) as long as the editor is passed the right kind of structure.

Let's recap what Andy has done so far...
After getting a little intro to groovy joint compilation (and groovy compiler structure) from Jochen Theodorou, I wondered how hard it would be to go about things slightly differently. Let the Eclipse JDT compiler take control of building a mixed code base but whenever it needed to deal with groovy - call the groovy compiler. If JDT could take control I could see all sorts of things 'just working' in Eclipse, problems upon which we are still expending large amounts of efforts to try and solve for AspectJ. If I could just plug groovy type reference resolution and classfile generation into JDT correctly then references between java and groovy artifacts would work and JDT would ensure incremental compilation worked, even across restarts of eclipse (this latter problem still hasn't been solved for AspectJ).
So originally, this worked great for joint compilation, but as soon as you try to open this in an editor, things just don't behave correctly...



The outline view is out of order, problems and errors don't show up where you expect them to, and source code navigation is broken.

After a bit of mucking around, I found that there is a single root to this problem: source locations. The JDT tooling and the editor is very specific about the source locations that it requires. JDT requires the start and end locations of identifiers, method and type bodies, and all declarations. However, the Groovy compiler (which builds its AST from antlr), only provides line and column information for AST nodes. Although it is possible to translate from line/column to file offset, offsets for identifiers are lost.

And, with a little bit of hacking, I was able to recreate source offsets for field, method, and class names and declarations. It turns out that this is enough to spring to like much of your favorite editor functionality:



Same program, same error as before, but now the outline view is correct, the error marker is in the proper location,* "Open Declaration" and searches navigate to the proper location. And even source hovers work:



So, let's take a deeper look at how this works.

The JDT uses three representations of abstract syntax of a Java file. There is the compiler AST, which is used to generate byte code and perform analysis. There is DOM (document object model) AST, which is used for refactoring and source code manipulation. And, there is the model (the IJavaElement hierarchy), which is more abstract, and is used for navigation and in places like the Package Explorer and Outline views.

There is also support to translate from the compiler AST to either of the other two ASTs. Earlier, I described (or rather Andy described) the way JDT can be used to drive the joint compilation of Groovy source code. The result is a JDT compiler AST with (crucially) the right source code offsets. This, when translated to the right kind of AST at the right time, can be fed to the different parts of JDT that require it. And, thereby, things just work.

A few caveats, of course. Right now, we are only generating a minimal compiler AST (class, method and field declarations only---method bodies are ignored for now). This, however, does seem to be enough for this basic functionality. Also, there are some major differences between Groovy and Java syntax (closures, regular expressions, and lists operations). We are currently sidestepping this problem by ignoring method bodies, statements and expressions.

At some point, however, we are going to have to tackle these loose ends. We will need to be more complete about our ASTs and fill in method bodies (this part is *just* lots of coding). And also, we will need to figure out how to translate Groovy-specific syntax into JDT AST, probably by sub-classing the existing API (this will require some thorough design work *and* lots of coding). So, there is still a lot of work to be done.

What does this mean? Well for one thing it means that the implementation of the Groovy editor is just this:


public class GroovyEditor extends CompilationUnitEditor {
private GroovyImageDecorator decorator = new GroovyImageDecorator();

@Override
public Image getTitleImage() {
return decorator.decorateImage(null, getEditorInput().getAdapter(IFile.class));
}
}

Yes, it is just a minimal subclass of the standard CompilationUnitEditor that merely ensures that the icon in the title bar indicates Groovy, not Java.

So, who cares if it's Groovy or Java? Certainly not the editor.

* There are currently Groovy compiler limitations with error reporting in that error locations only include the character or two of the error.

Tuesday, March 17, 2009

Aspects Everywhere: Using Equinox Aspects to Provide Language Developers with Deep Eclipse Integration...

...and other EclipseCon talks.

To shamelessly self promote, I wanted to mention my EclispeCon talk entitled Aspects Everywhere: Using Equinox Aspects to Provide Language Developers with Deep Eclipse Integration. It is about how we (the AJDT team) have used AspectJ and Equinox Aspects to overcome the limitations of JDT and make it more extensible to third parties.

You may want to check this out if:
  • You use AspectJ or AJDT
  • You are interested in using Equinox Aspects and want to find out how it can be used
  • You are a language developer who wants to integrate tooling support into Eclipse and JDT
No knowledge of AspectJ is required for this talk.

And if you are interested in anything Spring or AspectJ, please join us for our Bof: Spring on Eclipse: AspectJ, STS, and SpringIDE.

Some other talks that I am planning on attending are:
There are so many excellent talks that it is difficult to choose what to go to. It is going to be a great conference.

Monday, March 2, 2009

AJDT 1.7 pre-release for Eclipse 3.5

In time for EclipseCon, we have made a pre-release of AJDT 1.7 for Eclipse 3.5 available to install. You can find it at the following update site:

  http://download.eclipse.org/tools/ajdt/35/dev/update


Currently, AJDT 1.7 is largely the same as AJDT 1.6, with the exception of the changes required so that it will run on Eclispe 3.5. The two versions will continue to be developed in unison until well after the final release of AJDT 1.7, which will coincide with the Galileo release.

Please report any issues you have here or on bugzilla. Enjoy!

Wednesday, February 18, 2009

Extending JDT Part II

In my previous post, I described how we have been able to use AspectJ to make JDT more friendly to other Java-like languages such as AspectJ itself and Scala. In this post, I am going to go into a little more detail as to how this happens. This post will be brief because there's really not much more to it.

(If you are unsure about what AspectJ and aspect-oriented programming are, there are plenty of places to learn about it. For an excellent long introduction to AspectJ and aspect-oriented programming, see Ramnivas's I want my AOP, and for an excellent two paragraph introduction to AspectJ see Adrian Colyer's AOP without the buzzwords).

We've implemented the Eclipse weaving service on top of Equinox Aspects (EA). Eclipse is highly dynamic and uses different classloaders for each plugin/bundle. For this reason, regular compile-time and load-time weaving will not work. We need something that can hook into the Equinox OSGi framework and perform weaving as bundles are loaded into the framework. This is exactly how EA works: EA provides a weaving hook that inspects each bundle as it is being loaded to determine if it requires any weaving. If the bundle does, it is sent off to the AspectJ load-time weaver. Actually, it is a little more fine-grained than this since classes are only sent to the weaver as they are loaded.

The benefit of using EA is that it helps us stay robust through different Eclipse versions because we are not shipping woven code or feature patches. Currently, the Eclipse weaving service is available for Eclipse 3.4, 3.4.1, 3.4.2, and (soon to be released) 3.5M5.

Thursday, February 12, 2009

Extending JDT Part I

In my last post, I described some limitations of JDT, and how it is Java-centric and does not support other Java-like languages such as AspectJ, Scala, and Groovy that compile to Java byte code. In this post, I will show you how the AJDT project has made JDT more extensible by using AspectJ to weave into the Eclipse framework in a way that is elegant, generic, and Eclipse-friendly.

I will start by describing a particular limitation of JDT: that Java-like languages are not able to plug in their own variants of the ICompilationUnit class. Then I will describe the how AJDT uses AspectJ and some factory methods to produce custom ICompilationUnits for AspectJ files. Finally, I will describe how we packaged up this functionality in an extension point so in a way that is generic and consumable by other development tools.

ICompilationUnit


An ICompilationUnit "represents an entire Java compilation unit", according to its JavaDocs. It is part of the IJavaElement hierarchy. An IJavaElement supplies a "common protocol for all elements provided by the Java model." The last lines of both JavaDocs for these elements are crucial: "@noimplement This interface is not intended to be implemented by clients." ICompilationUnits are starting point for much of JDT's functionality, including refactoring, content assist, indexing, and eager parsing.

This leaves us in a quandry: Java-like languages must plug into the Java model through creating custom ICompilationUnits if they are to be compatible with JDT, but ICompilationUnit is not allowed to be sub-classed.

Well, just because the JavaDoc says @noimplement, doesn't mean that we can't implement. It really just means that we do so at our own risk, and this is something that AJDT has been doing for years in the AJCompilationUnit class. The problem is not the difficulty of implementing ICompilationUnit.

The Real Problem



The real problem is the lack of control over instantiation of ICompilationUnits. ICompilationUnits are instantiated deep within the framework, in ways that are opaque to third party plugins. What we really want is to have some kind of factory that provides the correct kind of ICompilationUnit for each file. For our purpose, it is sufficient that *.aj files correspond to AJCompiltionUnit and *.java files correspond to CompilationUnit.

This sounds like a job for AspectJ! AspectJ allows us to intercept creations of CompilationUnit objects and determine if a different kind of object should be created instead.

The Solution



As a first pass, we wrote this code:


pointcut compilationUnitCreations(PackageFragment parent, String name, WorkingCopyOwner owner) :
call(public CompilationUnit.new(PackageFragment, String, WorkingCopyOwner)) &&
within(org.eclipse.jdt..*) &&
args(parent, name, owner);

CompilationUnit around(PackageFragment parent, String name, WorkingCopyOwner owner) :
compilationUnitCreations(parent, name, owner) {

String extension = findExtension(name);
if (extension.equals(".aj") {
return new AJCompilationUnitProvider().create(parent, name, owner);
}
return proceed(parent, name, owner);
}



Don't worry if you are unfamiliar with AspectJ, what this snippet does is fairly straight forward. The compilationUnitCreations pointcut identifies a set of points in the execution of the program. In this case, compilationUnitCreations identifies all locations where CompilationUnit objects are constructed. Beneath that, is an around advice declaration. It describes what to do at the pointcut instead of creating a CompilationUnit object. If the file extension is *.aj, then an AJCompilationUnit object is created. Otherwise, a standard CompilationUnit is created.

This is very nice. Using this AspectJ code, the creation of CompilationUnits has been delegated to the aspect, which can now inject AspectJ elements into the Java model. This opens up a world of functionality to AJDT that had been closed. For example, with this simple aspect, the renaming, moving, organize imports, and other kinds of code clean up and refactorings just work. No more ugly exceptions when you try to do these kinds of things.

However, there are still a couple of considerations with this implementation:


  1. It is now our responsibility (i.e., AJDT's) to ensure that the pointcut matches through future versions. For example, a later version of Eclipse may add an argument to the CompilationUnit constructor. AJDT needs to stay on top of this. Not a problem. Unit tests are your friends. We have tests for each of our aspects that ensure they continue to provide the expected functionality as Eclipse evolves. This is no different from the rest of our test suite which helps ensure that AJDT's use of internal Eclipse APIs doesn't break in new versions of Eclipse.


  2. This implementation works for AJDT, but what about other Java-like languages that require the same kind of JDT integration? For this, we can utilize one of the Eclipse platform's strengths$mdash;its extensible plug-in architecture.



Our second (and current) implementation of this aspect is as follows:


pointcut compilationUnitCreations(PackageFragment parent, String name, WorkingCopyOwner owner) :
call(public CompilationUnit.new(PackageFragment, String, WorkingCopyOwner)) &&
within(org.eclipse.jdt..*) &&
args(parent, name, owner);

CompilationUnit around(PackageFragment parent, String name, WorkingCopyOwner owner) :
compilationUnitCreations(parent, name, owner) {

if (inWeavableProject(parent)) {
String extension = findExtension(name);
ICompilationUnitProvider provider =
CompilationUnitProviderRegistry.getInstance().getProvider(extension);
if (provider != null) {
try {
return provider.create(parent, name, owner);
} catch (Throwable t) {
JDTWeavingPlugin.logException(t);
}
}
return proceed(parent, name, owner);
}


The AspectJ part of the snippet is the same as before. The difference is in the advice body. Instead of calling the AJCompilationUnitProvider factory method directly, there is a call to the CompilationUnitProviderRegistry, which has a mapping from file extensions to ICompilationUnitProviders. A plugin can register its own ICompilationUnitProvider by using the org.eclipse.contribution.weaving.jdt.cuprovider extension point. It looks like this:




One benefit of this implementation is that it extends JDT behavior in a way that uses common Eclipse mechanisms—the extension point. This allows other plug-ins to extend in a well-defined and structured manner. This is the approach that the Scala development tools has chosen to follow, and other language development tools are on the way.

A second of the benefit of this approach is that the consumer of the org.eclipse.contribution.weaving.jdt.cuprovider extension point does not need to be aware of the underlying aspect-oriented implementation. From the consumer's point of view, this extension point can be extended just like any other extension point. No understanding of AspectJ or AOP is required to actually use this extension point.

This is how it is possible to make JDT extensible to other languages in a simple and structured way. However, experienced AspectJ programmers may be confused at this point. Eclipse is built on an OSGi framework. Compile time weaving is not possible because JDT is already installed on a user's machine by the time AJDT is installed. And load time weaving is not possible because it cannot handle an OSGi environment. In a future post, I will describe the Eclipse Weaving Service, which is built on top of Equinox Aspects, that allows plugins to weave into Eclipse.

Wednesday, February 4, 2009

JDT won't do that!

AspectJ is a language that is very close to Java, but it isn't Java. One might expect that since AspectJ is an extension of Java, Eclipse tooling for AspectJ would just be an extension of the Java Development Tools. But it's not that easy. This post describes the problems that we have faced when integrating AspectJ tooling into Eclipse, and motivates a solution that I will elaborate on in future posts.

JDT has not been engineered to be extensible, and rightly so. It is a set of tools that are finely tuned towards making the Java development experience as smooth and convenient as possible. Focusing on extensibility would be to the detriment of the JDT users who program in Java only.

However, this makes life difficult for us tool developers. Users expect high quality tooling for Java-like languages (i.e., languages that run on the JVM) because that is the bar set by JDT. Furthermore, they want tooling that integrates seamlessly with Java tooling. But, JDT won't do that! So, how can we provide good programmer experience in AJDT?

We want to make editing AspectJ look and feel as much like editing Java where it makes sense, and at the same time provide additional functionality so that the user is aware of AspectJ-specific issues. Up until AJDT 1.6.2, there have been limited options to integrate with JDT.

One example of successful integration is the AspectJEditor, which behaves mostly like JDT's CompilationUnitEditor, providing eager error detection, syntax highlighting, content assist, some refactoring support, etc.





The AspectJEditor should be a subclass of the CompilationUnitEditor, But the CompilationUnitEditor is not public API and AJDT really shouldn't be touching it. Or, at least, the responsibility is on us to maintain compatibility in future versions. In addition to using private APIs, the AspectJEditor uses some reflection to access private fields of its super class and copies some code, where reflection isn't practical. Through this combination of techniques, we have been able to largely implement JDT's behavior in AJDT. But, there is some functionality that we want AJDT to provide, but is not possible using private APIs, reflection, or code copying.

Let's take a look at the Open Type Dialog.




We would really like to see our aspect types in the standard Java Open Type Dialog. But this is unfortunately not possible. There is no API (public or private) that we can use to do this. Even the use of reflection or code copying will not help. What we really need is to gain access to JDT's Java indexer and somehow convince it to index aspect files.

This is a big problem, and one that we have run into many times during our development of AJDT. JDT is not engineered to be compatible with Java-like languages. But, tools for languages such as Groovy, Scala, and JRuby are gaining in popularity and require the same kind of JDT integration that is just not available. There is a long standing JDT bug to address this, but it does not seem like it will be fixed any time soon.

However, there is a solution! Since AJDT 1.6.2, we have been able to provide deep integration with JDT by using AspectJ on JDT itself. We are using Equinox Aspects to weave into the platform and expose functionality in a structured way through the use of Eclipse extension points. In future posts, I will describe how we have designed our JDT weaving service to expose otherwise inaccessible functionality in JDT and how this weaving service is generic enough to be used by tool developers for other languages.

You can also hear more about this at my EclipseCon talk: Aspects Everywhere: Using Equinox Aspects to Provide Language Developers with Deep Eclipse Integration.

Tuesday, February 3, 2009

Contraptions for programming

Hello Planet Eclipse!

I wanted to introduce my new blog, where I will be talking about the AJDT and AspectJ projects, aspect-oriented programming, OSGi, and all the nifty tools we are making so that these technologies can work better together.