Thursday, December 24, 2009

Extending Groovy-Eclipse for use with Domain-Specific Languages

One of the great things about Groovy is how easy it is to use the language to create domain specific languages (DSLs). And one of the great things about Eclipse is extensibility through its plugin architecture. Within Groovy-Eclipse, we are using the second to leverage the first. We have created a set of Eclipse extension points that allow Groovy programmers to create tool support for their own custom Groovy DSL. In this post, I will walk you through the extension points we created and show you how we use them to implement Grails support in STS.

We provide three extension points, one for adding new smarts to the inferencing engine: one for adding new content assist proposals, and one for adding new highlighting rules to the editor.

Extending Syntax Highlighting



The simplest way to extend Groovy-Eclipse is through the org.codehaus.groovy.eclipse.ui.syntaxHighlightingExtension extension point.

Here is what STS does to use the extension point:

<extension
point="org.codehaus.groovy.eclipse.ui.syntaxHighlightingExtension">
<highlightingExtender
extender="com.springsource.sts.grails.editor.groovy.GrailsSyntaxHighlighting"
natureID="com.springsource.sts.grails.core.nature">
</highlightingExtender>
</extension>


As you can see, an extender class (com.springsource.sts.grails.editor.groovy.GrailsSyntaxHighlighting) is associated with a project nature (com.springsource.sts.grails.core.nature). And now, whenever a Groovy Editor is opened for a Grails project, all of the syntax highlighting rules from the extender class is added to the Groovy Editor. For Grails, the extension is simple:

public class GrailsSyntaxHighlighting implements IHighlightingExtender {

public List getAdditionalGJDKKeywords() {
return Arrays.asList(
// domain fields
"constraints", "belongsTo", "hasMany", "nullable", "belongsTo", "mapping",
"hasMany", "embedded", "transients", "id", "tablePerHierarchy", "version",
// domain methods
"list", "save", "delete", "get",
// controller fields
"log", "actionName", "actionUri", "controllerName", "controllerUri",
"flash", "log", "params", "request", "response", "session",
"servletContext",
// controller methods
"render", "redirect"
);
}

public List getAdditionalRules() {
return null;
}
public List getAdditionalGroovyKeywords() {
return null;
}

}


New GJDK keywords are added and nothing else. Also, note that GrailsSyntaxHighlighting implements IHighlightingExtender. And here is what the additional syntax highlighting can give you:

syntax_highlighting

Notice that the special Grails domain class fields such as belongsTo and mapping are highlighted, and below in the controller class, keywords like params and render are highlighted.

Extending the inferencing engine



The Groovy-Eclipse inferencing engine is used to infer the types of expressions within a Groovy file. Because of its dynamic nature, determining the types of all Groovy expressions in a file is undecidable. The good news is that most programs are well-behaved and follow a simple set of rules through which we can infer the types of most expressions. Meta-programming in Groovy can add new members to Groovy objects and classes. This is a feature used by most Groovy DSLs.

Groovy-Eclipse allows DSL programmers to specify the meta-programming through the org.eclipse.jdt.groovy.core.typeLookup extension point. Here is what the extension point looks like in the Grails tool support in STS:

<extension
point="org.eclipse.jdt.groovy.core.typeLookup">
<lookup lookup="com.springsource.sts.grails.editor.groovy.types.GrailsTypeLookup">
<appliesTo projectNature="com.springsource.sts.grails.core.nature"/>
</lookup>
</extension>


Here the class com.springsource.sts.grails.editor.groovy.types.GrailsTypeLookup is defined to be a type lookup for projects that have the com.springsource.sts.grails.core.nature (i.e., this lookup is only activated for Grails projects).

Let's take a look at the GrailsTypeLookup class:

public class GrailsTypeLookup extends AbstractSimplifiedTypeLookup implements ITypeLookup {

private IGrailsElement element;
private GrailsProject gp;

public void initialize(GroovyCompilationUnit unit,
VariableScope topLevelScope) {
gp = GrailsCore.get().getGrailsProjectFor(unit);
if (gp != null) {
element = gp.getGrailsElement(unit);
element.initializeTypeLookup(topLevelScope);
}
}

@Override
protected TypeAndDeclaration lookupTypeAndDeclaration(
ClassNode declaringType, String name, VariableScope scope) {
IGrailsElement declaringElt = gp.getGrailsElement(declaringType);
return declaringElt.lookupTypeAndDeclaration(declaringType, name, scope);
}
}


According to the extension point specification, GrailsTypeLookup must extend ITypeLookup and we choose to let it extend AbstractSimplifiedTypeLookup in order to reduce the amount of coding required.

The initialize method is called when type inferencing is starting for a Groovy file. Here, it is possible to stuff things into the top level scope (such as global variables). For Grails, we determine what kind of Grails element we are performing inference on (e.g., a domain class, controller class, taglib, etc) and modify the top level scope appropriately.

More magic happens in the lookupTypeAndDeclaration method. Again, the type lookup delegates to the specific Grails element to determine what the type is.

Now, let's take a look at what this can do for us. Notice that hovering over Grails keywords will bring up a JavaDoc of the inferred type of that keyword:

hovers_in_controller

The Grails response field in controller classes is of type HttpServletResponse. Similarly, this allows us to get HttpServletResponse aware content assist proposals:

content_assist1

Extending Content Assist



The final step is to hook extensible content assist into the DSL. This can be tricky. For example, in Grails, there are certain fields that if defined have special meaning. There is the constraints field where constraints for domain classes are defined, and the mapping field where object-relational mappings are defined. The closure attached to each of these fields have special keywords that they expect.

This is possible to control through the org.codehaus.groovy.eclipse.codeassist.completion.completionProposalProvider extension point. Here is how it is used in STS:
<extension
point="org.codehaus.groovy.eclipse.codeassist.completion.completionProposalProvider">
<proposalProvider
proposalProvider=
"com.springsource.sts.grails.editor.groovy.contentassist.GrailsProposalProvider">
<appliesTo projectNature="com.springsource.sts.grails.core.nature"/>
</proposalProvider>
</extension>


This extension point wires a com.springsource.sts.grails.editor.groovy.contentassist.GrailsProposalProvider to the Grails project nature. And so (as with the other extension points), theis extra content logic will only occur when inside a Grails project.

com.springsource.sts.grails.editor.groovy.contentassist.GrailsProposalProvider implements org.codehaus.groovy.eclipse.codeassist.processors.IProposalProvider. This interface has three methods to implement:


  • getNewFieldProposals: Return a list of fields that can be defined at the content assist location. For example, here is where all special fields available in Grails domain classes are proposed. This method is only called when content assist is invoked when inside a class body (i.e., only where it is appropriate to define new fields).

  • getNewMethodProposals: Return a list of new methods that can be defined at the invocation location. As with the new fields method, this method is only called when content assist is invoked in a location that is possible to define new methods.

  • getStatementAndExpressionProposals: This method returns all possible special content assist proposals when in the context of an expression or statement. For example, here is where special controller class fields like params, request, and response are inserted.



Let's take a look at what this can do. In this screenshot, you can see that when performing content assist on a reference to a Grails domain class, you can access Grails specific methods like count:

extra_content_assist

And when inside the constraints block (and only when within that block) content assist is augmented with possible constraints to add:

content_assist_constraints

Conclusion



We have worked hard to make sure that Groovy-Eclipse is extensible. It is already being used by some DSLs such as EasyB and by BonitaSoft.

These extension points and APIs are still a work in progress if you have any questions, or require some changes to anything, please raise a bug or send a message to the mailing list.

Monday, December 14, 2009

Getting GMaven to play nicely with Groovy-Eclipse

There has been a lot of talk on the Groovy mailing list lately about how to get GMaven working with Groovy-Eclipse and coincidentally I saw my friend Mike last night who works at boats.com and he told me that his office has solved this problem. Here is what he told me:



Okay, the first things you will need in your POM file are the GMaven mojo and the GMaven runtime. The dependencies for the mojo seem to be a big buggered out of the box, so you will have to tweak the GMaven runtime dependencies. Here is how you do it:


<dependencies>
<dependency>
<groupId>org.codehaus.groovy.maven</groupId>
<artifactId>gmaven-mojo</artifactId>
<version>1.0</version>
<exclusions>
<exclusion>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all-minimal</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.groovy.maven.runtime</groupId>
<artifactId>gmaven-runtime-1.5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.codehaus.groovy.maven.runtime</groupId>
<artifactId>gmaven-runtime-1.6</artifactId>
<version>1.0</version>
</dependency>
</dependencies>


Next is your build configuration. Eclipse will need to be able to find your Groovy source files, so you will need to add your Groovy source as an explicit resource directory.

The mvn:eclipse plugin will also need to add the Groovy nature to your project.

You will also need to add directives for the GMaven plugin. This is where you will have to do a manual tweak: when using mvn to build your project, gmaven generates some stubs for your groovy code before the java code compiles, so that any java->groovy dependencies will be fulfilled. The Groovy code is then compiled and will overwrite the .class files from the previous step. This breaks things in eclipse, since if you attempt to execute code in your workbench, eclipse sees the stubs and somehow their .class files are what end up in your binary output directory.

Therefore, if you want to execute a gmaven project from Eclipse, you simply need to comment out the stub generation directive and delete any generated stub classes. This is what it looks like:


<build>
<resources>
<resource>
<directory>src/main/groovy</directory>
</resource>
</resources>
<plugins>
<plugin>
<artifactId>maven-eclipse-plugin</artifactId>
<configuration>
<additionalProjectnatures>
<projectnature>
org.eclipse.jdt.groovy.core.groovyNature
</projectnature>
</additionalProjectnatures>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.groovy.maven</groupId>
<artifactId>gmaven-plugin</artifactId>
<version>1.0</version>
<executions>
<execution>
<goals>
<!--
<goal>generateStubs</goal>
-->
<goal>compile</goal>
<!--
<goal>generateTestStubs</goal>
-->
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

The result is a groovy/java project that can be managed with maven but can be transparently developed (and more importantly, debugged!) in Eclipse.

There is one minor issue that has come up however: GMaven 1.6 stub generation is broken for enums: it generates public constructors, which do not compile. I submitted an issue to codehaus (GMAVEN-51), and apparently the workaround is to use the gmaven 1.7 runtime. I haven't tried it yet, since things are working okay for us thus far.

Mike


So, it does seem like there is a bit of a problem with stubs being recognized by Groovy-Eclipse when they should not be. The solution would be to generate the stubs in a directory that is not seen by Eclipse, rather than in the default output directory. But not knowing much about GMaven (or Maven for that matter) works, I don't know how feasible this is. Can anyone think of a better solution?

Friday, November 27, 2009

New cross-language refactoring support for Groovy-Eclipse plugin

Thanks to the work of Michael Klenk and his students at the Institute for Software in Switzerland, the Groovy-Eclipse plugin now has cross-language refactoring support for renaming. Its use is seamless. Let me give you an example.

Let's start with a simple Java file and a simple Groovy file:

before

You can rename the field in the Java class:

during1

This will perform the standard Java refactoring on all Java classes in your projects. Additionally, the refactoring will be performed on your groovy classes. If there are any ambiguous references to the Java field, the following dialog will open up allowing you to choose which references should be refactored:

choose

And the result is what you would expect:

after

The refactoring can be initiated from the Groovy side as well:

groovyrefactor_menu

As you can see, extract method an inline method are two other refactorings supported in Groovy code.

This functionality is currently available in the latest snapshot build of Groovy-Eclipse for Eclipse 3.5, available from the following update site:
   http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5/


Please send any questions or problems to the mailing list. We're happy to help. Enjoy!

Thursday, November 19, 2009

Getting AspectJ Pointcut matching timer information in AJDT

Andy Clement has just added pointcut match timing information to AspectJ. A number of people have been asking to have this included in AJDT, and so I did a little work and made it available in the latest dev builds of AJDT for 3.4 and 3.5.

Here's how to display timing information:


  1. Select Verbose and Pointcut matching timers from the AspectJ compiler preferences page (this can be either globally from Eclipse preferences or for a specific project from the project properties page):

    compiler prefs

  2. In the AJDT Event Trace View, ensure that you have Compiler / Task list messages selected:

    event trace



And now on every project build (incremental or full), you will see pointcut matching times spit out to the event trace:

output

Enjoy! Questions can be sent to the AJDT developer's mailing list.

Monday, November 16, 2009

How type inferencing for Groovy in Eclipse is leading to all sorts of Groovy goodness

Of all of the recent improvements of Groovy-Eclipse since M1 has been released, the groovy type inferencing engine is the most pervasive and powerful. This inferencing engine is able to take a groovy file or code snippet and infer the type of any or all of its expressions. It now forms the core of several of the most prominent features of Groovy-Eclipse. Everything that I describe here is currently available in the latest snapshot release of Groovy-Eclipse for Eclipse 3.5 and the update site is here:
http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5


In this post, I am going to describe how the inferencing engine is currently used in Groovy-Eclipse and then I will dive a deep and describe how it is implemented. I'll end the post by describing our plans for immediate the future.

What type inferencing is used for



Underlining statically indeterminate references


The Groovy editor now underlines all references that cannot be determined while editing. This happens in a background thread while you type. So, the following code (notice the mispelling of asImmutable):


def aList = []
def otherList = aList.asImmutabl()
otherList.asImmutable()



...will appear in the editor like this...

underline

...and when the typo is fixed, the underlines go away...

nounderline

Code is re-analyzed after every keystroke (well, actually, it is after every keystroke that is followed by 500ms of no keystrokes). The type of every reference in the file is looked up using the inferencing engine. Every reference that cannot be found is considered to be unknown and will be underlined.

Of course, Groovy being a dynamic language, not all references can be statically linked to a type (other than Object). This is especially the case when working with DSLs (such as Grails) that use metaprogramming to add new methods and fields to existing types. Later, I'll describe our plans for extensibility.

Search



Inferencing is used for Java Search as well. Using the search short cut (CTRL-Shift-G or CMD-Shift-G on Mac) or running search from the Java Search dialog, references from and to Groovy code snippets can be located. However, there are some difficulties here. Compared to Java, Groovy is quite a bit more flexible in the ways that fields and methods can be referenced. For example, this field is assigned a closure and referenced as if it were a method:

class Foo {
def x = { print it }
}

x("Hello!")


And this method is referenced in a way that is statically indistinguishable from a field:


class Foo {
def x() { print "Hello!" }
}

x


Furthermore, the spread operator and the use of default parameters makes the Java way of determining which method or field is referenced impossible. So, Groovy search is significantly more liberal than Java search in what it considers a search match. If a reference corresponds to any field, method, or property in the declaring type (regardless of parameter count and whether it is accessed as a field or method), then the match is considered successful. This leads to the following scenarios.

1. Performing search on the field declaration finds field references here:
searchForField

2. Performing search on the method declaration finds exactly the same references:
searchForMethod

And, notice that references attached to the wrong type are not considered a match.
searchNoResults

Content Assist



Most of the internals of Groovy-Eclipse has been rewritten since the previous 1.5.7 release. The most recent rewrite is of content assist. Although the previous version worked (usually), there was always a noticeable lag while waiting for the content assist window to appear. This previous implementation of content assist used a combination of Java classloading and Groovy AST matching in order to find all completion proposals available at a given location, which was both messy and slow. Now, we have hooked into the inferencing engine and have completely rewritten proposal generation with extensibility in mind. And we have made some remarkable improvements in performance.

Using the following code snippet I performed content assist 5 times under the old implementation and the new and took the average of the last 3 of each:


def x
x.hasProperty("foo").





Old implementation2315 milliseconds
New implementation8 milliseconds


That's right...there is a 3 orders of magnitude speed up under the new implementation, making content assist a much more pleasant feature to use. But please note that this is just the time for calculation of the proposals and it does not include the time required to actually pop up the window.

Open declaration


The implementation of open declaration (or code select as it is called in JDT because it is a general mechanism to determine what the current selection refers to and is also used for things like calculating hovers) also uses inferencing. Pressing F3 whenever the selection is any form of "navigateToMe" in the following code, the expected declaration will open:

navigateBefore ---> navigateAfter

Although Groovy-Eclipse M1 used type inferencing for code select, it was using the inferencing from content assist and was subject to the same mistakes, duplicate answers, and slowness that content assist had.

How it works


The inferencing engine has unified the implementation of several of Groovy-Eclipse's most widely used features. With this unification, we have been able to significantly reduce the amount of duplicated code, improve performance, and plan for extensibility.

There are four interacting components of the inferencing engine:


  1. The Abstract Syntax Tree (or AST): this is an abstract representation of the syntax of a groovy file expressed hierarchically. The code to work on can be obtained either from the disk, or it can be the latest in memory. The AST is generated by the Groovy compiler based on the current state of the editor whenever Eclipse is otherwise idle.

  2. The type inferencing visitor: this visitor walks the AST (i.e., uses a visitor pattern to visit every AST node) and delegates to the type lookups and the requestor described below. A new visitor is created for each call to the inferencing engine.

  3. The type lookups: these objects lookup the type information of a particular expression using whatever mechanism they require. They can store and retrieve information about types in a way that all other lookups can use. In this way, each lookup can do what it can and coordinate with other lookups where required. Several default lookups are provided including a simple lookup, an inferencing lookup (that stores information about types via assignment expressions), and a category lookup (that looks up types in Groovy categories). Type lookups are created from a registry on each call to the inferencing engine.

  4. The requestor: this is the object that collects the type information determined by the lookups. The requestor is general and can be used as a way to store the types of all expressions, lookup the type of a particular expression, or look for expressions with unknown types (just to name a few uses). The requestor is created by client code and past to the visitor to start the inferencing process.



The following diagram shows process and how the different pieces work together:

arch


  1. The visitor visits an AST expression node

  2. The visitor sends this expression to the type lookups.

    • each lookup responds with a type and its confidence in that type (ie, exact, potential, inferred...), or null, if it cannot find a type

    • a lookup may additionally store information in a scope object that is passed around to other lookups so that information can be shared between them


  3. The result of the lookup is sent to the requestor

  4. The requestor processes the result and may choose to end the visit or continue.

  5. If required, the visitor will continue to visit the rest of the abstract syntax of the Groovy file



What's next


One of the core design goals of the inferencing engine is extensibility. The inferencing engine must be extensible in two ways. First, it must be usable to help support new features such as quick fixes and refactoring. Also, DSLs, (most notably Grails) must be able to extend inferencing with their own type lookups. It should be apparent through the description of the variety of ways that the engine is currently used that it versatile and can become the core implementation of any number of new Groovy-Eclipse features. As for DSL extensibility, that has not yet been implemented, but the stubs are available. Other DSLs will need to implement their own type lookup, but there is currently no way for these new lookups to be plugged in. Fortunately, it will be possible to use Eclipse's plugin architecture here and this will be the subject of my next blog post (after it gets implemented).

Saturday, October 3, 2009

AJDT 2.0.1 is released!

Overview

In this service release of AJDT, we have included a number of performance enhancements, as well as a few new features. This is the last scheduled release of the AJDT stream targeting 3.4. The next release, in early 2010, will be targeting the 3.5 stream only. And our second 2010 release will target 3.5. At the same time, we will also release an early development version of AJDT targeting Eclipse 3.6 in time for EclipseCon 2010.

AJDT is available from the following update sites for Eclipse 3.5 and 3.4 respectively:
http://download.eclipse.org/tools/ajdt/35/update
http://download.eclipse.org/tools/ajdt/34/update

Export Feature With AspectJ

A new wizard is available to export features with AspectJ support. This wizard can be initiated from the File -> Export... command:



This wizard supports all of the options that the standard export feature supports. The only difference is that the AspectJ compiler is used to compile the projects instead of the Java compiler:


AJDT support for passing aop.xml file

AJDT now provides UI support to pass aop.xml files to the compiler. AspectJ bug 124460 describes how the AspectJ compiler can use aop.xml files for compile time weaving. And now AJDT provides UI support in the following way.

In project properties of your AspectJ project, select aop.xml management. Here you add and remove aop.xml files that will be sent to the compiler. Note that these files are highlighted using a special icon. Also, note that the name aop.xml is optional. Any xml file can be sent to the compiler in this way. See the following screenshot for an example


In order to use aop.xml files for compile-time configuring of the weaver, you must add the -xmlConfigured option under Non-Standard Compiler Options in the project's AspectJ Compiler options. This work is described in bug 287459.

This work is still experimental. We hope to receive some feedback so that compile time XML configuration of the weaver can be improved. Please send all comments to theAspectJ mailing list, or to bugzilla.

AspectJ 1.6.6

AJDT 2.0.1 contains AspectJ 1.6.6 and this version is available for download separately. See the release notes here.

Bug Fixes

Thirty-four bugs have been addressed in this release. See Bugzilla for the full report.

Thanks!

Thanks to those of you who have submitted patches, bug reports, and contributed to the mailing list for this release. We appreciate your help. Specifically, in this release:

Previous Releases

See here for information on the new features in AJDT 2.0.0.

Tuesday, September 29, 2009

Groovy-Eclipse plugin now supports Eclipse 3.5.1

The Groovy-Eclipse plugin can now be installed into Eclipse 3.5.1. As always, the update site is here:

http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5/

You can stop reading here if you don't care how I got this to work, or why I am a little bit surprised by this.

The Groovy-Eclipse plugin uses a feature patch on the JDT core bundle in order to achieve a high level of integration with JDT. Typically, feature patches can only apply to a specific version of a feature (e.g., 3.5.0.v20090527-2000-7r88FEeFJePyvYeA33DjZ_c1 or some ugliness). This means that Groovy-Eclipse, until recently could only install in a specific version of Eclipse, the one that ships the JDT feature version 3.5.0.v20090527-2000-7r88FEeFJePyvYeA33DjZ_c1. However, Andrew Niefer describes how to get around this by editing the content.xml file to widen the range that the patch applies to. Excellent stuff, and lucky for me, because I didn't want to branch the Groovy-Eclipse code base every time a new service release of Eclipse comes out.

Unfortunately, his instructions were not entirely accurate. Andrew mentions that the range attribute in the <patchscope> element needs to be widened in order allow the patch to be installable on multiple versions. I tried exactly what he suggested, but could not get my patch to install in both 3.5.0 and 3.5.1. After a little bit of exploration, I found that I also needed to change the range attribute in the <lifecycle> element.

All this meant was adding a single line to my ant script, to be executed after the p2 repository is generated:

    
<replace file="${updateSiteDir}/content.xml"
summary="yes" token="${orig.jdt.feature.version.range}"
value="${new.jdt.feature.version.range}"/>

And this turned out to be very simple. Thanks for the hint!