Tuesday, March 20, 2012

Light-weight IDE extensibility for custom DSLs in Groovy

This post gives a bit of a teaser for my EclipseCon talk next week.

Most of you reading this probably know that Groovy-Eclipse is the Eclipse tool support for Groovy and one of the major features it provides is type inferencing in the editor. For example, in the following screenshot:


Groovy-Eclipse knows that the type of myString is java.lang.String. That is why substring is not underlined, but uh_oh is. All references that cannot be statically resolved are underlined regardless whether or not the program will fail at runtime (Groovy is a dynamic language, and we can't be certain about what happens at runtime).

This works quite well in common situations. However, things get complicated because Groovy provides strong support for creating domain specific languages (DSLs). Typically, when a DSL is used, the editor will be confused. Consider, for example, this simple DSL to calculate distances and convert between units. When used in Groovy-Eclipse, the result looks like this:


Uh-oh, underlines. The references do not make sense to the editor, even though this executes without problem.

Thankfully, Groovy-Eclipse's inferencing engine was designed from the beginning to be extensible. There are three ways to extend the inferencing engine:

  1. Roll your own Eclipse plugin
  2. Inferencing suggestions
  3. DSL descriptors

And I'll talk about each of them in the next sections.

Roll your own Eclipse plugin


This is a powerful technique and gives you fine-grained control over exactly how your DSL integrates with Groovy-Eclipse. I have already written a full description of all the extension points available and how to use them to create a plugin that extends Groovy-Eclipse. This was the state of Groovy-Eclipse circa 2010.

Although there have been many successes with this approach, we found that overall this was quite a heavy-weight way for others to leverage the benefits of type inferencing:

  • Most Groovy DSL developers are not Eclipse experts and so do not have the experience or desire to learn Eclipse APIs, create plugins and features, produce an update site, and test it against various versions of Eclipse
  • Many Groovy programmers (the ones who consume the DSLs) are coming from a scripting world, use vi or textmate, and are used to a light-weight editor. Installing plugins was a big complaint for them. They want things to just work

Clearly, another approach was needed.

Inferencing suggestions (very light-weight extensibility)


Inferencing suggestions give the end user control to augment the inferencing engine with specific suggestions. Using a quick-assist (CTRL+1), the user is given an option to add a suggestion to the currently selected identifier in the editor:


This brings up the inferencing suggestions dialog, where the user can fill in as little or as many details as desired:


And now, back in the editor, the suggestion is now used for inferencing:


This feature has some big advantages over the plugin model:

  • it gives end-users more control over their own environment
  • it is dead simple to use and no other plugins are required
  • no need for any Eclipse API knowledge
  • little knowledge of Groovy required

But in some situations, this feature moves too far in the direction of simplicity over expressiveness.

  • can only target a single property/method at a time, which makes large DSLs painful to work with
  • not easy to share across users and projects

Inferencing suggestions was not sufficient and we needed yet another kind of extensibility.

DSL descriptors (light-weight extensibility)


A DSL descriptor (DSLD) is a Groovy script that describes your Groovy DSL. This script is compiled and executed by Groovy-Eclipse so that the inferencing engine can understand the DSL. They can be shipped with libraries and as long as Groovy-Eclipse can find these files on the classpath, they can contribute to type inferencing. I have already written a good introduction to DSLD.

We created the DSLD language so that library developers could have fine-grained control over how their library and DSL gets interpreted in Groovy-Eclipse and also so that end users can transparently use this functionality without needing to install a new plugin or do any extra work.

Taking our distances DSL from before, we can write a simple DSLD that encapsulates the DSL in a format consumable by Groovy-Eclipse:

contribute( currentType( subType( Number ) ) ) {
  [ “m”, “yd”, “cm”, “mi”, “km” ].each {
    property name:it, type:"Distance”, 
        doc: """Converts a {@link Number} to a {$it}
                @author Joe
                @since 1.0"""
  }
}

When this file is saved to something like distances.dsld and placed in the project or on the classpath, the distances expression is appropriately recognized in the editor:


(Which, you should notice looks the same as the inferencing suggestions hover above.)

What kind of extensibility is best?


That's a loaded question and of course, it depends. Here's a simple table to highlight the costs and benefits of each approach:

Extension pointsDSLDInferencing suggestions
Easeheavy-weightlight-weightvery light-weight
Flexibilitylotssomelittle
Eclipse knowledge required?lotsnonenone
Groovy knowledge required?lotslotslittle
How to use/installupdate siteincluded with librarypreferences/quick-assist

I'd still recommend that you come see the talk at EclipseCon (if you are lucky enough to be attending the conference). I'll go into more detail on implementation and I'll show some live demos on how this all works. See you at EclipseCon!

Friday, February 24, 2012

Better JavaScript content assist in Eclipse Orion

Even in the month since I started using and working on Eclipse Orion, there have been significant improvements to it. The UI is getting cleaned up, navigation is getting easier, git integration is progressing, and search is improving. It's great to be contributing to such a fast moving project, even if the rate of change can be dizzying sometimes. Also, I'm new to JavaScript and I have never worked on such a large code-base written in a dynamic language. One of the things that I have been missing the most is good tool support that really knows about the code you are working on. Traditional IDEs set the bar high in this area.

I'm happy to introduce a small step in the direction of making the JavaScript editor smarter. I've just released an Orion plugin that provides semantically aware content assist in the JavaScript editor. The plugin uses the Esprima JavaScript parser with some extra error recovery added by Andy Clement. I have found the Esprima parser to be fast, clean, and easy to use and using it as the core of the content assist plugin has been the right thing to do, even though this project, too, is fast moving and hard to keep up with.

How it works


Like content assist in Java editors in Eclipse, the Esprima content assist plugin is based off of a semantically rich abstract syntax tree (AST) and so content assist proposals are more likely to be relevant than if we were using a lexical approach to content assist. Here is what happens:

  1. On a content assist invocation, the contents of the buffer are parsed by Esprima.
  2. The resulting AST is walked by the content assist plugin.
  3. While walking the AST, the target type of any AST node is recorded as well as assignments and declarations. This information helps us keep track of what properties are available on each known type at any given point in the AST.
  4. After walking a sufficient amount of the AST (we don't need to walk the entire tree since parts of it are not going to be relevant for a given content assist invocation), all available proposals are calculated based on the target type of the invocation offset and the prefix.

The best way to understand how this works is through examples.

What it can do


Recognizing function scopes


As you can see in this screenshot, scoping is respected and identifiers that are not accessible in the current scope (vInnerInner, v3, v4,…) are not shown in content assist.


Object literals


The key/values of object literals are appropriately proposed:

Even nested object literals are recognized:


Simple control flow


Simple control flow is recorded by the plugin, so that assignments are remembered:


Pre-defined types


Some (but not all) predefined types are available in content assist.


Currently, the plugin recognizes JSON, MATH, Number, String, Boolean, and Date, but I will probably add more as it makes sense.

Constructors


Functions that start with capital letters are considered constructors


Parser recovery


Finally, Andy Clement has been doing some work on making the esprima parser recoverable from errors. Actually, some error recovery is already in esprima, but we need to tweak it a bit for content assist. Hopefully, this work can be contributed back to esprima after we have a good solution. Currently, the recovery work is focussed on errant dots. A common case is that you will type a variable name and then a '.' and expect content assist to provide all reasonable answers. Most JavaScript parsers will fail after the first error, which makes them quite useless when editing code.


As you can see in the screenshot, despite all of the funky dots, the plugin is able to realize that myVar is of type Number and is providing appropriate proposals.

What it can't do (yet)


This is still early for the content assist plugin and there is quite a bit of work to do. For example:

  1. There is no pre-defined window object, which probably should be there, along with possibly other predefined objects, like dojo, dijit, and $ (jquery).
  2. There is no analysis of function return types
  3. No inter-file type inferencing, which will be crucial for getting anything really smart working
  4. The plugin should recognize /*global */ comments
  5. The Esprima-based proposals are currently intermingled with proposals from the default JS content assist plugin and so duplicates appear. (Esprima proposals are always prefixed with a handle (Esprima) so you know where they come from, but they are always on the bottom).

I hope to deal with each of these issues eventually, but I also need to make sure that performance remains reasonable, which it currently seems to be, but is something I need to watch.

How to get it


Mark Macdonald has already added the pluign to the Orion plugin page, so after you log into Orion, click to the "Get Plugins" link and select the Install link for the Esprima content assist plugin:


The github page is located here: https://github.com/aeisenberg/esprimaContentAssist so try it out, have a look at the code and let me know what you think!

Wednesday, November 16, 2011

First ever Vancouver Eclipse Hackathon...success!

Last night we had our first Vancouver Eclipse Hackathon. It was a wonderful night, and we had 20 Eclipse hackers showing up with all kinds of experience. The goal of the evening was to introduce some programmers to Eclipse-based open source projects, to help them collaborate and learn about the project, and maybe even contribute some code. Some hackers had written Eclipse plugins before, others knew nothing about Eclipse (other than it being a great Java IDE), and a few didn't even know Java. It was a good range of people and we spent almost six hours working on our bugs.


Working on quick-assists inside of Groovy-Eclipse


Team working on mark-occurrences in the Groovy editor.


Some Mylyn Hackers working together.


More hacking.


Almost complete silence in the room as everyone concentrates on their bugs.


Happy birthday, Eclipse!

And here are the results of the evening:

  • 20 hackers (including two project leads)
  • 6 hours of hacking
  • 24 bottles of beer drunk
  • 6 pizzas eaten
  • Half of an Eclipse 10 year birthday cake eaten
  • 3 Eclipse-based projects worked on: Mylyn, AJDT, and Groovy-Eclipse
  • 12 bugs worked on
  • 3 patches submitted
  • 2 more patches in process
  • Fun had by all

It was encouraging to see the dedication of all the hackers. We will definitely be hosting another one soon. However, we did learn a few things about organizing hackathons that we will do differently next time:

  • Make it easy for hackers to set up their dev environments. Put up instructions a few days before so hackers can get ready before the hackathon and they don't have to spend 90 minutes setting up before starting.
  • Internet connections can be unreliable and slow. Have USB sticks available with Eclipse SDK tarballs for different environments. Also, have some workspaces set up with the source code of the projects checked out.
  • Try to have more projects and more project leads available. This time, we only had three projects and two project leads, but more variety would have been nicer. We were originally expecting more leads to show up, but they had to cancel.
  • Have a very concise list of bugs for hackers to work on. Be very specific about how to get started working on them.

Also, a huge thanks to the other co-organizers of this event Ducky Sherwood and Zoe Jong from Tasktop Technologies.

If you want to organize a hackathon in your area, drop me a line and I can help you get started. It is a fun time.

Thursday, September 22, 2011

Using Grails 2.0.0.M2, Cloud Foundry and the SpringSource Tool Suite

There are a lot of details in my VanDev talk tonight. Since most of what I showed you was from versions of our software not yet officially released, you are going to need to cobble together a few things to get everything working properly. Here are some instructions and links.

  1. First, here is a detailed guide on using Grails 2.0.0 from inside STS. It was written for M1, but it is essentially the same. Below, I'll summarize.
  2. Download STS 2.8.0.M1. To see 2.8.0.M1, you need to click on Other downloads.
  3. After downloading and unzipping/installing STS, you will need to install Groovy-Eclipse and Grails Tooling. Do not install Groovy-Eclipse or Grails from the dashboard since that only points to older releases. Instead, go to the Install manager (Help -> Install new software...), and add the nightly update sites for Groovy-Eclipse and Grails Tooling:
    • Nightly Grails Tooling update site:
      http://dist.springsource.com/snapshot/TOOLS/nightly/e3.7
    • Nightly Groovy-Eclipse update site. Make sure to also install the Groovy 1.8 compiler:
      http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.7/
    Use these sites to install the software.
  4. Next, download Grails 2.0.0.M2.
  5. Once downloaded and unzipped, you can install the Grails distribution into STS.
If you want to play around with Cloud Foundry:
  1. First install the CloudFoundry tools from the STS dashboard.
  2. Then sign up for a CloudFoundry account.
  3. There's lots of information on the cloudfoundry.com site as well as the community site. Here's more on using Cloud Foundry in STS
Here is some more information on DSL Descriptors in Groovy-Eclipse, and also a general description of our approach to integrating the Groovy and Java compilers inside of Groovy-Eclipse.

Support

If you have any questions of problems with STS or Groovy-Eclipse, please visit the forum. However, Groovy-Eclipse questions may be answered more quickly on the mailing list. Bugs can be reported here: Groovy-Eclipse and STS.

Wednesday, June 29, 2011

AJDT 2.1.3 Released

The AJDT team is proud to release AJDT 2.1.3. This release is the first one targeting Indigo (Eclipse 3.7) and it contains AspectJ 1.6.12.M1. The update sites are here:

For Indigo:
http://download.eclipse.org/tools/ajdt/37/update

For Helios:
http://download.eclipse.org/tools/ajdt/36/update

Please send any feedback to one of the mailing lists or raise an issue
on Bugzilla.

Groovy-Eclipse 2.5.1 Released

The Groovy-Eclipse team is proud to release Groovy-Eclipse 2.5.1. This release is now available for install using the update sites here:

for Eclipse Indigo:
http://dist.springsource.org/release/GRECLIPSE/e3.7/

for Eclipse Helios:
http://dist.springsource.org/release/GRECLIPSE/e3.6/

In this release, we have continued to work on our DSLD support (and we
ship with some useful DSLD scripts). Additionally, we have a GA
release of the groovy-eclipse-compiler-plugin for Maven. Now that
Indigo has been released, the 3.7 stream of Groovy-Eclipse will become
the main development branch. For now, we will continue to support the
3.6 stream of Groovy-Eclipse and will put out at least one more
release.

You can find all the details at our New & Noteworthy page.

Please send feedback here or raise an issue on our issuetracker.

Friday, June 24, 2011

Vancouver Eclipse Demo Camp 2011

In honor of the Indigo release of Eclipse, I will be hosting the annual Eclipse Demo camp in Vancouver on Monday, June 27. The demo camp is co-sponsored by the Eclipse Foundation, VMWare, and Tasktop.

We're expecting a full house as well as lots of exciting talks, including one on Groovy/Grails support in Eclipse and the SpringSource Tool Suite, a talk from Tasktop, and a talk about the Eclipse Modelling Framework from Ed Merks, the project lead.

If you are in the Vancouver area and interested in attending, please add your name to the wiki:

http://wiki.eclipse.org/index.php?title=Eclipse_DemoCamps_Indigo_2011/Vancouver

Alternatively, contact me directly (or on this blog) and I'll add you myself. Hope to see you there!