What is this cloud thing?

Here are the slides from my UBC alumni presentation on programming for the cloud.



You can also download the demo code that I wrote during the talk here on github.

Thank you all who came. It was a lot of fun to give the talk.

Lots of beer, pizza, and coding

Lots of beer, pizza, and coding. I'd say that our second Vancouver Eclipse hackathon was a huge success. We had committers from several Eclipse and Eclipse based projects: Mylyn, p2, JDT, Orion, and Scripted were all represented. We had 30 hackers spread across all of these projects. Currently, we have 7 patches that have been accepted into codebases, and three more that are working their way through the submission process.

The hackathon was a great and social way to be introduced into the open-source world and perhaps some of these new Eclipse contributors can become Eclipse committers. If you are interested in holding an Eclipse hackathon (or any hackathon) in your area, feel free to contact me. I now know what works and what doesn't.

Just getting started

All working hard

Nieraj and Leo take a moment to relax

Ian Skerret, flew all the way from Ottawa for us

A big thank you to all the project leads who helped (and are still helping) some newcomers fixing their first bug. And an even bigger thank you to everyone who worked hard and stayed up late, just to do some coding. It was a lot of fun.

For another look at the hackathon and more photos, see the Tasktop blog.

JSDoc is more than just documentation (in Scripted)

Recently, we released the Scripted editor for editing your JavaScript. It's still early stages, but our focus on a light-weight and semantically aware editor already has some compelling features. In this blog post, I'll show you how you can use JSDoc in your JavaScript files to provide Scripted with some inferencing help. Since Scripted uses the Orion editor, most of this JSDoc support is also available in Orion.

There are many flavors of JSDoc and there is no standard, but the Google Closure Compiler uses a style that is well-documented, self-consistent, and complete. We use Doctrine as the JSDoc parser, which has a complete understanding of the JSDoc annotations of the Closure Compiler.

Currently, Scripted recognizes a subset of JSDoc tags:

• @type {type-sig} {optional-doc} specifies the type of a variable declaration
• @param {type-sig} {param-name} {optional-doc} specifies the type of a function parameter. Parameter annotations are not positional. They must correspond to an actual named argument of the function.
• @return {type-sig} {optional-doc} specifies the return type of a function

The docs for the Closure Compiler specify many kinds of type signatures. Only a subset are relevant to and supported in Scripted. Scripted recognizes:

• Simple types, eg- {Window}, {boolean}, {Boolean}
• Record types, eg- {{myNum: number, myObject}}
• Function types, eg- {function(arg0:string,arg1:boolean)}
• Function return types, eg- {function():number}

And of course, type signatures can be combined to produce more complex signatures. The built-in types boolean, number, string, and object can optionally be capitalized, this breaks from the semantics of the closure compiler, but provides more flexibility to programmers. Other kinds of signatures are tolerated, but do not affect the inferred type.

Examples

With this information, we can start showing some examples. Scripted uses typing information for hovers, navigation (e.g., CTRL+Click to navigate to definitions), and for content assist.

Simple @type

Here is a simple JSDoc comment to define a string variable:

Record types

Using a record type signature, you can define the type of an object literal:

Functions

You can specify that a variable is a function, as well as include its parameter names and return types (parameter types are currently ignored for function type signatures). It looks like this:

If you want to declare the types of function parameters, then you should be using the @param annotation, like this:

User defined types

User defined types are recognized as well:

Union types

One possible point of confusion is with union types. They are interpreted by Scripted as whatever the first signature is. So, in the following is interpreted as a string:

Node support

And finally, Scripted's JSDoc support can be combined with its dependency analysis to infer types defined in other modules. This is extremely helpful for node projects where the use of callbacks can obscure the parameter types. In the following example, by specifying the types of the req an res parameters, you can get some meaningful content assist for them:

What's next

There are some more interesting cases that we should handle. A common idiom in JavaScript is to declare constructors using qualified names. Libraries like Dojo and Ext JS. For example, you can define a class name in Ext JS like this:

Ext.define('scripted.Widget', { ... });

And in Dojo:

var Widget = dojo.declare('scripted.Widget', [], { ... });

We could accomplish this using the @typedef annotation or possibly the @constructor annotation.

Additionally, the @lends annotation provides a nice way to specify options and configuration for constructing types and will likely make it into an upcoming version of Scripted.

So?

Using JSDoc in your JavaScript files is a good way to give some extra help to the inferencer, which often has trouble recognizing some of JavaScript's more dynamic features. Scripted is still in its early stages, and is moving ahead quickly. We like feedback. You can send a message to the Google group and fork us on Github.

Using Groovy-Eclipse to make your DSLs more friendly

Here are slides and source code for my talk at GR8Conf US 2012. The source code and slides are available as a zip.

You can also access the slides and source code from the GR8 Conf US 2012 github repository.

To play with the sample projects and DSL descriptors, you should start with a snapshot version of Groovy-Eclipse 2.7.1 and import the two projects from the zip file into your workspace.

The best place to get your questions answered is on the Groovy-Eclipse mailing list.

(Slides) Light-weight IDE extensibility for custom DSLs in Groovy

Here are my slides from my EclipseCon 2012 talk, Light-weight IDE extensibility for custom DSLs in Groovy.



Update: Uh-oh...looks like the conversion to slide rocket didn't work quite as well as I would have hoped and the animations as well as some of the text is missing. I'm going to see if I can get something better up here.

Light-weight IDE extensibility for custom DSLs in Groovy

This post gives a bit of a teaser for my EclipseCon talk next week.

Most of you reading this probably know that Groovy-Eclipse is the Eclipse tool support for Groovy and one of the major features it provides is type inferencing in the editor. For example, in the following screenshot:

Groovy-Eclipse knows that the type of myString is java.lang.String. That is why substring is not underlined, but uh_oh is. All references that cannot be statically resolved are underlined regardless whether or not the program will fail at runtime (Groovy is a dynamic language, and we can't be certain about what happens at runtime).

This works quite well in common situations. However, things get complicated because Groovy provides strong support for creating domain specific languages (DSLs). Typically, when a DSL is used, the editor will be confused. Consider, for example, this simple DSL to calculate distances and convert between units. When used in Groovy-Eclipse, the result looks like this:

Uh-oh, underlines. The references do not make sense to the editor, even though this executes without problem.

Thankfully, Groovy-Eclipse's inferencing engine was designed from the beginning to be extensible. There are three ways to extend the inferencing engine:

1. Roll your own Eclipse plugin
2. Inferencing suggestions
3. DSL descriptors

And I'll talk about each of them in the next sections.

Roll your own Eclipse plugin

This is a powerful technique and gives you fine-grained control over exactly how your DSL integrates with Groovy-Eclipse. I have already written a full description of all the extension points available and how to use them to create a plugin that extends Groovy-Eclipse. This was the state of Groovy-Eclipse circa 2010.

Although there have been many successes with this approach, we found that overall this was quite a heavy-weight way for others to leverage the benefits of type inferencing:

• Most Groovy DSL developers are not Eclipse experts and so do not have the experience or desire to learn Eclipse APIs, create plugins and features, produce an update site, and test it against various versions of Eclipse
• Many Groovy programmers (the ones who consume the DSLs) are coming from a scripting world, use vi or textmate, and are used to a light-weight editor. Installing plugins was a big complaint for them. They want things to just work

Clearly, another approach was needed.

Inferencing suggestions (very light-weight extensibility)

Inferencing suggestions give the end user control to augment the inferencing engine with specific suggestions. Using a quick-assist (CTRL+1), the user is given an option to add a suggestion to the currently selected identifier in the editor:

This brings up the inferencing suggestions dialog, where the user can fill in as little or as many details as desired:

And now, back in the editor, the suggestion is now used for inferencing:

This feature has some big advantages over the plugin model:

• it gives end-users more control over their own environment
• it is dead simple to use and no other plugins are required
• no need for any Eclipse API knowledge
• little knowledge of Groovy required

But in some situations, this feature moves too far in the direction of simplicity over expressiveness.

• can only target a single property/method at a time, which makes large DSLs painful to work with
• not easy to share across users and projects

Inferencing suggestions was not sufficient and we needed yet another kind of extensibility.

DSL descriptors (light-weight extensibility)

A DSL descriptor (DSLD) is a Groovy script that describes your Groovy DSL. This script is compiled and executed by Groovy-Eclipse so that the inferencing engine can understand the DSL. They can be shipped with libraries and as long as Groovy-Eclipse can find these files on the classpath, they can contribute to type inferencing. I have already written a good introduction to DSLD.

We created the DSLD language so that library developers could have fine-grained control over how their library and DSL gets interpreted in Groovy-Eclipse and also so that end users can transparently use this functionality without needing to install a new plugin or do any extra work.

Taking our distances DSL from before, we can write a simple DSLD that encapsulates the DSL in a format consumable by Groovy-Eclipse:

contribute( currentType( subType( Number ) ) ) {
  [ “m”, “yd”, “cm”, “mi”, “km” ].each {
    property name:it, type:"Distance”, 
        doc: """Converts a {@link Number} to a {$it}
                @author Joe
                @since 1.0"""
  }
}

When this file is saved to something like distances.dsld and placed in the project or on the classpath, the distances expression is appropriately recognized in the editor:

(Which, you should notice looks the same as the inferencing suggestions hover above.)

What kind of extensibility is best?

That's a loaded question and of course, it depends. Here's a simple table to highlight the costs and benefits of each approach:

	Extension points	DSLD	Inferencing suggestions
Ease	heavy-weight	light-weight	very light-weight
Flexibility	lots	some	little
Eclipse knowledge required?	lots	none	none
Groovy knowledge required?	lots	lots	little
How to use/install	update site	included with library	preferences/quick-assist

I'd still recommend that you come see the talk at EclipseCon (if you are lucky enough to be attending the conference). I'll go into more detail on implementation and I'll show some live demos on how this all works. See you at EclipseCon!

Better JavaScript content assist in Eclipse Orion

Even in the month since I started using and working on Eclipse Orion, there have been significant improvements to it. The UI is getting cleaned up, navigation is getting easier, git integration is progressing, and search is improving. It's great to be contributing to such a fast moving project, even if the rate of change can be dizzying sometimes. Also, I'm new to JavaScript and I have never worked on such a large code-base written in a dynamic language. One of the things that I have been missing the most is good tool support that really knows about the code you are working on. Traditional IDEs set the bar high in this area.

I'm happy to introduce a small step in the direction of making the JavaScript editor smarter. I've just released an Orion plugin that provides semantically aware content assist in the JavaScript editor. The plugin uses the Esprima JavaScript parser with some extra error recovery added by Andy Clement. I have found the Esprima parser to be fast, clean, and easy to use and using it as the core of the content assist plugin has been the right thing to do, even though this project, too, is fast moving and hard to keep up with.

How it works

Like content assist in Java editors in Eclipse, the Esprima content assist plugin is based off of a semantically rich abstract syntax tree (AST) and so content assist proposals are more likely to be relevant than if we were using a lexical approach to content assist. Here is what happens:

1. On a content assist invocation, the contents of the buffer are parsed by Esprima.
2. The resulting AST is walked by the content assist plugin.
3. While walking the AST, the target type of any AST node is recorded as well as assignments and declarations. This information helps us keep track of what properties are available on each known type at any given point in the AST.
4. After walking a sufficient amount of the AST (we don't need to walk the entire tree since parts of it are not going to be relevant for a given content assist invocation), all available proposals are calculated based on the target type of the invocation offset and the prefix.

The best way to understand how this works is through examples.

What it can do

Recognizing function scopes

As you can see in this screenshot, scoping is respected and identifiers that are not accessible in the current scope (vInnerInner, v3, v4,…) are not shown in content assist.

Object literals

The key/values of object literals are appropriately proposed:

Even nested object literals are recognized:

Simple control flow

Simple control flow is recorded by the plugin, so that assignments are remembered:

Pre-defined types

Some (but not all) predefined types are available in content assist.

Currently, the plugin recognizes JSON, MATH, Number, String, Boolean, and Date, but I will probably add more as it makes sense.

Constructors

Functions that start with capital letters are considered constructors

Parser recovery

Finally, Andy Clement has been doing some work on making the esprima parser recoverable from errors. Actually, some error recovery is already in esprima, but we need to tweak it a bit for content assist. Hopefully, this work can be contributed back to esprima after we have a good solution. Currently, the recovery work is focussed on errant dots. A common case is that you will type a variable name and then a '.' and expect content assist to provide all reasonable answers. Most JavaScript parsers will fail after the first error, which makes them quite useless when editing code.

As you can see in the screenshot, despite all of the funky dots, the plugin is able to realize that myVar is of type Number and is providing appropriate proposals.

What it can't do (yet)

This is still early for the content assist plugin and there is quite a bit of work to do. For example:

1. There is no pre-defined window object, which probably should be there, along with possibly other predefined objects, like dojo, dijit, and $ (jquery).
2. There is no analysis of function return types
3. No inter-file type inferencing, which will be crucial for getting anything really smart working
4. The plugin should recognize /*global */ comments
5. The Esprima-based proposals are currently intermingled with proposals from the default JS content assist plugin and so duplicates appear. (Esprima proposals are always prefixed with a handle (Esprima) so you know where they come from, but they are always on the bottom).

I hope to deal with each of these issues eventually, but I also need to make sure that performance remains reasonable, which it currently seems to be, but is something I need to watch.

How to get it

Mark Macdonald has already added the pluign to the Orion plugin page, so after you log into Orion, click to the "Get Plugins" link and select the Install link for the Esprima content assist plugin:

The github page is located here: https://github.com/aeisenberg/esprimaContentAssist so try it out, have a look at the code and let me know what you think!