This weekend I found some time to give it a spin: I hooked into Counterclockwise to generate a Graphviz DOT representation from the text selected in the Clojure editor, and pass that to the Zest graph view:

Using Graphviz DOT, we can create an image file for the visualized structure (e.g. for documentation):

Given the simple structure of Clojure code, supporting different nested data structures like maps, their entries, vectors, and sets was easy to implement (though I’m sure there are cases that won’t work yet):

This alone might already be useful for exploring and explaining parts of the code you’re working on, but I think a closer integration with Counterclockwise could be really interesting. One idea would be to link back from the tree, e.g. by highlighting the corresponding text in the editor, or even allowing real interaction with the tree to manipulate the code, like dragging and dropping nodes and branches to refactor the code. If the code you write really is the graphical structure you see, some kind of visual programming might actually make sense, even if it’s not exactly what people expect (like programming for non-programmers).

I’m not sure where or how to evolve this, but if you want to try it or contribute, you can check it out on GitHub.

The tag clouds are highly customizable. To demonstrate the possibilities, Cloudio contains a sample RCP application. It can load a text file, provides access to the visualization parameters, and supports image export:

Cloudio is part of the Zest Git repository. Its help content (like the other Zest documentation) is generated from the Eclipse wiki. Builds and p2 repos are available from the nightly and integration CI jobs. Integration builds can also be installed from the marketplace. You can get in touch in the GEF forum and report bugs on Bugzilla.

To hook the workflow into the Tycho build, I was basically able to use the info in the very good and detailed tutorial by Karsten Thoms. To make it work for our specific project setup and for the latest Xtext version I still had to tweak some minor things, so I thought I’ll do a quick writeup of what I have actually done to make it work. This assumes you already have an Xtext project and you’re already building with Tycho in general. If not, check out the tutorial mentioned above.

To add the MWE workflow to the Tycho build, all I had to do was add a plugin repository in the parent pom:

<pluginRepositories>
  <pluginRepository>
    <id>fornax-snapshots</id>
    <url>http://www.fornax-platform.org/archiva/repository/snapshots/</url>
    <snapshots><enabled>true</enabled></snapshots>
  </pluginRepository>
</pluginRepositories>

Add a plugin pointing to the MWE workflow file in the Xtext grammar project pom:

<build>
  <plugins>
    <plugin>
      <groupId>org.fornax.toolsupport</groupId>
        <artifactId>fornax-oaw-m2-plugin</artifactId>
        <version>3.3.0-SNAPSHOT</version>
        <configuration>
          <workflowEngine>mwe2</workflowEngine>
          <workflowDescriptor>src/parser/GenerateLang.mwe2</workflowDescriptor>
        </configuration>
        <executions>
          <execution>
          <phase>generate-sources</phase>
          <goals><goal>run-workflow</goal></goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

And define the Xtext grammar project source folder as a resource directory:

<build>
  <resources>
    <resource><directory>src</directory></resource>
  </resources>
</build>

When running mvn clean install, the MWE workflow now runs as part of the grammar project build.

Now something like Contracts for Java is only really useful if it is well integrated into the tools you use everyday – which in my case is Eclipse. Eclipse allows many levels of integration, and I quickly had it working by running javac as an external tool – which is OK, but not ideal, since it requires running the tool manually after every change to the annotations, and doesn’t give in-place error messages. I quickly found out Eclipse has support for custom annotation processing. I had to fiddle a little until I got that configured properly with Contracts for Java so I thought I do a quick write-up so other people can try this cool new tool in Eclipse.

Contracts for Java doesn’t provide any binary downloads yet, so my first step was to build a Jar. To use it, create a Java project, and add cofoja-latest.jar and asm-all-3.3.1.jar to the build path. Then add some code that uses contract annotations (copy and paste the code below into the src folder of the project):

import com.google.java.contract.Ensures;
import com.google.java.contract.Requires;

public class Contracts {
	public static void main(String[] args) {
		System.out.println(new Numbers().add(-10, 5));
	}
}

class Numbers {
	@Requires({ "c > 0", "b > 0" })
	@Ensures({ "result > a", "result > b" })
	int add(int a, int b) {
		return a - b;
	}
}

Next, configure the annotation processing in the project properties > Java Compiler > Annotation Processing. To make Contracts for Java work with the Eclipse Java compiler, currently some specific properties have to be set up: set the com.google.java.contract.classpath property to the location of your cofoja-latest.jar and the com.google.java.contract.classoutput property to the location of your project’s src or separate bin folder (both relative to the project root, using the %PROJECT.DIR% variable):

If your project uses multiple Jars they currently have to be given with absolute paths as the com.google.java.contract.classpath value, separated with ‘:’ on Unix or ‘;’ on Windows. Next, make sure you have set up your workspace to refresh automatically in Preferences > General > Workspace to make Eclipse see the resources generated by the annotation processor. Then, add the cofoja-latest.jar to the project properties > Java Compiler > Annotation Processing > Factory Path:

Now issues in the annotation values are reported in the editor, like variables that cannot be found, e.g. for our code above:

To have the contracts checked at runtime, add -javaagent:cofoja-current.jar to the VM arguments of your run configuration:

Now when running, we are made aware of the violated precondition in our code:

After we fix the call violating the precondition to new Numbers().add(10, 5); we now see our implementation of add does not fulfill the postcondition:

After fixing our implementation to return a + b; the code now runs without errors. For further information on using Contracts for Java check out the project wiki. General information on design by contract can be found elsewhere.

For the initial prototype implementation, we used XML files in zips to store the page content, combined with TrueZIP, a library which allows access to files in zips the way it should be: like files in a folder. So we could basically create and use the zip entries like this:

val zipEntry = new File("PPN345572629_0004.zip/0001.xml")

The XML is created using Scala’s XML support, e.g. the top class in our domain model (Page) is serialized by specifying the root element and calling toXml on each containing element (the words), like this:

def toXml:Elem = <page> { words.map(_.toXml) } </page>

The XML is deserialized with the reverse action, by passing the deserialized word elements contained in the root element to the Page factory method:

def fromXml(page:Elem) = Page( for(word <- (page \ "word")) 
  yield Word.fromXml(word) )

So that was basically our prototypical persistence mechanism. Of course, to actually collaboratively correct our digitized texts, we needed a central DB instance on some server. Or maybe not exactly: having heard good things about CouchDB (in german), it was the first option I looked into. The distributed nature of CouchDB sounded very interesting and seemed appropriate for our project – not for performance reasons, but to provide offline or decentralized editing as an option.

So I looked into CouchDB with scouchdb, a Scala API to access it (and a very welcoming project). After input from my colleague, I became aware that we’d lose quite a bit of installation ease if we pick a non-Java DB, so CouchDB was not really a client side option for offline storage, and therefore the strongest reason for us to consider it no longer there.

Next I considered an embedded relational Java DB with some ORM or ORM-like access. A JPA solution like EclipseLink or Hibernate seemed overkill and not ideal for Scala. Instead I took a closer look at ScalaQuery, which uses Scala’s for expressions for DB queries, which seems like the thing you want if you happen to use a language that basically has a query language built-in.

For instance, in ScalaQuery, we could write a query to get all pairs of pages and words where the page contains the word and the original form of the word is Bonifaci like this:

val bonifaci = for (page <- Pages; word <- Words;
  if page.id === word.pageId && word.original === "Bonifaci")
  yield (page, word)

To allow this kind of queries, we need to define tables named Pages and Words. Defining the tables with ScalaQuery is quite elegant, e.g. our main table for pages could be defined like this:

object Pages extends Table[(String, java.sql.Blob)]("pages") {
  def id = column[String]("id", O PrimaryKey)
  def image = column[java.sql.Blob]("image")
  def * = id ~ image
}

However, this leads to either a duplication of our domain objects (have a table object as above and the original class) or a complete rewrite (use table objects like above as the domain objects), which both seemed wrong.

Also, thinking about alternatives to our XML representation, I came to realize that we would want some form of XML export anyway to ensure long-term access to our data, which is in the public domain and a cultural asset. Of course it would be possible to implement this as some form of export, but if possible it would be just perfect if we could use a single persistence mechanism.

I did remember there are some XML DBs around, but initially abandoned that idea since XML is only half of what we store (the other thing being the images used to correct the text, see screenshot above). But after being unhappy with these other solutions, I took a closer look at eXist-db, an XML DB licensed under the LGPL which by default runs on Jetty.

And I learned that eXist actually supports both XML and binary data, which was just what we needed. With eXist, we also get XQuery support and various interfaces to the data. In our case, this is less of an immediate concern, given Scala’s XML support and for expressions are a bit like built-in XQuery capabilities. For instance, a query on XML that is equivalent to the ScalaQuery from above could look like this:

val bonifaci = for (page <- xml\"page"; word <- page\"word";
  if (word\"@original").text == "Bonifaci") yield (page, word)

This is not only about as concise as the query above, but also allows to express the hierarchical nature of our data in the query (‘each word in the page’), instead of having to map the relational representation (‘all words with a certain page ID’).

As we already deserialize the XML to page objects, what we actually do is query on these. For switching the serialization from entries in a zip file to the eXist DB and allow these queries, the only conceptual change was to store the XML and image files in the DB instead of using a File object, which allowed a really smooth transition.

And with Scala’s rich semantics and XML support, our DB wrapper in Scala can be very simple and precise about what it offers in its API, namely XML documents for a collection ID (if found in the DB):

val xmls:Option[List[Elem]] = Db.xml(collection)

Or binary data:

val imgs:Option[List[Array[Byte]]] = Db.bin(collection)

The elements provided by the DB in this way can be passed to the original deserialization method of Page:

val pages:List[Page] = for(page <- Db.xml(collection))
  yield Page.fromXml(page)

Given this, we can now use a plain Scala for expression on the deserialized objects to query our data like this:

val bonifaci = for (page <- pages; word <- page.words;
  if word.original == "Bonifaci") yield (page, word)

At this point we have replaced the file storage with the DB, but are still creating the XML manually, using Scala’s XML literals. To avoid any duplication (each class parameter or field is represented again in the serialization logic), and basically eliminate all direct XML manipulation, we could use an XML binding library like scalaxb, JAXB-RI or MOXy.

On the other hand, the type safety of the Scala XML serialization code makes the duplication feel much less dangerous than traditional, string-based XML serialization (which can break easily unnoticed by the compiler). And being about as concise as some JAXB annotations and marshalling code would be, the disadvantages seem almost entirely theoretical, but we’ll see – swapping the serialization logic should be as easy as swapping the storage implementation was.

I was amazed to see that Eclipse 4.0 (labeled Early Adopter Release) is already completely usable for me (even if it doesn’t yet offer all the things I’m used to in the current version – perspective customization and custom key bindings are things I hit). And I really like that fresh look – a whole new Eclipse, after all these years!

So here is a quick tour of some new dot4zest features, running on Eclipse 4.

A new feature in the DOT to Zest interpreter (which now uses the Xtext switch API generated for the DOT grammar) is support for ad-hoc DOT edge definitions, without having to declare the nodes before using them in an edge, e.g.

digraph { node[label="Node"] 1->2; 2->3; 2->4 }

Also new is support for the rankdir=LR attribute in the DOT input, e.g.

digraph { rankdir=LR; 1->2; 2->3; 2->4 }

The rankdir=LR setting is supported in Zest to DOT export, too: if a Zest graph using the horizontal layout is exported to DOT and rendered with Graphviz, the direction will be correct for the result, e.g. if we hit the export button in the view above, this is opened:

If the DOT input contains features that are not yet supported in the Zest representation created by dot4zest, the import will now handle the input better and try to display the parts it can make sense of, e.g. DOT with subgraphs can be rendered with Zest, but subgraphs are ignored:

digraph { subgraph cluster_1 {1->2} subgraph cluster_2 {1->3} }

The Graphviz image export can be customized (see toggle above) to use the original DOT input instead of a DOT representation generated from the Zest graph, e.g. with the view and the input above, the export will render the subgraphs from the original DOT input:

Finally, the graph view now contains a toggle button (enabled in all screenshots above) to switch between different modes: listen to changes made to DOT graphs in workspace files and visualize them with Zest automatically (useful for programs outputting DOT or for live preview during DOT editing), or visualize only files loaded explicitly.

The dot4zest code is part of Zest (a GEF component) and available from the Tools repository. It is not available from an eclipse.org update site yet, but can be easily installed (for Eclipse 3.6 and 4.0) from the marketplace (Help > Eclipse Marketplace).

I think the domain of this project is very interesting, as it not only aims at improving access to cultural assets, but also takes the community wiki approach, which leads to a situation that is probably not the most common case in academic software development: users are part of the plan.

But perhaps even more, I’m excited about the technical parts of the project, as we use both Scala for the core, non-UI components and e4 for the UI.

Our first milestone for the initial beta (which we plan to release in summer) provides a basic local, single-user implementation of our system. It features basic search and page selection, editing and persistence, word-based edit history and highlighting of the currently selected word in the original scan.

As expected, combining two cutting edge technologies (Scala 2.8 and e4) does cause some inconveniences (e.g. I currently can’t develop the Scala and the e4 code in the same Eclipse instance, as the Scala IDE for Eclipse does not yet support Helios and e4 won’t run on Galileo).

But once I had a proper Ant build system in place (compile Scala sources, deploy as Jars into the e4 workspace, etc.), I was amazed about the concise code base we were able to build both for the backend and the UI with this approach.

To give some examples of what I liked about developing using Scala and e4 I’ve picked out a few lines from both sides of the current (not very large) code base.

On the Scala side, I really liked having features like case classes, XML support, and concise anonymous functions.

For instance, the XML serialization method of a Page defines that the full structure of the XML is to be filled with the XML representation of each Word of the Page:

def toXml = <page> { words.toList.map(_.toXml) } </page>

We also needed a method to get a simple text representation of the Page, made only of the most recent form of each Word:

def toText = ("" /: words) (_ + " " + _.history.top.form)

And to implement a simple full text search for collecting pages in which the edit history of a Word contains the given term, I checked each Page like this:

page.words.exists(_.history.exists(_.form contains term))

On the e4 side, the whole development experience feels much more lightweight and modern than developing with Eclipse 3 thanks to e4 features like the Toolkit Workbench Model, dependency injection and CSS styling.

For instance, the view at the bottom of the application (showing the original scan) needs to listen to the selection of the Page in the top view (showing pages matching the search term). To listen to the selection of pages, the view defines a method with the following signature:

@Inject public void setSelection(@Optional @Named(SELECTION) Page page)

At the same time, the bottom view also needs to react to the selection of a word in the central edit view, in order to highlight the particular section in the page. Again, we define that we want to get called if a word is selected somewhere (which is represented by a Text widget here):

@Inject public void setSelection(@Optional @Named(SELECTION) Text word)

And I think it’s really awesome to be able to specify font style and size (and even gradients) for the Text widgets used in the edit view with CSS:

Text { font: Times 14px; background-color: #e8e8e8 #cccccc 60%; }

With the information on how the views are put together in the TM and the CSS file containing the styling information, the remaining UI code feels much lighter than before: there are basically just a few view classes and handlers, communicating loosely via DI and the TM.

So for me, combining Scala and e4 is a great new way to write a Java desktop app! And hopefully more… Our plans for the beta include multi-user support and web deployment with RAP – some challenges for sure, but with the great start I had into e4 development, I’m very confident it will work out!

We have been working on this in public from the start (after all, it’s publicly funded), so you can take a closer look at the code or check out this very early state of the application on GitHub.

I was a SOC student at Eclipse last year and it has been an awesome experience! After completing my project, I wanted to create and spread some SOC swag and had stickers printed from the official Eclipse SOC logo. Since it was the first time I did this, after sending multiple different sizes of the logo, the sticker shop gave up on me and just sent me the double amount I had ordered, but uncut.

So even after flooding both Ian Bull (my SOC mentor) and Gen Nishimura (the original logo creator) with real-world postal sticker spam, I still have plenty left! So if you’d like your own Eclipse Summer of Code stickers, hand-cut by a genuine Eclipse SOC student, just mail me your address, and I’ll be happy to ship you some.

Outdated and duplicated documentation is one of the topics I keep encountering in basically every project I work on. Given that I was already generating the dot4zest help with WikiText (but from a file, not straight from the wiki) I decided to give it a try.

Following the instructions in the crowdsourcing example, I first got the WikiText standalone Jars and set up a small Ant script:

<project name="dot4zest" default="doc" basedir=".">
 wikitext.tasks.classpath">
    <fileset dir="Graphviz_DOT_as_a_DSL_for_Zest">
        mylyn.wikitext.*core*.jar"/>
    </fileset>
 </path>
 <taskdef classpathref="wikitext.tasks.classpath" resource=
  "org/eclipse/mylyn/internal/wikitext/mediawiki/core/tasks/tasks.properties"/>
 <taskdef classpathref="wikitext.tasks.classpath" resource=
  "org/eclipse/mylyn/wikitext/core/util/anttask/tasks.properties"/>
 <target name="doc">
     <!-- Set up the base wiki location to pull content from: -->
     <mediawiki-to-eclipse-help wikiBaseUrl="http://wiki.eclipse.org"
        validate="true"
        failonvalidationerror="true"
        prependImagePrefix="images"
        formatoutput="true"
        defaultAbsoluteLinkTarget="doc_external"
        dest="${basedir}"
        title="Graphviz DOT as a DSL for Zest"
        generateUnifiedToc="false">
     	<!-- Set up which wiki page to pull from: -->
        <path name="Graphviz_DOT_as_a_DSL_for_Zest"
                title="Graphviz DOT as a DSL for Zest"
                generateToc="true"/>
     </mediawiki-to-eclipse-help>
 </target>
</project>

Running the Ant file (ant in a console or double-click after dragging it onto the Ant view in Eclipse) pulls the content from the wiki location specified in the script and generates the Eclipse help:

Next, I updated the help TOC extension in the plugin.xml:

<extension point="org.eclipse.help.toc">
      <toc file="Graphviz-DOT-as-a-DSL-for-Zest-toc.xml" primary="true"/>
</extension>

Then I added the generated help resources to the binary build in the build tab of the manifest editor to have them included in the deployed bundle:

With this, the wiki page content is included in the Eclipse online help of the bundle (Help > Help Contents).

Being based on Mylyn WikiText – which supports many wiki markup formats (textile, confluence, mediawiki, trac and twiki) as well as output to other formats than Eclipse help (e.g. DocBook and DITA) – I imagine this approach could be used in many project documentation setups.

I really like this WikiText feature both for technical and collaborative reasons: it reduces duplication (DRY for docs!) and eases collaboration on documentation. Instead of being part of the code repository, the help content is made available to a wider range of editors than the people committing code – which I believe is useful for projects of any size.

Say we are editing embedded DOT in wiki markup with the Mylyn WikiText editor and have an open dot4zest Zest graph view, which displays a Zest representation of the DOT graph (powered by the EMF model visualizer and Xtext – vote Xtext!):

Now upon saving, dot4zest asks if we want to add a reference to the Graphviz image export to our wiki markup file (requires a local Graphviz installation, for which we are prompted the first time the export happens):

If we say yes, dot4zest puts a reference to the exported image file into our plain text wiki markup file. This adds an image representation of the Zest graph displayed in the view to output generated from the plain text – like HTML export or the preview tab of the Mylyn WikiText editor:

Being sort of pre-incubation, this is all still fairly restricted and experimental, e.g. it currently supports only a single graph per file and only Textile markup. I’ve added these latest changes to the update site, so if you want to check it out update your existing dot4zest installation or install it from the update site at http://quui.de/updates (category Zest -> feature dot4zest).