Tuesday, 30 July 2013

Configuring vim in Windows

I like to use vim - I've learnt enough of the shortcuts for it to be powerful enough in my hands to be a serious alternative to Eclipse when editing non-project files. However, as a Windows user (yes, alright, why am I?) I often have trouble setting up my .vim directory and .vimrc file. There are a few helpful hints that I really ought to remember:

  • Instead of .vimrc, the filename is _vimrc - because Windows doesn't like filenames beginning with a dot.
  • Instead of .vim, the directory name it is looking for may be vimfiles - for the same reason.
  • Depending on the configuration of Windows, it may not be looking for your files where you think it is - to find out you can do :echo &rtp
Now I know where to put this useful snippet that makes the markdown editor work with Jekyll's YAML frontmatter. Phew.

Monday, 24 June 2013

Adventures with Scala Macros - Part 4

The Adventures with Scala Macros series was published on the ScottLogic company blog. It is reproduced here.

In the previous articles I built up a macro that generates a RESTful API from a case class model in a Play application. We now want to add a database to this API - in part 3 we used a simple map-based in memory data store, which wouldn’t really be that useful in the real world.

Step up MongoDB, a Scala library for interacting with it, Salat (which uses Casbah under the hood), and a Play plugin for making the integration with our Play application easier, Play-Salat. After following the install instructions, I’ve configured the play application to connect to Mongo:

We’ll use Play-Salat’s example Salat Context, and then we’ll create a set of traits that can be applied to the companion objects for our case classes. However, before we do that we need to change the case classes to use the Casbah ObjectId for their identifiers, so Author becomes:

However, when you try and compile this, Play now gets upset because it doesn’t know how to turn an ObjectId into JSON. A quick read of the documentation and we see that we need to include an implicit Format[ObjectId] to do the conversion to/from JSON. We can implement this as follows:

However, we want this value to be available in the implicit scope in the middle of our macro that creates the formatter using the Play JSON macro, so how can we do that? We’ll look at a way using the macro itself. The way you might do this if you weren’t writing a macro is put the implicit value in an object, and then import myObject._ to get the value into scope wherever we use it. Well, we can easily define a class that can be instantiated to form such an object:

Now we need to tell the macro what class it needs to use to get the implicit values - which we can use a type parameter for:

Which would be declared on the the macro:

By declaring the T on the implementation of the macro as c.WeakTypeTag, we can use it within the macro to get at the actual class that is being used as a type parameter using the implicitly function:

This gives us the ClassSymbol for the type parameter, which we can then use in a Ident to get at the class’s constructor, define a value for it, and then import all its values:

This is basically defining a value as the result of calling the no-argument constructor on the class from the type parameter. The result looks like this:

So now our Play application compiles again. Let’s now move on to creating the Mongo DAOs. First off, we need an abstract class for our companion objects to extend. From the Salat docs, we can see that the class needs to extend ModelCompanion for type parameter T, and needs an implicit Manifest[T]. The SalatDAO requires that T extends AnyRef, so we’ll add a bound for the type parameter. We can then create a dao value that is created using a mongoCollection. All we need then is an abstract collectionName to use for each class’s collection:

Now we can create Mongo traits for the CRUD operations. Let’s start with Read:

This is pretty simple - a template value is defined for the dao (that will be supplied by the MongoDb abstract class), and an Option for the object is found using the id. Let’s do another one - Delete is slightly more complicated, as we want to only delete if a record is found - a simple match expression should do the trick:

Now let’s look at Write. We’re need to take an object of type T, find the corresponding record, and update its state to that provided. To find the record we need the ID, but the object we have is of type T <: AnyRef, so we don’t have access to its ID. Simple enough - we introduced the WithId abstract class in the last article - we can update it to use ObjectId, and make sure the type parameter is bound to extend AnyRef:

Now we can use this in the MongoWrite trait to get the id:

I’m sure you’ve got the hang of these traits now - they’re pretty simple. Here’s the rest of those that we need:

Finally, we add the new traits to the companion objects, adding just the ones we want for each case class:

So that’s it - a RESTful API backed by MongoDB with the minimum of code using Scala def macros. I hope you’ve found this introduction interesting, and if you would like to browse the source code, please head over to Github.

Adventures with Scala Macros - Part 3

The Adventures with Scala Macros series was published on the ScottLogic company blog. It is reproduced here.


In parts 1 and 2 we used a macro to generate some regular expressions and match them against a path to extract IDs. Now we’re going to use that pattern matching to call an API.

First, we’ll define some traits for the RESTful actions that the client code can add to the companion object of the case class that is the entity being manipulated:

Then to make calling the implementations of these traits a bit more simple, I’m going to write a supertrait that does the Play work to handle the three basic types of requests:

  • Take string and return entity/entities (Read, ReadAll)
  • Take entities and return nothing or a simple response (Create, ReplaceAll, Write)
  • Take a string and return nothing (Delete, DeleteAll)

The first two of these require transformation of a string to/from JSON - this is done in Play by using the parse.json parser with the Action, and then validating the body as type T for read, and using Json.toJson on an object of type T for write. Each of these requires a Format[T] object in the implicit scope.

The actions that deal with single items will need an id field that can then be not provided for the other actions in the same function, id: Option[String]. The structure for the trait will therefore be:

Note that we don’t really need to have these different actions grouped together in those three functions - we could have a function for each - we’ll refactor the class later - but for the moment it will help us in implementing similar behaviour consistently.

So now we’ll go through implementing one of each of the types of requests.

Reading JSON (and writing data)

By using Action(parse.json) to construct our action, the body of the request is now an instance of play.api.libs.json.JsValue, which can then be validated as a type T. This then returns a play.api.libs.json.JsResult[T], from which we can either get an object of type T, or a validation error:

We know that this trait is a supertrait of the Create[T], so we can now easily implement the function to handle the valid case by casting this to an instance of that trait:

We’ve used two Play Result instances here - Ok is a standard HTTP 200 response with a body to be returned (the id of the new entity), and InternalServerError is a 500 response, that we could customise with a body if we wanted to.

Writing JSON (and reading data)

This is the inverse of the reading JSON - we use the play.api.libs.json.Json.toJson function to convert an object of type T into a JsValue:

This time we’re returning either a 200 response with the entity rendered as JSON, or we’re returning a NotFound result - a 404 response.

No Content

In the case of the delete actions, there is no inbound data to convert, and there is nothing to return either - a successful HTTP status indicates the delete happened - so we use the NoContent Play result:

We now have enough to complete the RequestHandler trait:

Modifying the macro to use the API

Now that we have a RESTful Scala API, we need to change our macro to generate calls to it for the different methods that have been implemented for each entity.

Finding implemented traits

Previously we’ve not bothered about what’s possible with the case classes we’ve found in the classpath of the macro call, but now we only want to generate endpoints for things that are possible - for example, a read-only entity would only have Read[T] and ReadAll[T] implemented on their companion object.

You’ll remember that we were previously finding case classes by looking for compilation units that match case ClassDef(mods, _, _, _) => mods.hasFlag(Flag.CASE). However, we now want to check the compilation unit’s companion object as well as the class itself, so we’ll need to use the class’s ClassSymbol and the corresponding ModuleSymbol to the companioned types. The ClassSymbol for a class can be found using c.mirror.staticClass with the fully qualified name as the argument.

So that’s given us a list of the names of classes that have one of our traits on their companion object, which is the list of classes we need to generate code for. Next we need to know exactly what those traits are, so that we can generate calls for the appropriate HTTP methods. Obviously we can do this by using similar code to that used in the isRestApiClass function above:

Matching against the request

Now that we’ve got a list of what classes have what traits, we can start generating our AST.

Static objects

The regex generation hasn’t changed from before, but for each class, T we now need a JSON formatter, Format[T]. Fortunately, Play provide a macro that can generate you a formatter for a given case class, so we can use that function:

The selectFullyQualified function here could probably do with a little explaining - it’s a helper function that turns a fully qualified class or function name into the required Select/Ident tree required to use it in the generated code. We then use that function to select the function we’re going to use, and call it, providing a type argument of the case class we’re processing. The Play format[T] macro function has a problem if you pass its type argument without that type being imported in the code that is calling it (i.e. our generated AST), as it generates code that refers to that type without fully qualifying it, so we include an import in our generated code.

Path and method matching

For a RESTful API, we need to match on both the path and the HTTP method (GET, PUT, POST, DELETE) - the easiest way to do this is using a tuple, so we’ll change our macro’s method signature to include a method parameter, and to return a Play Handler:

Now our case expressions will all be tuples of expected values - requests that are acting on collections will need to match the collection path, and the other requests will need to use the regex matching to extract the ID. We can generate a case statement for each of the traits that we find on the class’s companion object:

Similarly, for each trait we know what method we’re going to call with what arguments, so we can have another case statement to generate that:

And then we can combine the two functions into our case definition:

To finish off the changes, we just need to make the MatchDef also use a tuple, which is simple enough:

Testing it out

Now we need a Play application - downloading the latest Play build, and creating a new application, I’ve then copied the model for the previous posts, and then we need to use the macro to generate the REST endpoints. The place to do this is in a GlobalSettings object:

Now to test the macro out we need something to back our RESTful services. For this article I’m just going to use a simple Map-based in-memory data store:

Then all we need to do is add this trait to our data model. I’ve abstracted away the ID generation, so we end up with:

And there we have it - run the Play application, and you’ve got a working REST service. In the final article we’ll try using something a bit more useful than an in-memory map by implementing a MongoDB data accessor.

Footnote [1]: The term RESTful often leads to animated discussion about whether or not that which is being described properly implements the principles of REST. To be clear, by ‘RESTful’ I mean in-the-style-of-REST, and not necessarily strictly conforming to the principles of REST. Please let me know if you think there are some improvements I could make.

Adventures with Scala Macros - Part 2

The Adventures with Scala Macros series was published on the ScottLogic company blog. It is reproduced here.

Where next?

In part 1 we used a macro to generate some regular expressions and a case statement that used them:

What you’ve probably noticed though is that with this code, the regexes are parsed each time through the block, which isn’t going to be very good for performance. What we really want is to be able to put the regular expressions in an object so that they are parsed only once. But we’re using Scala 2.10, and all we have available are def (function) macros - we can’t generate any other types (although type macros do exist in the macro paradise).
Instead we need to create a value which then gets populated by a function that returns an object, like this:

So now we’ve got two functions for our generated code - and we know how to generate function bodies. However, we have another problem - each function will be generated by a separate macro, and we’ll need to know what the variables created in one macro are called when using them in the other macro. Fortunately we can just avoid that problem - each regular expression is generated from a class name, and each class name (with its package qualifier) is unique, so we can use the class name to generate the variable names (replacing . with $).

Code organisation

Our macro code is getting bigger, and I don’t want it to end up as a pair of unmanageble functions that go on for page after page - so I’m going to split the code up into two classes - the one that has the macro definitons in it and builds the resultant AST for the generated code, and another class that contains all the logic for analysing the compilation environment.
As documented on the def macros page on the Scala site, you can create other classes that use your macro context easily enough - you just have to give the compiler a little helping hand to work out where the context comes from. Of the two examples for doing this on that page, the second is (to me) much more readable, so we’ve got a new class:

And obviously we can initialise this just like any other class from within our macro implementations using val helper = new Helper(c). If, for tidyness, you want to import the functions defined in Helper, you can then do import helper.{c => cc, _}, which renames the c value from Helper so that it doesn’t collide with the c parameter from our function signature.
Moving the compilation unit processing code into the Helper, and adding some name processing so that the class name and package name are available to our macro, we end up with:

Object Creation

When you define an object in normal Scala, you just declare it, object myObject, because the syntactic sugar allows you to leave out a default constructor, and that it extends scala.AnyRef. In a macro you don’t have that luxury, so to define a new object, you do the following:

We already know how to put a block of code together from part 1, so all we need to do is merge the two together, and we get:

This can then be used by declaring val regexesUsingMacro = restapi.Generator.macroPaths in our unit test.
But wait - there’s a problem, isn’t there? That function is returning an object of type Any, so all our other generated code will know about it is that it’s an any - it won’t know anything about the vals we’ve added to it. Well actually, it turns out this isn’t a problem, but is intended that the return type should know more than its declaration specifies, if possible, as Eugene Burmako explains here.

Putting it all together

Now that we’ve got a macro that can be used as a val definition, we need to find that val and use it in our match expression. To find the val is simple enough - we just look for a ValDef that uses a Select for our function as the right hand side. However, if the user hasn’t defined such a val, we can’t continue - we need to tell the developer what they’ve done wrong. The macro Context includes functions to provide compiler warnings and errors, so we need to emit an error advising the developer how to fix it. The structure we end up with is as follows:

When integrated with the code we had for the match block from part 1, we end up with:

So now we’ve got our pattern matching working well, in the next article we can start calling an API to produce our endpoints.

Wednesday, 27 March 2013

Adventures with Scala Macros - Part 1

With the arrival of Scala 2.10 came two interesting new features: the reflection API and macros. The possibilities for macros are many, and one that I want to explore is how you can use a macro to create a Play Framework REST API from a data model represented as a set of case classes. In this and subsequent articles I'll be using a simple book store type application with case classes for Book, Author and Category in the model package.

Normally in the Play Framework you'd declare your rest API in the routes file, something like:

However, as you can't use custom Scala in routes (since Play 2.0, anyway), we'll be designing a macro for use in the onRouteRequest function of the Global object that extends the GlobalSettings trait. This function takes a request and returns an Action. The selection of what Action to return will be based on the URL, so we'll have to match the path to the right case class.

In this first article, we'll be working out how to do that matching based on an unknown number of case classes in the application's model.

If we weren't using a macro, we'd probably do something like:

Our macro will need to generate the same function by iterating over all the found case classes. Firstly we need to declare the macro, and then complete the implementation, as follows:

What we've done here is declare a macro as described on the documentation linked above. The macro context's universe is imported - it is important to make sure you import the universe exactly as above, as if you use a import like c.{universe => u} then for some reason erasure stops them working properly. We've then used the universe's reify function to return an expression for Some("TODO") - the reify function basically turns the enclosed scala into an AST tree expression.

Finally, we need a unit test to check we've implemented the method correctly:

The caseUsingMacro function will have its body replaced with the result of macro call - i.e. it is here that the function is going to be generated, replacing Macro.macroCase(s).

Now we're ready to start implementing the macroCaseImpl function from above - but what do we need to return, and how can we debug our macro's output to see how we're doing. Fortunately, there are two very useful functions in the universe - show and showRaw. Given an AST tree, the show function will return a pretty-formatted string of the equivalent scala code. The showRaw function will return a string of the AST making up the tree. As what we're trying to implement is something that for our 3 case classes will look like the caseStmt function above, we can get a good hint for what we need to end up with to look at its equivalent AST, which we can get by doing:

The results of this println will be output to the console for the compilation [1]. We get something like this:

We'll break this apart into useful fragments as we get to using them.

So, first we need to get a list of all the case classes in the model package. To do this, we can iterate over the context's enclosingRun.units property, matching on elements of the AST tree for each compilation unit to find out the package name, and then checking that we've got a case class. The rough code we need for this is:

We then need to implement the packageFinder and classFinder functions. To check that the compilation unit is in the desired package, the first of these needs to return true when the Tree instance is a PackageDef with a name equal to, or starting with, the second argument, so:

Then to check that the compilation unit contains a case class, the second function needs to return true if the Tree instance is a ClassDef with a CASE modifier:

So now we've got a list (well, in fact it's an Iterator) of the case class compilation units in the package we need to turn them into regular expressions to match against the URL, and into case statements to do the matching.

Firstly, the regular expressions. If we take the AST that we generated above, we can see that the value definition for one regex is:

Of the four parameters to the ValDef creator, the first, Modifiers(), and third, TypeTree(), we can take straight into our generated AST. The second parameter we can generate using a context function to create a new unique val name, c.fresh("regex$"). The last parameter is using the implicit augmentString function on a String so that we can then call the r function to turn it into a Regex. However, we can do this more simply using the reify function combined with a string expression for the regex we want to use, which we can construct in the macro using the class name:

Here we've created an Expr object of a constant that is the regex string, and then used the splice function, which is a special function that allows an Expr object to be grafted into the middle of the block being reified. We then just call the r function inside the reify block for it to be turned into a Regex.

The other bit of AST that we need to generate for each class name is the case statement of the match block. Again, from the above, we can see that this should look something like:

Reformatting, this becomes a little easier to understand:

Notice that again we've swapped the newTermName("bookRegex") for valName which is the fresh name we got using the "regex$" base name, the name of the defined value that is the Regex.

Now, we need to collect up all the val definitions and case statements so that they can be assembled into a block of scala code. We also need to add the default case:

Putting all this together, and getting the class name from the AST tree for the compilation unit, we get:

So, we assemble the cases and regexes into a block. The return of the block is going to be the result of the match statement, and the regexes are declared before that final statement, so (remember, s is the original parameter to the macroCaseImpl function):

To check that this AST tree really is the Scala code we hope it is, we can use the show function I mentioned before:

And finally, we return the Expr that contains our block of scala:

And when we compile, we see that our Block tree is equivalent to:

Here we can see the generated val names and the familiar regexes and match-case block returning the Option values that are expected. On running the unit test, we find it now passes.


In the next stage of this adventure, I'll be extending this to use persistence of case classes as declared in their companion object, and deducing what REST operations are available based on what data access functions are exposed.

[1] If you're using the Scala IDE it might take you a while to find the compilation output - what you need to do is turn up the logging to DEBUG in Window -> Preferences, Scala -> Logging. Your console output will then appear in [workspace]/.metadata/.plugins/org.scala-ide.sdt.core/scala-ide.log

Monday, 5 November 2012

Making Raspberry Jam

I made raspberry jam last night, and Danielle wanted to know how, so here you go:

Makes 5-6 1lb jars of jam.


4lbs Raspberries (frozen or freshly picked)
3lbs Jam Sugar (this is probably called something else in the US, where they call jam "jelly", but it's basically granulated sugar with added pectin and citric acid to aid setting)
A small knob of unsalted butter


  • Heat oven to GM2-3, 140C, 280F. Pop a saucer in the freezer.
  • Pick over your fruit and discard any leafy/mouldy bits. No need to wash the fruit as the boiling will sterilise it.
  • Put all fruit in a pan - ideally a jam pan with large bottom and plenty of surface area to help with boiling, but you can use a large normal saucepan if you don't have one.
  • Bring the pan to a simmer, and cook fruit for half an hour on the lowest heat you can, and at the same time, put the sugar in a metal container, cover and put in the oven to warm up. Wash up your jam jars if you haven't already.
  • After the fruit has cooked, take the sugar out of the oven and add to the fruit. While you're there, put the jam jars in the oven.
  • Stir the fruit and sugar as gently as you can (to minimise the fruit break-up that will happen) until the sugar has all dissolved. Once dissolved, a wooden spoon covered in jam will have no sugar crystals on the back.
  • Turn the heat up full and boil the jam for 5 minutes. Depending on your pan, you may need to scrape the bottom occasionally to stop the jam sticking. Take the saucer from the freezer and spoon a little bit of jam onto it - it should set quickly into a jam consistency. If it doesn't set, boil for another few minutes and repeat until it does.
  • Take the jam off the heat, and allow it to settle for 5 minutes. Get your jam covers and wax discs ready - I use a few sheets of soaked kitchen towel for wetting the cellophane covers, and remember - wax side of the paper discs faces down, wet side of the covers faces up.
  • Ladel the jam into the hot jam jars almost to the top (a jam funnel is really useful for doing this without mess), and cover immediately. Leave to cool.

Enjoy your jam with fresh toasted crumpets, oven-warmed croissants, scones with clotted cream or just a bit of bread and butter.

Tuesday, 25 September 2012

Kitchen Implements

Bold the ones you have and use at least once a year, italicize the ones you have and don't use, strike through the ones you have had but got rid of, underline the ones you'd like.

I wonder how many pasta machines, breadmakers, juicers, blendersdeep fat fryers, egg boilers, melon ballers, sandwich makers, pastry brushes, cheese boards, cheese knives, crepe makers, electric woks, salad spinners, griddle pans, jam funnels, pie funnels, meat thermometersfilleting knivesegg poachers, cake stands, garlic crushersmartini glassestea strainersbamboo steamers, pizza stones,coffee grinders, milk frothers, piping bags, banana stands, fluted pastry wheels, tagine dishesconical strainersrice cookerssteam cookers, pressure cookersslow cookers, spaetzle makers, cookie presses, gravy strainers, double boilers, sukiyaki stoves, ice cream makersfondue sets, healthy-grills, home smokers, tempura sets, tortilla presses, electric whisks, cherry stoners, sugar thermometers, food processors, stand mixers, mincers, bacon presses, bacon slicers, mouli mills, cake testers, pestle-and-mortarsgratin dishes, apple corers, mango stoners and sets of kebab skewers languish dustily at the back of the nation's cupboards.