Adventures with Scala Macros - Part 4

The Adventures with Scala Macros series was published on the ScottLogic company blog. It is reproduced here.

In the previous articles I built up a macro that generates a RESTful API from a case class model in a Play application. We now want to add a database to this API - in part 3 we used a simple map-based in memory data store, which wouldn’t really be that useful in the real world.

Step up MongoDB, a Scala library for interacting with it, Salat (which uses Casbah under the hood), and a Play plugin for making the integration with our Play application easier, Play-Salat. After following the install instructions, I’ve configured the play application to connect to Mongo:

We’ll use Play-Salat’s example Salat Context, and then we’ll create a set of traits that can be applied to the companion objects for our case classes. However, before we do that we need to change the case classes to use the Casbah ObjectId for their identifiers, so Author becomes:

However, when you try and compile this, Play now gets upset because it doesn’t know how to turn an ObjectId into JSON. A quick read of the documentation and we see that we need to include an implicit Format[ObjectId] to do the conversion to/from JSON. We can implement this as follows:

However, we want this value to be available in the implicit scope in the middle of our macro that creates the formatter using the Play JSON macro, so how can we do that? We’ll look at a way using the macro itself. The way you might do this if you weren’t writing a macro is put the implicit value in an object, and then import myObject._ to get the value into scope wherever we use it. Well, we can easily define a class that can be instantiated to form such an object:

Now we need to tell the macro what class it needs to use to get the implicit values - which we can use a type parameter for:

Which would be declared on the the macro:

By declaring the T on the implementation of the macro as c.WeakTypeTag, we can use it within the macro to get at the actual class that is being used as a type parameter using the implicitly function:

This gives us the ClassSymbol for the type parameter, which we can then use in a Ident to get at the class’s constructor, define a value for it, and then import all its values:

This is basically defining a value as the result of calling the no-argument constructor on the class from the type parameter. The result looks like this:

So now our Play application compiles again. Let’s now move on to creating the Mongo DAOs. First off, we need an abstract class for our companion objects to extend. From the Salat docs, we can see that the class needs to extend ModelCompanion for type parameter T, and needs an implicit Manifest[T]. The SalatDAO requires that T extends AnyRef, so we’ll add a bound for the type parameter. We can then create a dao value that is created using a mongoCollection. All we need then is an abstract collectionName to use for each class’s collection:

Now we can create Mongo traits for the CRUD operations. Let’s start with Read:

This is pretty simple - a template value is defined for the dao (that will be supplied by the MongoDb abstract class), and an Option for the object is found using the id. Let’s do another one - Delete is slightly more complicated, as we want to only delete if a record is found - a simple match expression should do the trick:

Now let’s look at Write. We’re need to take an object of type T, find the corresponding record, and update its state to that provided. To find the record we need the ID, but the object we have is of type T <: AnyRef, so we don’t have access to its ID. Simple enough - we introduced the WithId abstract class in the last article - we can update it to use ObjectId, and make sure the type parameter is bound to extend AnyRef:

Now we can use this in the MongoWrite trait to get the id:

I’m sure you’ve got the hang of these traits now - they’re pretty simple. Here’s the rest of those that we need:

Finally, we add the new traits to the companion objects, adding just the ones we want for each case class:

So that’s it - a RESTful API backed by MongoDB with the minimum of code using Scala def macros. I hope you’ve found this introduction interesting, and if you would like to browse the source code, please head over to Github.

Adventures with Scala Macros - Part 3

The Adventures with Scala Macros series was published on the ScottLogic company blog. It is reproduced here.

A RESTful¹ API

In parts 1 and 2 we used a macro to generate some regular expressions and match them against a path to extract IDs. Now we’re going to use that pattern matching to call an API.

First, we’ll define some traits for the RESTful actions that the client code can add to the companion object of the case class that is the entity being manipulated:

Then to make calling the implementations of these traits a bit more simple, I’m going to write a supertrait that does the Play work to handle the three basic types of requests:

• Take string and return entity/entities (Read, ReadAll)
• Take entities and return nothing or a simple response (Create, ReplaceAll, Write)
• Take a string and return nothing (Delete, DeleteAll)

The first two of these require transformation of a string to/from JSON - this is done in Play by using the parse.json parser with the Action, and then validating the body as type T for read, and using Json.toJson on an object of type T for write. Each of these requires a Format[T] object in the implicit scope.

The actions that deal with single items will need an id field that can then be not provided for the other actions in the same function, id: Option[String]. The structure for the trait will therefore be:

Note that we don’t really need to have these different actions grouped together in those three functions - we could have a function for each - we’ll refactor the class later - but for the moment it will help us in implementing similar behaviour consistently.

So now we’ll go through implementing one of each of the types of requests.

Reading JSON (and writing data)

By using Action(parse.json) to construct our action, the body of the request is now an instance of play.api.libs.json.JsValue, which can then be validated as a type T. This then returns a play.api.libs.json.JsResult[T], from which we can either get an object of type T, or a validation error:

We know that this trait is a supertrait of the Create[T], so we can now easily implement the function to handle the valid case by casting this to an instance of that trait:

We’ve used two Play Result instances here - Ok is a standard HTTP 200 response with a body to be returned (the id of the new entity), and InternalServerError is a 500 response, that we could customise with a body if we wanted to.

Writing JSON (and reading data)

This is the inverse of the reading JSON - we use the play.api.libs.json.Json.toJson function to convert an object of type T into a JsValue:

This time we’re returning either a 200 response with the entity rendered as JSON, or we’re returning a NotFound result - a 404 response.

No Content

In the case of the delete actions, there is no inbound data to convert, and there is nothing to return either - a successful HTTP status indicates the delete happened - so we use the NoContent Play result:

We now have enough to complete the RequestHandler trait:

Modifying the macro to use the API

Now that we have a RESTful Scala API, we need to change our macro to generate calls to it for the different methods that have been implemented for each entity.

Finding implemented traits

Previously we’ve not bothered about what’s possible with the case classes we’ve found in the classpath of the macro call, but now we only want to generate endpoints for things that are possible - for example, a read-only entity would only have Read[T] and ReadAll[T] implemented on their companion object.

You’ll remember that we were previously finding case classes by looking for compilation units that match case ClassDef(mods, _, _, _) => mods.hasFlag(Flag.CASE). However, we now want to check the compilation unit’s companion object as well as the class itself, so we’ll need to use the class’s ClassSymbol and the corresponding ModuleSymbol to the companioned types. The ClassSymbol for a class can be found using c.mirror.staticClass with the fully qualified name as the argument.

So that’s given us a list of the names of classes that have one of our traits on their companion object, which is the list of classes we need to generate code for. Next we need to know exactly what those traits are, so that we can generate calls for the appropriate HTTP methods. Obviously we can do this by using similar code to that used in the isRestApiClass function above:

Matching against the request

Now that we’ve got a list of what classes have what traits, we can start generating our AST.

Static objects

The regex generation hasn’t changed from before, but for each class, T we now need a JSON formatter, Format[T]. Fortunately, Play provide a macro that can generate you a formatter for a given case class, so we can use that function:

The selectFullyQualified function here could probably do with a little explaining - it’s a helper function that turns a fully qualified class or function name into the required Select/Ident tree required to use it in the generated code. We then use that function to select the function we’re going to use, and call it, providing a type argument of the case class we’re processing. The Play format[T] macro function has a problem if you pass its type argument without that type being imported in the code that is calling it (i.e. our generated AST), as it generates code that refers to that type without fully qualifying it, so we include an import in our generated code.

Path and method matching

For a RESTful API, we need to match on both the path and the HTTP method (GET, PUT, POST, DELETE) - the easiest way to do this is using a tuple, so we’ll change our macro’s method signature to include a method parameter, and to return a Play Handler:

Now our case expressions will all be tuples of expected values - requests that are acting on collections will need to match the collection path, and the other requests will need to use the regex matching to extract the ID. We can generate a case statement for each of the traits that we find on the class’s companion object:

Similarly, for each trait we know what method we’re going to call with what arguments, so we can have another case statement to generate that:

And then we can combine the two functions into our case definition:

To finish off the changes, we just need to make the MatchDef also use a tuple, which is simple enough:

Testing it out

Now we need a Play application - downloading the latest Play build, and creating a new application, I’ve then copied the model for the previous posts, and then we need to use the macro to generate the REST endpoints. The place to do this is in a GlobalSettings object:

Now to test the macro out we need something to back our RESTful services. For this article I’m just going to use a simple Map-based in-memory data store:

Then all we need to do is add this trait to our data model. I’ve abstracted away the ID generation, so we end up with:

And there we have it - run the Play application, and you’ve got a working REST service. In the final article we’ll try using something a bit more useful than an in-memory map by implementing a MongoDB data accessor.

---

Footnote [1]: The term RESTful often leads to animated discussion about whether or not that which is being described properly implements the principles of REST. To be clear, by ‘RESTful’ I mean in-the-style-of-REST, and not necessarily strictly conforming to the principles of REST. Please let me know if you think there are some improvements I could make.

Adventures with Scala Macros - Part 2

The Adventures with Scala Macros series was published on the ScottLogic company blog. It is reproduced here.

Where next?

In part 1 we used a macro to generate some regular expressions and a case statement that used them:

What you’ve probably noticed though is that with this code, the regexes are parsed each time through the block, which isn’t going to be very good for performance. What we really want is to be able to put the regular expressions in an object so that they are parsed only once. But we’re using Scala 2.10, and all we have available are def (function) macros - we can’t generate any other types (although type macros do exist in the macro paradise).
Instead we need to create a value which then gets populated by a function that returns an object, like this:

So now we’ve got two functions for our generated code - and we know how to generate function bodies. However, we have another problem - each function will be generated by a separate macro, and we’ll need to know what the variables created in one macro are called when using them in the other macro. Fortunately we can just avoid that problem - each regular expression is generated from a class name, and each class name (with its package qualifier) is unique, so we can use the class name to generate the variable names (replacing . with $).

Code organisation

Our macro code is getting bigger, and I don’t want it to end up as a pair of unmanageble functions that go on for page after page - so I’m going to split the code up into two classes - the one that has the macro definitons in it and builds the resultant AST for the generated code, and another class that contains all the logic for analysing the compilation environment.
As documented on the def macros page on the Scala site, you can create other classes that use your macro context easily enough - you just have to give the compiler a little helping hand to work out where the context comes from. Of the two examples for doing this on that page, the second is (to me) much more readable, so we’ve got a new class:

And obviously we can initialise this just like any other class from within our macro implementations using val helper = new Helper(c). If, for tidyness, you want to import the functions defined in Helper, you can then do import helper.{c => cc, _}, which renames the c value from Helper so that it doesn’t collide with the c parameter from our function signature.
Moving the compilation unit processing code into the Helper, and adding some name processing so that the class name and package name are available to our macro, we end up with:

Object Creation

When you define an object in normal Scala, you just declare it, object myObject, because the syntactic sugar allows you to leave out a default constructor, and that it extends scala.AnyRef. In a macro you don’t have that luxury, so to define a new object, you do the following:

We already know how to put a block of code together from part 1, so all we need to do is merge the two together, and we get:

This can then be used by declaring val regexesUsingMacro = restapi.Generator.macroPaths in our unit test.
But wait - there’s a problem, isn’t there? That function is returning an object of type Any, so all our other generated code will know about it is that it’s an any - it won’t know anything about the vals we’ve added to it. Well actually, it turns out this isn’t a problem, but is intended that the return type should know more than its declaration specifies, if possible, as Eugene Burmako explains here.

Putting it all together

Now that we’ve got a macro that can be used as a val definition, we need to find that val and use it in our match expression. To find the val is simple enough - we just look for a ValDef that uses a Select for our function as the right hand side. However, if the user hasn’t defined such a val, we can’t continue - we need to tell the developer what they’ve done wrong. The macro Context includes functions to provide compiler warnings and errors, so we need to emit an error advising the developer how to fix it. The structure we end up with is as follows:

When integrated with the code we had for the match block from part 1, we end up with:

So now we’ve got our pattern matching working well, in the next article we can start calling an API to produce our endpoints.