dejimata

Transcript of my EuRuKo 2018 Talk

2018-08-23

This is a very rough transcript of my talk, Metaprogramming for Generalists, given at this year’s EuRuKo conference in Vienna, Austria. There is also a video of the talk on YouTube.

Hi everybody, my name is Chris Salzberg and the title of this talk is “Metaprogramming for Generalists”. I know I know, you’re thinking, "He’s starting the morning of the first day with metaprogramming?!”

But don’t worry! This will be interesting.

About Me

Ok, so first about me. My handle is @shioyama and I live in Tokyo, in Japan. But don’t be fooled! I am not Japanese, in case that wasn’t obvious. I’m actually a Canadian originally from Montreal.

I work at a company called Degica based in Tokyo. We have a booth so be sure to check us out.

In the open source world, I’m the author of a few gems, the most well-known one is a gem called Mobility for managing model translations.

And I write about things like the Module Builder Pattern at my blog, dejimata.com.

Plan

This is going to be a two-part talk. In the first part, I’m going to introduce a different, more meaningful way to think about metaprogramming that will be useful for solving generic problems.

Then in the second part, we’ll solve a generic problem together in order to build what I will refer to as “generic software”.

Generalist Metaprogramming

Ok, so let’s start. “Generalist Metaprogramming”, what in the world is that?

What’s Metaprogramming?

Well, this year is Ruby’s 25th birthday, right?

And I think more than anything else, metaprogramming is an area where Ruby really stands out among programming languages. Because Ruby really embraces this concept. It underlies all these things we love, the DSLs, the human language-like syntax, all that stuff.

But we have a darn hard time defining this thing.

Defining Metaprogramming

Well ok, it’s a noun. I think we can agree on that. Unfortunately, that’s about all we can really agree on.

If “programming” is “writing code”, then the first obvious definition is “writing code that writes code”. But this doesn’t really cover everything we do with metaprogramming.

So the second definition is a “technique in which computer programs have the ability to treat programs as their data”. This one is from Wikipedia, it’s also something referred to as "reflection”.

So now we’re thinking more broadly. But if you look at the talk page on Wikipedia, you’ll see that not everybody agrees with this one either.

Another angle on metaprogramming is called “introspection”. A program is a subject, and it can find out things about other programs.

And then, at some point, we just throw up our hands and call it “a bag of tricks”. This by the way is from Metaprogramming Ruby, which by the way is a great book.

Bag of Tricks

And what are those tricks? Well, here they are. Some of you will recognize many of these. I think everyone will recognize “monkeypatching”.

The “What” View of Metaprogramming

But the thing is that, in practice, we tend to use a working definition that is much more basic. We label a subset of Ruby as “metaprogramming”, methods like eval, class_eval, define_method, etc.

And most talks about metaprogramming start there, with those methods, explaining how they work, what they do. They start with the “what” of metaprogramming.

But I don’t want to give that kind of talk today.

Metaprogramming as Fringe

And this is the reason why. It’s that for most programmers, this is what metaprogramming looks like. If they know about it at all, they just fence it off and label it as dangerous, or magical, or more likely just not useful to them.

But another part of the community, myself included — and I suspect many people here today — finds these methods absolutely essential, actually the core of Ruby.

And the reason for this divide is that most everything out there about metaprogramming with this “what” focus does not answer the obvious question: what on Earth is metaprogramming for?

And that’s the question I want to start with. To do that, I want to think about the structure of our ecosystem.

Ruby

So let’s start with a bit of fire down here, and put Ruby into the fire. Matz is feeling the heat!

So down here, this our programming language, where it is being forged out of fire and brimstone by Matz and his brave team of Ruby committers.

Applications

And then up here, we have where most of us spend our time: building applications. This is what pays our bills, this is what delivers value to our users.

And as users of the language, application developers know enough of the language to use it, but most don’t see or care much about its internals.

Libraries

Now, bridging these two worlds, we have the libraries. Or “gems”, but I’ll use the term “library” here.

Libraries fill the gaps in generic things we need - things that more than one application need - that are not supplied by the programming language.

And in Ruby in particular, these libraries are almost seamless in how they integrate with the programming language and with applications.

Metaprogramming Lives Here

And here’s the thing, right? It’s that metaprogramming overwhelmingly lives in this middle layer here, in the libraries.

Just taking Rails as an example, in Rails, it’s metaprogramming that is used to define everything you care about: attribute methods, associations, all model-specific stuff, view rendering logic, whatever.

But it’s not only ActiveRecord. Most of the widely-used gems we use heavily leverage metaprogramming to do what they do.

But applications don’t use metaprogramming much at all.

And that brings us to an obvious question, right?

Why?

Why? — Why is metaprogramming so much more prevalent in our dependencies than in our own application code?

This is such an important question. It’s telling us something very important about the nature of our programming language, and the ecosystem of components built out of it. But nobody is really asking this question, that I can see anyway.

I think part of the reason is maybe because it seems kind of obvious maybe, but I think it’s actually quite subtle. So let’s think about it a bit together now.

Generalization

Here’s another word, “generalization”. It’s the root of the second big word in the title of this talk.

Generalization is what libraries do. They generalize problems that many different applications have into generic components.

Generalization is the “formulation of general concepts from specific instances by abstracting common properties.” This is from Wikipedia again.

Abstracting Common Properties

This part, “abstracting common properties” from specific instances, is what we are going to do in the second part of this talk.

General Concepts

First, though, I want to focus on “general concepts”, and try to figure out the relation of general concepts with metaprogramming.

What’s a general concept that is relevant to everyone here?

Attribute

Well, I’m going to wager that this one is pretty relevant to almost everyone working with Ruby. The concept of “attribute”.

We want to figure out what role metaprogramming is playing in this kind of general concept, because it’s this kind of general concept that libraries implement for us.

Knockout Experiment

So to do this we’re going to take a cue from genetics. What does a geneticist do when they want to understand what a gene is doing?

Well, they knock it out. They just knock it out. And the they look to see what’s different about the mouse with the knocked out gene, compared to the mouse with the gene left in.

So we’re gong to do the same. We’ll knock out metaprogramming from the concept of “attribute”, and see what happens.

Control: Library View

So let’s start with the control. The control is the “normal” case, the one with metaprogramming.

This is the library view on the control. In each case, control and knockout, there’s a single class called “Model”, with a simple initializer that creates an empty hash to store attributes.

For the control here, we then have a class method, define_attribute, which takes an attribute name and defines getter and setter methods for that attribute. The getter just grabs the value for the key matching the method name, and the setter sets the value for the key matching the method name.

Control: Application View

And now here we’re switching to the application view, which uses this library.

In the application we define a class Talk which subclasses Model. And we call define_attribute twice to define two attributes, title and abstract.

And then here set and get the attributes like this, set the title and get the title, and set the abstract and get the abstract.

The point I want to make here is that the abstraction, remember the abstraction of “attribute”, is transparent, or you could say it’s invisible. It’s visible when we declare it of course, but after that it’s entirely transparent when we use it.

This is what we are used to and expect, but I want to highlight that.

Knockout: Library View

Ok now for the knockout. We start with the library view like we did for the control.

In the knockout, we have no metaprogramming, so we can’t use define_method like we did in the control. We can’t define methods dynamically like that.

So instead we define two instance methods, “get_attribute” and “set_attribute”, which is the only other way to really do it. Those methods do the same thing as the methods in the control, set and get attributes as keys on the @attributes hash.

Knockout: Application View

And here is the application view for the knockout.

In this case we don’t need to declare the attributes because our implementation will let you set and get anything you like (this is not really important).

Now in this case, as I mentioned, when we want to get and set attributes, we can’t just call “title” and “abstract” as methods because the library “Model” can’t dynamically define those methods, since it has no metaprogramming to do that.

So instead we call get_attribute and set_attribute to get and set attributes (keys on the @attributes hash).

Abstraction is Visible

So I want to highlight two things here. The first is that the abstraction is now visible. Remember that our abstraction is “attribute”, and in the control case once declared, we didn’t need to think about it, we could just talk about the talk’s title and abstract.

But here, it’s in our face. Every time you access an attribute, you need to actually write the word “attribute”.

Names are Arguments

And the other thing is that names are now arguments, in this case symbols, that we pass to these methods. They are not methods now, and so we don’t send them as messages to the objects but as arguments to methods on the object.

Results: Application View

Ok, so let’s tabulate the results of our experiment! We have two rows here, the first for the view from the application, the second for the view from the library. It’s really important to draw a distinction between these two.

And we have two columns, one for the control and one for the knockout. Recall that the control is with metaprogramming, and the knockout is without metaprogramming.

So from the application point of view, in the control the abstraction is transparent, or invisible, and we use the abstraction, i.e. we use attributes, by sending their names as messages to the object. And so we can say here that the library speaks in the language of the domain, i.e. you send the message “title” to the “talk” to get the title, nothing explicitly mentioning the abstraction itself there.

In the knockout, on the other hand, the abstraction is very much visible, remember we had those methods get_attribute and set_attribute. Also different: you send the attributes as values to a method, remember they were the symbols “title” and “abstract”.

So we can say here that the library speaks in the language of the abstraction. You really can’t avoid it as an application developer.

Results: Library View

Now let’s consider the library that implements this Model class. What’s going to be important here is the unknowns and what they refer to.

In the control, unknowns refer to code, to the attribute methods that we define with “define_attribute”. They are unknown because while they could be “title” and “abstract”, they could also be anything else. This makes library code hard to understand. That’s not obvious here because it’s a trivial example, but in the next section I’ll show how this can get more tricky.

In the knockout, the unknowns are the names of the attributes and their values. Attribute names are not code, they’re just keys on a hash, and that makes it easier to reason about than code with data that references code (like the control).

So if that’s the case then, why as library authors don’t we implement our code like the knockout?

Knockout ORM

Well, here’s the reason. Take the knockout example a bit further and imagine we implement an ORM like this, with associations. What would it look like?

Well, like this. We now have two general concepts here, attribute and association, but we can’t use metaprogramming for either of them, so the association abstraction is now also in your face.

So although this type of explicit abstraction is easier to handle at the library level, now it’s much more difficult to use at the application level.

Would you use this?

And so the question is: as an application developer, would you use this? Remember that at the application level, things like the name of the talk attributes talk and abstract are known to you.

This means that if you wanted to, you could build AR like code even without metaprogramming by defining these methods explicitly, with def. It might take time, but it might also be better when it comes to actually using those methods, because now your model would speak your domain language.

So the library would be at a disadvantage compared to the specific application. This is the really key point here.

ORM with metaprogramming

And now here’s what it looks like with metaprogramming. This looks more like what we’re used to.

Metaprogramming is Translating Unknowns into Code

So here’s the takeaway: metaprogramming levels the playing field by enabling libraries to translate unknowns into code. It levels the playing field with the applications using the library.

And seeing things this way — in terms of unknowns — really reveals a lot.

Think about it for a second. What are the unknowns for the application? Well, the classic one is user input, right? Do you want to translate user input into code? Hell no!

And so this explains the divide in perceptions of metaprogramming. From the library point of view, it’s a leveler. From the application point of view it’s a freaking menace!

The other thing this new definition does is reveal some problems with our previous working definition, the “what” definition. Let’s take a look at that now.

eval(“foo”)

Here we have eval("foo"), about the simplest kind of metaprogramming you can imagine, right?

Now, our working definition says anything with a metaprogramming method is metaprogramming, so this is metaprogramming then, right? But something feels wrong about this.

And the reason of course is that if we evaluate this, what do we get? We get foo, a call to the method foo or to a variable by that name. And that’s not terribly magical is it? We certainly aren’t “translating unknowns into code”.

eval(“foo#{str}”)

Ok, well what if we have a string, and we append “foo” to the string and eval that? Well, we still know the result here, which in this case would be calling foobar, whatever that is. So still not really interesting.

So how can we make this string unknown?

fooval

Well, wrap it in a method, call it fooval. Now by definition, it’s unknown, we can’t know the value of the string here. So this is translating unknowns to code. And it’s using eval, so in this case it lines up with both our expectation and our definition.

attr_writer

Ok, let’s take another case, attr_writer, with an argument “foo”.

This is pretty straightforward, right? attr_writer is definitely not metaprogramming.

attr_writer = def foo

And here’s what this evaluates to, defining a writer method foo. And that’s not translating unknowns to code, so ok this one lines up with our expectations.

define_writer

Ok, here’s another one. This is almost the same as the second half of the define_attribute method I showed you earlier in the control (except that it sets an instance variable instead of a hash value). This one uses define_method, so in our working definition this is metaprogramming. It also translates unknowns to code, i.e. translates name to a method, so this lines up with our expectations too.

define_writer aliasing attr_writer

But wait a second, that stuff in the method body can be re-written to look like this, right? It’s just aliasing attr_writer. And we said that attr_writer is definitely not metaprogramming.

But doing things like this, wrapping attr_writer in a method, or really just aliasing it, means that we’re translating unknowns to code. Because that’s really what attr_writer, and attr_reader and attr_accessor, do for us.

We don’t usually think about it this way because in 99% of cases we’re passing known values to these methods, but they can be used this way too.

So this is another case where our new definition does not line up with our working definition.

Data Points in Venn Diagram

So let’s visualize these results as Venn diagrams. Here we have our working definition, which I’ll call the “what” definition. And we have the control and the knockout here, which seem fine.

With additional data points

Now here are the other data points. Remember that eval("foo") is in this set, but it seemed like it didn’t belong. The second define_writer, which basically aliased to attr_writer, seemed like it did belong there, but it’s outside the set.

New definition

So what we’re going to do is to make a new definition, which will correspond to “translating unknowns into code”. And this definition is going to exclude the eval("foo") one, and include the second define_writer. It’s going to do that because it uses a definition based on what things reduce to.

eval("foo") reduces to what we call “normal programming”, i.e. a simple method call. attr_setter wrapped in a method does not reduce to anything we’d consider “normal programming”.

The point is that this new definition is about behaviour. It’s not about the what, it’s about the how.

So what’s normal programming then? Well, what if we invert this set on the right, and consider “normal programming” as the niche.

“Normal” Programming

So now what do we have. We have a set of methods and symbols and stuff that we consider “normal”. This is most of what you write as an application developer. Remember that this includes stuff that is normal and anything that reduces to stuff that’s normal.

In this world, code is code and data is data. Code can reference data, of course, but data never refers to code in this world. This is like the knockout example, where the attribute names did not refer to method names.

Normal Programming as Niche

And if we now zoom out and look at the big picture, this is what we have. It is “normal programming” in this picture, not "metaprogramming”, that is the niche. A very small niche in the much much larger space of computation, where we don’t draw any distinction between code and data.

Generic Software

And so where do the libraries that use metaprogramming live? Well, something like this, which I’ll refer to as “generic software”. They venture beyond the bounds of “normal programming” to make generalizations. And to do that effectively, they break that code/data barrier.

This way of understanding metaprogramming, as a means of generalizing from unknowns, is much more meaningful than any other definition out there that I have ever seen. And it really highlights what is so difficult, but also so powerful, about this concept.

To show that in more detail now, let’s look at an example of this thing I’ve referred to as “generic software”.

Building Generic Software

Ok, so “generic software”, what in the world is that, you say?

Generic Software: Definition

The term actually comes from a talk by Jeremy Evans, who is one of my heroes in Ruby. He gave a talk in 2012 about the development of Sequel, an ORM of which he’s the author, in which he said:

One of the best ways to write flexible software is to write generic software. Instead of designing a single API that completely handles a specific case, you write multiple APIs that handle smaller, more generic parts of that use case and then handling the entire case is just gluing those parts together.

And the key thing here is that the “glue”, in many cases and in particular in cases where the software needs to speak the domain language, will be metaprogramming, by the definition we came to in the last section.

So I want to explore this idea of “generic software”, but we need a general concept. We’ve already used “attribute”, so next I’m going to use…

Equality

Equality! Can’t get much more general than that, right?

Double Equals Equality

So there are a few methods for equality in Ruby, we’re going to be focusing here on “double-equals” equality. That’s the kind of equality that compares object content, not object identity.

So e.g., two strings which both have the characters f o o are double-equal equals even if they are different actual objects.

Finding a Use Case

Ok but we need some specific examples, like in the definition of generalization we saw earlier. So let’s go where everyone goes for examples, to Rails! And grep for it.

And we actually find a bunch of files that mention this type of equality.

To start we’ll use some classes defined in tests, just because they’ll be a bit easier to visualize, but we’ll come back afterward and apply it to models in other files here.

Address

So here’s the Address class, just has a few attr_readers for the attributes street, city and country, and an equality method that compares two addresses based on the values of those attributes.

Address

And we have this GpsLocation class, which is very similar. Here the methods are not defined using attr_reader, but that doesn’t really matter for us. We just care that there is a double-equals method and that it’s defined in terms of a list of methods.

Address + GpsLocation (1)

Ok, so let’s line them up together. We notice a small difference, which is that Address has this other.is_a?(self.class) and GpsLocation does not, but it’s not terribly important and could easily be added back, so we’ll just remove that one to simplify things. It won’t change anything about what I’m going to show you.

Address + GpsLocation (2)

Ok so now we need to extract what is common about these two methods. We’ll start by noticing that calling a method in Ruby is exactly the same thing as calling send with the method name as an argument. So let’s do that instead.

Extract with send

So you can see, in the first step here we’ve extracted something that was code - the method calls - to something that is data, the method names we’re passing to send. That’s important.

Extract Array

Now we’re going to refactor the sets of equality predicates using the enumerable method all?. This method takes a block and passes each element to the block, then takes the AND of the results.

In this case, the block will be the comparison of sending the attribute name to self and to the other object, and this will give us the same result as before.

(I’ve dropped GpsLocation here by the way since everything we’re doing now will be exactly the same for both classes, except the names will be different.)

Abstraction: key

Notice now that we have the first indication of our abstraction, the argument to the block here, “key”. This is an abstraction for the method names that define equality on each of these classes.

Keys

Ok now we’re going to name the array of keys as a variable “keys” and pull it out. Again, nothing has changed here, we’re just moving stuff around.

Replace def with define_method

Ok, so here’s an important step. We’re replacing def with define_method.

Remember that in our original working definition, this would mean that this was now metaprogramming. But in fact, nothing has changed, not yet anyway.

With our new definition, this is not metaprogramming because we are not translating any unknowns into code. We’re only translating knowns into code, the array of keys in each case. And we know what those keys are.

So you can see that our new definition is giving us more useful information about what is going on.

Hoist Keys

Ok, now we “hoist” the keys out of the define_method block. We can do that because blocks have an open scope, so what’s in the block can see the keys array, even if it’s outside of the block itself. (This is now a closure, if that says anything to you.)

Method Arguments

Now I want to look at these things differently. These keys we’re going to think of as arguments to a method. In GpsLocation the keys would be latitude and longitude, but for Address they are street, city, country.

Method Body

And now look at this code here. This is now independent of the keys, i.e. independent of what makes Address and GpsLocation different. So we can now extract it.

But how? By wrapping it in a method, like we saw earlier.

equalize

So we create a class method equalize, which takes arguments keys. The splat just means we can pass arguments and they will be captured in an array.

The code inside equalize is exactly the same as what we had in the last slide. But now since it’s in a method, we need to call it, which we do, just below. We call it with street, city, country.

Equalizer

Ok, and so now we can finally extract the module that will be our gem code. We extract that class method into a module, Equalizer, and we extend Address with it and call equalize with the keys.

That module is now our gem. And notice our unknowns? They are the keys to the equalize method.

require “equalizer”

And now that it’s a gem, a library, we can just require it, like this.

And now things are starting to look pretty nice and clean, right? The Address class is simpler than it was originally, but it’s very clear what is defining equality on each of them.

This is the sign of a good abstraction: it captures the essence of a general concept.

Equalizer in Big Picture

And now in that diagram I showed you earlier, here is where we are. We’ve extracted the generic component from Address (and GpsLocation would be the same), and this generic component, this generic software, now lives in this larger space outside of what we call “normal programming”.

Equalizer has data that refers to code, the keys that refer to method names defining equality.

Refactor other Classes with Equalizer

And now that we’ve done this, we can apply Equalizer to other classes. Here we have GpsLocation, but also a bunch of other classes from that list I showed you earlier. You can see we’ve hit on a very general pattern.

Equalizer Again

So here’s our Equalizer module again. This is a very compact little module, but there are a few important things going on.

Look at how define_method captures the keys in a block. Note that you couldn’t do this without define_method, because def has a closed scope, it doesn’t accept a block. Likewise, we could not build this method without calling the equalizer methods using send, because the keys here are unknown values.

Ruby provides methods like define_method and send for a very specific purpose, but that purpose is not clear unless you are solving a generic problem like this. Of course you could also use eval – eval is the “universal solvent” that dissolves anything, but it is an ultimate fallback and rarely used in practice.

Add inspect method

And the beauty of this is that once we have these keys, and these methods for dynamically defining methods and sending messages that Ruby provides, we can take this further.

We can observe that whatever defines equality basically defines identity. So we can use that fact to define an inspect method which shows the values of the key methods.

And you can see that this is useful in the example here.

Add hash method

And we can go further building on the abstraction, and add a hash method. The hash method is used to determine if two keys on a hash are the same or not, so by defining this method we allow instances of classes that extend Equalizer to be used as hash keys.

Here’s an example where GpsLocations are used as a hash of number of times a given location has been visited. We have two points, but they are the same latitude/longitude, so only one key in the hash is updated.

Dry Equalizer

Now, if all this seems useful to you, I have some good news! There’s a gem for it, called Dry Equalizer.

It’s one of the dry-rb gems, which are fantastic examples of generic software by the way. It’s a very very small gem, just a bit more than what we have here, done a little bit more cleverly, but basically the same idea.

DryEqualizer in dry-rb gem network

DryEqualizer may seem quite simple, but it plays a very important role in the network of dry-rb gems, which are being more and more heavily used in many applications. dry-equalizer is actually right at the core of this network, because it is so simple and so generic.

Being a Generalist (1)

Ok, so just before I close here, I want to come back to the title of this talk.

The title was “Metaprogramming for Generalists”, not “Metaprogramming and Generalization”, or “Metaprogramming and Generic Software”.

And there’s a reason for that.

“Generalist” usually means someone who has broad knowledge across different fields, and the ability to bridge ideas from various different areas.

And you can do that by jumping from one field to the other and learning high-level things about them, like this.

Being a Generalist (2)

But there’s another way to think of the “generalist”, as someone who digs deep down, to the general concepts that underlie all sorts of different problems. Concepts like “attribute” and “equality”.

Once you find these core concepts, and capture them in an abstraction, like the Equalizer module you saw, then you really grasp something very fundamental about a whole class of problems at once.

And then you can build on the abstraction, and everything you build will broaden the reach of the software.

Being a Generalist (3)

And as a community, we really need more of you to do this, to delve down to this world.

Solving generic problems, and solving them well, benefits everyone at all levels, by making every specific instance of that general problem easier to solve.

This is the metaprogramming magic that makes our ecosystem so powerful. It’s what makes Ruby the language it is.

Thanks!

Arel with Wharel

2018-05-30

Object Relational Mappers (ORM) are complex things. As the term implies, an ORM provides a mapping from objects in an object-oriented language like Ruby to a relational persistence layer. In theory, this makes it possible to abstract away details of this layer behind objects; in practice, of course, things are not so simple.

In this post, I’ll show you how I’ve leveraged a couple very simple features of Ruby’s object model to simplify building queries in ActiveRecord, in the form of a teeny tiny gem called Wharel. Wharel allows you to create Arel predicates in blocks passed to ActiveRecord query methods, without all the standard boilerplate.

Before I explain Wharel, though, let’s take a few steps back and consider the problem this gem tries to solve.

Querying with ActiveRecord

Part of the challenge of building and extending an ORM is striking a good balance between convenience and control. ActiveRecord offers you by default a fairly minimal interface for making queries which (I suppose) satisfies about 90% of users' basic needs. That interface is made up of the handful of methods like where and not, which allow you to create queries built up of simple equality predicates, optionally negated:

Post.where(title: "foo").where.not(content: "bar")
#=> SELECT "posts".* FROM "posts" WHERE "posts"."title" = 'foo' AND "posts"."content" != 'bar'

ActiveRecord also supports SELECT, HAVING, GROUP BY, along with the combinators AND and (recently) OR, as well as inner and outer JOIN, and a few others I forget just now.

Which is all fine and good, and may be adequate to power queries in your little CRUD app. But very quickly you will find that where and friends are not enough. Suppose, for example, you want to search for partial matches on string-valued data. A Rails developer would typically drop down to SQL to do this:

Post.where("title LIKE ?", "%#{params[:title]}%")
#=> SELECT `posts`.* FROM `posts` WHERE (title LIKE '%foo%')

… where params[:title] is a title from user input, and the application is using MySQL as its database.

In this case, we’re not really using the ORM much at all, other than to sanitize the data (params[:title]) interpolated into the SQL string. We can’t extend or re-use this query in other contexts. We can’t take it apart, build it up, or compose it. We can’t use it with a different database which uses a different name for LIKE (like PostgreSQL, which uses ILIKE for case-insensitive matching).

In short, we can either use this SQL snippet as-is, or re-write it to be something different.

Enter Arel

For one-off situations, this may well be enough. But for anything else, ActiveRecord offers another layer in the form of Arel, a relational algebra for building queries abstracted from the particulars of any particular database. If ActiveRecord is a mapping between Ruby model objects and a relational database, Arel is the fine-grained mapping between Ruby query objects and SQL query fragments.

Let’s get concrete. Convert the LIKE query above to Arel and we get:

posts = Post.arel_table
Post.where(posts[:title].matches("%#{params[:title]}%"))
#=> SELECT `posts`.* FROM `posts` WHERE (`posts`.`title` LIKE '%foo%')

Here, we needed to first call Post.arel_table to get an instance of Arel::Table corresponding to the posts table for the model. We can then call posts[:title] to get an arel node (actually, an Arel::Attribute) for the title column.

We can then use this title node to build a query using the matches method, which corresponds to LIKE in MySQL.

This may not seem like such a big win, since anyway you could have written the query in raw SQL with less lines of code. But what if you wanted to re-use that snippet in combination with another condition?

You could certainly do this with a new inline SQL, but a more scalable way is to use Arel. For example:

class Post < ApplicationRecord
  scope :with_title,             ->(str) { where(matches_title(str)) }
  scope :with_title_or_subtitle, ->(str) { where(matches_title(str).or(matches_subtitle(str))) }

  class << self
    def matches_title(title)
      arel_table[:title].matches("%#{title}%")
    end

    def matches_subtitle(subtitle)
      arel_table[:subtitle].matches("%#{subtitle}%")
    end
  end
end

In this case, we’ve used the arel predicates returned by matches_title and matches_subtitle to build a combined predicate using the pattern a.or(b), which generates SQL like:

# Post.with_title_or_subtitle("foo")
SELECT `posts`.* FROM `posts`
WHERE (`posts`.`title` LIKE '%foo%' OR `posts`.`subtitle` LIKE '%foo%')

As you can imagine, with more complex queries, using Arel building blocks can be quite useful. But it should also be apparent that Arel’s “wordiness” can be a real turn-off, and partly explains why Rails developers tend to prefer one-off SQL fragments over Arel predicates.

Where + Arel = Wharel

Recently I’ve been looking a lot at the internals of Sequel, a Ruby ORM developed by Jeremy Evans that is superficially similar in many ways to ActiveRecord, but which has a vastly different internal implementation and much broader feature set.

One of the things that fascinated me about Sequel is its interface for building queries, which allows you to use blocks in the form of so-called “virtual rows”, like this:

Artist.where{id > 5}

Here, we are querying on a column id. But notice that rather than passing a hash of keys and values to where, we are instead passing a block which references the method id and builds it into a predicate with id > 5.

This type of query would not be possible passing a hash to ActiveRecord’s where, since a hash can only generate equality predicates. Instead, we would need to build the appropriate predicate with Arel, like this:

artists = Artist.arel_table
Artist.where(artists[:id].gt(5))

For such a simple query, I find the ActiveRecord/Arel code to be much harder to read. Unsurprisingly, others have felt the same way.

Many years ago, Ernie Miller set out to improve ActiveRecord with an interface similar to Sequel, in the form of a gem called Squeel. Squeel was popular, but ultimately the monkey-patching magic it required to bend ActiveRecord into shape burnt out its maintainer, and the gem is now abandoned. More recently, a gem named BabySqueel has attempted to recreate some of this interface without Squeel’s monkey-patching baggage.

While working on Mobility, I recently had a need to incorporate Arel predicates with ActiveRecord relations in a way that went beyond AR’s standard query method interface. I decided to implement something like the block-based query interface from Sequel, but vastly simplified.

In the process, it became apparent to me how simple it is to do this, so I extracted the core logic into a separate gem I’ve called Wharel, a portmanteau of “Where” and “Arel”.

What does Wharel do? Well, add it to your Gemfile and you can simplify queries like the one above to:

Artist.where { id.gt(5) }

Wharel also supports combining nodes from different models by optionally using a block argument, like this:

Artist.joins(:posts).where { |a| Post.where { |p| p.title.matches(a.name) } }

(This query matches artists who have at least one post with a title that matches their name.)

As you can see, Wharel adds an optional block format like Sequel, within which any attributes called as methods are converted to their corresponding Arel node. In this case, id becomes the arel node Artist.arel_table[:id].

That’s it. There’s no additional magic to support predicates like id > 5, which Sequel, Squeel and BabySqueel offer, because I wanted to keep Wharel to an absolute minimum level of complexity.

And Wharel is indeed very simple: its core is a mere 31 lines of code. How can that be? Let’s have a look inside.

Virtual Rows and BasicObjects

Here is the Wharel module:

module Wharel
  module QueryMethods
    def where(opts = :chain, *rest, &block)
      block_given? ? super(VirtualRow.build_query(self, &block)) : super
    end

    module WhereChain
      def not(*args, &block)
        block_given? ? super(VirtualRow.build_query(@scope, &block)) : super
      end
    end
  end

  class VirtualRow < BasicObject
    def initialize(klass)
      @klass = klass
    end

    def method_missing(m, *)
      @klass.column_names.include?(m.to_s) ? @klass.arel_table[m] : super
    end

    class << self
      def build_query(klass, &block)
        row = new(klass)
        query = block.arity.zero? ? row.instance_eval(&block) : block.call(row)
        ::ActiveRecord::Relation === query ? query.arel.constraints.inject(&:and) : query
      end
    end
  end
end

There is really not much here. The QueryMethods module is required to patch ActiveRecord to handle blocks; nothing special there. What is special is the VirtualRow class, which subclasses BasicObject.

Why BasicObject? You may have seen this esoteric, minimalist object somewhere in a diagram of Ruby’s object model, and recall that every Object inherits from it. Indeed, BasicObject is a very simple object with almost no methods at all, and for most contexts not something you’d really want to lean on too much.

But in this case, the fact that BasicObject is so minimal happens to be very useful, because it allows us to create what’s called a Clean Room. Take a look at build_query, which you’ll see we call from where or not to convert a block to an Arel predicate:

VirtualRow.build_query(self, &block)

Here, self is the class or relation (for not it is @scope, which is basically the same), and block is the block passed to where or not.

In build_query, we first create an instance of VirtualRow with a reference to the class (or relation) (this is the new(klass) line). We then check the arity of the block (don’t be scared, “arity” is just another word for “number of arguments”).

If the block has no arguments (block.arity.zero?), like this:

Post.where { title.eq("foo") }

… then we call instance_eval on the virtual row and pass it the block.

Now what happens?

Well, instance_eval is a bit like that movie, Being John Malkovich, when you go through the door and enter the mind of actor John Malkovich. Except in this case, the door takes you to the mind of whatever object you’re calling instance_eval on. Any method you call from this context is received by that object, just like everything you see when you go though the door is what John Malkovich is seeing.

Here, the object is the virtual row. And this is where the fact that VirtualRow subclasses BasicObject comes in useful. Since VirtualRow has virtually no methods (no pun intended), anything you call on it will fall through to method_missing. So if you call title, method_missing will be called with :title as its first argument.

What do we do then? Well, just define a method_missing to catch these method calls and “route” them to the model’s Arel table, which Wharel does in three lines of code:

def method_missing(m, *)
  @klass.column_names.include?(m.to_s) ? @klass.arel_table[m] : super
end

Here, @klass would be Artist, so we’re calling Artist.arel_table[:title], which will get us the Arel node for title. Perfect! That’s exactly what we wanted.

Since this works for any attribute (as long as the attribute name does not overlap with a method on BasicObject, which is exceedingly unlikely), we can build up any arbitrary Arel predicate in this block without ever having to explicitly call arel_table on any model.

That Object Model Thing

Although this post has Arel in its title, the real point I’m trying to make with Wharel is to show how some of Ruby’s seemingly esoteric features, like BasicObject and instance_eval, can actually be very powerful. With only thirty lines of code, we’ve created an interface that eliminates a huge amount of Arel boilerplate code in any Rails app. I find this really remarkable.

Although I don’t think many Rubyists are very familiar with it, Ruby’s Object Model is actually very powerful and I highly recommend digging deeper into its details. Books like Metaprogramming Ruby by Paolo Perrotta are a great place to start, but it’s also a good idea to just pry open your favourite gem (like I did with Sequel) and see what it’s doing under the hood. If it’s doing something powerful, it’s probably leveraging this model to do it.

JSONify your Ruby Translations

2018-03-20

JSON is a ubiquitous format for storing and transmitting complex nested data, originally in Javascript but today in every major programming language. JSON is also increasingly used as a persisted data type, notably by PostgreSQL with its json and jsonb types, and more recently MySQL also supporting JSON natively. Ruby ORM like Sequel and ActiveRecord¹ offer interfaces for querying on this kind of JSON-formatted data.

Given its usefulness, I’ve naturally been interested in applying JSON to the storage of i18n data, a topic which I have written about before. As an arbitrary-depth hash format, JSON is a good fit for this kind of data, which tends to assume a similar nested key/value structure. With databases now providing ways to search JSON data quickly, we’ve got everything we need for a powerful translation storage and querying engine.

In this post I’ll introduce a new strategy for leveraging JSON data formats to store translations in a single table column. The strategy is implemented in Mobility, the i18n gem I’ve written about here before, and is inspired by (among other things) a library named Trans for Elixir.

Before jumping into this new JSON-based storage strategy, however, I’d like to first take a step back and think about the broader “translation mapping” problem, and how JSON fits into it.

Translation Mapping

When you think of translation as a Ruby developer, you may think of a YAML file with lots of nested translation keys and their corresponding translated values, popularized in Ruby by the I18n gem. Or you may think of gettext and .po (portable object) files. Or you may think of some other similar file-based key/value storage.

A key property of these kinds of internationalization and localization systems is that the context in which translations are written is fundamentally different from the context in which translations are used. Your users, for example, can’t edit the YAML i18n files for your application directly through the application itself.

Some YAML translations

That may be obvious to you, but it’s important because I want to contrast this role of i18n strings with another, fundamentally different role translation can play in a running application. The key characteristic of translation in this second role is that reading and writing happen in the same context.

An example of this second category would be a blog platform which allows authors to translate their articles into different languages, and query for articles in any language. Another example would be an event site that allows its users to translate their event listings. But it could also include any of many other contexts in which translation data is dynamically added, updated, deleted, and queried through the application itself.

Translation Interface on Doorkeeper.jp

If ActiveRecord and Sequel are “Object Relational Mappers” (ORM), I like to think of the libraries that help with this type of translation mapping as Translation Relational Mappers. They act as a layer between the ORM (data) and the interface for writing, reading and querying translations.

Mobility is unique among libraries I have seen (in Ruby and any other language) in that it takes the “mapping” part of this problem very seriously. While other libraries tend to support one storage strategy or another, Mobility supports many storage strategies through one common interface.

Which brings us to the question: what is a “storage strategy”? An example will help make things more concrete. In Ruby at least, there is one approach that is more popular than any other, and it does not use JSON.

Translations in Tables

Globalize is the most well-known gem for translating ActiveRecord i18n model data and has popularized a particular approach to storing persisted translation data.

In this strategy, translations for a model are stored in a model-specific table. If your model Post uses a table named posts, you would create a translation table named post_translations with a locale string column and other columns for each translated attribute. Then when you access post.title, the gem fetches the translation for the model in the current locale and returns the value of the title column on the translation.

Keeping all translations for a given model in the same place has some advantages. It means, among other things, that you can call post.translations.each and cycle through each of your translations as a separate record; SpinaCMS does this when generating its sitemap, for example. Mobility supports this type of translation strategy with its Table backend, which is what Spina uses.

Although it’s nice having translations in a separate table, there are also some significant downsides. Querying in particular becomes much more complicated since you need to constantly JOIN your translation table(s).

Here’s the kind of query you get with this strategy (taken from the Mobility Wiki):

SELECT "posts".* FROM "posts"
  INNER JOIN "post_translations" ON "post_translations"."post_id" = "posts"."id"
  AND "post_translations"."locale" = 'en'
  WHERE "post_translations"."title" = 'JSONify your Ruby Translations'

That’s not too bad, but you can imagine that once you start adding complexity you quickly hit edge cases which can push the limits of your ORM. This has prompted many people, myself included, to consider other options for storing translations. One of these is to store translations in JSON format.

Enter the B

A few years back, with the release of its version 9.4, PostgreSQL introduced a new datatype called JSONB. The “B” in “JSONB” signifies that JSON data are stored in this column format not as a string, but as binary data. This allows for fast lookups and more powerful search queries, through new operators like “contains” (@) and “exists” (?) which apply to JSON objects and keys, respectively. It also allows you to store a mix of integers, strings, arrays and hashes in the same column. This is a considerable step up from PostgreSQL’s Hstore format which, although also a key/value store, is limited to depth-1 string-valued data.

One of my original goals with Mobility was to support this type of innovative format alongside other strategies like the table-based one described in the last section. This, indeed, is Mobility’s killer feature: just like how you can swap MySQL for PostgreSQL as your database and keep your ActiveRecord code mostly unchanged, Mobility allows you to swap storage backends while retaining the same interface for accessing and querying translations.

The first implementation of JSON in Mobility was a backend called Jsonb. Using this backend looks like this:

class Post < ApplicationRecord
  extend Mobility
  translates :title, backend: :jsonb
end

Given that you have a title column of jsonb type, Mobility will allow you to call a virtual attribute named title and (if we’re in the English locale) will return to you the value on the en key of the JSON hash.

post = Post.create(title: "JSONify your Ruby Translations")
post.title
#=> "JSONify your Ruby Translations"
Mobility.with_locale(:ja) { post.title = "Rubyの翻訳をJSON化しよう" }
post.save
post = Post.first
post[:title]
#=> {"en"=>"JSONify your Ruby Translations", "ja"=>"Rubyの翻訳をJSON化しよう"}

It will also allow you to query on this attribute just like it was a table column (via a scope named i18n):

Post.i18n.where(title: "JSONify your Ruby Translations").to_sql
#=> SELECT "posts".* FROM "posts" WHERE (("posts"."title" -> 'en') = "JSONify your Ruby Translations")

So Mobility is mapping the query on title to a query format specific to jsonb, using the -> operator.²

Mobility actually does some more subtle things here. If you pass an array as the query value, for example, Mobility will treat this as a set of values to match, just like you would with a normal table column. So the query:

Post.i18n.find_by(title: ["foo", "bar"])

would generate the SQL:

SELECT "posts".* FROM "posts"
  WHERE (("posts"."title" -> 'en') = '"foo"' OR
         ("posts"."title" -> 'en') = '"bar"')

With the Table backend mentioned earlier, the same code would generate a very different query:

SELECT "posts".* FROM "posts"
  INNER JOIN "post_translations" ON "post_translations"."post_id" = "posts"."id"
  AND "post_translations"."locale" = 'en'
  WHERE "post_translations"."content" IN ('foo', 'bar')

Whereas the Jsonb backend uses OR, the Table backend uses IN to match translations that have a column value in ('foo', 'bar'). As a Mobility user you don’t need to really worry about these details; just write your ActiveRecord (or Sequel) code as you would for an untranslated attribute and the query should “just work”^TM.

Keeping ‘em together

One of the nice things about supporting many storage strategies in a single gem is that common implementation patterns can be leveraged to re-use code and minimize duplication. While there are gems offering Hstore-based model translation and gems offering JSON-based model translation, the implementation for each one is different in arbitrary ways (translates/translatable etc). Rather than re-invent the wheel, Mobility uses shared modules for accessing and querying PostgreSQL data types to eliminate duplication.

Which brings us to the new storage strategy mentioned at the top of this article. While playing around with Elixir recently, I came upon a package named Trans which uses a translation strategy I hadn’t previously considered. Like the Jsonb backend mentioned in the last section, this strategy uses a json or jsonb column in a PostgreSQL database to store translations. However, rather than store each set of attribute translations on its own column, Trans uses the fact that json columns can be deeply nested to store all translation on a single column, named translations. This would not be possible using Hstore since it is a depth-1 data type.

An example of this is shown in the figure below.

Model Translations as a depth-2 JSON hash

This is an interesting storage format with some notable advantages:

It minimizes the number of migrations needed to just one per translated model. Translated attributes can be added and removed from a model with only minimal code changes, since the database schema does not change when you add new keys to a json/jsonb hash.
It requires no complex JOIN queries since translations are all on the same table.
It allows cycling through model translations like with table translations (with post.translations.each), except without a JOIN.

Borrowing the name from Trans, I added this strategy as a new backend named “Container” in Mobility 0.4. In Mobility 0.5., I also added support for json-type columns in addition to jsonb (there are subtle differences between the two that are out of the scope of this post, but in general you probably want to use jsonb).

Just like the examples with other backends mentioned earlier, Mobility thus now supports both reading/writing and querying on translations stored in a single json or jsonb column³. Just specify the container backend in your model (or as a default in your Mobility configuration):

class Post < ApplicationRecord
  extend Mobility
  translates :title, backend: :container
  # Or if you want to set a different column name:
  # translates :title, backend: :container, column_name: :my_translations
end

Mobility will generate the appropriate query to find translations on the shared translations column. So:

Post.i18n.where(title: "JSONify your Ruby Translations")

will generate the SQL:

SELECT "posts".* FROM "posts"
  WHERE ((("posts"."translations" -> 'en') -> 'title') = "JSONify your Ruby Translations")

You will notice that this query is very similar to the earlier one for querying on a JSON/JSONB column for a single attribute. The difference is that we have changed the clause:

"posts"."title" -> 'en'

to the clause:

("post"."translations" -> 'en') -> 'title'

Mobility leverages this similarity such that only small tweaks are necessary to support querying on the Container backend. Indeed, although as of version 0.5 there are now three JSON-based backends in Mobility (Json, Jsonb and Container), the actual amount of code is relatively small given how many features are supported. This is exactly the benefits I was after when I designed Mobility in the first place.

JSONify it, even just a little bit

The JSON format offers many advantages in storing translations, but of course it’s not a silver bullet. Some downsides include:

Storing all translations on the same column can stress the database, triggering space reallocation and possibly slow down performance.
Parsing translation data may be more complex that simply grabbing the value on a string or text column.
The range of query options available is limited by database support for the column type (json or jsonb), which will generally lag query options for normal columns.
Only works on PostgreSQL currently (no Ruby ORM support yet for MySQL json columns).

If you look around the nets, you can find a lot of debate about the advantages and disadvantages of different storage strategies for i18n data. Some argue for storing each translation on its own column. Others argue for storing translations on a separate table. Others argue for the JSON data type strategies described here.

Well, Mobility supports all these strategies with the minimum code required to do so. It does so, moreover, in such a way that you can mix or match however you like. Want to give JSON a whirl on one attribute, but stick with another gem for the other attributes? You can do that. Want to use one backend for most attributes, but test a JSON backend on some other attributes? You can do that too.

The gem’s flexibility is why I gave Mobility its name. So if you’re considering storing translations in your application, I’d highly recommend giving JSON a try, alongside other options. No single solution will work best for everybody, but there are enough options to mix and match that something out there should fit your needs.

And of course, if you find JSON doesn’t work for you, translation tables don’t work for you, translatable columns don’t work for you, a polymorphic key/value store doesn’t work for you… well, you can always roll your own.

If you do, let me know what you find! 😉

Notes

¹ Actually, ActiveRecord’s support is pretty dismal compared to Sequel.
² Incidentally, this operator accepts any JSON object, so you can translate and query translated hash data in the same way, for example. See the Mobility Wiki for more details.
³ You don’t need to specify whether you are using a JSON column or a JSONB column, Mobility will check the db schema and figure it out based on the attribute name.

Mobility 0.3: Ready for Prime Time

2017-12-05

Just a little over six months ago, I released the first version of the Ruby translation framework I’ve called Mobility. With the third release of the gem – and a growing list of companies using it to power their production applications – I’m happy to say that Mobility is today more stable and reliable than ever before.

And we have a liftoff¹

As its name implies, Mobility is all about giving you choices. Whether you store your model translations in a set of model-specific tables, in a single shared table, as columns on each model table, or as values in a Postgres jsonb or hstore column, Mobility has you covered. Whether you use fallbacks, dirty change tracking, query support, or any of its other features, Mobility will do what you need it to do without getting in the way.

This week, I’ve released Mobility 0.3, and in the spirit of its name, this release (among other things) will keep you up to speed with your gem dependencies. Version 0.3 includes full support for the latest versions of its ORM dependencies: Rails/ActiveRecord 5.2 (literally just released in beta) as well as Sequel 5.0.

This release also comes with some important changes by a new contributor: Paul McMahon (@pwim), who has been also helping out commenting on and reviewing PRs. Having someone with Paul’s skill and experience helping out has been a huge boost; the fact that Paul is using Mobility in production on his own service, Doorkeeper, has also been great feedback for identifying important issues and fixing them quickly.

Paul and I have put together a short wiki page on migrating from Globalize to Mobility, which is what he did with Doorkeeper; be sure to check it out if you are using Globalize and considering switching to Mobility. For most users, Mobility will be a nearly 100% drop-in replacement for Globalize, and the migration process should be quite simple (as it was for Paul). However, for anyone with questions, I have setup a gitter community for discussions, and I also am watching for StackOverflow questions with the mobility tag.

So, without further ado, here are the changes…

Support for Rails 5.2 and Sequel 5

Yes, Rails 5.2.beta.2 was just released this week, and we’re already running tests against it. After a few small compatibility changes, those specs are now all passing and we are ready for ActiveRecord 5.2!

Sequel 5.0 was also recently released, and with a small update to Mobility released in 0.3.2, all features of the Sequel backends are now working with this version as well.

Support new ActiveRecord Dirty methods

In Rails 5.1, new dirty tracking methods were added to differentiate between changes that have been persisted (saved) to the database and those that have only been changed in-memory (set but not yet saved).

There is a new method saved_changes to return changes to attributes that have been saved, as well as attribute-specific methods (for an attribute title):

saved_change_to_title?
saved_change_to_title
title_before_last_save
will_save_change_to_title?
title_change_to_be_saved
title_in_database

Support for these methods has now been added. Just enable the dirty plugin and the methods will be available for translated attributes on your model.

Fallbacks when Locale is Specified

Paul noticed that the way locale accessors and fallbacks interacted was not very intuitive in version 0.2, and did not correspond to how Globalize works (and to users' typical expectations).

Here’s an example where we are setting the title in English, then calling title_ja:

class Event < ApplicationRecord
  extend Mobility
  translates :title, fallbacks: { en: :ja, ja: :en }
end

Event.new(title_en: "foo").title_ja
#=> "foo" instead of nil

Although fallbacks are enabled from Japanese to English, since we are explicitly specifying the locale we want (Japanese) in title_ja, one would reasonably expect the fallbacks to not take effect, but in fact they do.

This has been fixed in 0.3, so that you get the value in the locale you specified, without falling back to other locales:

Event.new(title_en: "foo").title_ja
#=> nil

This also applies when you fetch an attribute in a locale using the getter options hash:

Event.new(title_en: "foo").title(locale: :ja)
#=> nil

This is a really nice fix since as a byproduct it also makes the dirty plugin work more naturally with fallbacks (showing changes from the previous value without fallbacks).

Duplicate Translations when Duplicating Model

Paul noticed that the ActiveRecord Table backend does not duplicate translations when dup is called on the model. This is different from Globalize, which duplicates the translations as well.

After some discussion, Paul went ahead and implemented this feature by patching initialize_dup. So as of version 0.3, if you dup your translated model instance, you’ll get a copy both of the model and any translations.

Most other backends work without any such patch, since they map to columns on the model itself. However, Paul noticed that the KeyValue backends (which like the Table backend store translations in associations) were affected by the same issue. I have since followed-up and also fixed this issue for the AR KeyValue backend.

I have just released this fix in 0.3.3, so if you update to the latest release, you should have no dup-ing issues with ActiveRecord models for any backend. Sequel handles duplicating slightly differently, so will require more attention; a fix should be out in the next minor release.

Invalidate Cache Key

This is another one by Paul. Up to version 0.2, Mobility would not “touch” the associated translations for the Table and KeyValue backends. This results in problems with caching, since ActiveRecord doesn’t know that the attributes have changed.

The fix we decided on for this issue is simply to add a touch: true to the association defined for translations on the model. With this change, updating a translation will update the updated_at timestamp on the model. Unlike Globalize, which includes this setting as an option, as of version 0.3, Mobility sets touch: true by default, since there is not conceivable reason we could see that you would not want this enabled.

Convert AttributeMethods to a Plugin

One of the tricky parts of building a gem like Mobility is deciding how far to go in overriding core methods of the ORM.

There is a fine line you need to draw. Naturally, you want to make translated attributes behave just like normal attributes, since this is natural for the user and means they don’t need to think too much about them.

However, if you go to far with this, then the ORM (typically ActiveRecord) gets confused and thinks they are actually table columns, with unpredictable consequences. This has happened with Mobility and happens frequently with Globalize, which does much more patching. The results can be very hard to debug.

One of the ways that Mobility patched ActiveRecord previous to version 0.3 was to add translated attributes to the attributes hash:

# Version < 0.3
class Post < ApplicationRecord
  extend Mobility
  translates :title
end

post = Post.new
post.title = "foo"
post.attributes
#=> {"id"=>nil, "created_at"=>nil, "updated_at"=>nil, "title"=>"foo"}

Notice that the last key in the hash returned by post.attributes is title, although title is not actually a table column.

This may seem nice, but in practice it can cause problems both with ActiveRecord itself as well as with other gems. Since title is a key in this hash, a gem might reasonably assume that you can call read_attirbute("title") and get the value of the title, but Mobility (unlike Globalize) does not patch read_attribute to do this, so this fails.

To avoid compatibility issues, this feature of overriding attributes has now been extracted into a plugin, which by default is off, so the instance and model above would not by include title as a key in attributes:

post.attributes
#=> {"id"=>nil, "created_at"=>nil, "updated_at"=>nil}

You can enable it by passing attribute_methods: true to translates (or enabling it by default in the default_options configuration (see below).

# Version 0.3, with attribute_methods plugin enabled
class Post < ApplicationRecord
  extend Mobility
  translates :title, attribute_methods: true
end

post = Post.new
post.title = "foo"
post.attributes
#=> {"id"=>nil, "created_at"=>nil, "updated_at"=>nil, "title"=>"foo"}

Although you can do this with the plugin, in general unless you know you need it, my recommendation would be not to enable attribute methods since it can cause conflicts with other gems. For example, see this issue about a compatibility conflict with the strip_attributes gem.

Deprecate Default Options Setter

The last change is a deprecation, not always the most exciting thing for users of a gem. However, this is an important one. Reasons for the change are described in this issue.

In a nutshell, versions of Mobility prior to 0.3 would allow you to set a hash of default options, which would be merged with whatever options you pass to translates from any model:

Mobility.configure do |config|
  config.default_options = {
    dirty: true,
    fallbacks: true,
    # ...
  }
end

The problem with this is that, by directly setting the defaults, you override the “default defaults”, as it were. These defaults can be found here, and you can see some plugin settings that might not be familiar.

One of these is for a “presence” plugin which is on by default. This plugin simply filters any values set or fetched from the backend to filter out blank strings. I noticed that some users had blanks in their data, which should not happen if this plugin is enabled; it turns out, the reason is because they had set the default_options hash directly, without passing any value for the presence key, thus implicitly turning it off.

To avoid this from happening, default_options= (the setter) has been deprecated in version 0.3. If you are currently setting it, you will see a warning like this:

WARNING: The default_options= setter has been deprecated.
Set each option on the default_options hash instead, like this:

  config.default_options[:fallbacks] = { ... }
  config.default_options[:dirty] = true

This setter method will be removed in the next minor release. To get rid of the deprecation warning, set defaults for each key on the default_options hash, like this:

Mobility.configure do |config|
  config.default_options[:dirty] = true
  config.default_options[:fallbacks] = true
  # ...
end

Although this requires a few more keystrokes, it avoids the problem of overriding the “default defaults” stored initially in config.default_options. I think this is a saner way to set defaults, ensuring that you only opt-out of important plugins if you explicitly set the default.

Looking Ahead: A Christmas Release

The next release of Mobility will most likely be for the first major version (1.0.0), which I’m aiming to finish in time for Christmas. There will be a couple of breaking changes in this release, mostly just method renaming which should only affect some users and will be easy to fix.

As mentioned above, I’m eager to get any feedback and suggestions. Please join the gitter community or post your questions to StackOverflow. If you’re building a gem which depends on translations and are considering using Mobility, please contact me since this is a use case I’m particularly interested in.

Thanks for everyone’s support, and stay tuned for the next release!

Notes

¹ NASA Orbital ATK CRS-8 Launch (ref)

Mobility 0.2: Now with Plugins

2017-08-13

It’s been a little while since last I posted about Mobility, the pluggable translation framework for Ruby that I’ve been working on recently. I thought I’d take this opportunity with the release of the second version of the gem to highlight some of the important changes and new features, and to recap how far things have come.

Mobility and the Module Builder Pattern

In the time since I posted Translating with Mobility, one of the techniques I have used heavily in building the gem – the Module Builder Pattern, as I’ve termed it – has become a hot topic of its own. (I’ll actually be talking about module builders at the coming RubyKaigi in Hiroshima, check it out if you’ll be attending.)

In a nutshell, the Module Builder Pattern is a simple way to create customizable modules to mix into classes, one which falls naturally out of Ruby’s object model. It amounts to simply subclassing the Module class, defining module methods in an initializer or included hook, and including instances of the subclass in other classes. See the slide below from my recent talk at the Tokyo Rubyist Meetup, where I explain the basic elements of a module builder and how I use it in a core Mobility class, Mobility::Attributes.

From its first release, Mobility has used this pattern all over the place; I could never have achieved the flexibility the gem needed without it. However, in the latest release, I’ve pushed this yet further in order to achieve something just as important: extensibility.

Now with Plugins

Let’s recall that in Mobility, you define a translated attribute on a model by calling translates with one or more attribute names, and a hash to configure which options are applied: dirty-tracking, fallbacks, locale accessors, and so on. Under the hood, Mobility creates an instance of the Mobility::Attributes module builder and includes it in the class. I’ll use this form directly here to emphasize what is actually going on:

class Post
  extend Mobility
  include Mobility::Attributes.new("title", "content", locale_accessors: [:en, :ja], fallbacks: true)
end

Here, we are building a module instance of Mobility::Attributes for the attributes title and content, with locale accessors for English and Japanese and fallbacks enabled. (The format of arguments to this initializer have changed slightly since version 0.1, see this pull request for details).

What happens when we include the instance of this class into Post? Well, here’s the first change with version 0.2: I’ve made the magic here a whole lot clearer, and more extensible as a result (see PRs #50 and #62).

Previously there was a lot of hard-coded references to options directly in Attributes, but the code is now entirely free of such references. In fact, there are just a few lines in the builder which deal with options:

Mobility.plugins.each do |name|
  plugin = get_plugin_class(name)
  plugin.apply(self, options[name])
end

Here, Mobility.plugins is an alias for Mobility.config.plugins, a new configuration setting which defaults to this array (order of elements is important here¹):

[:cache, :dirty, :fallbacks, :presence, :default, :fallthrough_accessors, :locale_accessors]

The private get_plugin_class method fetches constants under Mobility::Plugins matching CamelCase-ed values of the plugin names from this array, then applies them with the attributes instance and option values as its arguments.

So locale_accessors: [:en, :ja] will fetch Mobility::Plugins::LocaleAccesors and call it like this:

Mobility::Plugins::LocaleAccessors.apply(self, [:en, :ja])

…where self is the instance of Mobility::Attributes being included into the model class, Post.

Notice that since we have no hard-coded references to option keys in this module now, any option key included in Mobility.plugins will apply a plugin with that name (as long as that plugin exists), with the attributes instance and option value passed into the plugin’s apply class method.

This makes is very easy to design new extensions: all they need is to have a class method, apply, which takes this Attributes instance (which itself has references to the backend class and attribute names), and an option value to configure how the plugin should be applied. (Incidentally, I borrow the term “plugins” here, and the method name apply, from Sequel, where plugins are defined in a somewhat similar way.)

Below is another slide from my recent talk, illustrating how these options trigger inclusion of yet more module builder instances into the backend class and into the Attributes instance itself.

Along with this introduction of plugins and the Mobility::Plugins namespace, I’ve also reorganized the backend namespace such that Mobility::Backends (plural) is now exclusively used for backend classes.

The result is that it becomes very straightforward to build custom backends and plugins: define a class in the appropriate namespace with the required structure, and call translates (or include an instance of Mobility::Attributes) in the model class, with any backend or option keys, and that’s it! You ready to go.

Default Options

Another minor annoyance some users of Mobility may have encountered in 0.1 is that while it was possible to set a default backend to use across all models (with the default_backend configuration option), there was no way to define a default set of options (to configure plugins and the backend). Along with the changes to enable plugins above, I also added this as a configuration setting, with a value that defaults to:

{ cache: true, dirty: false, fallbacks: nil, presence: true, default: nil }

These default option values were previously hard-coded, but are now explicitly set and isolated in the configuration instance, which makes them much easier to reason about and change.

Want to enable fallbacks by default on all your models? Just override the default for that key:

Mobility.default_options[:fallbacks] = true

You can also set backend options here, so for example if you want to set the default type for the KeyValue backend to be “string” instead of “text”, you could do this:

Mobility.default_options[:type] = :string

Note that how these option values are interpreted by actual plugins and backends depends on the plugin or backend, and there is some unavoidable variation there. But the values passed in to apply (plugin) and new (backend) are entirely determined from this hash.

Enumerable Backends

Backends as implemented prior to version 0.2 could only do two things: read and write values for a given locale. If you wanted to know what locales were available for a given attribute, you would be stuck.

This was raised as an issue, and I decided that rather than just add a method returning locales, I’d go a bit further and try to solve a more general problem: iterating through translations. I implemented this in PR #71 by adding a new each_locale method to each existing backend returning successive locales to a block.

From these backend-specific each_locale methods, an each method is defined in Mobility::Backend yielding instances of a Translation struct wrapping the locale and backend. We then include Ruby’s powerful Enumerable module, giving us with it a whole slew of traversal and search methods. A method locales is also added to Mobility::Backend which returns the available locales for the attribute (the original request).

With this change, you can now do things like select translations that match a condition:

post = Post.create(title_en: "English foo", title_de: "German foo", title_ja: "Something else")
post.title_backend.select { |t| t.read =~ /foo/ }.map &:locale
#=> [:en, :de]

Although this is a somewhat contrived example, it’s not hard to see that having the full arsenal of Enumerable methods at your disposal could be quite useful in a variety of contexts.

Autoload → Require

Mobility 0.1.x uses autoload to load nearly all its constants, but as Rubyists no doubt know, autoload has its issues, and may even be deprecated in a future release of Ruby. So I decided to bite the bullet and convert all autoload calls to require (here’s the PR). For this I followed the pattern (again) of Sequel, which only requires plugins when they are called with the plugin method.

So Mobility now does a similar thing for backends and plugins. Here is the code loading a backend when it is triggered from a call to translates:

def get_backend_class(backend)
  return backend if Module === backend
  require "mobility/backends/#{backend}"
  get_class_from_key(Mobility::Backends, backend)
end

Rather than simply finding the constant, we first require the backend at "mobility/backends/#{backend}", then once we have loaded the file we make a const_get call to actually fetch the constant (which previously would have triggered autoload to find the backend file in the same location).

Admittedly, this now requires the plugin/backend creator to put the backend/plugin in the requisite file location or else Mobility will not find it. But that seems to me a quite reasonable thing to do anyway.

It also means that you can’t just require "mobility", and then expect Mobility::Backends::KeyValue (or any backend class) to be loaded, because it will not be required until you actually use it. You need to either call translates, or require the class explicitly, to load the backend.

Mobility::Util

Continuing with the autoload clean-up, I also decided to fix something else that had been bothering me for a while: mobility/core_ext. When I originally decided to make Mobility support Sequel in addition to ActiveRecord, I found it hard to entirely give up ActiveSupport since (like many Rubyists) I had become so accustomed to using methods like present?, camelize, singularize, etc.

So by default, if ActiveSupport could not be loaded, I included a bare minimum implementation of just a few of these methods, filed under mobility/core_ext. In addition, for Sequel backends, I also just assumed that the Inflections module was loaded, otherwise the backends would not work (I required it in Mobility’s Sequel specs).

In this PR, I replaced mobility/core_ext, which (like ActiveSupport) monkeypatched core classes like Object and String, with a module Mobility::Util which contains all the methods needed. Util can be either included in a class, or its methods can be called as class methods: e.g. Util.present?(string) will call string.present? if the object responds to present?, otherwise it will do the work itself to get the result.

Now Sequel does not need to include the Inflections module to use Mobility, and the core parts of Mobility (e.g. Mobility::Attributes) do not require any special monkeypatching to work. This is important for anyone wanting to use Mobility for an application where they do not want to patch core Ruby classes (a good thing).

Super as Option

In some cases, Mobility overrides a method to redefine it as a translation accessor. This is the case, for example, with backends that store translations on a single column: the PostgreSQL (Jsonb + Hstore) backends, as well as the Serialized backend (which stores translations serialized as a json or yaml hash).

With these backends, the original attribute method, say title, returns a hash containing all translations as key/value pairs (where locales are the keys). Mobility overrides this column accessor to return value of the hash at the current locale, using (for AR models) read_attribute to avoid infinite recursion (Sequel is similar).

A few people had asked about how they could access the original (hash) column, so in this PR I added a new super option which bypasses the Mobility accessors (reader and writer) and continues up to the original method, in this case the column method.

So something like this:

post.title
#=> "Translating with Mobility"
post.title(super: true)
#=> { "en" => "Translating with Mobility", "ja" => "Traduction avec Mobilité" }

The writer can also be “super-ed”, although this is somewhat trickier to do:

post.send(:title=, {}, super: true)
#=> {}
post.title
nil

Other

The default name for associaitons in the KeyValue and Table backends has been simplified. KeyValue associations named mobility_string_translations and mobility_text_translations are now simply string_translations and text_translations, respectively. For the Table backend, mobility_model_translations has been simplified to simply translations (like Globalize).
In PR #58, I refactored a somewhat ugly part of the code dealing with caching. I’m still not completely happy with how this is done, but it’s at least an improvement over the earlier technique.
I now have some minimal object allocation specs, added in PR #57. I’d like to extend this to benchmark testing as well, but not quite sure how to proceed yet on that one (any suggestions welcome!)
The gem is now signed. You of course do not need to check this signature, but if you want to, there are instructions in the README on how to do this.
extend Mobility is now the preferred way to use Mobility in models (see this issue).
I’ve bumped Ruby support to minimum 2.2.7. This is pretty standard.

Finally, a few words

Mobility is a labour of love, and I think it’s worthwhile to mention that here. I’ve spent countless hours writing and re-writing Mobility’s code, and many more thinking and rethinking how I can refactor and improve that code.

I have both selfish and unselfish motivations for doing this: on the one hand, I want to show off the best that I can do with a clean slate and a clear problem (translation) that I happen to know reasonably well. On the other, I genuinely want to help people who are trying to share content across language barriers, a problem which has been an interest of mine for a long while.

So it’s been enormously gratifying for me to see the positive response Mobility has received so far in the form of encouraging comments and reactions, a growing list of stargazers, and as actual contributions to the gem code. I’ve had very thoughtful input on edge cases and subtle points that I would never have considered, as well as some great feedback at recent talks in Montreal and Tokyo (see slides from the latter of those talks below).

Slides from my recent talk at the Tokyo Rubyist Meetup

There are many horror stories about open source software maintainers getting exhausted from the demands that come with success. Mobility hasn’t reached anywhere near that level of success yet, but so far at least I don’t find myself exhausted at all. On the contrary, it’s been an extremely rewarding learning experience!

If you’re using Mobility for a project or application, please let me know. I’m eager to here about the kind of challenges people are tackling and how Mobility might be better able to help solve them.

Notes

¹ The order of plugins in Mobility.plugins is important: many of the plugin classes include modules into the backend overriding read and write, inclusion order determining the order of method composition. In the list, for example, cache comes before default (and included in that order) because we want to cache actual backend values, not default values.

The Ruby Module Builder Pattern

2017-05-20

There’s an interesting pattern I’ve discovered recently in Ruby that is very powerful, yet apparently not widely known or appreciated.¹ I call this pattern the Module Builder Pattern. I’ve used it heavily in designing Mobility, a pluggable translation framework I released a couple months ago, and it served me so well I thought I should share what I’ve learned.

At its core, the Module Builder is an extremely simple, elegant, and versatile way of dynamically defining named, customizable modules to mix into classes. This is done by subclassing the Module class, initializing these subclasses at runtime, and including the resulting modules into other classes.

If the idea of subclassing Module makes your head spin, read on. Despite its seemingly esoteric nature, the Module Builder does some very useful things. It is used extensively in projects such as dry-rb to great effect, but buried out of view among advanced coding styles that conceal its essential simplicity. It is also used sporadically in Rails², but only to a very limited extent. I think it’s fair to say that most Rubyists have never seen the pattern, let alone ever used it in their daily work.

To explain Module Builder, I will work through a series of progressively more complex examples, ending with some code extracted from my work on Mobility into a gem called MethodFound. The examples begin with some simple core concepts I think most readers will understand, then advance to a slightly more in-depth discussion exposing the pattern’s tangible benefits in real-life applications.

Modules in Ruby

Let’s first recall that a module in Ruby is basically a collection of methods and constants which you can include in any class, reducing duplication and encapsulating functionality in reusable, composable³ units. Callbacks like included and extended allow you to execute some custom code every time your module is included (or extended, etc) in a class or module.

So suppose we have a module MyModule with a method foo:

module MyModule
  def foo
    "foo"
  end
end

Now we can add this method to any class by including MyModule:

class MyClass
  include MyModule
end

a = MyClass.new
a.foo
#=> "foo"

That’s pretty straightforward, but also very limited: foo is pre-defined with a hard-coded return value.

Let’s take a slightly more “real-world” case. Say we’ve got a class, Point, with two attributes x and y, which for simplicity’s sake we’ll define using a Struct:

Point = Struct.new(:x, :y)

Now let’s create a module which can add two points together at their respective x and y values. Call it Addable:

module Addable
  def +(point)
    Point.new(point.x + x, point.y + y)
  end
end

Point.include Addable
p1 = Point.new(1, 2)
p2 = Point.new(2, 3)

p1 + p2
#=> #<Point:.. @x=3, @y=5>

The module is slightly more reusable now. But it still has rigid constraints: it will only work with a class called Point which has attributes x and y.

We can improve this a bit by removing the constant Point in the module, and replacing it with self.class:

module Addable
  def +(other)
    self.class.new(x + other.x, y + other.y)
  end
end

Now our Addable module can be used with any class that has accessors x and y,⁴ which makes it much more flexible. Also, thanks to Ruby’s very dynamic treatment of types, the module can be used to “add” a potentially wide variety of pairwise data types, as long as their elements have a + method.

In addition, note that modules in Ruby have the property that their methods can be composed through inclusion or inheritance by using the super keyword in the including or parent class. So if we wanted to log calls to our adder method, we could do this by defining + inside of the Point class, like this:

class Point < Struct.new(:x, :y)
  include Addable

  def +(other)
    puts "Enter Adder..."
    super.tap { puts "Exit Adder..." }
  end
end

And we can see that our logging indeed works as expected, calling the method on the class first, and then through super the method defined on the module:

p1 = Point.new(1, 2)
p2 = Point.new(2, 3)

p1 + p2
Enter Adder...
Exit Adder...
#=> #<Point:.. @x=3, @y=5>

There are many other things to say about modules; we’ve really only touched the tip of the iceberg here. However, the main topic of this post is module builders, not modules, so let’s take a look at that.

Building a Module that Builds Modules

Modules that Build

Modules are powerful because (when used effectively) they encapsulate certain shared patterns of functionality. In the example above, this functionality is the addition of variable pairs (points, or any other class with pairs of variables x and y). Identifying shared patterns like this and expressing them in the form of a module or set of modules makes it possible to tackle very complex problems with simple, modular solutions.

That said, the module above is still quite specific, constrained to classes with variables x and y. What if we wanted to loosen this constraint and have the module work for any set of variables? How would we do that?

Well, a module can’t be configured once defined, so we’ll need some other way to loosen this constraint. One way is to define a method on the module that itself adds the desired functionality to the class. Let’s call this method define_adder⁵:

module AdderDefiner
  def define_adder(*keys)
    define_method :+ do |other|
      self.class.new(*(keys.map { |key| send(key) + other.send(key) }))
    end
  end
end

Instead of defining a + method like we did in the Addable module, we instead define a method which defines the method. We’ll call this “method-defining method” the bootstrap method here, since it “bootstraps” the module’s instance methods dynamically into the class.

This bootstrap method takes an arbitrary number of arguments, mapped to an array keys using the splat operator. These “keys” are the names of the variables to add (or, more precisely, the names of the methods which return the variable values). It then defines a method, called +, which adds the values of these variables at each key and creates an instance from the results using self.class.new.

Let’s see if it works. We’ll create a new class, LineItem, with different reader names, and extend the module in this case rather than include it (since define_adder needs to be a class method):

class LineItem < Struct.new(:amount, :tax)
  extend AdderDefiner
  define_adder(:amount, :tax)
end

Now we can add line item pairs, just like we did point pairs earlier:

l1 = LineItem.new(9.99, 1.50)
l2 = LineItem.new(15.99, 2.40)

l1 + l2
#=> #<LineItem:... @amount=25.98, @tax=3.9>

So it works! And we now have no dependency on variable names (or even on number of variables), so our module is more flexible.

But there’s a problem. When we copy over our earlier logging code from the Point class to log calls to +, like this:

class LineItem < Struct.new(:amount, :tax)
  extend AdderDefiner
  define_adder(:amount, :tax)

  def +(other)
    puts "Enter Adder..."
    super.tap { puts "Exit Adder..." }
  end
end

… and then add two line items again, instead of getting the expected logs, things blow up with an exception:

NoMethodError: super: no superclass method `+' for #<struct LineItem amount=9.99, tax=1.5>

What’s going on here?

Take a good look at how define_adder bootstraps the + method: it uses define_method to add the method directly to the class, in this case LineItem. A class can only have one definition of any given method, so when, later in the class, we again define +, we clobber the earlier definition. There is no “superclass” here since we have not included or inherited anything when defining the method.

Solving this problem will require some “module building” magic. Let’s look at the solution first, and then see how it solves this issue:

module AdderIncluder
  def define_adder(*keys)
    adder = Module.new do
      define_method :+ do |other|
        self.class.new(*(keys.map { |key| send(key) + other.send(key) }))
      end
    end
    include adder
  end
end

Now we replace AdderDefiner in the logged LineItem code above with AdderIncluder and we find that it works:

l1 = LineItem.new(9.99, 1.50)
l2 = LineItem.new(15.99, 2.40)

l1 + l2
Enter Adder...
Exit Adder...
#=> #<LineItem:... @amount=25.98, @tax=3.9>

The trick that makes this work is the use of an anonymous module in define_adder to define the + method, which is different from the previous technique of defining the method directly on the calling class.

Recall that in Ruby, any module (small “m”) is simply an instance of a class, and this class is Module (big “M”). Like other classes, Module has a new method, in this case one which can take a block argument and evaluate the block in the context of the newly-created module.

So to generate a “module on the fly”, you can simply call Module.new, pass a block with some method definitions, and use it just as you would any other (named) module:

mod = Module.new do
  def with_foo
    self + " with Foo"
  end
end

title = "The Ruby Module Builder Pattern"
title.extend mod
title.with_foo
#=> "The Ruby Module Builder Pattern with Foo"

This is a handy trick, often used in testing when you don’t want to clutter the global namespace with a one-off module used only in a single test (works for Class too).⁶

The method defined in the block in AdderIncluder is the same + method defined earlier in AdderDefiner, and in this sense, the two modules perform a very similar function. The key difference is that whereas AdderDefiner defines the method on the class, AdderIncluder includes a module with the method into the class.

If you look up the chain of ancestors, you will see this module just after the LineItem class itself:

LineItem.ancestors
#=> [LineItem,
#<Module:0x...>,
...

Given this ancestor chain, it is clear why super works here: when you call + on LineItem, the method calls super, which continues up the class hierarchy to the anonymous module where our + method is defined. It then calculates and returns the result, which gets composed in the LineItem adder with the logging code.

Although at first glance it might seem somewhat strange and unfamiliar, this technique of defining a bootstrap method to include a dynamically-generated anonymous module is in fact extremely useful, and widely used for this reason. This is particularly true for a flexible framework like Rails, where simply defining a model or setting up routes triggers calls to many bootstrap methods like define_adder. The technique can also be found in other gems that require a similar level of flexiblity (including Mobility).

Bootstrapped module building can thus be considered a very important metaprogramming technique in Ruby. Indeed, it underpins the very dynamic nature that enables frameworks like Rails to work the way they do.

That said, I hope to convince you here that the potential of module building goes well beyond the bootstrapping technique shown above, and as such that it is a pattern that deserves more attention. To do this, I’m going to return to the Adder module one last time, and define it in a slightly different way.

The Module Builder

So far we’ve looked at three “adder” modules:

Addable, which adds a + method on two variables x and y,
AdderDefiner, which adds a class method define_adder that defines a + method on an arbitrary set of variables, and
AdderIncluder, which also adds a class method define_adder that defines the + method, but in such a way that the method can be composed using super.

Although not very flexible, note that in hindsight, the first Addable module had some good things going for it:

It only included the + method, and left no other bootstrapping “artifacts” like define_adder (unlike both AdderDefiner and AdderIncluder)
It allowed the parent class to access its methods using super (like AdderIncluder but not AdderDefiner)
It had a name, and was therefore very easy to recognize in the chain of ancestors. While AdderDefiner and AdderIncluder themselves appear in the chain of ancestors, the fact that the bootstrap method was called, and how it was called, is hard or impossible to discern.

In this section, I will show how we can solve the adder problem in a way that retains these benefits, while also offering the flexibility we sought in the last section.

Module Builders

Let’s start by defining an AdderBuilder class which inherits from Module:

class AdderBuilder < Module
  def initialize(*keys)
    define_method :+ do |other|
      self.class.new(*(keys.map { |key| send(key) + other.send(key) }))
    end
  end
end

This is a short but very dense little class. Before going into the details of what it’s doing here, let’s first check that it works. We’ll include (not extend) an instance of this class (a module) into LineItem this time:

class LineItem < Struct.new(:amount, :tax)
  include AdderBuilder.new(:amount, :tax)
end

l1 = LineItem.new(9.99, 1.50)
l2 = LineItem.new(15.99, 2.40)
l1 + l2
#=> #<LineItem:... @amount=25.98, @tax=3.9>

So, as we can see, this little class does the same thing that we achieved with the define_adder bootstrap method in AdderDefiner, except without ever defining a class method to do it.

How does it do this? First, it subclasses Module. Since we can call new on Module, it’s not much of a stretch to imagine that we can also subclass it and define our own methods such as initialize. And indeed, as shown above, we can do this.

Here’s that initializer again:

def initialize(*keys)
  define_method :+ do |other|
    self.class.new(*(keys.map { |key| send(key) + other.send(key) }))
  end
end

There are two levels at play here, and it’s important to recognize both of them. On one level, we are initializing a new instance of a class, Module. This instance is itself a module, so it’s a collection of methods we will mix into other classes. We are then, as we initialize this module, defining one of these methods, a method called +.

The method + itself is being defined in terms of the method names we have passed as arguments to Module.new (the “keys”, which were :amount and :tax in the example above).⁷ When we call self.class.new in the method, this new will be evaluated in the context of the class including this new module, and thus would evaluate to LineItem (or Point, or whatever class the module was included into).

So AdderBuilder.new(:amount, :tax) evaluates to a module functionally equivalent to this:

Module.new do
  def +(other)
    self.class.new(amount + other.amount, tax + other.tax)
  end
end

… which is what we had in Addable, with x and y swapped for amount and tax. But the power here comes from the fact that we can define our dynamic module in terms of whatever variable names we want.

And herein lies the essential ingenuity of this pattern: that it allows you to define modules not as fixed collections of methods and constants, but as configurable prototypes for building such collections. There is no need for class methods or other metaprogramming tricks to bootstrap the module-inclusion step; the builder builds modules directly, so they can be included in one step.

Not only that, but the modules created in this process, unlike the anonymous modules generated by techniques described in the last section, actually have a name, so you can clearly see them in the chain of ancestors:

LineItem.ancestors
[LineItem, #<AdderBuilder:0x...>, Object, ... ]

This is a nice by-product of the encapsulation we have achieved by subclassing Module.⁸ With an anonymous module, this is much harder to debug:

LineItem.ancestors
[LineItem, #<Module:0x...>, Object, ... ]

Module Builders thus have the benefits of anonymous modules (their instances can be defined “on the fly” and do not clutter the global constant namespace), are easily traceable since their instances have a name (like “normal” modules), and yet do not require defining (and calling) new methods on the including class to bootstrap them.

We can go a step further. We previously added some logging code to the including class. This logging code can also be rewritten as a module builder, which we will call LoggerBuilder:

class LoggerBuilder < Module
  def initialize(method_name, start_message: "", end_message: "")
    define_method method_name do |*args, &block|
      print start_message
      super(*args, &block).tap { print end_message }
    end
  end
end

Now we can log any included or inherited method by creating an instance of LoggerBuilder with the name of the method to be logged:

class LineItem < Struct.new(:amount, :tax)
  include AdderBuilder.new(:amount, :tax)
  include LoggerBuilder.new(:+, start_message: "Enter Adder...\n",
                                end_message: "Exit Adder...\n")
end

And, presto, we’ve got a logged addable line item:

l1 = LineItem.new(9.99, 1.50)
l2 = LineItem.new(15.99, 2.40)

l1 + l2
Enter Adder...
Exit Adder...
#=> #<LineItem:... @amount=25.98, @tax=3.9>

But of course, we can now also log any other included or inherited method we like:

LineItem.include(LoggerBuilder.new(:to_s, start_message: "Stringifying...\n"))

l1.to_s
Stringifying...
=> "#<struct LineItem amount=9.99, tax=1.5>"

There are many other directions you can take this; I’ve only scratched the surface here. For a relatively simple (yet powerful) example of this “in the wild”, take a look at Dry Equalizer, which dynamically defines equality methods on a class using an instance of an Equalizer class that, like AdderBuilder, subclasses Module.

Module Builders and Composition

You may have noticed that I have used the word “compose” quite a lot in this post with respect to including modules into a class. Any time you include a module which overrides a method and calls super, you are in effect using function (method) composition, although it is not generally referred to as such.

The same is true of our logging code in the last section. When we include an instance of both AdderBuilder and LoggerBuilder, we are building + as a composite function, which we could alternatively write as:

class LineItem < Struct.new(:amount, :tax)
  def add(other)
    self.class.new(amount + other.amount, tax + other.tax)
  end

  def log(result)
    print "Enter Adder..."
    result.tap { print "Exit Adder..." }
  end

  def +(other)
    log(add(other))
  end
end

So super is essentially taking the output of the last method call and incorporating it into the result of the current module’s method.

Composition happens to be a very interesting way to “combine” different instances of a module builder, like we combined logging and adding modules in the example above. With module builders, we can configure and combine many modules to build more complex methods and class behaviours.

One way to do this is via a “branching” method flow. We define multiple modules that each override the same method, but only trigger some special processing if a condition on the method and its arguments matches the module’s state; otherwise, the control flow continues up the class hierarchy to the next module via super.

If you’ve worked with Ruby for any length of time, this “branching composition” should make you think of one method: method_missing. Rarely does anyone override method_missing without a call to super somewhere, since doing so would catch any method not defined on the class, which is generally not what you want.

Instead, a typical method_missing override looks something like this:

def method_missing(method_name, *arguments, &block)
  if method_name =~ METHOD_NAME_REGEX
    # ... do some special stuff ...
  else
    super
  end
end

… where METHOD_NAME_REGEX is a regex used to determine if we should “do some special stuff”.

I actually use exactly this branching flow in Mobility in a module builder called FallthroughAccessors. Each instance of FallthroughAccessors is initialized with a set of attributes (its “state”) and intercepts method calls where the method name is made up of one of these attributes plus a locale suffix (e.g. title_fr for the title attribute in French), which I refer to as an “I18n accessor”. If the method name matches, the attribute value in the suffix locale is returned, otherwise control continues up the class hierarchy.

Composing method_missing in module builders this way results in an sequence of nested regex conditionals spread across a series of modules, which I call “interceptors”. The fact that these interceptors are entirely encapsulated, with no external dependencies on the class itself, enables Mobility to “plug in” i18n accessors for each translated attribute independently, at any time and in complete isolation from each other.

Having found this pattern useful in Mobility, I extracted it into a gem called MethodFound. MethodFound is essentially one module builder, MethodFound::Interceptor, which intercepts method calls that match a regex or proc (the “state” in this case) and passes the method name, any matches captured in the regex (or the return value of the proc), and any method arguments to a block (the “do some special stuff” in the conditional above). This block is evaluated in the context of the instance of the class including the module.

Here’s an example:

class Greeter < Struct.new(:name)
  include MethodFound::Interceptor.new(/\Asay_([a-zA-Z_]+)\Z/) { |method_name, matches|
    "#{name} says: #{matches[1].gsub('_', ' ').capitalize}."
  }
  include MethodFound::Interceptor.new(/\Ascream_([a-zA-Z_]+)\Z/) { |method_name, matches|
    "#{name} screams: #{matches[1].gsub('_', ' ').capitalize}!!!"
  }
  include MethodFound::Interceptor.new(/\Aask_([a-zA-Z_]+)\Z/) { |method_name, matches|
    "#{name} asks: #{matches[1].gsub('_', ' ').capitalize}?"
  }
end

Here we have composed three interceptors on method_missing. No other trace is left in Greeter, but you can peek at its ancestors to see directly the three module builders along with their regex matchers (interceptors override Module#inspect to show this):

Greeter.ancestors
=> [Greeter,
#<MethodFound::Builder:0x...>,
#<MethodFound::Interceptor: /\Aask_([a-zA-Z_]+)\Z/>,
#<MethodFound::Interceptor: /\Ascream_([a-zA-Z_]+)\Z/>,
#<MethodFound::Interceptor: /\Asay_([a-zA-Z_]+)\Z/>,
Object, ...]

So when a method is called that the class does not know, it will bubble up first to the “ask” interceptor and see if it matches the regex. If it does, it will return a value; if not, it continues up to the “scream” interceptor and checks against the second regex, and so on.

Here is the result:

greeter = Greeter.new("Bob")
greeter.say_hello_world
#=> "Bob says: Hello world."
greeter.scream_good_morning
#=> "Bob screams: Good morning!!!"
greeter.ask_how_do_you_do
#=> "Bob asks: How do you do?"

Given this example using module builders and method_missing, here’s a problem for you: how would you implement what we have shown above without module builders, but with the same level of flexibility?

The first thing that happens when you take away the module builder is that you lose the home for your module state – “normal” modules have no place to put it. So those regexes and blocks in the interceptor will need to go somewhere else, and the only other natural place to put them is the including class itself.

There are problems with this, which a real-world example should make clear.

Module Builders and Encapsulation

Let’s now take everything we’ve learned so far and apply it to some real living code at the core of Ruby’s most popular application framework.

Continuing from our discussion in the last section, I want to focus here on how modules are typically used to configure state, where that state is stored, and the problems with this approach to module configuration. I hope to convince you that module builders offer a much more natural way to encapsulate this state in the place where it best belongs: in the module itself.

ActiveModel::AttributeMethods

The module we will look at is ActiveModel’s AttributeMethods, which adds support for prefixed and suffixed attribute methods to Rails models¹⁰. If you have worked with Rails for any length of time, you are probably familiar with these prefix/suffix patterns through their use in change-tracking methods like name_changed? in ActiveModel::Dirty. The module implements both module bootstrapping (like AdderIncluder) and message interception using method_missing (like MethodFound::Interceptor).

Let’s start with a simple class that implements a method prefix and a method affix:¹¹

class Person
  include ActiveModel::AttributeMethods

  attribute_method_affix  prefix: 'reset_', suffix: '_to_default!'
  attribute_method_prefix 'clear_'
  define_attribute_methods :name

  attr_accessor :name, :title

  def attributes
    { 'name' => @name, 'title' => @title }
  end

  private

  def clear_attribute(attr)
    send("#{attr}=", nil)
  end

  def reset_attribute_to_default!(attr)
    send("#{attr}=", "Default #{attr.capitalize}")
  end
end

This is how we would use these attribute methods:

person = Person.new
person.name = "foo"
person.name
#=> "foo"
person.clear_name
person.name
#=> nil
person.reset_name_to_default!
person.name
#=> "Default Name"

Just looking at the code above, it is clear that the module is storing state somewhere, since it needs to know what prefixes, suffixes and affixes have been defined for a given class. We can see where this state is stored in the code for attribute_method_prefix, which is called in Person to setup a method prefix:

def attribute_method_prefix(*prefixes)
  self.attribute_method_matchers += prefixes.map! { |prefix| AttributeMethodMatcher.new prefix: prefix }
  undefine_attribute_methods
end

So the array of prefix strings are mapped to instances of a class, AttributeMethodMatcher, stored in an array attribute_method_matchers on the Person class. (The affix and suffix methods are similar.) These matchers are the module’s core configuration, and they are stored outside the module itself.

What’s in one of these matchers? Let’s have a look at the initialize method of the class to get an idea:

def initialize(options = {})
  @prefix, @suffix = options.fetch(:prefix, ""), options.fetch(:suffix, "")
  @regex = /^(?:#{Regexp.escape(@prefix)})(.*)(?:#{Regexp.escape(@suffix)})$/
  @method_missing_target = "#{@prefix}attribute#{@suffix}"
  @method_name = "#{prefix}%s#{suffix}"
end

So essentially, a matcher is two things: a prefix and suffix pair, and a regex generated from them (the rest is mostly for convenience). This internal state is used for two different but related purposes.

The first of these two purposes is important but not well-documented at all; you can find it explained inline in a comment here. The purpose is to provide a way to convert a hash of keys and values returned from an attributes method into a set of first-class attribute methods, supporting all defined prefix, suffix and affix method patterns. In the example above, attributes returns a hash with keys name and title, so these become attributes that dispatch to methods like clear_attribute and reset_attribute_to_default!.

Like MethodFound, this is implemented using method_missing: any method call which matches a matcher regex and whose target (captured from the regex) is a key in attributes is intercepted and dispatched to an attribute handler. So reset_title_to_default! is dispatched to reset_attribute_to_default!, with the name of the attribute ("title") as its argument.

Method interception is great for an open-ended, changing hash of attributes, but method_missing is slow as molasses. If you know what attributes you want, it’s generally better to define the methods explicitly.

This is the second purpose of the method matchers, triggered by calling define_attribute_methods with one or more attribute names after setting up prefixes, suffixes and affixes: to define methods for each combination of prefix and/or suffix on each attribute. Person defines methods for name, so clear_name and reset_name_to_default! dispatch to clear_attribute and reset_attribute_to_default!, respectively.

These methods are bound to an anonymous module defined and included in another method added to the Person class, named generated_attribute_methods. As instance methods, they take precedence over method interception, so while both clear_title and clear_name will return the same result, the former falls through to method_missing, whereas the latter is handled (much more quickly) by the attribute method.

ActiveModel's AttributeMethods

You can see the defined attribute methods if you grep the methods on an Person instance:

person = Person.new
person.methods.grep /name/
#=> [:name, :name=, :reset_name_to_default!, :clear_name, ...]

You can confirm these generated methods are defined on the module, and not somewhere else, by grabbing the module and checking its instance methods:

Person.ancestors
#=> [Person, #<Module:Ox...>, ActiveModel::AttributeMethods, ...]
Person.ancestors[1].instance_methods
#=> [:name, :name=, :reset_name_to_default!, :clear_name]

You can also find the method_missing override described earlier here on ActiveModel::AttributeMethods, just after the anonymous module in the class hierarchy.

These two implementations of attribute methods, one using method interception with method_missing and the other using method definition, complement each other since they are suited to different access patterns (one for a large open-ended set of attributes, the other for a smaller fixed set of attributes). Mobility actually uses the same approach in defining i18n accessors.¹² However, whereas AttributeMethods adds a dozen class methods and variables, Mobility adds none, instead hiding all state in its modules.

Let’s now look at an alternative implementation of AttributeMethods which, like Mobility, uses module builders to encapsulate state in the modules it builds.

MethodFound::AttributeMethods

This is where we bring all the ideas discussed so far together and apply them to a real application. Before jumping into more code, let’s review the elements implementing attribute methods on a class:

a class variable (attribute_method_matchers) holding an array of matchers for prefixes/suffixes/affixes
a method_missing override to dispatch method calls matching the attribute matchers and the attributes hash to handlers accordingly
an included anonymous module (generated_attribute_methods) holding defined attribute methods

Now consider that each of these core elements is located in a different place:

the method matchers are defined on the including class
the generated attribute methods are defined on an anonymous module included in this class
method_missing is defined on AttributeMethods itself, also included in the class

So we have the three elements of our implementation, coupled to each other through the method matchers class variable, spread out across different parts of the implementing system. Not only that, but the method matchers and generated methods, which (as we will see below) are essentially functionally independent, are stored together in the same place (an array and an anonymous module, respectively).

Let’s consider an alternative implementation which distributes state in a different way. In this implementation, we will group the following elements together into a single module using a module builder:

a single prefix/suffix pair, from which a matcher regex is generated
a single method_missing override which uses the regex to intercept methods
a method to define attribute methods for the given prefix/suffix pair

The module builder is named AttributeInterceptor. It inherits from MethodFound::Interceptor, which we saw in the last section, customizing the initializer so that it accepts a prefix and/or suffix instead of the more open-ended format of the default interceptor builder. Although it is not very big, I won’t include the actual code here so we can instead focus on what the class actually does.

Implementing AttributeMethods with MethodFound interceptors

Let’s first take a look at intercepting. We can reproduce this by replacing calls to define_attribute_prefix and define_attribute_affix in the Person class above with attribute interceptors, like this:

class Person
  include MethodFound::AttributeInterceptor.new(prefix: 'clear_')
  include MethodFound::AttributeInterceptor.new(prefix: 'reset_', suffix: '_to_default!')

  attr_accessor :name, :title

  def attributes
    { 'name' => @name, 'title' => @title }
  end

  # ...
end

We are instantiating two attribute interceptors and including each of them into the class. There is no special case here for “affix”; we simply build an interceptor with both a prefix and suffix.

With these modules included, Person behaves externally exactly the same as the earlier version using the ActiveModel module, but the internal implementation is quite different. This can be seen from its ancestors:

Person.ancestors
#=> [Person,
 #<MethodFound::AttributeInterceptor: /\A(?:reset_)(.*)(?:_to_default!)\z/>,
 #<MethodFound::AttributeInterceptor: /\A(?:clear_)(.*)(?:)\z/>,
 ...  ]

Like the MethodFound interceptors described earlier, these two modules each override method_missing and branch the code path if the method name matches their regex (stored on the module).

The resulting set of distributed nested conditionals is equivalent to this line from ActiveModel, which iterates through attribute method matchers checking for matches:

matchers.map { |method| method.match(method_name) }.compact

The key difference between these two implementations is that whereas this line is executed in ActiveModel::AttributeMethods (a module), using a class variable stored in Person (a class), the MethodFound version is executed across independent modules through method composition, using super.

The implementation of generated attribute methods is similar. AttributeInterceptor has a method define_attribute_methods which takes one or more attribute names and defines attribute methods for each of them with the module’s prefix and/or suffix, on the module itself. Again, because the module already contains its own prefix and suffix, it has all the information it needs to do this.

So functionality is truly encapsulated: a single module contains its own prefix and suffix, the method_missing override to catch attribute method calls, and a method to generate its own attribute method.¹³

Given this module builder, we can reproduce ActiveModel’s implementation without any class methods or class variables, like this:

class Person
  [ MethodFound::AttributeInterceptor.new(prefix: 'clear_'),
    MethodFound::AttributeInterceptor.new(prefix: 'reset_', suffix: '_to_default!')
  ].each do |mod|
    mod.define_attribute_methods(:name)
    include mod
  end

  #...
end

MethodFound includes another module, MethodFound::AttributeMethods, which adds class methods do simplify the code above, so it can be used as a drop-in replacement for ActiveModel::AttributeMethods. The implementation of define_attribute_methods in this module is interesting:

def define_attribute_methods(*attr_names)
  ancestors.each do |ancestor|
    ancestor.define_attribute_methods(*attr_names) if ancestor.is_a?(AttributeInterceptor)
  end
end

Since the class no longer stores the matchers in its state, the module cannot simply iterate through these matchers to define methods. Instead, it iterates through its own ancestors, which are module instances each with a prefix/suffix pair, and calls define_attribute_methods if the ancestor is an attribute interceptor. This generates attribute methods on each interceptor module, which can then be called by instances of the class.

The resulting implementation encapsulates related elements together into separate modules without coupling via class variables or methods. This encapsulation means that we could now design totally different types of attribute interceptors and include them in the same way; as long as they have the same interface (for generating attribute methods), the internals can be entirely different and nothing else will need to change.

This, it seems to me, is what a “module” should really be: an independent, interchangeable unit containing everything it needs to execute only one aspect of some desired functionality. Not only that, but the interface for building these modules falls directly out of Ruby’s own object model, a model that has been around as long as the language itself. Ruby has had this trick up its sleeve for years, most of us just never realized it.

What’s in a Name?

You may argue that the “Module Builder Pattern” I’ve introduced is just a Ruby subclass where the class happens to be Module, and what’s the big deal? And technically, you would be right. I’m not the first to notice this idea, nor am I the first to write about it. Just Foo < Module, and I could be done with it.

But programming languages are written and read by and for humans, and to humans, names mean a lot. If you can’t find the module with the methods on it buried in a list of ancestors, you won’t notice it, and if you can’t remember that subclassing thing some blogger wrote about, you won’t use it. The fact that this pattern has been around for this long without almost anybody knowing about it is testament to this fact.

So I gave it a name. A catchy one.

Because as Rubyists, I think we need this pattern. Take a look at the code of a large Ruby project and how modules are used (and abused) in practice. Our modules trigger callbacks that bootstrap all kinds of extra state into our classes, coupling them to the modules they include. A module should be self-contained, but we use the excuse that we cannot configure modules at runtime to justify storing configuration all over the place.

Ruby can do better than this, and we can do better than this. So take another look at that bloated module, the one with the tentacles that seem to have crept over every corner of your application, and give it a chance to be free. I think you will find that it — and you — will be all the better for it.

Update (2017/10/20)

This post has been translated into Japanese: see part 1, part 2 and part 3.

Update (2017/09/27)

Since writing this blog post, I have also presented the Module Builder Pattern at RubyKaigi 2017 in Hiroshima. Slides from that talk can be found on Speaker Deck, and a video has also been posted to YouTube.

Notes

¹ The first reference I can find to the idea of “subclassing Module” is in an article in 2012 by Piotr Solnica. Eric Anderson and Andreas Robecke have written about it more recently, but hardly anything else even mentions it.
² At the time of writing this post, it is used here and here.
³ By this I mean you can include modules on over another and use super to build complex composed method chains. I do a lot of this in Mobility.
⁴ And, to be precise, which also has an initializer which takes the values of these attributes as its arguments.
⁵ I’ve used a trick here to add a module’s methods as class methods to a class using include, by extending the class when the module is included. This is a common trick, if it looks a bit mysterious you might want to read up on the technique before continuing.
⁶ This “no-clutter” property of anonymous modules is also what makes them so handy in metaprogramming, since it means you don’t need to worry about constant name collisions.
⁷ It is actually being defined from these keys using a closure.
⁸ Defining an inspect method in the module builder class which includes the variables (e.g. amount and tax) which define makes this ancestor chain even easier to read.
⁹ MethodFound actually does a bit more than this: it also “caches” the method name matched in the regex or proc and defines it on the interceptor module, so that future calls will be much faster. You can see this if you look at the methods on an interceptor after it has matched a method using method_missing.
¹⁰ ActiveRecord models in fact include ActiveRecord::AttributeMethods, which itself includes ActiveModel::AttributeMethods and overrides some of the methods discussed here, specifically define_attribute_methods. The relationship between these two modules, and between how persisted and non-persisted attribute methods are handled, is somewhat complex, but the ideas discussed here are relevant to both.
¹¹ Note that I have slightly modified and simplified this example class from the one in the inline documentation to highlight some things that the standard docs do not mention.
¹² Mobility handles these two cases with the module builders FallthroughAccessors and LocaleAccessors. I also extracted these from Mobility into a gem called I18nAccessors.
¹³ It actually also has a method alias_attribute to alias attributes, like ActiveModel::AttributeMethods.

Figures

^a “Drawing Hands” by M.C. Escher. [reference]
^b “Automated space exploration and industrialization using self-replicating systems.” From Advanced Automation for Space Missions: 5. REPLICATING SYSTEMS CONCEPTS: SELF-REPLICATING LUNAR FACTORY AND DEMONSTRATION. [reference]
^c “Proposed demonstration of simple robot self-replication.” [reference]

Translating with Mobility

2017-03-03

In the first real post on this blog, I described my experience hacking globalize, a gem that calls itself the “Rails I18n de-facto standard library for ActiveRecord model/data translation.”

Globalize was the first gem that I contributed to, and the experience was very important to me as a step toward becoming a better and more knowledgeable programmer.

I am now one of the authors of that gem, having made many commits to the repository. In the time since I first contributed to Globalize, I’ve become keenly aware of the limitations of the gem, and of problems with existing solutions for managing translations in Rails projects.

I’d like to look back in this post to the problem of translation and to solutions devised so far, as motivation to introduce a gem I’ve just released called Mobility. Mobility attempts to solve some core problems with the multitude of Ruby-based translation gems out there right now.

The Problem

You have an application with user-generated content. You want to translate that content into multiple languages. And – importantly – you want to do it in such a way that you don’t need to recode all your existing presentation logic (views, mailers, controllers, etc.) to handle the new languages.

In other words, things should just work – more or less the same as they do with one language, except that switching the language should tell the application to switch context to read or write content stored in that language.

So this:

post = Post.first
post.title

… should return the title of the post, in the language I am currently viewing.

I18n.locale = :en
post.title
#=> "Mobility (noun): quality of being changeable, adaptable or versatile"
I18n.locale = :ja
post.title
#=> "Mobility(名詞):動きやすさ、可動性"

Likewise, I should be able to write translated values the same way:

I18n.locale = :en
post.title = "Translating with Mobility"
post.save

Now the English title has changed, but the Japanese one has not:

post = Post.first
post.title
#=> "Translating with Mobility"
I18n.locale = :ja
post.title
#=> "Mobility(名詞):動きやすさ、可動性"

For bonus points, this solution should also work for querying attributes, so something similar to this:

Post.find_by(title: "Translating with Mobility")

… should work even if title is a translated attribute.

For this to happen, something has to hook in to attribute accessors, to detect changes in the locale and return language-specific content accordingly. This may seem easy to do at first glance, but it gets tricky.

Notice that what we’re asking for is similar but different from the problem solved by localization solutions such as the Rails I18n API, which provide namespaced translation lookup (generally) for interface translation, typically with static data.

With application content, we want translations at the class level, in the form of persisted attributes of a model. Ultimately we’re talking about a storage strategy.

As it turns out, there are a number of different strategies that different people have come up with to solve this problem. Let’s take a look at them.

Solutions

Strategy 1: Translatable Columns

The most obvious solution to this problem is to create columns on your model table in each locale you want to support, with some convention for identifying the locale of the column.

Typically, this is done by appending a suffix to the column name made up of an underscore plus the locale, with any dashes in the locale converted to underscores.

So title becomes title_en, title_fr, title_ja, and so on. Adding a new locale to an application then requires that you add columns for that locale to each and every translated model table (see figure below.)

Table structure of column strategy

For a small number of models, and a small number of locales, this is a reasonable solution. It has the advantage that the column structure is easy to understand, and querying by translated attribute is straightforward:

Post.find_by(title_en: "Mobility")
#=> #<Post:0x... id: 123 content_en: "Mobility" content_ja: "モビリティ"...>
Post.find_by(title_ja: "モビリティ")
#=> #<Post:0x... id: 123 content_en: "Mobility" content_ja: "モビリティ"...>

Some key disadvantages of this approach:

For many models and many locales, this approach requires many migrations, each time a locale or model is added.
Space is not used very efficiently: every record has a column for every locale, even if the locale is not used.

Despite these disadvantages, this strategy is appropriate for applications with a fixed number of locales. For a very nice implementation of this approach, see Traco.

Mobility supports this strategy with its :column backend.

Strategy 2: Model Translation Tables

Now consider that we have an application where we have many translated models, with a potentially open-ended list of locales we want to support. A service with user-generated content offered to users around the world would be a good example of this pattern. In this case, the first strategy becomes impractical as the number of locales grows; with, say, 50 locales, and only a small number of actually translations, the vast majority of translated columns would be null. On top of which, adding new locales to all the translated models would be a pain, not to mention error-prone.

This brings us to the next strategy, shown in the database schema below. Here, each model table is assigned its own translations table. Each row in the translations table has:

A column for the locale (locale)
A foreign key pointing to the model table (post_id)
One or more columns for translations (title, content)

With this structure, we can now join the translation table (e.g. post_translations) to the model table (e.g. posts) on the locale we want to get the desired result: a model with translations in that locale.

Table structure of model translation strategy

In ActiveRecord, we would have an association on the model Post to the model PostTranslation, such that we can fetch translations with post.translations and pick out the one we want:

post = Post.first
translation = post.translations.find do |translation|
  translation.locale == "en"
end
#=> #<PostTranslation:0x... id: 123 locale: "en" ...>
translation.title
#=> "Translating with Mobility"
translation.content
#=> "In the first real post on this blog..."

With this approach, we’ve nicely solved some of the problems with the first strategy:

Since translations now have their own model and table, and we only create rows in that table when we save a translation in that locale, space usage is more efficient (no wasted columns/rows for locales that have no translation).
We do not need any migrations to add locales: simply save the translation for a new locale and it becomes available. So scalability on the “locale axis” is very good.

However, there are some clear disadvantages with this approach that should be kept in mind:

Having translations on a different table implies a lot of joining, which (if not managed very carefully) can lead to performance degradation.
Joining can also add significant complexity to the code dealing with translations. In particular, querying on translated attributes is much more complicated than it was with the first strategy, where we queried on the model column directly.

Note also that although this strategy scales nicely along the “locale axis”, it nonetheless requires a table for every translated model, and thus a migration every time we add a new model or a new translated attribute to an existing model. So this strategy may not scale well if an application has many different forms of translated content, spread across a large and growing list of models.

This strategy is implemented by Globalize, the gem mentioned at the beginning of this post (it is also implemented by many other gems). For the reasons mentioned above, Globalize has become the preferred option for model translation in Rails. Like the first strategy, it is a very useful approach for many (but not all) use cases.

Mobility supports this strategy with its :table backend.

Strategy 3: Shared Translation Tables

My experience working with Globalize, and in particular with scalability issues mentioned above, prompted me to consider another strategy which would scale better with model creation. I wanted a strategy which would be usable immediately at the code level, without having to add a table or column every time a new model or translated attribute is added.

This led me to the third strategy, which I’ve called the “key value” strategy. The basic idea is simple: rather than using a dedicated table for each translated model, instead create shared tables to store translations across all models (as shown in the figure below). Two tables are required since we will be storing two types of values: strings and text. (If you wanted to store other column types, you could add more tables for each of those types, or cast strings to some other type).

Table structure of key value strategy

The key thing here is that rather than using a normal association as we did in the last strategy, where post_id on the post_translations table pointed to the posts table, we will instead use a polymorphic association. By doing this, we can share the translations tables with as many models as we like, migrating just once to create the shared tables.

The new tables have the following columns:

A column for the locale (locale)
A foreign key pointing to the translated model id (translatable_id)
A column holding the model class (translatable_type)
A column holding the name of the attribute being translated (key)
A column holding the value of the translated attribute (value)

With these tables in place, we define a polymorphic relationship on the model, something like this:

class Post < ActiveRecord::Base
  has_many :text_translations,
    as: :translatable,
    class_name: TextTranslation,
    inverse_of: :translatable
  #...
end

and a translation class:

class TextTranslation
  self.table_name = "mobility_text_translations"
  belongs_to :translatable, polymorphic: true
end

Now that we have these models in place, we can easily fetch and set translations of a model class:

post = Post.first
post.text_translations.find do |translation|
  translation.key == "title" && translation.value == "Translating with Mobility"
end
#=> #<TextTranslation:0x...
      id: 123
      locale: "en"
      key="title"
      value="Translating with Mobility"
      ... >

To add translations to a new model, we simply add a polymorphic relationship like the one above on Post, and we can immediately add translations to the model.

In practice, there are some other issues that need to be handled to make this work effectively, but done correctly this strategy has some notable benefits:

No more migrations (after the first one), so an application can quickly add new translated models.
As with Strategy 2, no extra effort to add new locales.
Relatively efficient in terms of storage, in the sense that we only create records for translations in locales where they actually exist. However, since each attribute translation gets a row in the table, for multiple attributes on a single translation table, Strategy 2 may be more space efficient since it stores translated attributes together in a single row.

Now for the disadvantages:

The join complexity described in the last section is even worse now, because we have to join every attribute separately when searching for matches. This is quite tricky but can be done. I’ll probably write about this in a future post.
Since we have each attribute stored separately (as a key/value pair) it becomes trickier to maintain our database in a consistent state. For example, if we remove a model, its translations will remain in the translations table(s) unless we explicitly remove them. Likewise, if we decide to rename a translated attribute, we may end up with translations for the previous attribute name leftover in the table.

Despite these disadvantages, I consider this strategy to be applicable to the widest variety of situations. It is the default strategy in Mobility, implemented as the :key_value backend.

Strategy 4: Serialized Data

Yet another set of strategies take the approach of jamming all the translations for an attribute into a single column. The first strategy following this approach is one which serializes translations as a hash and stores it on a text column.

At the model layer, this is pretty trivial in Rails:

class Post < ActiveRecord::Base
  serialize :title_translations, Hash
end

post = Post.new
post.title_translations = { "en" => "Translating with Mobility" }
post.save

We can now fetch translations simply by fetching the value for the key matching a given locale:

post.title_translations["en"]
#=> "Translating with Mobility"

This strategy has the obvious advantage of simplicity: all our translations are stored together on the attribute column. Like the model translations and key-value strategies, locale scalability is not an issue since we can just add new hash keys as we need them.

In addition, no additional migrations are necessary; simply adding a text column to every model for each translated attribute is enough to support any number of translations.

Despite these advantages, the serialized approach is not generally recommended, for the key reason that querying on serialized data is not possible. Indeed, Rails itself has discouraged the use of serialized attributes in favour of database-specific storage solutions (described below).

Nonetheless, there are valid use cases for this strategy. One gem that implements this approach is Multilang.

Mobility supports this strategy with its :serialized backend.

Strategy 5/6: PostgreSQL Hstore/Jsonb

The last two strategies, Hstore and jsonb, require the use of PostgreSQL as your database. Hstore columns store depth-1 hashes, where keys and values of the hash must be of string type. Jsonb (“binary json”) columns store data in JSON format, which supports a much broader range of primitives (strings, numbers, booleans and null) as well as arrays, and unlimited depth. Jsonb is generally preferred over Hstore since it is more flexible.

Further details on these data types can be found in the links above; for our purposes, the main point is that they both store complex data types in a single column, in a form that can be easily and quickly queried.

An example of such a query (on a Jsonb column) looks like this:

SELECT "posts".* FROM "posts"
WHERE (posts.title @> ('{"en":"Translating with Mobility"}')::jsonb)

Querying on multiple locales at once is also simple given the flexibility of the data format.

Gems that use Hstore as a storage format for translations include Trasto, Multilang Hstore and Hstore Translate. I only know of one gem (as of yet) that uses Jsonb for storage, Json Translate.

Mobility supports Hstore with its :hstore backend and Jsonb with its :jsonb backend.

Common Patterns

Having worked extensively on Globalize and having delved into the code of many of the other gems mentioned above, it became clear to me that each translation solution ended up implementing many of the same things, over and over again.

The concept of a translation cache, for example, is implemented for translatable columns in Traco, for model table translations in Globalize, and for hstore translations in Multilang Hstore.

The concept of fallbacks (originally described by Sven Fuchs in this blog post) is likewise implemented again, and again, and again.

Locale accessors, methods which fetch an attribute in a particular locale with a dedicated method like title_en, is yet another repeated pattern found in many gems, for example Globalize Accessors in the case of Globalize.

Finally, tracking of changes (i.e. Active Model Dirty in Rails) is another common pattern, found here in Globalize and here in Multilang Hstore, for example.

What is important to note here is that while each of these is implemented in their own way, the repeated patterns (caching, fallbacks, locale accessors, dirty tracking, etc.) are not in general dependent on the storage implementation strategies described earlier.

In fact, the core pattern of fetching and updating translations using getter and setter methods, which behind the scenes map to some storage format, is itself similar across translation gems, independent of the storage strategy.

This code from Trasto captures this core pattern succinctly:

def define_localized_attribute(column)
  define_method(column) do
    read_localized_value(column)
  end

  define_method("#{column}=") do |value|
    write_localized_value(column, value)
  end
end

(Here, read_localized_value fetches the value of the translated attribute column in the current locale.)

This commonality between translation solutions motivated me to consider whether it would be possible to separate the implementation of translation as a high-level pattern from the lower-level implementation of storing translations (an idea also discussed here). This led me to start developing Mobility.

Introducing Mobility

So what is Mobility? Well, I’ve spent most of this post explaining a bunch of storage strategies and some things which are similar about them. Mobility takes this whole idea to the next level and turns the problem of translating attributes into the problem of mapping translated attribute accessors (readers and writers) to an abstract pluggable “backend”, which encapsulates (and isolates) all the storage logic.

Rather than jump into the details of how these backends work (which would require a blog post of its own), let’s look at how we can use Mobility to quickly implement any of the storage strategies above.

Getting Started

First, add mobility to the Gemfile:

gem 'mobility', '> 0.1.3'

In Rails, you can use a generator to create an initializer file and migrations for the shared translation tables for the key value strategy:

rails generate mobility:install

The initializer file has the line:

Mobility.config.default_backend = :key_value

With this configuration, the key value backend will be our default backend. Run the two migrations to generate string and text translation tables with rake db:migrate, and we’re ready to go.

Creating a translated attribute in a model is as easy as including (or extending) the Mobility module and declaring translated attributes (similar to Globalize and other gems):

class Post < ActiveRecord::Base
  include Mobility
  translates :title,   type: :string
  translates :content, type: :text
end

Now we can use the attributes:

post = Post.new
#=> #<Post id: nil>
post.title = "Mobility"
post.title
#=> "Mobility"
I18n.locale = :ja
post.title
#=> nil
post.title = "モビリティ"
post.title
#=> "モビリティ"
post.save

When you call post.title, you will see the SQL:

SELECT "mobility_string_translations".* FROM "mobility_string_translations"
WHERE "mobility_string_translations"."translatable_id" = 1
AND "mobility_string_translations"."translatable_type" = 'Post'
AND "mobility_string_translations"."key" = 'title'

post.content looks similar, but instead fetches from the mobility_text_translations table:

SELECT "mobility_text_translations".* FROM "mobility_text_translations"
WHERE "mobility_text_translations"."translatable_id" = 1
AND "mobility_text_translations"."translatable_type" = 'Post'
AND "mobility_text_translations"."key" = 'content'

It’s also possible to query for a post with finder methods using the i18n scope:

Post.i18n.find_by(title: "Mobility")
#=> #<Post id: 1>

The SQL for this query is a somewhat tricky table join, which is necessary since translations for this backend are stored on the shared translations table. (I’ll leave the details of that for another blog post.)

What is important to note here is that although the format for declaring translated attributes (with translates) is similar in form to other translation gems, the implementation is totally different. In particular, Mobility strives to keep the accessor and setup code for a given set of attributes totally isolated from attributes declared separately. This is what allows us to call translates twice in the model code above, one for each type of attribute (string and text).

Translation Options

Behind the scenes, when translates is called, Mobility creates an anonymous backend class which optionally includes modules for caching, fallbacks, and dirty accessors. Which of these modules is included can be customized by passing options:

class Post < ActiveRecord::Base
  include Mobility
  translates :title,   type: :string, fallbacks: { en: :ja }
  translates :content, type: :text,   fallbacks: { en: :fr }
end

I wouldn’t want to set up fallbacks like this in a real application, but regardless, it actually works: if we call post.title and the title is not set in English, it will fall back to Japanese. However, post.content will fall back to French, since this is how we have configured the attributes.

The fallbacks option is an example of a shared option which can be included for any backend. Others of this kind are cache (true by default), dirty (limited to classes which include ActiveModel::Dirty) and locale_accessors.

Here is an example of adding locale accessors to attributes:

class Post < ActiveRecord::Base
  include Mobility
  translates :title,   type: :string, locale_accessors: [:en, :ja]
  translates :content, type: :text,   locale_accessors: [:en, :ja]
end

Now we can use the accessors to access attribute values in English or Japanese:

post = Post.first
post.title_en
#=> "Mobility"
post.title_ja
#=> "モビリティ"

For more details on how these options are set, see the API documentation on the Mobility::Attributes class.

Translation Backends

As mentioned in Getting Started, the default backend is set in the initializer file to :key_value. But we can override it to use any backend, for any attribute:

class Post < ActiveRecord::Base
  include Mobility
  translates :title,   backend: :column
  translates :content, backend: :table
end

In order for this to work, we will need to create the necessary columns for the column backend (as described in Strategy 1, here for three locales):

class AddColumnsToPosts < ActiveRecord::Migration[5.0]
  def change
    add_column :posts, :title_en, :string
    add_column :posts, :title_ja, :string
    add_column :posts, :title_fr, :string
  end
end

…and translations table for the table backend (as described in Strategy 2):

class CreatePostTranslations < ActiveRecord::Migration[5.0]
  def change
    create_table :post_translations do |t|
      t.string :locale
      t.integer :post_id
      t.text :content
    end
  end
end

With these columns and table, we can use the attributes:

post.title = "Translating with Mobility"
=> "Translating with Mobility"
post.content = "In the first real post on this blog..."
=> "In the first real post on this blog..."

If we now save, this is the generated SQL:

INSERT INTO "posts" ("title_en") VALUES ('Translating with Mobility')
INSERT INTO "post_translations" ("locale", "post_id", "content") VALUES ('en',5,'In the first real post on this blog...')

So each attribute is treated entirely in isolation, and we can have multiple backends operating on the same model without any conflicts.

This works as well for queries:

Post.i18n.find_by(title: "Translating with Mobility", content: "In the first real post on this blog...")
#=> #<Post id: 5, title_en: "Translating with Mobility", title_ja: nil, title_fr: nil>

Note the SQL:

SELECT  "posts".* FROM "posts"
INNER JOIN "post_translations"
ON "post_translations"."post_id" = "posts"."id"
AND "post_translations"."locale" = 'en'
WHERE "posts"."title_en" = 'Translating with Mobility'
AND "post_translations"."content" = 'In the first real post on this blog...'

So components for each backend are combined to generate the final query. This applies to any combination of backends.

Mobility = Freedom

Why go to the trouble to implement things this way? There are a few reasons, but the driving motivation is to free the developer from having to constantly think about the details of the underlying translation implementation when working with translations. As its name implies, Mobility also strives to make it easy to change these details (e.g. change strategy) without having to rewrite an entire code base.

This in fact goes further than the examples described here. If you look at Mobility’s gemspec, you will see that it does not depend on activerecord or on activesupport. These are optional dependencies, and indeed the basic framework for setting up translated attributes with a backend (including caching, fallbacks and locale accessors) is totally independent of Rails. (Dirty tracking can also be used on a non-persisted class as long as it inherits from ActiveModel::Dirty.)

Thanks to this isolation of dependencies, Mobility can also support other ORM, and currently does: all strategies described here are also implemented for Sequel models. So this will work (provided translation tables have been created):

class Post < ::Sequel::Model
  include Mobility
  translates :title,   type: :string
  translates :content, type: :text
end

Mobility detects the parent class and generates a backend instance using a class namespaced accordingly.

Aside from keeping with the “DRY” philosophy of not repeating yourself, having a shared framework which works with many different configurations (ActiveRecord models, Sequel models, PORO, etc.) creates an opportunity to build something more interesting on top of the translation framework. I think there are some really interesting possibilities in this space, but current solutions being spread across more than a dozen different gems makes it impossible to explore those possibilities.

Try it!

So, try it and let me know what you think! I’d be really eager to hear from anyone interested in building interesting projects using Mobility for translation. While this blog has been nearly dead for a long time, I plan to follow up this post with more details on how Mobility works, other use cases, and on further developments of the gem. Stay tuned!

Debugging Spree

2014-04-17

Spree is the most popular e-commerce platform built on Ruby on Rails. It’s made up of a set of different gems implementing different functions of the platform: a front and back end, an API, command-line tools and a core powering a cart and payment system.

Spree: e-commerce solution for Ruby on Rails

Until recently although I had heard of it, I’d never had the chance to deploy an application with Spree, let alone dig into its internals.

That all changed with my new job at Degica, where Spree is a core element of almost everything we do. Not only do we use Spree to power our carts, we’ve also extended it, built on it, transformed it and twisted it to make it do things it was probably never intended to do.

Putting aside the question of whether this kind of “adaptation” is a good thing (and you could fairly argue that in some cases it is not), as a programmer it presents an interesting challenge: to ensure that the core Spree engine and everything we’ve built on top of it are working smoothly together. The more complex our adaptations become, the more difficult this can be.

One bug we hit on recently brought this challenge home to me, and got me thinking about the whole process of programming and how it is affected by our environment.

A transaction in limbo

The bug was brought to our attention by our customer support team, who reported that some customers would make an order, get a confirmation email with payment instructions, pay for the product, but then never receive the expected license key(s) from us.

When we checked our system, we saw that the payment attached to the order was in a “failed” state. Spree’s cart makes extensive use of the state_machine gem to manage transitions between the sequence of events in the cart checkout and payment process. The “failed” state in this machine is a final state with no transitions – once payment reached this state, the order was effectively killed on our side.

Transitions between payment states in the Spree payment flow

The mystery deepened when we checked with the gateway and learned that, while the order had failed on our side, the payment had gone through on theirs without any issues. Customer support made a note of the bug on the order and manually triggered delivery of the keys as we began the process of combing our logs to figure out what had happened.

Eventually we tracked the problem down to “double submission” by the client: for whatever reason (perhaps an accidental double-click or javascript bug), the customer sent two requests to our server with the same order number. Since our servers run multiple workers (unicorns), each request triggered a parallel request to the payment gateway with the same information.

The gateway accepted the first request, returning a success response and triggering our system to send out the first confirmation email and mark payment as “pending”. But when the second request hit the gateway, it returned an error complaining that the payment had already been processed. This in turn caused the second worker to overwrite the previous pending state with a “failed” state.

Through this sequence of events, the order was left in limbo: its payment had been accepted by the gateway but marked as failed on our system, so it was unable to transition to the “complete” state and deliver the requested keys.

The ghost in the (state) machine

At this point, based on log data and a bit of guesswork, we had narrowed down the what of the bug, but not the why. I found that I could actually reproduce the bug running multiple unicorns on my laptop, but the deep integration of the state_machine gem into Spree – not to mention the commits we had added in our fork – made it hard to pinpoint where the bug was actually coming from.

The Spree documentation described a “payment processing” state, “intended to prevent double submission” (see diagram above). The state is triggered by the started_processing method, which is called in two key places: in the authorize! and purchase! methods of the Spree::Payment::Processing module, included in Spree::Payment. Payment processing is wrapped in the process! method in a conditional which checks that it is not already in that state – this is the mechanism to prevent double submission mentioned in the documentation.

The conditional should block double submission, but when I dug deep enough, I found that this was not happening. By placing a binding.pry call in the authorize! method, I could get two parallel processes simultaneously into the processing state, which by all accounts should not be possible.

Running slower through the code with parallel pry sessions, I found that although the first process would see the payment state as processing, the second process would see it as pending until it too entered the processing state. With both processes processing, I then opened a mysql session and found that – contrary to what both pry sessions were telling me – the state in the database was still pending.

The processes, in other words, were running in bubbles. Or, in database-speak, “transactions”.

Transactions in transactions

A transaction is a unit of work performed in a database which can be rolled-back at any point prior to completion. Two processes running in transactions are isolated until the point when the transaction ends and data is written to the database.

A quick check from either pry session confirmed that the payment processing code was indeed wrapped in a transaction:

ActiveRecord::Base.connection.open_transactions
#=> 1

Before moving on, I wrote a quick failing test isolate the problem:

it "does not process payment within transaction" do
  # Make sure we are not already in a transaction
  ActiveRecord::Base.connection.open_transactions.should == 0

  Spree::Payment.any_instance.should_receive(:authorize!) do
    ActiveRecord::Base.connection.open_transactions.should == 0
  end

  order.payments.create!({
    :amount => order.outstanding_balance,
    :payment_method => payment_method,
    :source => creditcard})
  order.next!
end

Now all I had to do was get this test to pass. To do this, I had to dig into state_machine, particularly the 1.1.2 release which Spree 1-3-stable uses. As it turns out, there are some inline comments that were a big help:

# To turn off transactions:
#
#   class Vehicle < ActiveRecord::Base
#     state_machine :initial => :parked, :use_transactions => false do
#       ...
#     end
#   end
#
# If using the +save+ action for the machine, this option will be ignored as
# the transaction will be created by ActiveRecord within +save+. To avoid
# this, use a different action like so:
#
#   class Vehicle < ActiveRecord::Base
#     state_machine :initial => :parked, :use_transactions => false, :action => :save_state do
#       ...
#     end
#
#     alias_method :save_state, :save
#   end

So here was my answer: add the :use_transactions => false and :action => :save_state options and alias save to save_state. (The details of why we have to alias save to save_state are pretty involved, but the basic point is that state_machine runs its transition callbacks as ActiveRecord callbacks if the action is save, and these callbacks are called within the transaction. State transition callbacks on any other action are run around the action, so are not within the transaction.)

With these small changes, my test passed and the processing state actually began to do its job. I’m happy to say that those changes have now been incorporated into the master (PR) and 1-3-stable (PR) branches of Spree, and I can count myself as one of the contributors to the project. (Thanks in particular to @huoxito who read and responded to my rambling comments on the pull request.)

Epilogue

This little “bug hunting” expedition was quite an eye-opening experience for me. The bug itself of course taught me a lot about the internals of Spree. More than that, though, it highlighted to me how essential it is that programmers have the space and freedom to dive deep into code – as deep as necessary – to solve difficult, often unexpected problems.

Here is a case where a silently failing feature in the core of a widely-used library emerges as an annoying double-click bug, one that could easily be ignored or patched at the surface level. To actually fix the bug required the patience to yank it out by its roots, delving through application code, to a dependency on a forked core library (spree), to the interaction of that library with its own dependency (state machine).

At any point, it would be very easy for something (an impatient manager, impending deadline or unexpected event) to interrupt the bug hunt and leave the core library broken. No disaster would have resulted had that happened in this case, and I could have spared myself days of frustrating debugging.

But in the long run, this would leave us with a broken payment system at the core of all our e-commerce applications. If anyone underestimated the importance of having reliable open-source libraries, recent events have provided ample evidence to convince them otherwise.

Disregarding the bug would also have left me, the programmer, with a gaping hole in my understanding. It would have robbed me of a great chance to understand how the system works. And it would have left me with the disempowering sense that the bug was beyond my understanding, when in fact it was not.

I’m glad that didn’t happen to me in this case. But I have to wonder how often it does, and how that type of failure – coming back empty-handed after hours or even days of fruitless bug hunting – must affect programmers and the code they write. The experience has given me a greater appreciation of how much a work environment can contribute to code quality. And it has taught me to slow down: to look at problems carefully before rushing in to solve them, to think carefully before rushing to write code.

Here’s hoping those lessons will lead to more successful bug hunts, and less bugs.

Software, the Moving Target

2013-11-27

Yukihiro “Matz” Matsumoto, creator of the Ruby programming language, gave a fascinating keynote talk at this year’s RubyWorld conference the other day that left me thinking. It was a talk about Ruby, about the recent history of software, but also about much more than that.

Yukihiro "Matz" Matsumoto at RubyWorld 2013
(Photo by Osamu Fukui)

In his trademark style, Matz recounted this history through the story of his own life. That story begins just before he first created Ruby, 20 years ago, in an era where hardware was expensive, software took years to create, and documentation was measured not in number of pages but in centimeters of thickness.

“20 years ago, we should have admitted our ignorance.”

Software was like nothing mankind had ever created before, a digital embodiment of human thought. But too much money was invested in hardware and infrastructure for companies to admit to clients that they knew nothing about how to build it. Failure was just too costly.

Now, looking back from an era where hardware is cheap, software is commonly developed in weeks or months, not years, and documentation is measured in commits, not pages, some things become clear.

The first is that our basic assumptions were wrong, foremost among them the assumption that “we know what we should make”. Were this true, software development would simply be a matter of specifying requirements, setting out a plan based on those requirements, and executing the plan.

Except, that doesn’t work. Or at least, it doesn’t work very well.

Moody skies at RubyWorld 2013
(Photo by Hiroshi Shibata)

Before creating Ruby, Matz himself worked at a company producing enterprise software this way, with large projects executed over many years and code written to specification, waterfall-style. Clients said what they wanted, specifications were drawn up, and programmers wrote code to satisfy those specs.

And in end, something was produced. It may not have been very satisfactory, or even satisfactory at all, but there was something to show for the many years of time and resources invested, and that something was produced on schedule. That was the minimum requirement.

But twenty years ago, Matz already felt that something was wrong. The big projects at his company treated software just like automobiles, or cabinets, or toasters, something to be designed, manufactured and delivered. But code is not physical, it’s digital, and it follows a different set of laws.

The challenge of grappling with those laws was vastly underestimated then just as it is now. The word “software” itself, coinded to describe the stuff being coded, evoked the wrong image: software was not “soft” at all, but hard. Very hard.

Once upon a time, computing was expensive

“Software may be the most complex thing ever created by human beings.”

It is fascinating to me to think that software is at once one of the most complex things we have ever created, and also something that defies our basic instincts about what it is to “create”. Thousands of years of blood and sweat have been spent learning to build things in the physical world, so it’s not exactly surprising that we want to apply these lessons to the digital world.

But they fail us, Matz said, and not in a good way – at high cost, without teaching us anything. They fail us because we are not developing software in a predictable, cartesian world like the one we live in, but in an abstract world that is unfamiliar and constantly changing.

In this world, where our real-world instincts are often wrong, we need to try and fail often, in a repeatable way, and cheaply. Large projects of the kind Matz had been working on 20 years ago do exactly the opposite.

The message was not exactly new, but Matz expressed it with such simplicity that it seemed self-evident. He did not profess to have the solution to the problem, or even suggest that there was one solution to the problem, but only offered some hints drawn from experience.

“Keep moving,” he said, advising against the conservative “ostrich” strategy of putting your head in the sand in the hope that things will return to normal. (They won’t.)

“Work hard to code less,” he said. Collaborate, leverage others' work, use existing tools. In other words, stand on the shoulder of giants. This made a lot of sense to me.

Write great code, change the world

“Write great code. Change the world.”

Ultimately though, this was the most powerful message. Ruby was designed to make this possible, by maximizing programmer happiness. But Matz, to his credit, hardly mentioned Ruby. He was thinking more broadly.

And when thinking more broadly, something struck me as curious about the whole gathering at RubyWorld.

Here we were, brought together by a programming language named Ruby. Ruby was designed by one programmer in his free time at a large corporation producing enterprise software to specification. Twenty years later, this programming language is used around the world, forms the foundation for one of the world’s most popular web frameworks, and has arguably elevated Japan’s status and visibility in the global technology sphere.

The fact that this programmer had the free time to develop Ruby was pure luck. Certainly nobody at his company ever allocated time to develop a quirky programming language to build better software by increasing programmer happiness.

The situation today is not so different: industry and government invest massive sums of money in large, expensive projects destined to the dustbin of history, while largely ignoring a homegrown success story right at their doorstep.

Now here was the creator of Ruby telling us that the existing approach to building software is wrong. Were Matz not such a gifted storyteller, this message might come off as arrogant or paternalistic. But it didn’t.

And yet.

I watch everybody nodding their heads in agreement. But are they really listening?

For a core group of hackers and entrepreneurs mostly in Tokyo, early adopters of every new technology, what he’s saying is old news. They’ve heard the message before – by now, many have internalized it. They’re looking into their laptops as he speaks, acting on his advice without even hearing it.

Matz isn’t really speaking to them. He’s speaking to everyone else. And he’s trying, in his very personable, self-effacing style, to get a message across to them.

The Red Queen from Alice in Wonderland

“If you want to get somewhere else, you must run at least twice as fast as that! ”

Software is not just something which affects software developers. Not anymore. It’s not just in the computers, it’s in the automobiles and the toasters, too. It won’t be long before it’s in almost everything.

And just as it is hard for software developers to build software, so it is hard for everyone else to understand software, and the potential of software. People look to lessons from the past. When those lessons do not provide useful clues, they stick their head in the sand, hoping things eventually return to normal.

But they won’t.

Software is fundamentally different. It’s a moving target. Simply to keep up takes enormous effort – to actually push the boundaries, you need to run twice as fast, think twice as hard, stretch your imagination twice as far – so most people just stick to doing the same thing, and demanding more of the same thing.

We need to change more than just how we create software. We need to change how people understand software.

And in that sense, watching his talk, it seemed to me that Matz the storyteller was at least as influential in today’s world as Matz the programmer was 20 years ago. Ruby is a language that changed how programmers write code. The lesson of Ruby is a story that could change how people understand code.

That’s a story that I want to read.

Bundler, Meet Bower

2013-11-04

Bundler is a package management tool for ruby gems. It takes in a ruby file with a list of gems and version numbers, like this:

source 'https://rubygems.org'

gem 'rails', '4.0.0'
gem 'sqlite3'
gem 'sass-rails', '~> 4.0.0'
gem 'uglifier', '>= 1.3.0'
gem 'coffee-rails', '~> 4.0.0'
gem 'jquery-rails'

and a single command:

bundle install

and magically downloads all the dependencies for your project, so you don’t have to hunt them down yourself or store them in your git repository.

That means that when you update a dependency, rather than see all the changes to that dependency in your git history, you see this:

-gem 'rails', '4.0.0'
+gem 'rails', '4.0.1'

Beautiful! Elegant and succinct – everything that I want to know expressed in one line.

Bundler: The best way to package your application's dependencies.

But now wait, what about all those pesky assets? What do we do when we want to update our version of Backbone.js? Well, without a package manager, we’ll end up with an ugly commit like this.

What? Did I make all those changes? No, but it sure looks like I did.

And how do I know that Backbone.js depends on jQuery and Underscore.js? I just need to know. And if the dependencies change, I’ll have to go hunt down the documentation to see what the new ones are.

Yuck. Not elegant at all.

It’s amazing to me that most of the Rails community seems satisfied with this sorry state of affairs. I supect that this is one of many factors contributing to the slow uptake of Javascript frameworks like Backbone.js by Rails developers.

If assets are first-class citizens, then they deserve to be managed as such.

Enter Bower, a package management tool for javascript libraries. It takes in a JSON file with a list of javascript dependencies and version numbers, like this:

{
  "name": "bower dependencies",
  "dependencies": {
    "backbone": "1.0.0",
    "backbone-relational": "0.8.5",
    "jquery": "1.10.1",
    "jquery-timeago": "https://raw.github.com/rmm5t/jquery-timeago/b311508ee019650200f552015a48f69de18c5857/jquery.timeago.js",
    "underscore": "1.4.4",
    "jquery-masonry": "http://www.fishspotr.com/jquery.masonry.js",
    "jquery-autosize": "latest"
  },
  "devDependencies": {
    "sinon": "http://sinonjs.org/releases/sinon-1.4.2.js",
    "jasmine-sinon": "0.1.0"
  }
}

and a single command:

bower install

and magically downloads all the dependencies for your project, so you don’t have to hunt them down yourself or store them in your git repository.

That means that when you update a dependency, rather than see all the changes to that dependency in your git history, you see this:

-    "backbone": "1.0.0",
+    "backbone": "1.1.0",

Beautiful! Elegant and succinct – everything that I want to know expressed in one line.

Bower: A package manager for the web.

The amazing thing is that these two package managers work beautifully in tandem: bundler for backend packages (gems) and bower for front-end assets.

I recently added bower to our pipeline and zapped 8,935 lines in one commit. Those were all lines of code from assets that should never have been in our repository in the first place. All of that code is now expressed in a single bower.json file, used at build-time to download all packages and dependencies, just like you would with bundler.

Now that we’ve switched to bower, I can’t believe that anyone would manage assets without it.

For those interested in trying out bower with a Rails stack, here are a few steps to get you going. First, assuming you have node and npm installed (they’re packaged together nowadays, so installing once will get you both), you’ll can install bower with:

npm install -g bower

Next, create a directory in vendor/assets called bower_components. Then add a .bowerrc file to your root directory to tell bower to install assets in that directory:

{
  "directory": "vendor/assets/bower_components"
}

You’ll probably want to ignore that directory in your .gitignore file since you the whole point is not to commit these dependencies to the git repository.

Now you’ll need to describe your dependencies in a JSON file named bower.json. The key element of that file is the dependencies hash, which contains all your apps dependencies and their version numbers (see the example above).

Once you have that file, just run: bower install to pull them all in with whatever their dependencies.

There are a couple other caveats. You have to add bower_components to the assets.paths config to get the asset pipeline to pick it up. In config/application.rb, add the line:

config.assets.paths << Rails.root.join('vendor', 'assets', 'bower_components')

Also, since bower packages are each given their own directory, you will need to require them with jquery/jquery rather than just jquery as you normally would, e.g.:

#= require jquery/jquery

Incidentally, if you dig into Sprockets, the Rack-based asset packaging system used by the Rails asset pipeline, you’ll see that it has in fact recently added support for bower.json configuration files in assets and if it finds one, will let you load the package with its name rather than the full path. However, many bower packages continue to name their configuration file package.json instead of bower.json, which sprockets will not find, so this setup will not work universally. (I submitted a pull request to change this but it was rejected. Bah.)

Also, somewhat annoyingly, the bower assets will be given a low priority in the asset search paths, so if you have gems that include assets, those assets will take precedence. Better just to use the full path (i.e. jquery/jquery) and avoid any potentially confusion bugs down the line.

That being said, using bower with a Rails stack is easy peasy and brings the full ecosystem of asset packages to your fingertips. Take the plunge, you won’t regret it!

Further reading: How to manage front-end packages in Rails with Bower