dejimatahttps://dejimata.com2013-02-13Chris SalzbergTranscript of my EuRuKo 2018 Talkhttps://dejimata.com/2018/8/23/transcript-of-my-euruko-2018-talk2018-08-232018-08-23Chris Salzberg<p>This is a very rough transcript of my talk, <a href="https://speakerdeck.com/shioyama/metaprogramming-for-generalists">Metaprogramming for
Generalists</a>,
given at this year’s <a href="https://euruko2018.org/">EuRuKo</a> conference in Vienna,</p>
<p>This is a very rough transcript of my talk, <a href="https://speakerdeck.com/shioyama/metaprogramming-for-generalists">Metaprogramming for
Generalists</a>,
given at this year’s <a href="https://euruko2018.org/">EuRuKo</a> conference in Vienna,
Austria. There is also a <a href="https://www.youtube.com/watch?v=1fIlcnrJHxs&feature=youtu.be&t=286">video of the talk on
YouTube</a>.</p>
<script async class="speakerdeck-embed" data-id="d451cf97ffa5413c9b79d2c73f3feb51" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<hr />
<p>Hi everybody, my name is Chris Salzberg and the title of this talk is
“Metaprogramming for Generalists”. I know I know, you’re thinking, "He’s
starting the morning of the first day with metaprogramming?!”</p>
<p>But don’t worry! This will be interesting.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="About Me" src="/img/201808/slide-1.png" />
</div>
<h2>About Me</h2>
<p>Ok, so first about me. My handle is @shioyama and I live in Tokyo, in Japan.
But don’t be fooled! I am <em>not</em> Japanese, in case that wasn’t obvious. I’m
actually a Canadian originally from Montreal.</p>
<p>I work at a company called <a href="https://degica.com">Degica</a> based in Tokyo. We
have a booth so be sure to check us out.</p>
<p>In the open source world, I’m the author of a few gems, the most well-known
one is a gem called <a href="https://github.com/shioyama/mobility">Mobility</a> for
managing model translations.</p>
<p>And I write about things like the <a href="/2017/5/20/the-ruby-module-builder-pattern">Module Builder Pattern</a> at my blog,
dejimata.com.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Plan" src="/img/201808/slide-2.png" />
</div>
<h2>Plan</h2>
<p>This is going to be a two-part talk. In the first part, I’m going to introduce
a different, more meaningful way to think about metaprogramming that will be
useful for solving generic problems.</p>
<p>Then in the second part, we’ll solve a generic problem together in order to
build what I will refer to as “generic software”.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Generalist Metaprogramming" src="/img/201808/slide-3.png" />
</div>
<h2>Generalist Metaprogramming</h2>
<p>Ok, so let’s start. “Generalist Metaprogramming”, what in the world is that?</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="What's Metaprogramming?" src="/img/201808/slide-4.png" />
</div>
<h2>What’s Metaprogramming?</h2>
<p>Well, this year is Ruby’s 25th birthday, right?</p>
<p>And I think more than anything else, metaprogramming is an area where Ruby
really stands out among programming languages. Because Ruby really embraces
this concept. It underlies all these things we love, the DSLs, the human
language-like syntax, all that stuff.</p>
<p>But we have a darn hard time defining this thing.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Defining Metaprogramming" src="/img/201808/slide-9.png" />
</div>
<h2>Defining Metaprogramming</h2>
<p>Well ok, it’s a noun. I think we can agree on that. Unfortunately, that’s
about all we can really agree on.</p>
<p>If “programming” is “writing code”, then the first obvious definition is
“writing code that writes code”. But this doesn’t really cover everything we
do with metaprogramming.</p>
<p>So the second definition is a “technique in which computer programs have the
ability to treat programs as their data”. This one is from Wikipedia, it’s
also something referred to as "reflection”.</p>
<p>So now we’re thinking more broadly. But if you look at the talk page on
Wikipedia, you’ll see that not everybody agrees with this one either.</p>
<p>Another angle on metaprogramming is called “introspection”. A program is a
subject, and it can find out things about other programs.</p>
<p>And then, at some point, we just throw up our hands and call it “a bag of
tricks”. This by the way is from <a href="https://pragprog.com/book/ppmetr/metaprogramming-ruby">Metaprogramming
Ruby</a>, which by the way
is a great book.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Defining Metaprogramming"
src="/img/201808/slide-10.png" />
</div>
<h2>Bag of Tricks</h2>
<p>And what are those tricks? Well, here they are. Some of you will recognize
many of these. I think everyone will recognize “monkeypatching”.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Working Definition"
src="/img/201808/slide-11.png" />
</div>
<h2>The “What” View of Metaprogramming</h2>
<p>But the thing is that, in practice, we tend to use a working definition that
is much more basic. We label a subset of Ruby as “metaprogramming”, methods
like eval, class_eval, define_method, etc.</p>
<p>And most talks about metaprogramming start there, with those methods,
explaining how they work, what they do. They start with the “what” of
metaprogramming.</p>
<p>But I don’t want to give that kind of talk today.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Metaprogramming as Fringe" src="/img/201808/slide-12.png" />
</div>
<h2>Metaprogramming as Fringe</h2>
<p>And this is the reason why. It’s that for most programmers, this is what
metaprogramming looks like. If they know about it at all, they just fence it
off and label it as dangerous, or magical, or more likely just not useful to
them.</p>
<p>But another part of the community, myself included — and I suspect many
people here today — finds these methods absolutely essential, actually
the core of Ruby.</p>
<p>And the reason for this divide is that most everything out there about
metaprogramming with this “what” focus does not answer the obvious question:
what on Earth is metaprogramming for?</p>
<p>And that’s the question I want to start with. To do that, I want to think
about the structure of our ecosystem.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Metaprogramming as Fringe" src="/img/201808/slide-14.png" />
</div>
<h2>Ruby</h2>
<p>So let’s start with a bit of fire down here, and put Ruby into the fire. Matz
is feeling the heat!</p>
<p>So down here, this our programming language, where it is being forged out of
fire and brimstone by Matz and his brave team of Ruby committers.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Applications" src="/img/201808/slide-15.png" />
</div>
<h2>Applications</h2>
<p>And then up here, we have where most of us spend our time: building
applications. This is what pays our bills, this is what delivers value to our
users.</p>
<p>And as users of the language, application developers know enough of the
language to use it, but most don’t see or care much about its internals.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Applications" src="/img/201808/slide-16.png" />
</div>
<h2>Libraries</h2>
<p>Now, bridging these two worlds, we have the libraries. Or “gems”, but I’ll use
the term “library” here.</p>
<p>Libraries fill the gaps in generic things we need - things that more than one
application need - that are not supplied by the programming language.</p>
<p>And in Ruby in particular, these libraries are almost seamless in how they
integrate with the programming language and with applications.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Metaprogramming lives here" src="/img/201808/slide-17.png" />
</div>
<h2>Metaprogramming Lives Here</h2>
<p>And here’s the thing, right? It’s that metaprogramming overwhelmingly lives in
this middle layer here, in the libraries.</p>
<p>Just taking Rails as an example, in Rails, it’s metaprogramming that is used
to define everything you care about: attribute methods, associations, all
model-specific stuff, view rendering logic, whatever.</p>
<p>But it’s not only ActiveRecord. Most of the widely-used gems we use heavily
leverage metaprogramming to do what they do.</p>
<p>But applications don’t use metaprogramming much at all.</p>
<p>And that brings us to an obvious question, right?</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Why?" src="/img/201808/slide-18.png" />
</div>
<h2>Why?</h2>
<p><em>Why?</em> — Why is metaprogramming so much more prevalent in our dependencies than in our
own application code?</p>
<p>This is such an important question. It’s telling us something very important
about the nature of our programming language, and the ecosystem of components
built out of it. But nobody is really asking this question, that I can see
anyway.</p>
<p>I think part of the reason is maybe because it seems kind of obvious maybe,
but I think it’s actually quite subtle. So let’s think about it a bit together
now.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Generalization"
src="/img/201808/slide-20.png" />
</div>
<h2>Generalization</h2>
<p>Here’s another word, “generalization”. It’s the root of the second big word in
the title of this talk.</p>
<p>Generalization is what libraries do. They generalize problems that many
different applications have into generic components.</p>
<p>Generalization is the “formulation of general concepts from specific
instances by abstracting common properties.” This is from Wikipedia again.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Abstracting Common Properties"
src="/img/201808/slide-21.png" />
</div>
<h2>Abstracting Common Properties</h2>
<p>This part, “abstracting common properties” from specific instances, is what we
are going to do in the second part of this talk.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="General Concepts" src="/img/201808/slide-22.png" />
</div>
<h2>General Concepts</h2>
<p>First, though, I want to focus on “general concepts”, and try to figure out
the relation of general concepts with metaprogramming.</p>
<p>What’s a general concept that is relevant to everyone here?</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Attribute" src="/img/201808/slide-23.png" />
</div>
<h2>Attribute</h2>
<p>Well, I’m going to wager that this one is pretty relevant to almost everyone
working with Ruby. The concept of “attribute”.</p>
<p>We want to figure out what role metaprogramming is playing in this kind of
general concept, because it’s this kind of general concept that libraries
implement for us.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Knockout Experiment" src="/img/201808/slide-24.png" />
</div>
<h2>Knockout Experiment</h2>
<p>So to do this we’re going to take a cue from genetics. What does a geneticist
do when they want to understand what a gene is doing?</p>
<p>Well, they knock it out. They just knock it out. And the they look to see
what’s different about the mouse with the knocked out gene, compared to the
mouse with the gene left in.</p>
<p>So we’re gong to do the same. We’ll knock out metaprogramming from the concept
of “attribute”, and see what happens.</p>
<hr />
<div class="img-center">
<img alt="Control: Library View" src="/img/201808/slide-25.png" />
</div>
<h2>Control: Library View</h2>
<p>So let’s start with the control. The control is the “normal” case, the one
with metaprogramming.</p>
<p>This is the library view on the control. In each case, control and knockout,
there’s a single class called “Model”, with a simple initializer that creates
an empty hash to store attributes.</p>
<p>For the control here, we then have a class method, define_attribute, which
takes an attribute name and defines getter and setter methods for that
attribute. The getter just grabs the value for the key matching the method
name, and the setter sets the value for the key matching the method name.</p>
<hr />
<div class="img-center">
<img alt="Control: Application View" src="/img/201808/slide-26.png" />
</div>
<h2>Control: Application View</h2>
<p>And now here we’re switching to the application view, which uses this library.</p>
<p>In the application we define a class <code>Talk</code> which subclasses <code>Model</code>. And we
call define_attribute twice to define two attributes, <code>title</code> and <code>abstract</code>.</p>
<p>And then here set and get the attributes like this, set the title and get the
title, and set the abstract and get the abstract.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Abstraction is Transparent"
src="/img/201808/slide-27.png" />
</div>
<p>The point I want to make here is that the abstraction, remember the
abstraction of “attribute”, is <em>transparent</em>, or you could say it’s
<em>invisible</em>. It’s visible when we declare it of course, but after that it’s
entirely transparent when we use it.</p>
<p>This is what we are used to and expect, but I want to highlight that.</p>
<hr />
<div class="img-center">
<img alt="Knockout: Library View"
src="/img/201808/slide-28.png" />
</div>
<h2>Knockout: Library View</h2>
<p>Ok now for the knockout. We start with the library view like we did for the
control.</p>
<p>In the knockout, we have no metaprogramming, so we can’t use define_method
like we did in the control. We can’t define methods dynamically like that.</p>
<p>So instead we define two instance methods, “get_attribute” and
“set_attribute”, which is the only other way to really do it. Those methods do
the same thing as the methods in the control, set and get attributes as keys
on the @attributes hash.</p>
<hr />
<div class="img-center">
<img alt="Knockout: Application View"
src="/img/201808/slide-29.png" />
</div>
<h2>Knockout: Application View</h2>
<p>And here is the application view for the knockout.</p>
<p>In this case we don’t need to declare the attributes because our
implementation will let you set and get anything you like (this is not really
important).</p>
<p>Now in this case, as I mentioned, when we want to get and set attributes, we
can’t just call “title” and “abstract” as methods because the library “Model”
can’t dynamically define those methods, since it has no metaprogramming to do
that.</p>
<p>So instead we call get_attribute and set_attribute to get and set attributes
(keys on the @attributes hash).</p>
<hr />
<div class="img-center">
<img alt="Abstraction is Visible"
src="/img/201808/slide-30.png" />
</div>
<h2>Abstraction is <em>Visible</em></h2>
<p>So I want to highlight two things here. The first is that <em>the abstraction is
now visible</em>. Remember that our abstraction is “attribute”, and in the control
case once declared, we didn’t need to think about it, we could just talk about
the talk’s title and abstract.</p>
<p>But here, it’s in our face. Every time you access an attribute, you need to
actually write the word “attribute”.</p>
<hr />
<div class="img-center">
<img alt="Names are Arguments"
src="/img/201808/slide-31.png" />
</div>
<h2>Names are Arguments</h2>
<p>And the other thing is that names are now arguments, in this case symbols, that
we pass to these methods. They are not methods now, and so we don’t send them
as messages to the objects but as arguments to methods on the object.</p>
<hr />
<div class="img-center">
<img alt="Results: Application View"
src="/img/201808/slide-33.png" />
</div>
<h2>Results: Application View</h2>
<p>Ok, so let’s tabulate the results of our experiment! We have two rows here,
the first for the view from the application, the second for the view from the
library. It’s really important to draw a distinction between these two.</p>
<p>And we have two columns, one for the control and one for the knockout. Recall
that the control is with metaprogramming, and the knockout is without
metaprogramming.</p>
<p>So from the application point of view, in the control the abstraction is
transparent, or invisible, and we use the abstraction, i.e. we use attributes,
by sending their names as messages to the object. And so we can say here that
the library speaks in the language of the domain, i.e. you send the message
“title” to the “talk” to get the title, nothing explicitly mentioning the
abstraction itself there.</p>
<p>In the knockout, on the other hand, the abstraction is very much visible,
remember we had those methods get_attribute and set_attribute. Also different:
you send the attributes as values to a method, remember they were the symbols
“title” and “abstract”.</p>
<p>So we can say here that <strong>the library speaks in the language of the
abstraction</strong>. You really can’t avoid it as an application developer.</p>
<hr />
<div class="img-center">
<img alt="Results: Library View"
src="/img/201808/slide-34.png" />
</div>
<h2>Results: Library View</h2>
<p>Now let’s consider the library that implements this Model class. What’s going
to be important here is the unknowns and what they refer to.</p>
<p>In the control, unknowns refer to code, to the attribute methods that we
define with “define_attribute”. They are unknown because while they could be
“title” and “abstract”, they could also be anything else. This makes library
code hard to understand. That’s not obvious here because it’s a trivial
example, but in the next section I’ll show how this can get more tricky.</p>
<p>In the knockout, the unknowns are the names of the attributes and their
values. Attribute names are not code, they’re just keys on a hash, and that
makes it easier to reason about than code with data that references code (like
the control).</p>
<p>So if that’s the case then, why as library authors don’t we implement our code
like the knockout?</p>
<hr />
<div class="img-center">
<img alt="Knockout ORM" src="/img/201808/slide-35.png" />
</div>
<h2>Knockout ORM</h2>
<p>Well, here’s the reason. Take the knockout example a bit further and imagine
we implement an ORM like this, with associations. What would it look like?</p>
<p>Well, like this. We now have two general concepts here, attribute and
association, but we can’t use metaprogramming for either of them, so the
association abstraction is now also in your face.</p>
<p>So although this type of explicit abstraction is easier to handle at the
library level, now it’s much more difficult to use at the application level.</p>
<hr />
<div class="img-center">
<img alt="Would you use this?" src="/img/201808/slide-36.png" />
</div>
<h2>Would you use this?</h2>
<p>And so the question is: as an application developer, would you use this?
Remember that at the application level, things like the name of the talk
attributes <code>talk</code> and <code>abstract</code> are known to you.</p>
<p>This means that if you wanted to, you could build AR like code even without
metaprogramming by defining these methods explicitly, with <code>def</code>. It might
take time, but it might also be better when it comes to actually using those
methods, because now your model would speak your domain language.</p>
<p>So the library would be at a disadvantage compared to the specific
application. This is the really key point here.</p>
<hr />
<div class="img-center">
<img alt="ORM with metaprogramming" src="/img/201808/slide-37.png" />
</div>
<h2>ORM with metaprogramming</h2>
<p>And now here’s what it looks like with metaprogramming. This looks more like
what we’re used to.</p>
<hr />
<div class="img-center">
<img alt="Metaprogramming is translating unknowns into code"
src="/img/201808/slide-38.png" />
</div>
<h2>Metaprogramming is Translating Unknowns into Code</h2>
<p>So here’s the takeaway: metaprogramming levels the playing field by enabling
libraries to translate unknowns into code. It levels the playing field with
the applications using the library.</p>
<p>And seeing things this way — in terms of unknowns — really reveals a lot.</p>
<p>Think about it for a second. What are the unknowns for the application?
Well, the classic one is user input, right? Do you want to translate user
input into code? Hell no!</p>
<p>And so this explains the divide in perceptions of metaprogramming. From the
library point of view, it’s a leveler. From the application point of view it’s
a freaking menace!</p>
<p>The other thing this new definition does is reveal some problems with our
previous working definition, the “what” definition. Let’s take a look at that
now.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="eval('foo')" src="/img/201808/slide-40.png" />
</div>
<h2>eval(“foo”)</h2>
<p>Here we have <code>eval("foo")</code>, about the simplest kind of metaprogramming you can
imagine, right?</p>
<p>Now, our working definition says anything with a metaprogramming method is
metaprogramming, so this is metaprogramming then, right? But something feels
wrong about this.</p>
<p>And the reason of course is that if we evaluate this, what do we get? We get
<code>foo</code>, a call to the method foo or to a variable by that name. And that’s not
terribly magical is it? We certainly aren’t “translating unknowns into code”.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="eval with string" src="/img/201808/slide-41.png" />
</div>
<h2>eval(“foo#{str}”)</h2>
<p>Ok, well what if we have a string, and we append “foo” to the string and eval
that? Well, we still know the result here, which in this case would be calling
foobar, whatever that is. So still not really interesting.</p>
<p>So how can we make this string unknown?</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="fooval" src="/img/201808/slide-42.png" />
</div>
<h2>fooval</h2>
<p>Well, wrap it in a method, call it <code>fooval</code>. Now by definition, it’s unknown,
we can’t know the value of the string here. So this is translating unknowns to
code. And it’s using <code>eval</code>, so in this case it lines up with both our
expectation and our definition.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="attr_writer" src="/img/201808/slide-43.png" />
</div>
<h2>attr_writer</h2>
<p>Ok, let’s take another case, <code>attr_writer</code>, with an argument “foo”.</p>
<p>This is pretty straightforward, right? <code>attr_writer</code> is definitely not
metaprogramming.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="attr_writer = def foo" src="/img/201808/slide-44.png" />
</div>
<h2>attr_writer = def foo</h2>
<p>And here’s what this evaluates to, defining a writer method <code>foo</code>. And that’s
not translating unknowns to code, so ok this one lines up with our
expectations.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="define_writer" src="/img/201808/slide-45.png" />
</div>
<h2>define_writer</h2>
<p>Ok, here’s another one. This is almost the same as the second half of the
<code>define_attribute</code> method I showed you earlier in the control (except that it
sets an instance variable instead of a hash value). This one uses
<code>define_method</code>, so in our working definition this is metaprogramming. It also
translates unknowns to code, i.e. translates <code>name</code> to a method, so this lines
up with our expectations too.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="define_writer aliasing attr_writer"
src="/img/201808/slide-46.png" />
</div>
<h2>define_writer aliasing attr_writer</h2>
<p>But wait a second, that stuff in the method body can be re-written to look
like this, right? It’s just aliasing <code>attr_writer</code>. And we said that
<code>attr_writer</code> is definitely not metaprogramming.</p>
<p>But doing things like this, wrapping <code>attr_writer</code> in a method, or really just
aliasing it, means that we’re translating unknowns to code. Because that’s
really what <code>attr_writer</code>, and <code>attr_reader</code> and <code>attr_accessor</code>, do for us.</p>
<p>We don’t usually think about it this way because in 99% of cases we’re passing
known values to these methods, but they can be used this way too.</p>
<p>So this is another case where our new definition does not line up with our
working definition.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="what definition as Venn diagrams"
src="/img/201808/slide-47.png" />
</div>
<h2>Data Points in Venn Diagram</h2>
<p>So let’s visualize these results as Venn diagrams. Here we have our working
definition, which I’ll call the “what” definition. And we have the control and
the knockout here, which seem fine.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="With additional data points"
src="/img/201808/slide-48.png" />
</div>
<h2>With additional data points</h2>
<p>Now here are the other data points. Remember that <code>eval("foo")</code> is in this set,
but it seemed like it didn’t belong. The second <code>define_writer</code>, which basically
aliased to <code>attr_writer</code>, seemed like it did belong there, but it’s outside
the set.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="New definition"
src="/img/201808/slide-49.png" />
</div>
<h2>New definition</h2>
<p>So what we’re going to do is to make a new definition, which will correspond
to “translating unknowns into code”. And this definition is going to exclude
the <code>eval("foo")</code> one, and include the second define_writer. It’s going to do
that because it uses a definition based on what things reduce to.</p>
<p><code>eval("foo")</code> reduces to what we call “normal programming”, i.e. a simple method
call. <code>attr_setter</code> wrapped in a method does not reduce to anything we’d
consider “normal programming”.</p>
<p>The point is that this new definition is about behaviour. It’s not about the
what, it’s about the <em>how</em>.</p>
<p>So what’s normal programming then? Well, what if we invert this set on the
right, and consider “normal programming” as the niche.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Normal Programming"
src="/img/201808/slide-50.png" />
</div>
<h2>“Normal” Programming</h2>
<p>So now what do we have. We have a set of methods and symbols and stuff that we
consider “normal”. This is most of what you write as an application developer.
Remember that this includes stuff that is normal and anything that reduces to
stuff that’s normal.</p>
<p>In this world, code is code and data is data. Code can reference data, of
course, but data never refers to code in this world. This is like the knockout
example, where the attribute names did not refer to method names.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Normal Programming as Niche" src="/img/201808/slide-51.png" />
</div>
<h2>Normal Programming as Niche</h2>
<p>And if we now zoom out and look at the big picture, this is what we have. It
is “normal programming” in this picture, not "metaprogramming”, that is the
niche. A very small niche in the much much larger space of computation, where
we don’t draw any distinction between code and data.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Generic Software" src="/img/201808/slide-52.png" />
</div>
<h2>Generic Software</h2>
<p>And so where do the libraries that use metaprogramming live? Well, something
like this, which I’ll refer to as “generic software”. They venture beyond the
bounds of “normal programming” to make generalizations. And to do that
effectively, they break that code/data barrier.</p>
<p>This way of understanding metaprogramming, as a means of generalizing from
unknowns, is much more meaningful than any other definition out there that I
have ever seen. And it really highlights what is so difficult, but also so
powerful, about this concept.</p>
<p>To show that in more detail now, let’s look at an example of this thing I’ve
referred to as “generic software”.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Building Generic Software" src="/img/201808/slide-53.png" />
</div>
<h2>Building Generic Software</h2>
<p>Ok, so “generic software”, what in the world is that, you say?</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Generic Software: Definition" src="/img/201808/slide-54.png" />
</div>
<h2>Generic Software: Definition</h2>
<p>The term actually comes from a talk by Jeremy Evans, who is one of my heroes
in Ruby. He gave a talk in 2012 about the development of Sequel, an ORM of
which he’s the author, in which <a href="http://code.jeremyevans.net/presentations/heroku201205/index.html#68">he
said</a>:</p>
<blockquote><p>One of the best ways to write flexible software is to write generic
software. Instead of designing a single API that completely handles a specific
case, you write multiple APIs that handle smaller, more generic parts of that
use case and then handling the entire case is just gluing those parts
together.</p></blockquote>
<p>And the key thing here is that the “glue”, in many cases and in particular in
cases where the software needs to speak the domain language, will be
metaprogramming, by the definition we came to in the last section.</p>
<p>So I want to explore this idea of “generic software”, but we need a general
concept. We’ve already used “attribute”, so next I’m going to use…</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Equality" src="/img/201808/slide-55.png" />
</div>
<h2>Equality</h2>
<p>Equality! Can’t get much more general than that, right?</p>
<hr />
<div class="img-center">
<img alt="Double Equals Equality" src="/img/201808/slide-56.png" />
</div>
<h2>Double Equals Equality</h2>
<p>So there are a few methods for equality in Ruby, we’re going to be focusing
here on “double-equals” equality. That’s the kind of equality that compares
object content, not object identity.</p>
<p>So e.g., two strings which both have the characters <code>f</code> <code>o</code> <code>o</code> are
double-equal equals even if they are different actual objects.</p>
<hr />
<div class="img-center">
<img alt="Finding a Use Case" src="/img/201808/slide-59.png" />
</div>
<h2>Finding a Use Case</h2>
<p>Ok but we need some specific examples, like in the definition of
generalization we saw earlier. So let’s go where everyone goes for examples,
to Rails! And grep for it.</p>
<p>And we actually find a bunch of files that mention this type of equality.</p>
<p>To start we’ll use some classes defined in tests, just because they’ll be a
bit easier to visualize, but we’ll come back afterward and apply it to models
in other files here.</p>
<hr />
<div class="img-center">
<img alt="Address" src="/img/201808/slide-60.png" />
</div>
<h2>Address</h2>
<p>So here’s the <code>Address</code> class, just has a few <code>attr_reader</code>s for the attributes
street, city and country, and an equality method that compares two addresses
based on the values of those attributes.</p>
<hr />
<div class="img-center">
<img alt="GpsLocation" src="/img/201808/slide-61.png" />
</div>
<h2>Address</h2>
<p>And we have this <code>GpsLocation</code> class, which is very similar. Here the methods
are not defined using <code>attr_reader</code>, but that doesn’t really matter for us. We
just care that there is a double-equals method and that it’s defined in terms
of a list of methods.</p>
<hr />
<div class="img-center">
<img alt="Address + GpsLocation (1)" src="/img/201808/slide-62.png" />
</div>
<h2>Address + GpsLocation (1)</h2>
<p>Ok, so let’s line them up together. We notice a small difference, which is
that <code>Address</code> has this <code>other.is_a?(self.class)</code> and <code>GpsLocation</code> does not, but
it’s not terribly important and could easily be added back, so we’ll just
remove that one to simplify things. It won’t change anything about what I’m
going to show you.</p>
<hr />
<div class="img-center">
<img alt="Address + GpsLocation (2)" src="/img/201808/slide-63.png" />
</div>
<h2>Address + GpsLocation (2)</h2>
<p>Ok so now we need to extract what is common about these two methods. We’ll
start by noticing that calling a method in Ruby is exactly the same thing as
calling send with the method name as an argument. So let’s do that instead.</p>
<hr />
<div class="img-center">
<img alt="Extract with send" src="/img/201808/slide-64.png" />
</div>
<h2>Extract with send</h2>
<p>So you can see, in the first step here we’ve extracted something that was code
- the method calls - to something that is data, the method names we’re passing
to <code>send</code>. That’s important.</p>
<hr />
<div class="img-center">
<img alt="Use all?" src="/img/201808/slide-65.png" />
</div>
<h2>Extract Array</h2>
<p>Now we’re going to refactor the sets of equality predicates using the
enumerable method <code>all?</code>. This method takes a block and passes each element to
the block, then takes the AND of the results.</p>
<p>In this case, the block will be the comparison of sending the attribute name
to self and to the other object, and this will give us the same result as
before.</p>
<p>(I’ve dropped <code>GpsLocation</code> here by the way since everything we’re doing now
will be exactly the same for both classes, except the names will be
different.)</p>
<hr />
<div class="img-center">
<img alt="Abstraction: key" src="/img/201808/slide-66.png" />
</div>
<h2>Abstraction: key</h2>
<p>Notice now that we have the first indication of our abstraction, the argument
to the block here, “key”. This is an abstraction for the method names that
define equality on each of these classes.</p>
<hr />
<div class="img-center">
<img alt="Keys" src="/img/201808/slide-67.png" />
</div>
<h2>Keys</h2>
<p>Ok now we’re going to name the array of keys as a variable “keys” and pull it
out. Again, nothing has changed here, we’re just moving stuff around.</p>
<hr />
<div class="img-center">
<img alt="Replace def with define_method" src="/img/201808/slide-68.png" />
</div>
<h2>Replace def with define_method</h2>
<p>Ok, so here’s an important step. We’re replacing <code>def</code> with <code>define_method</code>.</p>
<p>Remember that in our original working definition, this would mean that this
was now metaprogramming. But in fact, nothing has changed, not yet anyway.</p>
<p>With our new definition, this is not metaprogramming because we are not
translating any unknowns into code. We’re only translating knowns into code,
the array of keys in each case. And we know what those keys are.</p>
<p>So you can see that our new definition is giving us more useful information
about what is going on.</p>
<hr />
<div class="img-center">
<img alt="Hoist Keys" src="/img/201808/slide-69.png" />
</div>
<h2>Hoist Keys</h2>
<p>Ok, now we “hoist” the <code>keys</code> out of the <code>define_method</code> block. We can do that
because blocks have an open scope, so what’s in the block can see the keys
array, even if it’s outside of the block itself. (This is now a closure, if
that says anything to you.)</p>
<hr />
<div class="img-center">
<img alt="Method Arguments" src="/img/201808/slide-70.png" />
</div>
<h2>Method Arguments</h2>
<p>Now I want to look at these things differently. These keys we’re going to
think of as arguments to a method. In <code>GpsLocation</code> the keys would be latitude
and longitude, but for Address they are street, city, country.</p>
<hr />
<div class="img-center">
<img alt="Method Body" src="/img/201808/slide-71.png" />
</div>
<h2>Method Body</h2>
<p>And now look at this code here. This is now independent of the keys, i.e.
independent of what makes Address and GpsLocation different. So we can now
extract it.</p>
<p>But how? By wrapping it in a method, like we saw earlier.</p>
<hr />
<div class="img-center">
<img alt="Equalize" src="/img/201808/slide-72.png" />
</div>
<h2>equalize</h2>
<p>So we create a class method <code>equalize</code>, which takes arguments <code>keys</code>. The splat
just means we can pass arguments and they will be captured in an array.</p>
<p>The code inside equalize is exactly the same as what we had in the last slide.
But now since it’s in a method, we need to call it, which we do, just below.
We call it with street, city, country.</p>
<hr />
<div class="img-center">
<img alt="Equalizer" src="/img/201808/slide-73.png" />
</div>
<h2>Equalizer</h2>
<p>Ok, and so now we can finally extract the module that will be our gem code. We
extract that class method into a module, <code>Equalizer</code>, and we extend <code>Address</code> with
it and call <code>equalize</code> with the keys.</p>
<p>That module is now our gem. And notice our unknowns? They are the keys to the
equalize method.</p>
<hr />
<div class="img-center">
<img alt="require 'equalizer'" src="/img/201808/slide-74.png" />
</div>
<h2>require “equalizer”</h2>
<p>And now that it’s a gem, a library, we can just require it, like this.</p>
<p>And now things are starting to look pretty nice and clean, right? The Address
class is simpler than it was originally, but it’s very clear what is defining
equality on each of them.</p>
<p>This is the sign of a good abstraction: it captures the essence of a general
concept.</p>
<hr />
<div class="img-center">
<img alt="Equalizer in Big Picture" src="/img/201808/slide-75.png" />
</div>
<h2>Equalizer in Big Picture</h2>
<p>And now in that diagram I showed you earlier, here is where we are. We’ve
extracted the generic component from <code>Address</code> (and <code>GpsLocation</code> would be the
same), and this generic component, this generic software, now lives in this
larger space outside of what we call “normal programming”.</p>
<p><code>Equalizer</code> has data that refers to code, the keys that refer to method names
defining equality.</p>
<hr />
<div class="img-center">
<img alt="Refactor other Classes with Equalizer" src="/img/201808/slide-76.png" />
</div>
<h2>Refactor other Classes with Equalizer</h2>
<p>And now that we’ve done this, we can apply <code>Equalizer</code> to other classes. Here we
have <code>GpsLocation</code>, but also a bunch of other classes from that list I showed
you earlier. You can see we’ve hit on a very general pattern.</p>
<hr />
<div class="img-center">
<img alt="Equalizer Again" src="/img/201808/slide-77.png" />
</div>
<h2>Equalizer Again</h2>
<p>So here’s our <code>Equalizer</code> module again. This is a very compact little module,
but there are a few important things going on.</p>
<p>Look at how define_method captures the keys in a block. Note that you couldn’t
do this without <code>define_method</code>, because <code>def</code> has a closed scope, it doesn’t
accept a block. Likewise, we could not build this method without calling the
equalizer methods using <code>send</code>, because the <code>keys</code> here are unknown values.</p>
<p>Ruby provides methods like <code>define_method</code> and <code>send</code> for a very specific purpose,
but that purpose is not clear unless you are solving a generic problem like
this. Of course you could also use <code>eval</code> – <code>eval</code> is the “universal solvent”
that dissolves anything, but it is an ultimate fallback and rarely used in
practice.</p>
<hr />
<div class="img-center">
<img alt="Add inspect method" src="/img/201808/slide-78.png" />
</div>
<h2>Add inspect method</h2>
<p>And the beauty of this is that once we have these keys, and these methods for
dynamically defining methods and sending messages that Ruby provides, we can
take this further.</p>
<p>We can observe that whatever defines equality basically defines identity. So
we can use that fact to define an inspect method which shows the values of the
key methods.</p>
<p>And you can see that this is useful in the example here.</p>
<hr />
<div class="img-center">
<img alt="Add hash method" src="/img/201808/slide-79.png" />
</div>
<h2>Add hash method</h2>
<p>And we can go further building on the abstraction, and add a <code>hash</code> method. The
<code>hash</code> method is used to determine if two keys on a hash are the same or not, so
by defining this method we allow instances of classes that extend <code>Equalizer</code>
to be used as hash keys.</p>
<p>Here’s an example where <code>GpsLocation</code>s are used as a hash of number of times a
given location has been visited. We have two points, but they are the same
latitude/longitude, so only one key in the hash is updated.</p>
<hr />
<div class="img-center">
<img alt="Dry Equalizer" src="/img/201808/slide-80.png" />
</div>
<h2>Dry Equalizer</h2>
<p>Now, if all this seems useful to you, I have some good news! There’s a gem for
it, called <a href="https://github.com/dry-rb/dry-equalizer">Dry Equalizer</a>.</p>
<p>It’s one of the dry-rb gems, which are fantastic examples of generic software
by the way. It’s a very very small gem, just a bit more than what we have
here, done a little bit more cleverly, but basically the same idea.</p>
<hr />
<div class="img-center">
<img alt="Dry Equalizer in dry-rb gem network" src="/img/201808/slide-80.png" />
</div>
<h2>DryEqualizer in dry-rb gem network</h2>
<p>DryEqualizer may seem quite simple, but it plays a very important role in the
network of dry-rb gems, which are being more and more heavily used in many
applications. dry-equalizer is actually right at the core of this network,
because it is so simple and so generic.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Being a Generalist (1)"
src="/img/201808/slide-83.png" />
</div>
<h2>Being a Generalist (1)</h2>
<p>Ok, so just before I close here, I want to come back to the title of this
talk.</p>
<p>The title was “Metaprogramming for Generalists”, not “Metaprogramming and
Generalization”, or “Metaprogramming and Generic Software”.</p>
<p>And there’s a reason for that.</p>
<p>“Generalist” usually means someone who has broad knowledge across different
fields, and the ability to bridge ideas from various different areas.</p>
<p>And you can do that by jumping from one field to the other and learning
high-level things about them, like this.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Being a Generalist (2)"
src="/img/201808/slide-84.png" />
</div>
<h2>Being a Generalist (2)</h2>
<p>But there’s another way to think of the “generalist”, as someone who digs deep
down, to the general concepts that underlie all sorts of different problems.
Concepts like “attribute” and “equality”.</p>
<p>Once you find these core concepts, and capture them in an abstraction, like
the Equalizer module you saw, then you really grasp something very fundamental
about a whole class of problems at once.</p>
<p>And then you can build on the abstraction, and everything you build will
broaden the reach of the software.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Being a Generalist (3)"
src="/img/201808/slide-85.png" />
</div>
<h2>Being a Generalist (3)</h2>
<p>And as a community, we really need more of you to do this, to delve down to
this world.</p>
<p>Solving generic problems, and solving them well, benefits everyone at all
levels, by making every specific instance of that general problem easier to
solve.</p>
<p>This is the metaprogramming magic that makes our ecosystem so powerful. It’s
what makes Ruby the language it is.</p>
<hr />
<div class="img-center">
<img style="padding-bottom: 20px" alt="Thanks" src="/img/201808/slide-86.png" />
</div>
<h2>Thanks!</h2>
Arel with Wharelhttps://dejimata.com/2018/5/30/arel-with-wharel2018-05-302018-05-30Chris Salzberg<p>Object Relational Mappers (ORM) are complex things. As the term implies,
an ORM provides a mapping from objects in an object-oriented language like Ruby</p>
<p>Object Relational Mappers (ORM) are complex things. As the term implies,
an ORM provides a mapping from objects in an object-oriented language like Ruby
to a relational persistence layer. In theory, this makes it possible to
abstract away details of this layer behind objects; in practice, of course,
things are not so simple.</p>
<p>In this post, I’ll show you how I’ve leveraged a couple very simple features
of Ruby’s object model to simplify building queries in ActiveRecord, in the
form of a teeny tiny gem called <a href="https://github.com/shioyama/wharel">Wharel</a>.
Wharel allows you to create Arel predicates in blocks passed to ActiveRecord
query methods, without all the standard boilerplate.</p>
<p>Before I explain Wharel, though, let’s take a few steps back and consider the
problem this gem tries to solve.</p>
<h2>Querying with ActiveRecord</h2>
<p>Part of the challenge of building and extending an ORM is striking a good
balance between convenience and control. ActiveRecord offers you by default a
fairly minimal interface for making queries which (I suppose) satisfies about
90% of users' basic needs. That interface is made up of the handful of
methods like <code>where</code> and <code>not</code>, which allow you to create queries built up of
simple equality predicates, optionally negated:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.where(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>).where.not(<span style="color:#606">content</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">bar</span><span style="color:#710">"</span></span>)
<span style="color:#777">#=> SELECT "posts".* FROM "posts" WHERE "posts"."title" = 'foo' AND "posts"."content" != 'bar'</span>
</pre></div>
</div>
<p>ActiveRecord also supports <code>SELECT</code>, <code>HAVING</code>, <code>GROUP BY</code>, along with the
combinators <code>AND</code> and (recently) <code>OR</code>, as well as inner and outer <code>JOIN</code>, and a few
others I forget just now.</p>
<p>Which is all fine and good, and may be adequate to power queries in your little CRUD
app. But very quickly you will find that <code>where</code> and friends are not enough.
Suppose, for example, you want to search for partial matches on string-valued
data. A Rails developer would typically drop down to SQL to do this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.where(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title LIKE ?</span><span style="color:#710">"</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">%</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>params[<span style="color:#A60">:title</span>]<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%</span><span style="color:#710">"</span></span>)
<span style="color:#777">#=> SELECT `posts`.* FROM `posts` WHERE (title LIKE '%foo%')</span>
</pre></div>
</div>
<p>… where <code>params[:title]</code> is a title from user input, and the application is
using MySQL as its database.</p>
<p>In this case, we’re not really using the ORM much at all, other than to
sanitize the data (<code>params[:title]</code>) interpolated into the SQL string. We
can’t extend or re-use this query in other contexts. We can’t take it apart,
build it up, or compose it. We can’t use it with a different database which
uses a different name for <code>LIKE</code> (like PostgreSQL, which uses <code>ILIKE</code> for
case-insensitive matching).</p>
<p>In short, we can either use this SQL snippet as-is, or re-write it to be
something different.</p>
<h2>Enter Arel</h2>
<p>For one-off situations, this may well be enough. But for anything else,
ActiveRecord offers another layer in the form of Arel, a <a href="https://en.wikipedia.org/wiki/Relational_algebra">relational
algebra</a> for building
queries abstracted from the particulars of any particular database. If
ActiveRecord is a mapping between Ruby model objects and a relational
database, Arel is the fine-grained mapping between Ruby query objects and
SQL query fragments.</p>
<p>Let’s get concrete. Convert the <code>LIKE</code> query above to Arel and we get:</p>
<div class="CodeRay">
<div class="code"><pre>
posts = <span style="color:#036;font-weight:bold">Post</span>.arel_table
<span style="color:#036;font-weight:bold">Post</span>.where(posts[<span style="color:#A60">:title</span>].matches(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">%</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>params[<span style="color:#A60">:title</span>]<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%</span><span style="color:#710">"</span></span>))
<span style="color:#777">#=> SELECT `posts`.* FROM `posts` WHERE (`posts`.`title` LIKE '%foo%')</span>
</pre></div>
</div>
<p>Here, we needed to first call <code>Post.arel_table</code> to get an instance of
<code>Arel::Table</code> corresponding to the <code>posts</code> table for the model. We can then
call <code>posts[:title]</code> to get an arel node (actually, an <code>Arel::Attribute</code>) for
the <code>title</code> column.</p>
<p>We can then use this <code>title</code> node to build a query using the <code>matches</code> method,
which corresponds to <code>LIKE</code> in MySQL.</p>
<p>This may not seem like such a big win, since anyway you could have written the
query in raw SQL with less lines of code. But what if you wanted to re-use
that snippet in combination with another condition?</p>
<p>You could certainly do this with a new inline SQL, but a more scalable way is
to use Arel. For example:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ApplicationRecord</span>
scope <span style="color:#A60">:with_title</span>, ->(str) { where(matches_title(str)) }
scope <span style="color:#A60">:with_title_or_subtitle</span>, ->(str) { where(matches_title(str).or(matches_subtitle(str))) }
<span style="color:#080;font-weight:bold">class</span> << <span style="color:#B06;font-weight:bold">self</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">matches_title</span>(title)
arel_table[<span style="color:#A60">:title</span>].matches(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">%</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>title<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%</span><span style="color:#710">"</span></span>)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">matches_subtitle</span>(subtitle)
arel_table[<span style="color:#A60">:subtitle</span>].matches(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">%</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>subtitle<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%</span><span style="color:#710">"</span></span>)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>In this case, we’ve used the arel predicates returned by <code>matches_title</code> and
<code>matches_subtitle</code> to build a combined predicate using the pattern <code>a.or(b)</code>,
which generates SQL like:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#777"># Post.with_title_or_subtitle("foo")</span>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">posts</span><span style="color:#710">`</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">posts</span><span style="color:#710">`</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">posts</span><span style="color:#710">`</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">title</span><span style="color:#710">`</span></span> <span style="color:#080;font-weight:bold">LIKE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">%foo%</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">OR</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">posts</span><span style="color:#710">`</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">subtitle</span><span style="color:#710">`</span></span> <span style="color:#080;font-weight:bold">LIKE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">%foo%</span><span style="color:#710">'</span></span>)
</pre></div>
</div>
<p>As you can imagine, with more complex queries, using Arel building blocks can
be quite useful. But it should also be apparent that Arel’s “wordiness” can be
a real turn-off, and partly explains why Rails developers tend to prefer
one-off SQL fragments over Arel predicates.</p>
<h2>Where + Arel = Wharel</h2>
<p>Recently I’ve been looking a lot at the internals of
<a href="http://sequel.jeremyevans.net/">Sequel</a>, a Ruby ORM developed by <a href="https://github.com/jeremyevans">Jeremy
Evans</a> that is superficially similar in many ways to
ActiveRecord, but which has a vastly different internal implementation and
much broader feature set.</p>
<p>One of the things that fascinated me about
Sequel is its interface for building queries, which allows you to use blocks
in the form of so-called “<a href="http://sequel.jeremyevans.net/rdoc/files/doc/querying_rdoc.html#label-Virtual+Row+Blocks">virtual
rows</a>”,
like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Artist</span>.where{id > <span style="color:#00D">5</span>}
</pre></div>
</div>
<p>Here, we are querying on a column <code>id</code>. But notice that rather than passing a hash of
keys and values to <code>where</code>, we are instead passing a <em>block</em> which references
the <em>method</em> <code>id</code> and builds it into a predicate with <code>id > 5</code>.</p>
<p>This type of query would not be possible passing a hash to ActiveRecord’s
<code>where</code>, since a hash can only generate equality predicates. Instead, we would
need to build the appropriate predicate with Arel, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
artists = <span style="color:#036;font-weight:bold">Artist</span>.arel_table
<span style="color:#036;font-weight:bold">Artist</span>.where(artists[<span style="color:#A60">:id</span>].gt(<span style="color:#00D">5</span>))
</pre></div>
</div>
<p>For such a simple query, I find the ActiveRecord/Arel code to be much harder
to read. Unsurprisingly, others have felt the same way.</p>
<p>Many years ago, <a href="https://github.com/ernie">Ernie Miller</a> set out to improve ActiveRecord with an
interface similar to Sequel, in the form of a gem called
<a href="https://github.com/activerecord-hackery/squeel">Squeel</a>. Squeel was popular,
but ultimately the monkey-patching magic it required to bend ActiveRecord into
shape burnt out its maintainer, and the gem is now abandoned.
More recently, a gem named
<a href="https://github.com/rzane/baby_squeel">BabySqueel</a> has attempted to recreate some
of this interface without Squeel’s monkey-patching baggage.</p>
<p>While working on <a href="https://github.com/shioyama/mobility">Mobility</a>, I recently
had a need to incorporate Arel predicates with ActiveRecord relations in a way
that went beyond AR’s standard query method interface. I decided to implement
something like the block-based query interface from Sequel, but vastly
simplified.</p>
<p>In the process, it became apparent to me how <em>simple</em> it is to do this, so I
extracted the core logic into a separate gem I’ve called
<a href="https://github.com/shioyama/wharel">Wharel</a>, a portmanteau of “Where” and
“Arel”.</p>
<p>What does Wharel do? Well, add it to your Gemfile and you can simplify queries
like the one above to:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Artist</span>.where { id.gt(<span style="color:#00D">5</span>) }
</pre></div>
</div>
<p>Wharel also supports combining nodes from different models by optionally using
a block argument, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Artist</span>.joins(<span style="color:#A60">:posts</span>).where { |a| <span style="color:#036;font-weight:bold">Post</span>.where { |p| p.title.matches(a.name) } }
</pre></div>
</div>
<p>(This query matches artists who have at least one post with a title that
matches their name.)</p>
<p>As you can see, Wharel adds an optional block format like Sequel, within which
any attributes called as methods are converted to their corresponding Arel
node. In this case, <code>id</code> becomes the arel node <code>Artist.arel_table[:id]</code>.</p>
<p>That’s it. There’s no additional magic to support predicates like <code>id >
5</code>, which Sequel, Squeel and BabySqueel offer, because I wanted to keep Wharel
to an absolute minimum level of complexity.</p>
<p>And Wharel is indeed <em>very</em> simple: its core is a mere <em>31 lines</em> of code. How
can that be? Let’s have a look inside.</p>
<h2>Virtual Rows and BasicObjects</h2>
<p>Here is the <code>Wharel</code> module:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Wharel</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">QueryMethods</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">where</span>(opts = <span style="color:#A60">:chain</span>, *rest, &block)
block_given? ? <span style="color:#080;font-weight:bold">super</span>(<span style="color:#036;font-weight:bold">VirtualRow</span>.build_query(<span style="color:#069">self</span>, &block)) : <span style="color:#080;font-weight:bold">super</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">WhereChain</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">not</span>(*args, &block)
block_given? ? <span style="color:#080;font-weight:bold">super</span>(<span style="color:#036;font-weight:bold">VirtualRow</span>.build_query(<span style="color:#33B">@scope</span>, &block)) : <span style="color:#080;font-weight:bold">super</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">VirtualRow</span> < <span style="color:#036;font-weight:bold">BasicObject</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(klass)
<span style="color:#33B">@klass</span> = klass
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">method_missing</span>(m, *)
<span style="color:#33B">@klass</span>.column_names.include?(m.to_s) ? <span style="color:#33B">@klass</span>.arel_table[m] : <span style="color:#080;font-weight:bold">super</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">class</span> << <span style="color:#B06;font-weight:bold">self</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">build_query</span>(klass, &block)
row = new(klass)
query = block.arity.zero? ? row.instance_eval(&block) : block.call(row)
::<span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Relation</span> === query ? query.arel.constraints.inject(&<span style="color:#A60">:and</span>) : query
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>There is really not much here. The <code>QueryMethods</code> module is required to patch
ActiveRecord to handle blocks; nothing special there. What <em>is</em> special is the
<code>VirtualRow</code> class, which subclasses <code>BasicObject</code>.</p>
<p>Why <code>BasicObject</code>? You may have seen this esoteric, minimalist object somewhere in a
diagram of Ruby’s object model, and recall that every <code>Object</code> inherits
from it. Indeed, <code>BasicObject</code> is a <em>very</em> simple object with almost no
methods at all, and for most contexts not something you’d really want to lean
on too much.</p>
<p>But in this case, the fact that <code>BasicObject</code> is so minimal happens to be very
useful, because it allows us to create what’s called a <a href="https://www.sethvargo.com/the-cleanroom-pattern/">Clean
Room</a>. Take a look at
<code>build_query</code>, which you’ll see we call from <code>where</code> or <code>not</code> to convert a
block to an Arel predicate:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">VirtualRow</span>.build_query(<span style="color:#069">self</span>, &block)
</pre></div>
</div>
<p>Here, <code>self</code> is the class or relation (for <code>not</code> it is <code>@scope</code>, which is
basically the same), and <code>block</code> is the block passed to <code>where</code> or <code>not</code>.</p>
<p>In <code>build_query</code>, we first create an instance of <code>VirtualRow</code> with a reference
to the class (or relation) (this is the <code>new(klass)</code> line). We then check the
<code>arity</code> of the block (don’t be scared, “arity” is just another word for
“number of arguments”).</p>
<p>If the block has no arguments (<code>block.arity.zero?</code>), like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.where { title.eq(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>) }
</pre></div>
</div>
<p>… then we call <code>instance_eval</code> on the virtual row and pass it the block.</p>
<p>Now what happens?</p>
<p>Well, <code>instance_eval</code> is a bit like that movie, <a href="https://en.wikipedia.org/wiki/Being_John_Malkovich">Being John
Malkovich</a>, when you go
through the door and enter the mind of actor John Malkovich. Except in this
case, the door takes you to the mind of whatever object you’re calling
<code>instance_eval</code> on. Any method you call from this context is received by that
object, just like everything you see when you go though the door is what John
Malkovich is seeing.</p>
<p>Here, the object is the virtual row. And this is where the fact that
<code>VirtualRow</code> subclasses <code>BasicObject</code> comes in useful. Since <code>VirtualRow</code> has
virtually no methods (no pun intended), anything you call on it will fall through
to <code>method_missing</code>. So if you call <code>title</code>, <code>method_missing</code> will be called
with <code>:title</code> as its first argument.</p>
<p>What do we do then? Well, just define a <code>method_missing</code> to catch these method
calls and “route” them to the model’s Arel table, which Wharel does in three
lines of code:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">method_missing</span>(m, *)
<span style="color:#33B">@klass</span>.column_names.include?(m.to_s) ? <span style="color:#33B">@klass</span>.arel_table[m] : <span style="color:#080;font-weight:bold">super</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Here, <code>@klass</code> would be <code>Artist</code>, so we’re calling
<code>Artist.arel_table[:title]</code>, which will get us the Arel node for <code>title</code>.
Perfect! That’s exactly what we wanted.</p>
<p>Since this works for any attribute (as
long as the attribute name does not overlap with a method on <code>BasicObject</code>,
which is exceedingly unlikely), we can build up any arbitrary Arel predicate
in this block without ever having to explicitly call <code>arel_table</code> on any
model.</p>
<h2>That Object Model Thing</h2>
<p>Although this post has Arel in its title, the real point I’m trying to make
with Wharel is to show how some of Ruby’s seemingly esoteric features, like
<code>BasicObject</code> and <code>instance_eval</code>, can actually be very powerful. With only
thirty lines of code, we’ve created an interface that eliminates a huge amount
of Arel boilerplate code in any Rails app. I find this really remarkable.</p>
<p>Although I don’t think many Rubyists are very familiar with it, Ruby’s Object
Model is actually very powerful and I highly recommend digging deeper into its
details. Books like <a href="https://pragprog.com/book/ppmetr/metaprogramming-ruby">Metaprogramming
Ruby</a> by Paolo Perrotta
are a great place to start, but it’s also a good idea to just pry open your
favourite gem (like I did with Sequel) and see what it’s doing under the hood.
If it’s doing something powerful, it’s probably leveraging this model to do
it.</p>
JSONify your Ruby Translationshttps://dejimata.com/2018/3/20/jsonify-your-ruby-translations2018-03-202018-03-20Chris Salzberg<p><a href="https://en.wikipedia.org/wiki/JSON">JSON</a> is a
ubiquitous format for storing and
transmitting complex nested data, originally in Javascript but today</p>
<p><a href="https://en.wikipedia.org/wiki/JSON">JSON</a> is a
ubiquitous format for storing and
transmitting complex nested data, originally in Javascript but today
in every major programming language. JSON is also
increasingly used as a persisted data type, notably by
<a href="https://en.wikipedia.org/wiki/PostgreSQL">PostgreSQL</a> with its <a href="https://www.postgresql.org/docs/9.4/static/datatype-json.html">json and jsonb
types</a>, and
more recently <a href="https://dev.mysql.com/doc/refman/5.7/en/json.html">MySQL also supporting JSON
natively</a>. Ruby
ORM like Sequel and ActiveRecord<a href="#note-1"><sup>1</sup></a> offer interfaces for
querying on this kind of JSON-formatted data.</p>
<div class="img-center" style="max-width: 100%; margin: 40px 0 40px 0">
<img alt="JSON Translation Data" src="/img/json_locales.png"></a>
</div>
<p>Given its usefulness, I’ve naturally been interested in applying JSON to the
storage of i18n data, a topic which I have <a href="/2017/3/3/translating-with-mobility">written about
before</a>. As an arbitrary-depth hash
format, JSON is a good fit for this kind of data, which tends to assume a
similar nested key/value structure. With databases now providing ways to
search JSON data quickly, we’ve got everything we need for a powerful
translation storage and querying engine.</p>
<p>In this post I’ll introduce a new strategy for leveraging JSON data formats to
store translations in a single table column. The strategy is implemented in
<a href="https://github.com/shioyama/mobility">Mobility</a>, the i18n gem I’ve written
about here before, and is inspired by (among other things) a library named
<a href="https://github.com/belaustegui/trans">Trans</a> for Elixir.</p>
<p>Before jumping into this new JSON-based storage strategy, however, I’d like to
first take a step back and think about the broader “translation mapping”
problem, and how JSON fits into it.</p>
<h2>Translation Mapping</h2>
<p>When you think of translation as a Ruby developer, you may think of a YAML
file with lots of nested translation keys and their corresponding translated
values, popularized in Ruby by the <a href="https://github.com/svenfuchs/i18n">I18n</a>
gem. Or you may think of <a href="https://en.wikipedia.org/wiki/Gettext">gettext</a> and
.po (portable object) files. Or you may think of some other similar file-based
key/value storage.</p>
<p>A key property of these kinds of internationalization and localization systems
is that the context in which translations are <em>written</em> is fundamentally
different from the context in which translations are <em>used</em>. Your users,
for example, can’t edit the YAML i18n files for your application directly
through the application itself.</p>
<div class="img-center" style="max-width: 100%; margin: 40px 0 20px 0">
<img alt="JSON Translation Data" src="/img/yamlfile.png"></a>
<p class="caption">Some YAML translations</p>
</div>
<p>That may be obvious to you, but it’s important because I want to contrast this
role of i18n strings with another, fundamentally different role translation
can play in a running application. The key characteristic of translation in
this second role is that <em>reading and writing happen in the same context</em>.</p>
<p>An example of this second category would be a blog platform which allows
authors to translate their articles into different languages, and query for
articles in any language. Another example would be an <a href="https://www.doorkeeper.jp/">event
site</a> that allows its users to translate their
event listings. But it could also include any of <a href="https://github.com/shioyama/mobility#companies-using-mobility">many other
contexts</a> in
which translation data is dynamically added, updated, deleted, and queried
through the application itself.</p>
<div class="img-right" style="margin: 20px 0 20px 20px; overflow: hidden">
<img alt="Event Traslations on Doorkeeper.jp" src="/img/doorkeeper-translation.png"></a>
<p class="caption">Translation Interface on <a href="https://doorkeeper.jp/" target="_blank">Doorkeeper.jp</a></p>
</div>
<p>If ActiveRecord and Sequel are “Object Relational Mappers” (ORM), I like to
think of the libraries that help with this type of translation mapping as
<strong>Translation Relational Mappers</strong>. They act as a layer between the ORM (data)
and the interface for writing, reading and querying translations.</p>
<p>Mobility is unique among libraries I have seen (in Ruby and any other
language) in that it takes the “mapping” part of this problem very seriously.
While other libraries tend to support one storage strategy or another, Mobility
supports many storage strategies through one common interface.</p>
<p>Which brings us to the question: what <em>is</em> a “storage strategy”? An example
will help make things more concrete. In Ruby at least, there is one approach
that is more popular than any other, and it does not use JSON.</p>
<h2>Translations in Tables</h2>
<p><a href="/2013/10/11/hacking-globalize">Globalize</a> is the most well-known gem for
translating ActiveRecord i18n model data and has
popularized a particular approach to storing persisted translation data.</p>
<p>In this strategy, translations for a model are stored in a model-specific
table. If your model <code>Post</code> uses a table named <code>posts</code>, you would create a
translation table named <code>post_translations</code> with a <code>locale</code> string column and
other columns for each translated attribute. Then when you access
<code>post.title</code>, the gem fetches the translation for the model in the current
locale and returns the value of the <code>title</code> column on the translation.</p>
<p>Keeping all
translations for a given model in the same place has some
advantages. It means, among other things, that you can call
<code>post.translations.each</code> and cycle through each of your translations as a
separate record; <a href="http://www.spinacms.com/">SpinaCMS</a> does this <a href="https://github.com/SpinaCMS/Spina/blob/376e659a2fa76331821dfbb90814fdec79f07966/app/views/spina/sitemaps/show.xml.builder#L9-L15">when
generating its
sitemap</a>,
for example. Mobility supports this type of translation strategy with its <a href="https://github.com/shioyama/mobility/wiki/Table-Backend">Table
backend</a>,
which is what Spina uses.</p>
<p>Although it’s nice having translations in a separate table, there are also
some significant downsides. Querying in particular becomes much more
complicated since you need to constantly <code>JOIN</code> your translation table(s).</p>
<p>Here’s the kind of query you get with this strategy (taken from the <a href="https://github.com/shioyama/mobility/wiki/Table-Backend">Mobility
Wiki</a>):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">INNER</span> <span style="color:#080;font-weight:bold">JOIN</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span> <span style="color:#080;font-weight:bold">ON</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_id</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">id</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">locale</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">'</span></span>
</pre></div>
</div>
<p>That’s not too bad, but you can imagine that once you start adding complexity
you quickly hit edge cases which can push the limits of your ORM. This
has prompted many people, myself included, to consider other options for
storing translations. One of these is to store translations in JSON format.</p>
<h2>Enter the B</h2>
<p>A few years back, with the release of its version 9.4, PostgreSQL <a href="https://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.4">introduced a
new datatype called
JSONB</a>. The
“B” in “JSONB” signifies that JSON data are stored in this
column format not as a string, but as binary data. This allows for fast
lookups and more powerful search queries, through new operators like
“contains” (<code>@</code>) and “exists” (<code>?</code>) which apply to JSON objects and keys,
respectively. It also allows you to store a mix of integers, strings, arrays
and hashes in the same column. This is a considerable step up from PostgreSQL’s <a href="https://www.postgresql.org/docs/current/static/hstore.html">Hstore
format</a> which,
although also a key/value store, is limited to depth-1 string-valued data.</p>
<p>One of my original goals with Mobility was to support this type of innovative
format <em>alongside</em> other strategies like the table-based one described in the
last section. This, indeed, is Mobility’s killer feature:
just like how you can swap MySQL for PostgreSQL as your database and keep your
ActiveRecord code mostly unchanged, Mobility allows you to swap
storage backends while retaining the same interface for accessing and
querying translations.</p>
<p>The first implementation of JSON in Mobility was a backend called <code>Jsonb</code>.
Using this backend looks like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ApplicationRecord</span>
extend <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">backend</span>: <span style="color:#A60">:jsonb</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Given that you have a <code>title</code> column of <code>jsonb</code> type, Mobility will allow you
to call a virtual attribute named <code>title</code> and (if we’re in the English locale)
will return to you the value on the <code>en</code> key of the JSON hash.</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.create(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">"</span></span>)
post.title
<span style="color:#777">#=> "JSONify your Ruby Translations"</span>
<span style="color:#036;font-weight:bold">Mobility</span>.with_locale(<span style="color:#A60">:ja</span>) { post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Rubyの翻訳をJSON化しよう</span><span style="color:#710">"</span></span> }
post.save
post = <span style="color:#036;font-weight:bold">Post</span>.first
post[<span style="color:#A60">:title</span>]
<span style="color:#777">#=> {"en"=>"JSONify your Ruby Translations", "ja"=>"Rubyの翻訳をJSON化しよう"}</span>
</pre></div>
</div>
<p>It will also allow you to query on this attribute just like it was a table
column (via a scope named <code>i18n</code>):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.i18n.where(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">"</span></span>).to_sql
<span style="color:#777">#=> SELECT "posts".* FROM "posts" WHERE (("posts"."title" -> 'en') = "JSONify your Ruby Translations")</span>
</pre></div>
</div>
<p>So Mobility is mapping the query on <code>title</code> to a query format specific to
<code>jsonb</code>, using the <code>-></code> operator.<sup><a href="#note-2">2</a></sup></p>
<p>Mobility actually does some more subtle things here. If you pass an array as
the query value, for example, Mobility will treat this as a set of values to
match, just like you would with a normal table column. So the query:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.i18n.find_by(<span style="color:#606">title</span>: [<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">bar</span><span style="color:#710">"</span></span>])
</pre></div>
</div>
<p>would generate the SQL:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> ((<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span> -> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>) = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">"foo"</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">OR</span>
(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span> -> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>) = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">"bar"</span><span style="color:#710">'</span></span>)
</pre></div>
</div>
<p>With the Table backend mentioned earlier, the same code would generate a very different query:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">INNER</span> <span style="color:#080;font-weight:bold">JOIN</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span> <span style="color:#080;font-weight:bold">ON</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_id</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">id</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">locale</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">content</span><span style="color:#710">"</span></span> <span style="color:#080;font-weight:bold">IN</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">foo</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">bar</span><span style="color:#710">'</span></span>)
</pre></div>
</div>
<p>Whereas the Jsonb backend uses <code>OR</code>, the Table backend uses <code>IN</code>
to match translations that have a column value in <code>('foo', 'bar')</code>. As a
Mobility user you don’t need to really worry about these details; just write
your ActiveRecord (or Sequel) code as you would for an untranslated attribute and the query should
“just work”<sup>TM</sup>.</p>
<h2>Keeping ‘em together</h2>
<p>One of the nice things about supporting many storage strategies in a single
gem is that common implementation patterns can be leveraged to re-use code and
minimize duplication.
While there are <a href="https://github.com/cfabianski/hstore_translate">gems offering Hstore-based model
translation</a> and <a href="https://github.com/olegantonyan/translateable">gems
offering JSON-based model
translation</a>, the
implementation for each one is different in arbitrary ways
(<code>translates</code>/<code>translatable</code> etc). Rather than re-invent the wheel, Mobility uses shared modules for
<a href="https://github.com/shioyama/mobility/blob/aa52e89e50ffc0cb817db4ca9e4623ed69b6bb84/lib/mobility/backends/active_record/pg_hash.rb">accessing</a>
and
<a href="https://github.com/shioyama/mobility/blob/aa52e89e50ffc0cb817db4ca9e4623ed69b6bb84/lib/mobility/backends/active_record/pg_query_methods.rb">querying</a>
PostgreSQL data types to eliminate duplication.</p>
<p>Which brings us to the new storage strategy mentioned at the top of this
article. While playing around with <a href="https://elixir-lang.org/">Elixir</a>
recently, I came upon a package named
<a href="https://hex.pm/packages/trans">Trans</a> which uses a translation strategy I
hadn’t previously considered. Like the Jsonb backend mentioned in the last
section, this strategy uses a <code>json</code> or <code>jsonb</code> column in a PostgreSQL
database to store translations. However, rather than store each set of
attribute translations on its own column, Trans uses the fact that json
columns can be deeply nested to store all translation on a single column,
named <code>translations</code>. This would not be possible using Hstore since it is
a depth-1 data type.</p>
<p>An example of this is shown in the figure below.</p>
<div class="img-center" style="margin: 20px 0 40px 0; max-width: 100%;
overflow: hidden">
<img alt="JSON Translation Data" src="/img/json_translations.png" style="min-width: 650px"></a>
<p class="caption">Model Translations as a depth-2 JSON hash</p>
</div>
<p>This is an interesting storage format with some notable advantages:</p>
<ul>
<li>It <strong>minimizes the number of migrations</strong> needed to just one per translated model. Translated attributes can be added and removed from a model with only minimal code changes, since the database schema does not change when you add new keys to a json/jsonb hash.</li>
<li>It <strong>requires no complex JOIN queries</strong> since translations are all on the same table.</li>
<li>It allows <strong>cycling through model translations</strong> like with table translations (with <code>post.translations.each</code>), except without a <code>JOIN</code>.</li>
</ul>
<p>Borrowing the name from Trans, I added this strategy as a new backend named
“<a href="https://github.com/shioyama/mobility/wiki/Container-Backend">Container</a>” in
Mobility 0.4. In Mobility 0.5., I also added
support for <code>json</code>-type columns in addition to <code>jsonb</code> (there are subtle
differences between the two that are out of the scope of this post, but
in general you probably want to use <code>jsonb</code>).</p>
<p>Just like the examples with other backends mentioned earlier, Mobility thus
now supports both reading/writing <em>and</em> querying on translations stored in a
single <code>json</code> or <code>jsonb</code> column<sup><a href="#note-3">3</a></sup>. Just specify the
<code>container</code> backend in your model (or as a default in your Mobility
configuration):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ApplicationRecord</span>
extend <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">backend</span>: <span style="color:#A60">:container</span>
<span style="color:#777"># Or if you want to set a different column name:</span>
<span style="color:#777"># translates :title, backend: :container, column_name: :my_translations</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Mobility will generate the appropriate query to find translations on the
shared <code>translations</code> column. So:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.i18n.where(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">"</span></span>)
</pre></div>
</div>
<p>will generate the SQL:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> (((<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">translations</span><span style="color:#710">"</span></span> -> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>) -> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">title</span><span style="color:#710">'</span></span>) = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">"</span></span>)
</pre></div>
</div>
<p>You will notice that this query is very similar to the earlier one for
querying on a JSON/JSONB column for a single attribute. The difference is that
we have changed the clause:</p>
<pre>
"posts"."title" -> 'en'
</pre>
<p>to the clause:</p>
<pre>
("post"."translations" -> 'en') -> 'title'
</pre>
<p>Mobility leverages this similarity such that only small tweaks are necessary
to support querying on the Container backend. Indeed, although as of version
0.5 there are now <em>three</em> JSON-based backends in Mobility (<code>Json</code>, <code>Jsonb</code> and
<code>Container</code>), the actual amount of code is relatively small given how many
features are supported. This is exactly the benefits I was after when I
designed Mobility in the first place.</p>
<h2>JSONify it, even just a little bit</h2>
<p>The JSON format offers many advantages in storing translations, but of
course it’s not a silver bullet. Some downsides include:</p>
<ul>
<li>Storing all translations on the same column can stress the database, triggering space reallocation and possibly slow down performance.</li>
<li>Parsing translation data may be more complex that simply grabbing the value on a string or text column.</li>
<li>The range of query options available is limited by database support for the
column type (<code>json</code> or <code>jsonb</code>), which will generally lag query options for normal columns.</li>
<li>Only works on PostgreSQL currently (no Ruby ORM support yet for MySQL json columns).</li>
</ul>
<p>If you look around the nets, you can find <a href="https://elixirforum.com/t/trans-embedded-translations-for-elixir/840/18">a lot of
debate</a>
about the advantages and disadvantages of different storage strategies for
i18n data. Some argue for storing <a href="https://elixirforum.com/t/trans-embedded-translations-for-elixir/840/13">each translation on its own
column</a>.
Others argue for <a href="https://groups.google.com/forum/#!topic/sequel-talk/qPQTxFcikeI">storing translations on a separate
table</a>.
Others <a href="https://www.bounga.org/ruby/2017/11/15/translating-activerecord-data-using-json-datatype/">argue for the JSON data type strategies described
here</a>.</p>
<p>Well, Mobility supports <em>all these strategies</em> with the minimum code required
to do so. It does so, moreover, in such a way that you can mix or match
however you like. Want to give JSON a whirl on one attribute, but stick with
another gem for the other attributes? You can do that. Want to use
one backend for most attributes, but test a JSON backend on some other
attributes? You can do that too.</p>
<p>The gem’s flexibility is why I gave Mobility its name. So if you’re
considering storing translations in your application, I’d highly recommend
giving JSON a try, alongside other options. No single solution will work best
for everybody, but there are enough options to mix and match that something
out there should fit your needs.</p>
<p>And of course, if you find JSON doesn’t work for you, translation tables don’t
work for you, <a href="https://github.com/shioyama/mobility/wiki/Column-Backend">translatable
columns</a> don’t work for you, a <a href="https://github.com/shioyama/mobility/wiki/KeyValue-Backend">polymorphic key/value
store</a> doesn’t
work for you… well, you can always <a href="https://github.com/shioyama/mobility/wiki/Introduction-to-Mobility-Backends">roll your
own</a>.</p>
<p>If you do, let me know what you find! 😉</p>
<h4>Notes</h4>
<p><small><a name="note-1"><sup>1</sup></a> Actually, ActiveRecord’s support is
pretty dismal compared to Sequel.</small><br />
<small><a name="note-2"><sup>2</sup></a> Incidentally, this operator accepts
any JSON object, so you can translate and query translated hash data in the
same way, for example. See the <a
href="https://github.com/shioyama/mobility/wiki/Postgres-Backends-(Column-Attribute)"
target="_blank">Mobility
Wiki</a> for more details.</small><br />
<small><a name="note-3"><sup>3</sup></a> You don’t need to specify whether you
are using a JSON column or a JSONB column, Mobility will check the db schema
and figure it out based on the attribute name.<small></p>
Mobility 0.3: Ready for Prime Timehttps://dejimata.com/2017/12/5/mobility-0-3-ready-for-prime-time2017-12-052017-12-05Chris Salzberg<p>Just a little over six months ago, I released the first version of the
Ruby translation framework I’ve called
<a href="https://github.com/shioyama/mobility">Mobility</a>. With the third release of</p>
<p>Just a little over six months ago, I released the first version of the
Ruby translation framework I’ve called
<a href="https://github.com/shioyama/mobility">Mobility</a>. With the third release of
the gem – and a <a href="https://github.com/shioyama/mobility#companies-using-mobility">growing list of
companies</a>
using it to power their production applications – I’m happy to say that
<strong>Mobility is today more stable and reliable than ever before</strong>.</p>
<div class="img-center" style="margin: 60px 0 40px 0">
<img alt="Column Strategy" src="/img/rocket.png"></a>
<p class="caption">
And we have a liftoff<a href="#note-1"><sup>1</sup></a>
</p>
</div>
<p>As its name implies, <strong>Mobility is all about <a href="/2017/3/3/translating-with-mobility">giving you
choices</a></strong>. Whether you
store your model translations in a <a href="https://github.com/shioyama/mobility/wiki/Table-Backend">set of model-specific
tables</a>, in a <a href="https://github.com/shioyama/mobility/wiki/KeyValue-Backend">single
shared table</a>, as
<a href="https://github.com/shioyama/mobility/wiki/Column-Backend">columns on each model
table</a>, or as values
in a Postgres <a href="https://github.com/shioyama/mobility/wiki/Postgres-Backends">jsonb or hstore
column</a>,
Mobility has you covered. Whether you use
<a href="https://github.com/shioyama/mobility#fallbacks">fallbacks</a>, <a href="https://github.com/shioyama/mobility#dirty-tracking">dirty change
tracking</a>, <a href="https://github.com/shioyama/mobility#querying">query
support</a>, or any of its other
features, Mobility will do what you need it to do without getting in the way.</p>
<p>This week, I’ve released <a href="https://rubygems.org/gems/mobility">Mobility 0.3</a>,
and in the spirit of its name, this release (among other things) will keep you
up to speed with your gem dependencies.
Version 0.3 includes full support
for the latest versions of its ORM dependencies: Rails/ActiveRecord 5.2
(literally just released in beta) as well as
<a href="http://sequel.jeremyevans.net/">Sequel</a> 5.0.</p>
<p>This release also comes with some <a href="https://github.com/shioyama/mobility/commits?author=pwim">important
changes</a> by a new contributor: Paul
McMahon (<a href="https://github.com/pwim">@pwim</a>), who has been also helping out
commenting on and reviewing PRs. Having someone with Paul’s skill and experience
helping out has been a huge boost; the fact that Paul is using
Mobility in production on his own service,
<a href="https://www.doorkeeper.jp/">Doorkeeper</a>, has also been great feedback for
identifying important issues and fixing them quickly.</p>
<p>Paul and I have put together a short wiki page on <a href="https://github.com/shioyama/mobility/wiki/Migrating-from-Globalize">migrating from Globalize to
Mobility</a>,
which is what he did with Doorkeeper; be sure to check it out if you are using
Globalize and considering switching to Mobility. For most users, Mobility will
be a nearly 100% drop-in replacement for Globalize, and the migration process
should be quite simple (as it was for Paul). However, for anyone with
questions, I have setup a <a href="https://gitter.im/mobility-ruby/mobility">gitter
community</a> for discussions, and I
also am watching for StackOverflow questions with the
<a href="https://stackoverflow.com/questions/tagged/mobility">mobility tag</a>.</p>
<p>So, without further ado, here are the changes…</p>
<h2>Support for Rails 5.2 and Sequel 5</h2>
<p>Yes, Rails 5.2.beta.2 was <a href="https://weblog.rubyonrails.org/2017/11/27/Rails-5-2-Active-Storage-Redis-Cache-Store-HTTP2-Early-Hints-Credentials/">just released this
week</a>,
and we’re <a href="https://github.com/shioyama/mobility/pull/114">already running tests
against it</a>. After a few small
<a href="https://github.com/shioyama/mobility/pull/116">compatibility changes</a>, those
specs are now all passing and we are ready for ActiveRecord 5.2!</p>
<p>Sequel 5.0 was <a href="https://github.com/jeremyevans/sequel/blob/master/doc/release_notes/5.0.0.txt">also recently
released</a>, and with a <a href="https://github.com/shioyama/mobility/pull/121">small
update to Mobility</a> released in
<a href="https://rubygems.org/gems/mobility/versions/0.3.2">0.3.2</a>, all features of
the Sequel backends are now working with this version as well.</p>
<h2>Support new ActiveRecord Dirty methods</h2>
<p>In Rails 5.1, <a href="https://github.com/rails/rails/pull/25337">new dirty tracking methods were
added</a> to differentiate between
changes that have been persisted (saved) to the database and those that have
only been changed in-memory (<em>set</em> but not yet <em>saved</em>).</p>
<p>There is a new method <code>saved_changes</code> to return changes to attributes that
have been saved, as well as attribute-specific methods (for an attribute <code>title</code>):</p>
<ul>
<li><code>saved_change_to_title?</code></li>
<li><code>saved_change_to_title</code></li>
<li><code>title_before_last_save</code></li>
<li><code>will_save_change_to_title?</code></li>
<li><code>title_change_to_be_saved</code></li>
<li><code>title_in_database</code></li>
</ul>
<p>Support for these methods <a href="https://github.com/shioyama/mobility/pull/111">has now been
added</a>. Just enable the <code>dirty</code>
plugin and the methods will be available for translated attributes on your
model.</p>
<h2>Fallbacks when Locale is Specified</h2>
<p>Paul <a href="https://github.com/shioyama/mobility/issues/85">noticed</a> that the way
locale accessors and fallbacks interacted was not very intuitive in version 0.2, and
did not correspond to how Globalize works (and to users' typical expectations).</p>
<p>Here’s an example where we are setting the title in English, then calling <code>title_ja</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Event</span> < <span style="color:#036;font-weight:bold">ApplicationRecord</span>
extend <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">fallbacks</span>: { <span style="color:#606">en</span>: <span style="color:#A60">:ja</span>, <span style="color:#606">ja</span>: <span style="color:#A60">:en</span> }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#036;font-weight:bold">Event</span>.new(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>).title_ja
<span style="color:#777">#=> "foo" instead of nil</span>
</pre></div>
</div>
<p>Although fallbacks are enabled from Japanese to English, since we are
explicitly specifying the locale we want (Japanese) in <code>title_ja</code>, one
would reasonably expect the fallbacks to not take effect, but in fact they do.</p>
<p>This has been fixed in 0.3, so that you get the value in the locale you
specified, without falling back to other locales:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Event</span>.new(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>).title_ja
<span style="color:#777">#=> nil</span>
</pre></div>
</div>
<p>This also applies when you fetch an attribute in a locale using the getter
options hash:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Event</span>.new(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>).title(<span style="color:#606">locale</span>: <span style="color:#A60">:ja</span>)
<span style="color:#777">#=> nil</span>
</pre></div>
</div>
<p>This is a really nice fix since as a byproduct it <a href="https://github.com/shioyama/mobility/pull/86/files#r137799779">also makes the dirty plugin
work more naturally with
fallbacks</a>
(showing changes from the previous value <em>without</em> fallbacks).</p>
<h2>Duplicate Translations when Duplicating Model</h2>
<p>Paul noticed that the ActiveRecord Table backend does not duplicate
translations when <code>dup</code> is called on the model. This is different from
Globalize, which duplicates the translations <a href="https://github.com/globalize/globalize/blob/ef1e362f3144b2276d89f47407c67ffa3fc1f9f2/lib/globalize/active_record/instance_methods.rb#L98-L105">as
well</a>.</p>
<p>After some discussion, Paul went ahead and <a href="https://github.com/shioyama/mobility/pull/84">implemented this
feature</a> by patching
<code>initialize_dup</code>. So as of version 0.3, if you <code>dup</code> your translated model
instance, you’ll get a copy both of the model and any translations.</p>
<p>Most other backends work without any such patch, since they map to columns on
the model itself. However, Paul noticed that the KeyValue backends (which like
the Table backend store translations in associations) were affected by the
same issue. I have since followed-up and <a href="https://github.com/shioyama/mobility/pull/126">also fixed this
issue for the AR KeyValue
backend</a>.</p>
<p>I have just released this fix in
<a href="https://rubygems.org/gems/mobility/versions/0.3.3">0.3.3</a>, so if you update
to the latest release, you should have no dup-ing issues with ActiveRecord
models for any backend. Sequel handles duplicating slightly differently, so
will require more attention; a fix should be out in the next minor release.</p>
<h2>Invalidate Cache Key</h2>
<p>This is <a href="https://github.com/shioyama/mobility/pull/104">another one by Paul</a>.
Up to version 0.2, Mobility would not “touch” the associated translations for
the Table and KeyValue backends. This results in problems with caching, since
ActiveRecord doesn’t know that the attributes have changed.</p>
<p>The fix we decided on for this issue is simply to add a <code>touch: true</code> to the
association defined for translations on the model. With this change, updating
a translation will update the <code>updated_at</code> timestamp on the model. Unlike
Globalize, which includes this setting as an option, as of version 0.3,
Mobility sets <code>touch: true</code> by default, since there is not conceivable reason
we could see that you would <em>not</em> want this enabled.</p>
<h2>Convert AttributeMethods to a Plugin</h2>
<p>One of the tricky parts of building a gem like Mobility is deciding how far to
go in overriding core methods of the ORM.</p>
<p>There is a fine line you need to draw. Naturally, you want to make translated
attributes behave just like normal attributes, since this is natural for the
user and means they don’t need to think too much about them.</p>
<p>However, if you go to far with this, then the ORM (typically
ActiveRecord) gets confused and thinks they are actually table columns, with
unpredictable consequences. This has happened <a href="https://github.com/shioyama/mobility/issues/97">with
Mobility</a> and happens
frequently with Globalize, which does much more patching. The results can be
very hard to debug.</p>
<p>One of the ways that Mobility patched ActiveRecord previous to version 0.3 was
to add translated attributes to the <code>attributes</code> hash:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#777"># Version < 0.3</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ApplicationRecord</span>
extend <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>
<span style="color:#080;font-weight:bold">end</span>
post = <span style="color:#036;font-weight:bold">Post</span>.new
post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>
post.attributes
<span style="color:#777">#=> {"id"=>nil, "created_at"=>nil, "updated_at"=>nil, "title"=>"foo"}</span>
</pre></div>
</div>
<p>Notice that the last key in the hash returned by <code>post.attributes</code> is <code>title</code>,
although <code>title</code> is not actually a table column.</p>
<p>This may seem nice, but in practice it can cause problems both with
ActiveRecord itself as well as with other gems. Since <code>title</code>
is a key in this hash, a gem might reasonably assume that you can call
<code>read_attirbute("title")</code> and get the value of the title, but Mobility (unlike
Globalize) does not patch <code>read_attribute</code> to do this, so this fails.</p>
<p>To avoid compatibility issues, this feature of overriding <code>attributes</code> has now
been extracted into a plugin, which by default is <em>off</em>, so the instance and
model above would not by include <code>title</code> as a key in <code>attributes</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
post.attributes
<span style="color:#777">#=> {"id"=>nil, "created_at"=>nil, "updated_at"=>nil}</span>
</pre></div>
</div>
<p>You can enable it by passing <code>attribute_methods: true</code> to <code>translates</code> (or
enabling it by default in the <code>default_options</code> configuration (see below).</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#777"># Version 0.3, with attribute_methods plugin enabled</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ApplicationRecord</span>
extend <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">attribute_methods</span>: <span style="color:#069">true</span>
<span style="color:#080;font-weight:bold">end</span>
post = <span style="color:#036;font-weight:bold">Post</span>.new
post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>
post.attributes
<span style="color:#777">#=> {"id"=>nil, "created_at"=>nil, "updated_at"=>nil, "title"=>"foo"}</span>
</pre></div>
</div>
<p>Although you can do this with the plugin, in general unless you know you need
it, my recommendation would be not to enable attribute methods since it can
cause conflicts with other gems. For example, see <a href="https://github.com/shioyama/mobility/issues/97">this
issue</a> about a
compatibility conflict with the <a href="https://github.com/rmm5t/strip_attributes">strip_attributes
gem</a>.</p>
<h2>Deprecate Default Options Setter</h2>
<p>The last change is a deprecation, not always the most exciting thing for users
of a gem. However, this is an important one. Reasons for the change are
described in <a href="https://github.com/shioyama/mobility/issues/92">this issue</a>.</p>
<p>In a nutshell, versions of Mobility prior to 0.3 would allow you to set a hash
of default options, which would be merged with whatever options you pass to
<code>translates</code> from any model:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.configure <span style="color:#080;font-weight:bold">do</span> |config|
config.default_options = {
<span style="color:#606">dirty</span>: <span style="color:#069">true</span>,
<span style="color:#606">fallbacks</span>: <span style="color:#069">true</span>,
<span style="color:#777"># ...</span>
}
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>The problem with this is that, by directly setting the defaults, you
<em>override</em> the “default defaults”, as it were. These defaults can be found
<a href="https://github.com/shioyama/mobility/blob/6e9c8168b324c4ca12bc63b8a8af53c17c31e02c/lib/mobility/configuration.rb#L76-L83">here</a>,
and you can see some plugin settings that might not be familiar.</p>
<p>One of these is for a <a href="http://www.rubydoc.info/gems/mobility/Mobility/Plugins/Presence">“presence”
plugin</a> which
is on by default. This plugin simply filters any values set or fetched from
the backend to filter out blank strings. I noticed that some users had blanks
in their data, which should not happen if this plugin is enabled; it turns
out, the reason is because they had set the <code>default_options</code> hash directly,
without passing any value for the <code>presence</code> key, thus implicitly turning it
off.</p>
<p>To avoid this from happening, <code>default_options=</code> (the setter) has been
deprecated in version 0.3. If you are currently setting it, you will see a
warning like this:</p>
<pre>
WARNING: The default_options= setter has been deprecated.
Set each option on the default_options hash instead, like this:
config.default_options[:fallbacks] = { ... }
config.default_options[:dirty] = true
</pre>
<p>This setter method will be removed in the next minor release. To get rid of
the deprecation warning, set defaults for each key on the <code>default_options</code>
hash, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.configure <span style="color:#080;font-weight:bold">do</span> |config|
config.default_options[<span style="color:#A60">:dirty</span>] = <span style="color:#069">true</span>
config.default_options[<span style="color:#A60">:fallbacks</span>] = <span style="color:#069">true</span>
<span style="color:#777"># ...</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Although this requires a few more keystrokes, it avoids the problem of
overriding the “default defaults” stored initially in
<code>config.default_options</code>. I think this is a saner way to set defaults,
ensuring that you only opt-out of important plugins if you explicitly set the
default.</p>
<h2>Looking Ahead: A Christmas Release</h2>
<p>The next release of Mobility will most likely be for the first major version
(1.0.0), which I’m aiming to finish in time for Christmas. There will be a
couple of breaking changes in this release, mostly just method renaming which
should only affect some users and will be easy to fix.</p>
<p>As mentioned above, I’m eager to get any feedback and suggestions. Please join
the <a href="https://gitter.im/mobility-ruby/mobility">gitter</a> community or post your
questions to StackOverflow. If you’re building a gem which depends on
translations and are considering using Mobility, please contact me since this
is a use case I’m particularly interested in.</p>
<p>Thanks for everyone’s support, and stay tuned for the next release!</p>
<h3>Notes</h3>
<p><small><a name="note-1"><sup>1</sup></a> NASA Orbital ATK CRS-8 Launch
(<a href="https://www.flickr.com/photos/nasahqphoto/37648937724">ref</a>)</small><br></p>
Mobility 0.2: Now with Pluginshttps://dejimata.com/2017/8/13/mobility-0-2-now-with-plugins2017-08-132017-08-13Chris Salzberg<p>It’s been a little while since last I
<a href="/2017/3/3/translating-with-mobility">posted</a> about
<a href="https://github.com/shioyama/mobility">Mobility</a>, the pluggable translation
framework for Ruby that I’ve been working on recently. I thought I’d take this
opportunity with the release of the second version of the gem to
highlight some of the important changes and new features, and to recap how far
things have come.</p>
<h2>Mobility and the Module Builder Pattern</h2>
<p>In the time since I posted <a href="/2017/3/3/translating-with-mobility">Translating with
Mobility</a>, one of the
techniques I have used heavily in building the gem – the <a href="/2017/5/20/the-ruby-module-builder-pattern">Module
Builder
Pattern</a>, as
I’ve termed it – has become a <a href="https://news.ycombinator.com/item?id=14386723">hot topic of its
own</a>. (I’ll actually be <a href="http://rubykaigi.org/2017/presentations/shioyama.html">talking
about module builders</a>
at the coming RubyKaigi in Hiroshima, check it out if you’ll be attending.)</p>
<p>In a nutshell, the Module Builder Pattern is a simple way to create
customizable modules to mix into classes, one which falls naturally out of
Ruby’s object model. It amounts to simply subclassing the <code>Module</code> class,
defining module methods in an initializer or <code>included</code> hook, and including
instances of the subclass in other classes. See the slide below from my recent talk
at the <a href="https://trbmeetup.doorkeeper.jp/events/63110">Tokyo Rubyist Meetup</a>,
where I explain the basic elements of a module builder and how I use it in a
core Mobility class, <code>Mobility::Attributes</code>.</p>
<div class="img-center" style="margin: 40px">
<img alt="Elements of a Module Builder" src="/img/tokyorubyist-modulebuilder.png"/>
</div>
<p>From its first release, Mobility has used this pattern all over the place; I
could never have achieved the flexibility the gem needed without it. However,
in the latest release, I’ve pushed this yet further in order to
achieve something just as important: <em>extensibility</em>.</p>
<h2>Now with Plugins</h2>
<p>Let’s recall that in Mobility, you define a translated attribute on a model by
calling <code>translates</code> with one or more attribute names, and a hash to configure
which options are applied:
<a href="https://github.com/shioyama/mobility#dirty-tracking">dirty-tracking</a>,
<a href="https://github.com/shioyama/mobility">fallbacks</a>, <a href="https://github.com/shioyama/mobility#getting-and-setting-translations">locale accessors</a>,
and so on. Under
the hood, Mobility creates an instance of the
<a href="https://github.com/shioyama/mobility/blob/93af67072bb69b10818923a03ca73aa7b8eb0cc2/lib/mobility/attributes.rb#L100">Mobility::Attributes</a>
module builder and includes it in the class. I’ll use this form directly here
to emphasize what is actually going on:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span>
extend <span style="color:#036;font-weight:bold">Mobility</span>
include <span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Attributes</span>.new(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">content</span><span style="color:#710">"</span></span>, <span style="color:#606">locale_accessors</span>: [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>], <span style="color:#606">fallbacks</span>: <span style="color:#069">true</span>)
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Here, we are building a module instance of <code>Mobility::Attributes</code> for the
attributes <code>title</code> and <code>content</code>, with locale accessors for English and
Japanese and fallbacks enabled. (The format of arguments to this initializer
have changed slightly since version 0.1, see this <a href="https://github.com/shioyama/mobility/pull/72">pull
request</a> for details).</p>
<p>What happens when we include the instance of this class into <code>Post</code>? Well,
here’s the first change with version 0.2: I’ve made the magic here a
whole lot clearer, and more extensible as a result (see PRs <a href="https://github.com/shioyama/mobility/pull/50">#50</a> and
<a href="https://github.com/shioyama/mobility/pull/64">#62</a>).</p>
<p>Previously there was a lot of hard-coded references to options
directly in <code>Attributes</code>, but the code is now entirely free of such
references. In fact, there are just a few lines in the builder which deal with
options:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.plugins.each <span style="color:#080;font-weight:bold">do</span> |name|
plugin = get_plugin_class(name)
plugin.apply(<span style="color:#069">self</span>, options[name])
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Here, <code>Mobility.plugins</code> is an alias for <code>Mobility.config.plugins</code>, a new
configuration setting which defaults to <a href="https://github.com/shioyama/mobility/blob/e1f2987d67dd113b8c69dc82d80fb55ec32807f4/lib/mobility/configuration.rb#L73-L81">this
array</a>
(order of elements is important here<a href="#note-1"><sup>1</sup></a>):</p>
<div class="CodeRay">
<div class="code"><pre>
[<span style="color:#A60">:cache</span>, <span style="color:#A60">:dirty</span>, <span style="color:#A60">:fallbacks</span>, <span style="color:#A60">:presence</span>, <span style="color:#A60">:default</span>, <span style="color:#A60">:fallthrough_accessors</span>, <span style="color:#A60">:locale_accessors</span>]
</pre></div>
</div>
<p>The private <code>get_plugin_class</code> method fetches constants under
<code>Mobility::Plugins</code> matching CamelCase-ed values of the plugin names from this
array, then applies them with the attributes instance and option values as its arguments.</p>
<p>So <code>locale_accessors: [:en, :ja]</code> will fetch
<code>Mobility::Plugins::LocaleAccesors</code> and call it like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Plugins</span>::<span style="color:#036;font-weight:bold">LocaleAccessors</span>.apply(<span style="color:#069">self</span>, [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>])
</pre></div>
</div>
<p>…where <code>self</code> is the instance of <code>Mobility::Attributes</code> being included into
the model class, <code>Post</code>.</p>
<p>Notice that since we have no hard-coded references to option keys in this
module now, <em>any</em> option key included in <code>Mobility.plugins</code> will
apply a plugin with that name (as long as that plugin exists), with the
attributes instance and option value passed into the plugin’s <code>apply</code> class
method.</p>
<p>This makes is very easy to design new extensions: all they need is to
have a class method, <code>apply</code>, which
takes this <code>Attributes</code> instance (which itself has references to the
backend class and attribute names), and an option value to
configure how the plugin should be applied. (Incidentally, I borrow the term
“plugins” here, and the method name <code>apply</code>, from
<a href="http://sequel.jeremyevans.net/">Sequel</a>, where plugins are defined in a
somewhat similar way.)</p>
<p>Below is another slide from my recent talk, illustrating how these options
trigger inclusion of yet more module builder instances into the backend class
and into the Attributes instance itself.</p>
<div class="img-center" style="margin: 40px">
<img alt="Mobility Plugins" src="/img/tokyorubyist-plugins.png" />
</div>
<p>Along with this introduction of plugins and the <code>Mobility::Plugins</code> namespace,
I’ve also reorganized the backend namespace such that <code>Mobility::Backends</code>
(plural) is now exclusively used for backend classes.</p>
<p>The result is that it becomes very
straightforward to build custom backends and plugins: define a class in the
appropriate namespace with the required structure, and call <code>translates</code> (or
include an instance of <code>Mobility::Attributes</code>) in the model class, with any
backend or option keys, and that’s it! You ready to go.</p>
<h2>Default Options</h2>
<p>Another minor annoyance some users of Mobility may have encountered in 0.1 is
that while it was possible to set a default backend to use across all models
(with the <code>default_backend</code> configuration option), there was no way to define
a default set of options (to configure plugins and the backend). Along with
the changes to enable plugins above, I also <a href="https://github.com/shioyama/mobility/pull/50">added this as a
configuration setting</a>, with a
value that defaults to:</p>
<div class="CodeRay">
<div class="code"><pre>
{ <span style="color:#606">cache</span>: <span style="color:#069">true</span>, <span style="color:#606">dirty</span>: <span style="color:#069">false</span>, <span style="color:#606">fallbacks</span>: <span style="color:#069">nil</span>, <span style="color:#606">presence</span>: <span style="color:#069">true</span>, <span style="color:#606">default</span>: <span style="color:#069">nil</span> }
</pre></div>
</div>
<p>These default option values were previously hard-coded, but are
now explicitly set and isolated in the configuration instance, which makes
them much easier to reason about and change.</p>
<p>Want to enable fallbacks by default on all your models? Just override the
default for that key:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.default_options[<span style="color:#A60">:fallbacks</span>] = <span style="color:#069">true</span>
</pre></div>
</div>
<p>You can also set backend options here, so for example if you want to set the
default <code>type</code> for the KeyValue backend to be “string” instead of “text”, you
could do this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.default_options[<span style="color:#A60">:type</span>] = <span style="color:#A60">:string</span>
</pre></div>
</div>
<p>Note that <em>how</em> these option values are interpreted by actual plugins and
backends depends on the plugin or backend, and there is some unavoidable
variation there. But the values passed in to <code>apply</code> (plugin) and <code>new</code>
(backend) are entirely determined from this hash.</p>
<h2>Enumerable Backends</h2>
<p>Backends as implemented prior to version 0.2 could only do two things: read
and write values for a given locale. If you wanted to know <em>what</em> locales were
available for a given attribute, you would be stuck.</p>
<p>This was raised as an <a href="https://github.com/shioyama/mobility/issues/39">issue</a>,
and I decided that rather than just add a method returning locales, I’d go a
bit further and try to solve a more general problem: iterating through
translations. I implemented this in <a href="https://github.com/shioyama/mobility/pull/71">PR #71</a>
by adding a new <code>each_locale</code> method to each existing backend returning
successive locales to a block.</p>
<p>From these backend-specific <code>each_locale</code> methods, an <code>each</code> method
is defined in <code>Mobility::Backend</code> yielding instances of a
<code>Translation</code> struct wrapping the locale and backend. We then include Ruby’s
powerful <a href="https://ruby-doc.org/core-2.4.1/Enumerable.html">Enumerable</a> module,
giving us with it a whole slew of traversal and search methods. A method
<code>locales</code> is also added to <code>Mobility::Backend</code> which returns the available
locales for the attribute (the original request).</p>
<p>With this change, you can now do things like select translations that match a
condition:</p>
<p><pre type="ruby">
post = Post.create(title_en: “English foo”, title_de: “German foo”, title_ja: “Something else”)
post.title_backend.select { |t| t.read =</p>
<p>It’s been a little while since last I
<a href="/2017/3/3/translating-with-mobility">posted</a> about
<a href="https://github.com/shioyama/mobility">Mobility</a>, the pluggable translation
framework for Ruby that I’ve been working on recently. I thought I’d take this
opportunity with the release of the second version of the gem to
highlight some of the important changes and new features, and to recap how far
things have come.</p>
<h2>Mobility and the Module Builder Pattern</h2>
<p>In the time since I posted <a href="/2017/3/3/translating-with-mobility">Translating with
Mobility</a>, one of the
techniques I have used heavily in building the gem – the <a href="/2017/5/20/the-ruby-module-builder-pattern">Module
Builder
Pattern</a>, as
I’ve termed it – has become a <a href="https://news.ycombinator.com/item?id=14386723">hot topic of its
own</a>. (I’ll actually be <a href="http://rubykaigi.org/2017/presentations/shioyama.html">talking
about module builders</a>
at the coming RubyKaigi in Hiroshima, check it out if you’ll be attending.)</p>
<p>In a nutshell, the Module Builder Pattern is a simple way to create
customizable modules to mix into classes, one which falls naturally out of
Ruby’s object model. It amounts to simply subclassing the <code>Module</code> class,
defining module methods in an initializer or <code>included</code> hook, and including
instances of the subclass in other classes. See the slide below from my recent talk
at the <a href="https://trbmeetup.doorkeeper.jp/events/63110">Tokyo Rubyist Meetup</a>,
where I explain the basic elements of a module builder and how I use it in a
core Mobility class, <code>Mobility::Attributes</code>.</p>
<div class="img-center" style="margin: 40px">
<img alt="Elements of a Module Builder" src="/img/tokyorubyist-modulebuilder.png"/>
</div>
<p>From its first release, Mobility has used this pattern all over the place; I
could never have achieved the flexibility the gem needed without it. However,
in the latest release, I’ve pushed this yet further in order to
achieve something just as important: <em>extensibility</em>.</p>
<h2>Now with Plugins</h2>
<p>Let’s recall that in Mobility, you define a translated attribute on a model by
calling <code>translates</code> with one or more attribute names, and a hash to configure
which options are applied:
<a href="https://github.com/shioyama/mobility#dirty-tracking">dirty-tracking</a>,
<a href="https://github.com/shioyama/mobility">fallbacks</a>, <a href="https://github.com/shioyama/mobility#getting-and-setting-translations">locale accessors</a>,
and so on. Under
the hood, Mobility creates an instance of the
<a href="https://github.com/shioyama/mobility/blob/93af67072bb69b10818923a03ca73aa7b8eb0cc2/lib/mobility/attributes.rb#L100">Mobility::Attributes</a>
module builder and includes it in the class. I’ll use this form directly here
to emphasize what is actually going on:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span>
extend <span style="color:#036;font-weight:bold">Mobility</span>
include <span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Attributes</span>.new(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">content</span><span style="color:#710">"</span></span>, <span style="color:#606">locale_accessors</span>: [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>], <span style="color:#606">fallbacks</span>: <span style="color:#069">true</span>)
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Here, we are building a module instance of <code>Mobility::Attributes</code> for the
attributes <code>title</code> and <code>content</code>, with locale accessors for English and
Japanese and fallbacks enabled. (The format of arguments to this initializer
have changed slightly since version 0.1, see this <a href="https://github.com/shioyama/mobility/pull/72">pull
request</a> for details).</p>
<p>What happens when we include the instance of this class into <code>Post</code>? Well,
here’s the first change with version 0.2: I’ve made the magic here a
whole lot clearer, and more extensible as a result (see PRs <a href="https://github.com/shioyama/mobility/pull/50">#50</a> and
<a href="https://github.com/shioyama/mobility/pull/64">#62</a>).</p>
<p>Previously there was a lot of hard-coded references to options
directly in <code>Attributes</code>, but the code is now entirely free of such
references. In fact, there are just a few lines in the builder which deal with
options:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.plugins.each <span style="color:#080;font-weight:bold">do</span> |name|
plugin = get_plugin_class(name)
plugin.apply(<span style="color:#069">self</span>, options[name])
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Here, <code>Mobility.plugins</code> is an alias for <code>Mobility.config.plugins</code>, a new
configuration setting which defaults to <a href="https://github.com/shioyama/mobility/blob/e1f2987d67dd113b8c69dc82d80fb55ec32807f4/lib/mobility/configuration.rb#L73-L81">this
array</a>
(order of elements is important here<a href="#note-1"><sup>1</sup></a>):</p>
<div class="CodeRay">
<div class="code"><pre>
[<span style="color:#A60">:cache</span>, <span style="color:#A60">:dirty</span>, <span style="color:#A60">:fallbacks</span>, <span style="color:#A60">:presence</span>, <span style="color:#A60">:default</span>, <span style="color:#A60">:fallthrough_accessors</span>, <span style="color:#A60">:locale_accessors</span>]
</pre></div>
</div>
<p>The private <code>get_plugin_class</code> method fetches constants under
<code>Mobility::Plugins</code> matching CamelCase-ed values of the plugin names from this
array, then applies them with the attributes instance and option values as its arguments.</p>
<p>So <code>locale_accessors: [:en, :ja]</code> will fetch
<code>Mobility::Plugins::LocaleAccesors</code> and call it like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Plugins</span>::<span style="color:#036;font-weight:bold">LocaleAccessors</span>.apply(<span style="color:#069">self</span>, [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>])
</pre></div>
</div>
<p>…where <code>self</code> is the instance of <code>Mobility::Attributes</code> being included into
the model class, <code>Post</code>.</p>
<p>Notice that since we have no hard-coded references to option keys in this
module now, <em>any</em> option key included in <code>Mobility.plugins</code> will
apply a plugin with that name (as long as that plugin exists), with the
attributes instance and option value passed into the plugin’s <code>apply</code> class
method.</p>
<p>This makes is very easy to design new extensions: all they need is to
have a class method, <code>apply</code>, which
takes this <code>Attributes</code> instance (which itself has references to the
backend class and attribute names), and an option value to
configure how the plugin should be applied. (Incidentally, I borrow the term
“plugins” here, and the method name <code>apply</code>, from
<a href="http://sequel.jeremyevans.net/">Sequel</a>, where plugins are defined in a
somewhat similar way.)</p>
<p>Below is another slide from my recent talk, illustrating how these options
trigger inclusion of yet more module builder instances into the backend class
and into the Attributes instance itself.</p>
<div class="img-center" style="margin: 40px">
<img alt="Mobility Plugins" src="/img/tokyorubyist-plugins.png" />
</div>
<p>Along with this introduction of plugins and the <code>Mobility::Plugins</code> namespace,
I’ve also reorganized the backend namespace such that <code>Mobility::Backends</code>
(plural) is now exclusively used for backend classes.</p>
<p>The result is that it becomes very
straightforward to build custom backends and plugins: define a class in the
appropriate namespace with the required structure, and call <code>translates</code> (or
include an instance of <code>Mobility::Attributes</code>) in the model class, with any
backend or option keys, and that’s it! You ready to go.</p>
<h2>Default Options</h2>
<p>Another minor annoyance some users of Mobility may have encountered in 0.1 is
that while it was possible to set a default backend to use across all models
(with the <code>default_backend</code> configuration option), there was no way to define
a default set of options (to configure plugins and the backend). Along with
the changes to enable plugins above, I also <a href="https://github.com/shioyama/mobility/pull/50">added this as a
configuration setting</a>, with a
value that defaults to:</p>
<div class="CodeRay">
<div class="code"><pre>
{ <span style="color:#606">cache</span>: <span style="color:#069">true</span>, <span style="color:#606">dirty</span>: <span style="color:#069">false</span>, <span style="color:#606">fallbacks</span>: <span style="color:#069">nil</span>, <span style="color:#606">presence</span>: <span style="color:#069">true</span>, <span style="color:#606">default</span>: <span style="color:#069">nil</span> }
</pre></div>
</div>
<p>These default option values were previously hard-coded, but are
now explicitly set and isolated in the configuration instance, which makes
them much easier to reason about and change.</p>
<p>Want to enable fallbacks by default on all your models? Just override the
default for that key:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.default_options[<span style="color:#A60">:fallbacks</span>] = <span style="color:#069">true</span>
</pre></div>
</div>
<p>You can also set backend options here, so for example if you want to set the
default <code>type</code> for the KeyValue backend to be “string” instead of “text”, you
could do this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.default_options[<span style="color:#A60">:type</span>] = <span style="color:#A60">:string</span>
</pre></div>
</div>
<p>Note that <em>how</em> these option values are interpreted by actual plugins and
backends depends on the plugin or backend, and there is some unavoidable
variation there. But the values passed in to <code>apply</code> (plugin) and <code>new</code>
(backend) are entirely determined from this hash.</p>
<h2>Enumerable Backends</h2>
<p>Backends as implemented prior to version 0.2 could only do two things: read
and write values for a given locale. If you wanted to know <em>what</em> locales were
available for a given attribute, you would be stuck.</p>
<p>This was raised as an <a href="https://github.com/shioyama/mobility/issues/39">issue</a>,
and I decided that rather than just add a method returning locales, I’d go a
bit further and try to solve a more general problem: iterating through
translations. I implemented this in <a href="https://github.com/shioyama/mobility/pull/71">PR #71</a>
by adding a new <code>each_locale</code> method to each existing backend returning
successive locales to a block.</p>
<p>From these backend-specific <code>each_locale</code> methods, an <code>each</code> method
is defined in <code>Mobility::Backend</code> yielding instances of a
<code>Translation</code> struct wrapping the locale and backend. We then include Ruby’s
powerful <a href="https://ruby-doc.org/core-2.4.1/Enumerable.html">Enumerable</a> module,
giving us with it a whole slew of traversal and search methods. A method
<code>locales</code> is also added to <code>Mobility::Backend</code> which returns the available
locales for the attribute (the original request).</p>
<p>With this change, you can now do things like select translations that match a
condition:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.create(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">English foo</span><span style="color:#710">"</span></span>, <span style="color:#606">title_de</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">German foo</span><span style="color:#710">"</span></span>, <span style="color:#606">title_ja</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Something else</span><span style="color:#710">"</span></span>)
post.title_backend.select { |t| t.read =~ <span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#808">foo</span><span style="color:#404">/</span></span> }.map &<span style="color:#A60">:locale</span>
<span style="color:#777">#=> [:en, :de]</span>
</pre></div>
</div>
<p>Although this is a somewhat contrived example, it’s not hard to see that
having the full arsenal of Enumerable methods at your disposal could be quite
useful in a variety of contexts.</p>
<h2><del>Autoload</del> → Require</h2>
<p>Mobility 0.1.x uses
<a href="https://ruby-doc.org/core-2.4.1/Kernel.html#method-i-autoload">autoload</a> to
load nearly all its constants, but as Rubyists no doubt know, autoload <a href="https://bugs.ruby-lang.org/issues/921">has its
issues</a>, and <a href="http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/41149">may even be
deprecated</a>
in a future release of Ruby. So I decided to bite the bullet and convert all
<code>autoload</code> calls to <code>require</code>
(<a href="https://github.com/shioyama/mobility/pull/65">here’s the PR</a>). For this I
followed the pattern (again) of Sequel, which only requires plugins when they
are called with the <code>plugin</code> method.</p>
<p>So Mobility now does a similar thing for backends and plugins. Here is the
code loading a backend when it is triggered from a call to <code>translates</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">get_backend_class</span>(backend)
<span style="color:#080;font-weight:bold">return</span> backend <span style="color:#080;font-weight:bold">if</span> <span style="color:#036;font-weight:bold">Module</span> === backend
require <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility/backends/</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>backend<span style="font-weight:bold;color:#666">}</span></span><span style="color:#710">"</span></span>
get_class_from_key(<span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Backends</span>, backend)
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Rather than simply finding the constant, we first require the backend at
<code>"mobility/backends/#{backend}"</code>, then once we have loaded the file we make a
<code>const_get</code> call to actually fetch the constant (which previously would have
triggered <code>autoload</code> to find the backend file in the same location).</p>
<p>Admittedly, this now requires the plugin/backend creator to put the
backend/plugin in the requisite file location or else Mobility will not find
it. But that seems to me a quite reasonable thing to do anyway.</p>
<p>It also means that you can’t just <code>require "mobility"</code>, and then expect
<code>Mobility::Backends::KeyValue</code> (or any backend class) to be loaded, because it will not
be required until you actually <em>use it</em>. You need to either call <code>translates</code>,
or require the class explicitly, to load the backend.</p>
<h2>Mobility::Util</h2>
<p>Continuing with the autoload clean-up, I also decided to fix something
else that had been bothering me for a while: <code>mobility/core_ext</code>. When I originally
decided to make Mobility support Sequel in addition to ActiveRecord, I found
it hard to entirely give up ActiveSupport since (like many Rubyists) I had
become so accustomed to using methods like <code>present?</code>, <code>camelize</code>, <code>singularize</code>, etc.</p>
<p>So by default, if ActiveSupport could not be loaded, I included a bare minimum
implementation of just a few of these methods, filed under
<code>mobility/core_ext</code>. In addition, for Sequel
backends, I also just assumed that the
<a href="http://www.rubydoc.info/gems/sequel/4.26.0/Sequel/Inflections">Inflections</a>
module was loaded, otherwise the backends would not work (I required it in
Mobility’s Sequel specs).</p>
<p>In <a href="https://github.com/shioyama/mobility/pull/60">this PR</a>, I replaced
<code>mobility/core_ext</code>, which (like ActiveSupport) monkeypatched core classes
like <code>Object</code> and <code>String</code>, with a module <code>Mobility::Util</code> which contains all
the methods needed. <code>Util</code> can be either included in a class, or its methods
can be called as class methods: e.g. <code>Util.present?(string)</code> will call
<code>string.present?</code> if the object responds to <code>present?</code>, otherwise it will
do the work itself to get the result.</p>
<p>Now Sequel does not need to include the Inflections module to use Mobility,
and the core parts of Mobility (e.g. <code>Mobility::Attributes</code>) do not require
any special monkeypatching to work. This is important for anyone wanting
to use Mobility for an application where they do <em>not</em> want to patch core
Ruby classes (a good thing).</p>
<h2>Super as Option</h2>
<p>In some cases, Mobility overrides a method to redefine it as a translation
accessor. This is the case, for example, with backends that store translations
on a single column: the PostgreSQL (Jsonb + Hstore) backends, as well as the
Serialized backend (which stores translations serialized as a json or yaml
hash).</p>
<p>With these backends, the original attribute method, say <code>title</code>, returns a
hash containing all translations as key/value pairs (where locales are the
keys). Mobility overrides this column accessor to return value of the hash at
the current locale, using (for AR models) <code>read_attribute</code> to avoid infinite
recursion (Sequel is similar).</p>
<p>A few people had asked about how they could access the original (hash) column,
so in <a href="https://github.com/shioyama/mobility/pull/62">this PR</a> I added a new
<code>super</code> option which bypasses the Mobility accessors (reader and writer) and
continues up to the original method, in this case the column method.</p>
<p>So something like this:</p>
<div class="CodeRay">
<div class="code"><pre>
post.title
<span style="color:#777">#=> "Translating with Mobility"</span>
post.title(<span style="color:#606">super</span>: <span style="color:#069">true</span>)
<span style="color:#777">#=> { "en" => "Translating with Mobility", "ja" => "Traduction avec Mobilité" }</span>
</pre></div>
</div>
<p>The writer can also be “super-ed”, although this is somewhat trickier to do:</p>
<div class="CodeRay">
<div class="code"><pre>
post.send(<span style="color:#A60">:title=</span>, {}, <span style="color:#606">super</span>: <span style="color:#069">true</span>)
<span style="color:#777">#=> {}</span>
post.title
<span style="color:#069">nil</span>
</pre></div>
</div>
<h2>Other</h2>
<ul>
<li>The default name for associaitons in the KeyValue and Table backends has
been simplified. KeyValue associations named
<code>mobility_string_translations</code> and <code>mobility_text_translations</code> are now
simply <code>string_translations</code> and <code>text_translations</code>, respectively. For the Table
backend, <code>mobility_model_translations</code> has been simplified to simply
<code>translations</code> (like Globalize).</li>
<li>In <a href="https://github.com/shioyama/mobility/pull/58">PR #58</a>, I refactored a
somewhat ugly part of the code dealing with caching. I’m still not
completely happy with how this is done, but it’s at least an improvement
over the earlier technique.</li>
<li>I now have some minimal object allocation specs, added in <a href="https://github.com/shioyama/mobility/pull/57">PR #57</a>. I’d like to extend this
to benchmark testing as well, but not quite sure how to proceed yet on that
one (any suggestions welcome!)</li>
<li>The gem is now <em>signed</em>. You of course do not need to check this signature,
but if you want to, there are instructions in the README on how to do this.</li>
<li><code>extend Mobility</code> is now the preferred way to use Mobility in models (see
<a href="https://github.com/shioyama/mobility/issues/69">this issue</a>).</li>
<li>I’ve bumped Ruby support to minimum 2.2.7. This is pretty standard.</li>
</ul>
<h2>Finally, a few words</h2>
<p>Mobility is a <strong>labour of love</strong>, and I think it’s worthwhile to mention that
here. I’ve spent countless hours writing and re-writing Mobility’s code, and
many more thinking and rethinking how I can refactor and improve that code.</p>
<p>I have both selfish and unselfish motivations for doing this: on the one hand,
I want to show off the best that I can do with a clean slate and a clear
problem (translation) that I happen to know reasonably well. On the other, I
genuinely want to help people who are trying to share content across language
barriers, a problem which has been an interest of mine for a <a href="http://translationjournal.net/journal/45global.htm">long
while</a>.</p>
<p>So it’s been enormously gratifying for me to see the positive response
Mobility has received so far in the form of encouraging comments and
reactions, a
growing list of
<a href="https://github.com/shioyama/mobility/stargazers">stargazers</a>, and as actual
<a href="https://github.com/shioyama/mobility/graphs/contributors">contributions</a> to
the gem code. I’ve had very thoughtful input on <a href="https://github.com/shioyama/mobility/issues/69">edge
cases</a> and <a href="https://github.com/shioyama/mobility/pull/71#issuecomment-321836122">subtle
points</a>
that I would never have considered, as well as some great feedback
at recent talks in
<a href="https://www.montrealrb.com/events/19-june-20th-meetup">Montreal</a> and
<a href="https://trbmeetup.doorkeeper.jp/events/63110">Tokyo</a> (see slides from the
latter of those talks below).</p>
<div style="width: 100%; max-width: 500px; margin: auto; margin-top: 20px; margin-bottom: 20px">
<script async class="speakerdeck-embed" data-id="38720207c49c4d8da954e6a26499ed0a" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<p class="caption">Slides from my recent talk at the Tokyo Rubyist Meetup</p>
</div>
<p>There are many <a href="https://groups.google.com/forum/#!topic/capistrano/nmMaqWR1z84">horror
stories</a> about
open source software maintainers getting exhausted from the demands that come
with success. Mobility hasn’t reached anywhere near that level of success yet, but
so far at least I don’t find myself exhausted at all. On the contrary, it’s
been an extremely rewarding learning experience!</p>
<p>If you’re using Mobility for a project or application, please let me know. I’m
eager to here about the kind of challenges people are tackling and how
Mobility might be better able to help solve them.</p>
<h4>Notes</h4>
<p><small><a name="note-1"><sup>1</sup></a> The order of plugins in <code>Mobility.plugins</code>
is important: many of the plugin classes include modules into the backend
overriding <code>read</code> and <code>write</code>, inclusion order determining the order of method
composition. In the list, for example, <code>cache</code> comes before <code>default</code> (and
included in that order) because we want to cache actual backend values, not
default values.</small></p>
The Ruby Module Builder Patternhttps://dejimata.com/2017/5/20/the-ruby-module-builder-pattern2017-05-202017-05-20Chris Salzberg<p>There’s an interesting pattern I’ve discovered recently in Ruby that is very
powerful, yet apparently not widely known or
appreciated.<a href="#note-1"><sup>1</sup></a> I call this pattern the <em>Module Builder
Pattern</em>. I’ve used it heavily in designing
<a href="https://github.com/shioyama/mobility">Mobility</a>, a pluggable translation
framework I released a couple months ago, and it served me so well I thought I
should share what I’ve learned.</p>
<div class="img-right">
<img alt="Drawing Hands" src="/img/DrawingHands.jpg" />
</div>
<p>At its core, the Module Builder is an extremely simple, elegant, and versatile
way of dynamically defining named, customizable modules to mix into classes.
This is done by <a href="http://solnic.codes/2012/08/13/subclassing-module-for-fun-and-profit">subclassing the <code>Module</code>
class</a>,
initializing these subclasses at runtime, and including the resulting modules
into other classes.</p>
<p>If the idea of <em>subclassing</em> <code>Module</code> makes your head spin, read on.
Despite its seemingly esoteric nature, the Module Builder does some very
useful things. It is used extensively in projects such as
<a href="http://dry-rb.org/">dry-rb</a> to great effect, but buried out of view among
advanced coding styles that conceal its essential simplicity. It is also used
sporadically in Rails<a href="#note-2"><sup>2</sup></a>, but only to a very limited
extent. I think it’s fair to say that most Rubyists have never seen the
pattern, let alone ever used it in their daily work.</p>
<p>To explain Module Builder, I will work through a series of progressively
more complex examples, ending with some code extracted from my work on
<a href="https://github.com/shioyama/mobility">Mobility</a> into a gem called
<a href="https://github.com/shioyama/method_found">MethodFound</a>. The
examples begin with some simple core concepts I think most readers will
understand, then advance to a slightly more in-depth discussion exposing the
pattern’s tangible benefits in real-life applications.</p>
<h2>Modules in Ruby</h2>
<p>Let’s first recall that a module in Ruby is basically a collection of methods
and constants which you can include in any class, reducing duplication and
encapsulating functionality in reusable, composable<a href="#note-3"><sup>3</sup></a>
units. Callbacks like
<a href="http://ruby-doc.org/core-2.4.1/Module.html#method-i-included"><code>included</code></a>
and <a href="http://ruby-doc.org/core-2.4.1/Module.html#method-i-extended"><code>extended</code></a>
allow you to execute some custom code every time your module is
included (or extended, etc) in a class or module.</p>
<p>So suppose we have a module <code>MyModule</code> with a method <code>foo</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">MyModule</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">foo</span>
<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we can add this method to any class by including <code>MyModule</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">MyClass</span>
include <span style="color:#036;font-weight:bold">MyModule</span>
<span style="color:#080;font-weight:bold">end</span>
a = <span style="color:#036;font-weight:bold">MyClass</span>.new
a.foo
<span style="color:#777">#=> "foo"</span>
</pre></div>
</div>
<p>That’s pretty straightforward, but also very limited: <code>foo</code> is pre-defined
with a hard-coded return value.</p>
<p>Let’s take a slightly more “real-world” case. Say we’ve got a class, <code>Point</code>,
with two attributes <code>x</code> and <code>y</code>, which for simplicity’s sake we’ll define
using a <a href="http://ruby-doc.org/core-2.4.1/Struct.html">Struct</a>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Point</span> = <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:x</span>, <span style="color:#A60">:y</span>)
</pre></div>
</div>
<p>Now let’s create a module which can add two points together at their
respective <code>x</code> and <code>y</code> values. Call it <code>Addable</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Addable</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(point)
<span style="color:#036;font-weight:bold">Point</span>.new(point.x + x, point.y + y)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#036;font-weight:bold">Point</span>.include <span style="color:#036;font-weight:bold">Addable</span>
p1 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">1</span>, <span style="color:#00D">2</span>)
p2 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">2</span>, <span style="color:#00D">3</span>)
p1 + p2
<span style="color:#777">#=> #<Point:.. @x=3, @y=5></span>
</pre></div>
</div>
<p>The module is slightly more reusable now. But it still has rigid constraints:
it will only work with a class called <code>Point</code> which has attributes <code>x</code> and
<code>y</code>.</p>
<p>We can improve this a bit by removing the constant <code>Point</code> in the module, and
replacing it with <code>self.class</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Addable</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
<span style="color:#069">self</span>.class.new(x + other.x, y + other.y)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now our <code>Addable</code> module can be used with <em>any class</em> that has accessors <code>x</code> and
<code>y</code>,<a href="#note-4"><sup>4</sup></a> which makes it much more flexible. Also, thanks to
Ruby’s very dynamic treatment of types, the module can be used to “add” a
potentially wide variety of pairwise data types, as long as their elements
have a <code>+</code> method.</p>
<p>In addition, note that modules in Ruby have the property that their methods can be
composed through inclusion or inheritance by using the <code>super</code> keyword in
the including or parent class. So if we wanted to log calls to our adder method, we
could do this by defining <code>+</code> inside of the <code>Point</code> class, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Point</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:x</span>, <span style="color:#A60">:y</span>)
include <span style="color:#036;font-weight:bold">Addable</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">super</span>.tap { puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">"</span></span> }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>And we can see that our logging indeed works as expected, calling the method
on the class first, and then through <code>super</code> the method defined on the module:</p>
<div class="CodeRay">
<div class="code"><pre>
p1 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">1</span>, <span style="color:#00D">2</span>)
p2 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">2</span>, <span style="color:#00D">3</span>)
p1 + p2
<span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#777">#=> #<Point:.. @x=3, @y=5></span>
</pre></div>
</div>
<p>There are many other things to say about modules; we’ve really only touched
the tip of the iceberg here. However, the main topic of this post is module
<em>builders</em>, not modules, so let’s take a look at that.</p>
<div class="img-center" style="margin: 60px 0 70px 0">
<img alt="Automated space exploration and industrialization using self-replicating systems" src="/img/selfreplicatingsystem.png" />
<p class="caption">Building a Module that Builds Modules</p>
</div>
<h2>Modules that Build</h2>
<p>Modules are powerful because (when used effectively) they encapsulate certain
shared patterns of functionality. In the example above, this functionality is
the addition of variable pairs (points, or any other class with pairs of
variables <code>x</code> and <code>y</code>). Identifying shared patterns like this and expressing
them in the form of a module or set of modules makes it possible to tackle very
complex problems with simple, modular solutions.</p>
<p>That said, the module above is still quite specific, constrained to classes
with variables <code>x</code> and <code>y</code>. What if we wanted to loosen this constraint and
have the module work for any set of variables? How would we do that?</p>
<p>Well, a module can’t be configured once defined, so we’ll need some other way to loosen
this constraint. One way is to define a method
on the module that <em>itself</em> adds the desired functionality to the class. Let’s
call this method <code>define_adder</code><a href="#note-5"><sup>5</sup></a>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">AdderDefiner</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_adder</span>(*keys)
define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other|
<span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) }))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Instead of defining a <code>+</code> method like we did in the <code>Addable</code> module, we
instead define <em>a method which defines the method</em>. We’ll call this
“method-defining method” the <em>bootstrap method</em> here, since it “bootstraps”
the module’s instance methods dynamically into the class.</p>
<p>This bootstrap method takes an arbitrary number of arguments, mapped to an
array <code>keys</code> using the <a href="https://renugasaraswathy.wordpress.com/2014/09/03/splat-operator-in-ruby-2/">splat
operator</a>.
These “keys” are the <em>names</em> of the variables to add (or, more precisely, the
names of the methods which return the variable values). It then defines a
method, called <code>+</code>, which adds the values of these variables at each key and
creates an instance from the results using <code>self.class.new</code>.</p>
<p>Let’s see if it works. We’ll create a new class, <code>LineItem</code>, with different
reader names, and <code>extend</code> the module in this case rather than include it
(since <code>define_adder</code> needs to be a class method):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
extend <span style="color:#036;font-weight:bold">AdderDefiner</span>
define_adder(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we can add line item pairs, just like we did point pairs earlier:</p>
<div class="CodeRay">
<div class="code"><pre>
l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>)
l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>)
l1 + l2
<span style="color:#777">#=> #<LineItem:... @amount=25.98, @tax=3.9></span>
</pre></div>
</div>
<p>So it works! And we now have no dependency on variable names (or even on
number of variables), so our module is more flexible.</p>
<p>But there’s a problem. When we copy over our earlier logging code from the
<code>Point</code> class to log calls to <code>+</code>, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
extend <span style="color:#036;font-weight:bold">AdderDefiner</span>
define_adder(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">super</span>.tap { puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">"</span></span> }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>… and then add two line items again, instead of getting the expected logs,
things blow up with an exception:</p>
<div class="CodeRay">
<div class="code"><pre>
NoMethodError: super: no superclass method `+' for #<struct LineItem amount=9.99, tax=1.5>
</pre></div>
</div>
<p>What’s going on here?</p>
<p>Take a good look at how <code>define_adder</code> bootstraps the <code>+</code> method: it
uses <code>define_method</code> to add the method <em>directly to the class</em>, in this case
<code>LineItem</code>. A class can only have one definition of any given method, so when,
later in the class, we again define <code>+</code>, we clobber the earlier definition.
There is no “superclass” here since we have not included or inherited anything
when defining the method.</p>
<p>Solving this problem will require some “module building” magic. Let’s look at the
solution first, and then see how it solves this issue:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">AdderIncluder</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_adder</span>(*keys)
adder = <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span>
define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other|
<span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) }))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
include adder
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we replace <code>AdderDefiner</code> in the logged <code>LineItem</code> code above with
<code>AdderIncluder</code> and we find that it works:</p>
<div class="CodeRay">
<div class="code"><pre>
l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>)
l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>)
l1 + l2
<span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#777">#=> #<LineItem:... @amount=25.98, @tax=3.9></span>
</pre></div>
</div>
<p>The trick that makes this work is the use of an <em>anonymous module</em> in
<code>define_adder</code> to define the <code>+</code> method, which is different from the previous
technique of defining the method directly on the calling class.</p>
<p>Recall that in Ruby, any module (small “m”) is simply an instance of a class,
and this class is <a href="http://ruby-doc.org/core-2.4.1/Module.html">Module</a> (big
“M”). Like other classes, <code>Module</code> has a <code>new</code> method, in this case one which
<a href="http://ruby-doc.org/core-2.4.1/Module.html#method-c-new">can take a block
argument</a> and
evaluate the block in the context of the newly-created module.</p>
<p>So to generate a “module on the fly”, you can simply call <code>Module.new</code>, pass a
block with some method definitions, and use it just as you would any other
(named) module:</p>
<div class="CodeRay">
<div class="code"><pre>
mod = <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">with_foo</span>
<span style="color:#069">self</span> + <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20"> with Foo</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">The Ruby Module Builder Pattern</span><span style="color:#710">"</span></span>
title.extend mod
title.with_foo
<span style="color:#777">#=> "The Ruby Module Builder Pattern with Foo"</span>
</pre></div>
</div>
<p>This is a handy trick, often used in testing when you don’t want to clutter
the global namespace with a one-off module used only in a single test (works
for <code>Class</code> too).<a href="#note-6"><sup>6</sup></a></p>
<p>The method defined in the block in <code>AdderIncluder</code> is the same <code>+</code> method
defined earlier in <code>AdderDefiner</code>, and in this sense, the two modules perform
a very similar function. The key difference is that whereas <code>AdderDefiner</code>
defines the method <em>on</em> the class, <code>AdderIncluder</code> <em>includes a module</em> with the
method <em>into</em> the class.</p>
<p>If you look up the chain of ancestors, you will see this module just after the
<code>LineItem</code> class itself:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">LineItem</span>.ancestors
<span style="color:#777">#=> [LineItem,</span>
<span style="color:#777">#<Module:0x...>,</span>
...
</pre></div>
</div>
<p>Given this ancestor chain, it is clear why <code>super</code> works here: when you
call <code>+</code> on <code>LineItem</code>, the method calls <code>super</code>, which continues up the
class hierarchy to the anonymous module where our <code>+</code> method is defined. It
then calculates and returns the result, which gets composed in the <code>LineItem</code>
adder with the logging code.</p>
<p>Although at first glance it might seem somewhat strange and unfamiliar, this
technique of defining a bootstrap method to include a dynamically-generated
anonymous module is in fact extremely useful, and widely used for this reason.
This is particularly true for a flexible framework like Rails, where simply
<a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activerecord/lib/active_record/attribute_methods.rb#L13">defining a model</a>
or <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/actionpack/lib/action_dispatch/routing/mapper.rb#L652">setting up routes</a>
triggers calls to many bootstrap methods like <code>define_adder</code>. The technique
can also be found in other gems that require a similar level of flexiblity
(including <a href="https://github.com/shioyama/mobility">Mobility</a>).</p>
<p>Bootstrapped module building can thus be considered a very important
metaprogramming technique in Ruby. Indeed, it underpins the very dynamic
nature that enables frameworks like Rails to work the way they do.</p>
<p>That said, I hope to convince you here that the potential of module building
goes well beyond the bootstrapping technique shown above, and as such that it
is a pattern that deserves more attention. To do this, I’m going to return to
the Adder module one last time, and define it in a slightly different way.</p>
<h2>The Module Builder</h2>
<p>So far we’ve looked at three “adder” modules:</p>
<ol>
<li><code>Addable</code>, which adds a <code>+</code> method on two variables <em>x</em> and <em>y</em>,</li>
<li><code>AdderDefiner</code>, which adds a class method <code>define_adder</code> that
defines a <code>+</code> method on an arbitrary set of variables, and</li>
<li><code>AdderIncluder</code>, which also adds a class method <code>define_adder</code> that
defines the <code>+</code> method, but in such a way that the method can be composed
using <code>super</code>.</li>
</ol>
<p>Although not very flexible, note that in hindsight, the first <code>Addable</code> module
had some good things going for it:</p>
<ul>
<li>It only included the <code>+</code> method, and left no other bootstrapping
“artifacts” like <code>define_adder</code> (unlike both <code>AdderDefiner</code> and
<code>AdderIncluder</code>)</li>
<li>It allowed the parent class to access its methods using
<code>super</code> (like <code>AdderIncluder</code> but not <code>AdderDefiner</code>)</li>
<li>It had a <em>name</em>, and was therefore very easy to recognize in the chain of
ancestors. While <code>AdderDefiner</code> and <code>AdderIncluder</code> <em>themselves</em> appear in
the chain of ancestors, the fact that the bootstrap method was called, and
<em>how</em> it was called, is hard or impossible to discern.</li>
</ul>
<p>In this section, I will show how we can solve the adder problem in a way that
retains these benefits, while also offering the flexibility we sought in the last section.</p>
<div class="img-center" style="margin: 70px 0 40px 0">
<img style="padding-bottom: 20px" alt="A Self-Replicating Machine"
src="/img/selfreplicatingmachine-medium.png" />
<p class="caption">Module Builders</p>
</div>
<p>Let’s start by defining an <code>AdderBuilder</code> class which inherits from <code>Module</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">AdderBuilder</span> < <span style="color:#036;font-weight:bold">Module</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(*keys)
define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other|
<span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) }))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>This is a short but very dense little class. Before going into the details of
what it’s doing here, let’s first check that it works. We’ll include (not
extend) an <em>instance</em> of this class (a module) into <code>LineItem</code> this time:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
include <span style="color:#036;font-weight:bold">AdderBuilder</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
<span style="color:#080;font-weight:bold">end</span>
l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>)
l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>)
l1 + l2
<span style="color:#777">#=> #<LineItem:... @amount=25.98, @tax=3.9></span>
</pre></div>
</div>
<p>So, as we can see, this little class does the same thing that we achieved with
the <code>define_adder</code> bootstrap method in <code>AdderDefiner</code>, except <em>without ever
defining a class method to do it</em>.</p>
<p>How does it do this? First, it subclasses <code>Module</code>. Since we can call <code>new</code> on
<code>Module</code>, it’s not much of a stretch to imagine
that we can also subclass it and define our own methods such as
<code>initialize</code>. And indeed, as shown above, we <em>can</em> do this.</p>
<p>Here’s that initializer again:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(*keys)
define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other|
<span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) }))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>There are two levels at play here, and it’s important to recognize both of
them. On one level, we are initializing a new instance of a class, <code>Module</code>.
This instance is itself a module, so it’s a collection of methods we will mix
into other classes. We are then, as we initialize this module, defining one
of these methods, a method called <code>+</code>.</p>
<p>The method <code>+</code> itself is being defined <em>in terms of</em> the method names we have
passed as arguments to <code>Module.new</code> (the “keys”, which were <code>:amount</code> and
<code>:tax</code> in the example above).<a href="#note-7"><sup>7</sup></a> When we call
<code>self.class.new</code> <em>in</em> the method, <em>this</em> <code>new</code> will be evaluated in the context
of the class including this new module, and thus would evaluate to <code>LineItem</code>
(or <code>Point</code>, or whatever class the module was included into).</p>
<p>So <code>AdderBuilder.new(:amount, :tax)</code> evaluates to a module
functionally equivalent to this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
<span style="color:#069">self</span>.class.new(amount + other.amount, tax + other.tax)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>… which is what we had in <code>Addable</code>, with <code>x</code> and <code>y</code> swapped for <code>amount</code>
and <code>tax</code>. But the power here comes from the fact that we can define our
dynamic module in terms of <em>whatever variable names we want.</em></p>
<p>And herein lies the essential ingenuity of this pattern: that <strong>it allows you
to define modules not as fixed collections of methods and constants, but as
configurable prototypes for building such collections</strong>. There is no need for
class methods or other metaprogramming tricks to bootstrap the
module-inclusion step; the builder builds modules directly, so they can be
included in one step.</p>
<p>Not only that, but the modules created in this process, unlike the anonymous
modules generated by techniques described in the last section, actually have a
name, so you can clearly see them in the chain of ancestors:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">LineItem</span>.ancestors
[<span style="color:#036;font-weight:bold">LineItem</span>, <span style="color:#777">#<AdderBuilder:0x...>, Object, ... ]</span>
</pre></div>
</div>
<p>This is a nice by-product of the encapsulation we have achieved by subclassing
<code>Module</code>.<a href="#note-8"><sup>8</sup></a> With an anonymous module, this is much harder
to debug:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">LineItem</span>.ancestors
[<span style="color:#036;font-weight:bold">LineItem</span>, <span style="color:#777">#<Module:0x...>, Object, ... ]</span>
</pre></div>
</div>
<p>Module Builders thus have the benefits of anonymous modules (their instances
can be defined “on the fly” and do not clutter the global constant namespace),
are easily traceable since their instances have a name (like “normal”
modules), and yet do not require defining (and calling) new methods on the
including class to bootstrap them.</p>
<p>We can go a step further. We previously added some logging code to the
including class. This logging code can also be rewritten as a module builder,
which we will call <code>LoggerBuilder</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LoggerBuilder</span> < <span style="color:#036;font-weight:bold">Module</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(method_name, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#710">"</span></span>, <span style="color:#606">end_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#710">"</span></span>)
define_method method_name <span style="color:#080;font-weight:bold">do</span> |*args, &block|
print start_message
<span style="color:#080;font-weight:bold">super</span>(*args, &block).tap { print end_message }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we can log <em>any included or inherited method</em> by creating an instance of
<code>LoggerBuilder</code> with the name of the method to be logged:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
include <span style="color:#036;font-weight:bold">AdderBuilder</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
include <span style="color:#036;font-weight:bold">LoggerBuilder</span>.new(<span style="color:#A60">:+</span>, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Enter Adder...</span><span style="color:#b0b">\n</span><span style="color:#710">"</span></span>,
<span style="color:#606">end_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Exit Adder...</span><span style="color:#b0b">\n</span><span style="color:#710">"</span></span>)
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>And, presto, we’ve got a logged addable line item:</p>
<div class="CodeRay">
<div class="code"><pre>
l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>)
l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>)
l1 + l2
<span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#777">#=> #<LineItem:... @amount=25.98, @tax=3.9></span>
</pre></div>
</div>
<p>But of course, we can now also log any other included or inherited method we
like:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">LineItem</span>.include(<span style="color:#036;font-weight:bold">LoggerBuilder</span>.new(<span style="color:#A60">:to_s</span>, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Stringifying...</span><span style="color:#b0b">\n</span><span style="color:#710">"</span></span>))
l1.to_s
<span style="color:#036;font-weight:bold">Stringifying</span>...
=> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">#<struct LineItem amount=9.99, tax=1.5></span><span style="color:#710">"</span></span>
</pre></div>
</div>
<p>There are many other directions you can take this; I’ve only scratched the
surface here. For a relatively simple (yet powerful) example of this “in the
wild”, take a look at <a href="https://github.com/dry-rb/dry-equalizer">Dry
Equalizer</a>, which dynamically defines
equality methods on a class using an instance of an <code>Equalizer</code> class that,
like <code>AdderBuilder</code>, subclasses <code>Module</code>.</p>
<h2>Module Builders and Composition</h2>
<p>You may have noticed that I have used the word “compose” quite a lot in this
post with respect to including modules into a class. Any time you include a
module which overrides a method and calls <code>super</code>, you are in effect using
function (method) composition, although it is not generally referred to as
such.</p>
<p>The same is true of our logging code in the last section. When we include an
instance of both <code>AdderBuilder</code> <em>and</em> <code>LoggerBuilder</code>, we are building <code>+</code> as
a composite function, which we could alternatively write as:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">add</span>(other)
<span style="color:#069">self</span>.class.new(amount + other.amount, tax + other.tax)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">log</span>(result)
print <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">"</span></span>
result.tap { print <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">"</span></span> }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
log(add(other))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>So <code>super</code> is essentially taking the output of the last method call and
incorporating it into the result of the current module’s method.</p>
<p>Composition happens to be a very interesting way to “combine” different
instances of a module builder, like we combined logging and adding modules in
the example above. With module builders, we can configure and combine many
modules to build more complex methods and class behaviours.</p>
<p>One way to do this is via a “branching” method flow. We define multiple
modules that each override the same method, but only trigger some special
processing if a condition on the method and its arguments matches the module’s
state; otherwise, the control flow continues up the class hierarchy to the
next module via <code>super</code>.</p>
<p>If you’ve worked with Ruby for any length of time, this “branching
composition” should make you think of one method:
<a href="http://ruby-doc.org/core-2.4.1/BasicObject.html#method-i-method_missing">method_missing</a>.
Rarely does anyone override <code>method_missing</code> without a call to <code>super</code>
somewhere, since doing so would catch <em>any</em> method not defined on the class, which is
generally not what you want.</p>
<p>Instead, a typical <code>method_missing</code> override looks something like this:</p>
<p><pre type="ruby">
def method_missing(method_name, *arguments, &block)
if method_name =</p>
<p>There’s an interesting pattern I’ve discovered recently in Ruby that is very
powerful, yet apparently not widely known or
appreciated.<a href="#note-1"><sup>1</sup></a> I call this pattern the <em>Module Builder
Pattern</em>. I’ve used it heavily in designing
<a href="https://github.com/shioyama/mobility">Mobility</a>, a pluggable translation
framework I released a couple months ago, and it served me so well I thought I
should share what I’ve learned.</p>
<div class="img-right">
<img alt="Drawing Hands" src="/img/DrawingHands.jpg" />
</div>
<p>At its core, the Module Builder is an extremely simple, elegant, and versatile
way of dynamically defining named, customizable modules to mix into classes.
This is done by <a href="http://solnic.codes/2012/08/13/subclassing-module-for-fun-and-profit">subclassing the <code>Module</code>
class</a>,
initializing these subclasses at runtime, and including the resulting modules
into other classes.</p>
<p>If the idea of <em>subclassing</em> <code>Module</code> makes your head spin, read on.
Despite its seemingly esoteric nature, the Module Builder does some very
useful things. It is used extensively in projects such as
<a href="http://dry-rb.org/">dry-rb</a> to great effect, but buried out of view among
advanced coding styles that conceal its essential simplicity. It is also used
sporadically in Rails<a href="#note-2"><sup>2</sup></a>, but only to a very limited
extent. I think it’s fair to say that most Rubyists have never seen the
pattern, let alone ever used it in their daily work.</p>
<p>To explain Module Builder, I will work through a series of progressively
more complex examples, ending with some code extracted from my work on
<a href="https://github.com/shioyama/mobility">Mobility</a> into a gem called
<a href="https://github.com/shioyama/method_found">MethodFound</a>. The
examples begin with some simple core concepts I think most readers will
understand, then advance to a slightly more in-depth discussion exposing the
pattern’s tangible benefits in real-life applications.</p>
<h2>Modules in Ruby</h2>
<p>Let’s first recall that a module in Ruby is basically a collection of methods
and constants which you can include in any class, reducing duplication and
encapsulating functionality in reusable, composable<a href="#note-3"><sup>3</sup></a>
units. Callbacks like
<a href="http://ruby-doc.org/core-2.4.1/Module.html#method-i-included"><code>included</code></a>
and <a href="http://ruby-doc.org/core-2.4.1/Module.html#method-i-extended"><code>extended</code></a>
allow you to execute some custom code every time your module is
included (or extended, etc) in a class or module.</p>
<p>So suppose we have a module <code>MyModule</code> with a method <code>foo</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">MyModule</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">foo</span>
<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we can add this method to any class by including <code>MyModule</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">MyClass</span>
include <span style="color:#036;font-weight:bold">MyModule</span>
<span style="color:#080;font-weight:bold">end</span>
a = <span style="color:#036;font-weight:bold">MyClass</span>.new
a.foo
<span style="color:#777">#=> "foo"</span>
</pre></div>
</div>
<p>That’s pretty straightforward, but also very limited: <code>foo</code> is pre-defined
with a hard-coded return value.</p>
<p>Let’s take a slightly more “real-world” case. Say we’ve got a class, <code>Point</code>,
with two attributes <code>x</code> and <code>y</code>, which for simplicity’s sake we’ll define
using a <a href="http://ruby-doc.org/core-2.4.1/Struct.html">Struct</a>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Point</span> = <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:x</span>, <span style="color:#A60">:y</span>)
</pre></div>
</div>
<p>Now let’s create a module which can add two points together at their
respective <code>x</code> and <code>y</code> values. Call it <code>Addable</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Addable</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(point)
<span style="color:#036;font-weight:bold">Point</span>.new(point.x + x, point.y + y)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#036;font-weight:bold">Point</span>.include <span style="color:#036;font-weight:bold">Addable</span>
p1 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">1</span>, <span style="color:#00D">2</span>)
p2 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">2</span>, <span style="color:#00D">3</span>)
p1 + p2
<span style="color:#777">#=> #<Point:.. @x=3, @y=5></span>
</pre></div>
</div>
<p>The module is slightly more reusable now. But it still has rigid constraints:
it will only work with a class called <code>Point</code> which has attributes <code>x</code> and
<code>y</code>.</p>
<p>We can improve this a bit by removing the constant <code>Point</code> in the module, and
replacing it with <code>self.class</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Addable</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
<span style="color:#069">self</span>.class.new(x + other.x, y + other.y)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now our <code>Addable</code> module can be used with <em>any class</em> that has accessors <code>x</code> and
<code>y</code>,<a href="#note-4"><sup>4</sup></a> which makes it much more flexible. Also, thanks to
Ruby’s very dynamic treatment of types, the module can be used to “add” a
potentially wide variety of pairwise data types, as long as their elements
have a <code>+</code> method.</p>
<p>In addition, note that modules in Ruby have the property that their methods can be
composed through inclusion or inheritance by using the <code>super</code> keyword in
the including or parent class. So if we wanted to log calls to our adder method, we
could do this by defining <code>+</code> inside of the <code>Point</code> class, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Point</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:x</span>, <span style="color:#A60">:y</span>)
include <span style="color:#036;font-weight:bold">Addable</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">super</span>.tap { puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">"</span></span> }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>And we can see that our logging indeed works as expected, calling the method
on the class first, and then through <code>super</code> the method defined on the module:</p>
<div class="CodeRay">
<div class="code"><pre>
p1 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">1</span>, <span style="color:#00D">2</span>)
p2 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">2</span>, <span style="color:#00D">3</span>)
p1 + p2
<span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#777">#=> #<Point:.. @x=3, @y=5></span>
</pre></div>
</div>
<p>There are many other things to say about modules; we’ve really only touched
the tip of the iceberg here. However, the main topic of this post is module
<em>builders</em>, not modules, so let’s take a look at that.</p>
<div class="img-center" style="margin: 60px 0 70px 0">
<img alt="Automated space exploration and industrialization using self-replicating systems" src="/img/selfreplicatingsystem.png" />
<p class="caption">Building a Module that Builds Modules</p>
</div>
<h2>Modules that Build</h2>
<p>Modules are powerful because (when used effectively) they encapsulate certain
shared patterns of functionality. In the example above, this functionality is
the addition of variable pairs (points, or any other class with pairs of
variables <code>x</code> and <code>y</code>). Identifying shared patterns like this and expressing
them in the form of a module or set of modules makes it possible to tackle very
complex problems with simple, modular solutions.</p>
<p>That said, the module above is still quite specific, constrained to classes
with variables <code>x</code> and <code>y</code>. What if we wanted to loosen this constraint and
have the module work for any set of variables? How would we do that?</p>
<p>Well, a module can’t be configured once defined, so we’ll need some other way to loosen
this constraint. One way is to define a method
on the module that <em>itself</em> adds the desired functionality to the class. Let’s
call this method <code>define_adder</code><a href="#note-5"><sup>5</sup></a>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">AdderDefiner</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_adder</span>(*keys)
define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other|
<span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) }))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Instead of defining a <code>+</code> method like we did in the <code>Addable</code> module, we
instead define <em>a method which defines the method</em>. We’ll call this
“method-defining method” the <em>bootstrap method</em> here, since it “bootstraps”
the module’s instance methods dynamically into the class.</p>
<p>This bootstrap method takes an arbitrary number of arguments, mapped to an
array <code>keys</code> using the <a href="https://renugasaraswathy.wordpress.com/2014/09/03/splat-operator-in-ruby-2/">splat
operator</a>.
These “keys” are the <em>names</em> of the variables to add (or, more precisely, the
names of the methods which return the variable values). It then defines a
method, called <code>+</code>, which adds the values of these variables at each key and
creates an instance from the results using <code>self.class.new</code>.</p>
<p>Let’s see if it works. We’ll create a new class, <code>LineItem</code>, with different
reader names, and <code>extend</code> the module in this case rather than include it
(since <code>define_adder</code> needs to be a class method):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
extend <span style="color:#036;font-weight:bold">AdderDefiner</span>
define_adder(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we can add line item pairs, just like we did point pairs earlier:</p>
<div class="CodeRay">
<div class="code"><pre>
l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>)
l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>)
l1 + l2
<span style="color:#777">#=> #<LineItem:... @amount=25.98, @tax=3.9></span>
</pre></div>
</div>
<p>So it works! And we now have no dependency on variable names (or even on
number of variables), so our module is more flexible.</p>
<p>But there’s a problem. When we copy over our earlier logging code from the
<code>Point</code> class to log calls to <code>+</code>, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
extend <span style="color:#036;font-weight:bold">AdderDefiner</span>
define_adder(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">super</span>.tap { puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">"</span></span> }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>… and then add two line items again, instead of getting the expected logs,
things blow up with an exception:</p>
<div class="CodeRay">
<div class="code"><pre>
NoMethodError: super: no superclass method `+' for #<struct LineItem amount=9.99, tax=1.5>
</pre></div>
</div>
<p>What’s going on here?</p>
<p>Take a good look at how <code>define_adder</code> bootstraps the <code>+</code> method: it
uses <code>define_method</code> to add the method <em>directly to the class</em>, in this case
<code>LineItem</code>. A class can only have one definition of any given method, so when,
later in the class, we again define <code>+</code>, we clobber the earlier definition.
There is no “superclass” here since we have not included or inherited anything
when defining the method.</p>
<p>Solving this problem will require some “module building” magic. Let’s look at the
solution first, and then see how it solves this issue:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">AdderIncluder</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_adder</span>(*keys)
adder = <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span>
define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other|
<span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) }))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
include adder
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we replace <code>AdderDefiner</code> in the logged <code>LineItem</code> code above with
<code>AdderIncluder</code> and we find that it works:</p>
<div class="CodeRay">
<div class="code"><pre>
l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>)
l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>)
l1 + l2
<span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#777">#=> #<LineItem:... @amount=25.98, @tax=3.9></span>
</pre></div>
</div>
<p>The trick that makes this work is the use of an <em>anonymous module</em> in
<code>define_adder</code> to define the <code>+</code> method, which is different from the previous
technique of defining the method directly on the calling class.</p>
<p>Recall that in Ruby, any module (small “m”) is simply an instance of a class,
and this class is <a href="http://ruby-doc.org/core-2.4.1/Module.html">Module</a> (big
“M”). Like other classes, <code>Module</code> has a <code>new</code> method, in this case one which
<a href="http://ruby-doc.org/core-2.4.1/Module.html#method-c-new">can take a block
argument</a> and
evaluate the block in the context of the newly-created module.</p>
<p>So to generate a “module on the fly”, you can simply call <code>Module.new</code>, pass a
block with some method definitions, and use it just as you would any other
(named) module:</p>
<div class="CodeRay">
<div class="code"><pre>
mod = <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">with_foo</span>
<span style="color:#069">self</span> + <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20"> with Foo</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">The Ruby Module Builder Pattern</span><span style="color:#710">"</span></span>
title.extend mod
title.with_foo
<span style="color:#777">#=> "The Ruby Module Builder Pattern with Foo"</span>
</pre></div>
</div>
<p>This is a handy trick, often used in testing when you don’t want to clutter
the global namespace with a one-off module used only in a single test (works
for <code>Class</code> too).<a href="#note-6"><sup>6</sup></a></p>
<p>The method defined in the block in <code>AdderIncluder</code> is the same <code>+</code> method
defined earlier in <code>AdderDefiner</code>, and in this sense, the two modules perform
a very similar function. The key difference is that whereas <code>AdderDefiner</code>
defines the method <em>on</em> the class, <code>AdderIncluder</code> <em>includes a module</em> with the
method <em>into</em> the class.</p>
<p>If you look up the chain of ancestors, you will see this module just after the
<code>LineItem</code> class itself:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">LineItem</span>.ancestors
<span style="color:#777">#=> [LineItem,</span>
<span style="color:#777">#<Module:0x...>,</span>
...
</pre></div>
</div>
<p>Given this ancestor chain, it is clear why <code>super</code> works here: when you
call <code>+</code> on <code>LineItem</code>, the method calls <code>super</code>, which continues up the
class hierarchy to the anonymous module where our <code>+</code> method is defined. It
then calculates and returns the result, which gets composed in the <code>LineItem</code>
adder with the logging code.</p>
<p>Although at first glance it might seem somewhat strange and unfamiliar, this
technique of defining a bootstrap method to include a dynamically-generated
anonymous module is in fact extremely useful, and widely used for this reason.
This is particularly true for a flexible framework like Rails, where simply
<a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activerecord/lib/active_record/attribute_methods.rb#L13">defining a model</a>
or <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/actionpack/lib/action_dispatch/routing/mapper.rb#L652">setting up routes</a>
triggers calls to many bootstrap methods like <code>define_adder</code>. The technique
can also be found in other gems that require a similar level of flexiblity
(including <a href="https://github.com/shioyama/mobility">Mobility</a>).</p>
<p>Bootstrapped module building can thus be considered a very important
metaprogramming technique in Ruby. Indeed, it underpins the very dynamic
nature that enables frameworks like Rails to work the way they do.</p>
<p>That said, I hope to convince you here that the potential of module building
goes well beyond the bootstrapping technique shown above, and as such that it
is a pattern that deserves more attention. To do this, I’m going to return to
the Adder module one last time, and define it in a slightly different way.</p>
<h2>The Module Builder</h2>
<p>So far we’ve looked at three “adder” modules:</p>
<ol>
<li><code>Addable</code>, which adds a <code>+</code> method on two variables <em>x</em> and <em>y</em>,</li>
<li><code>AdderDefiner</code>, which adds a class method <code>define_adder</code> that
defines a <code>+</code> method on an arbitrary set of variables, and</li>
<li><code>AdderIncluder</code>, which also adds a class method <code>define_adder</code> that
defines the <code>+</code> method, but in such a way that the method can be composed
using <code>super</code>.</li>
</ol>
<p>Although not very flexible, note that in hindsight, the first <code>Addable</code> module
had some good things going for it:</p>
<ul>
<li>It only included the <code>+</code> method, and left no other bootstrapping
“artifacts” like <code>define_adder</code> (unlike both <code>AdderDefiner</code> and
<code>AdderIncluder</code>)</li>
<li>It allowed the parent class to access its methods using
<code>super</code> (like <code>AdderIncluder</code> but not <code>AdderDefiner</code>)</li>
<li>It had a <em>name</em>, and was therefore very easy to recognize in the chain of
ancestors. While <code>AdderDefiner</code> and <code>AdderIncluder</code> <em>themselves</em> appear in
the chain of ancestors, the fact that the bootstrap method was called, and
<em>how</em> it was called, is hard or impossible to discern.</li>
</ul>
<p>In this section, I will show how we can solve the adder problem in a way that
retains these benefits, while also offering the flexibility we sought in the last section.</p>
<div class="img-center" style="margin: 70px 0 40px 0">
<img style="padding-bottom: 20px" alt="A Self-Replicating Machine"
src="/img/selfreplicatingmachine-medium.png" />
<p class="caption">Module Builders</p>
</div>
<p>Let’s start by defining an <code>AdderBuilder</code> class which inherits from <code>Module</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">AdderBuilder</span> < <span style="color:#036;font-weight:bold">Module</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(*keys)
define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other|
<span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) }))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>This is a short but very dense little class. Before going into the details of
what it’s doing here, let’s first check that it works. We’ll include (not
extend) an <em>instance</em> of this class (a module) into <code>LineItem</code> this time:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
include <span style="color:#036;font-weight:bold">AdderBuilder</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
<span style="color:#080;font-weight:bold">end</span>
l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>)
l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>)
l1 + l2
<span style="color:#777">#=> #<LineItem:... @amount=25.98, @tax=3.9></span>
</pre></div>
</div>
<p>So, as we can see, this little class does the same thing that we achieved with
the <code>define_adder</code> bootstrap method in <code>AdderDefiner</code>, except <em>without ever
defining a class method to do it</em>.</p>
<p>How does it do this? First, it subclasses <code>Module</code>. Since we can call <code>new</code> on
<code>Module</code>, it’s not much of a stretch to imagine
that we can also subclass it and define our own methods such as
<code>initialize</code>. And indeed, as shown above, we <em>can</em> do this.</p>
<p>Here’s that initializer again:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(*keys)
define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other|
<span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) }))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>There are two levels at play here, and it’s important to recognize both of
them. On one level, we are initializing a new instance of a class, <code>Module</code>.
This instance is itself a module, so it’s a collection of methods we will mix
into other classes. We are then, as we initialize this module, defining one
of these methods, a method called <code>+</code>.</p>
<p>The method <code>+</code> itself is being defined <em>in terms of</em> the method names we have
passed as arguments to <code>Module.new</code> (the “keys”, which were <code>:amount</code> and
<code>:tax</code> in the example above).<a href="#note-7"><sup>7</sup></a> When we call
<code>self.class.new</code> <em>in</em> the method, <em>this</em> <code>new</code> will be evaluated in the context
of the class including this new module, and thus would evaluate to <code>LineItem</code>
(or <code>Point</code>, or whatever class the module was included into).</p>
<p>So <code>AdderBuilder.new(:amount, :tax)</code> evaluates to a module
functionally equivalent to this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
<span style="color:#069">self</span>.class.new(amount + other.amount, tax + other.tax)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>… which is what we had in <code>Addable</code>, with <code>x</code> and <code>y</code> swapped for <code>amount</code>
and <code>tax</code>. But the power here comes from the fact that we can define our
dynamic module in terms of <em>whatever variable names we want.</em></p>
<p>And herein lies the essential ingenuity of this pattern: that <strong>it allows you
to define modules not as fixed collections of methods and constants, but as
configurable prototypes for building such collections</strong>. There is no need for
class methods or other metaprogramming tricks to bootstrap the
module-inclusion step; the builder builds modules directly, so they can be
included in one step.</p>
<p>Not only that, but the modules created in this process, unlike the anonymous
modules generated by techniques described in the last section, actually have a
name, so you can clearly see them in the chain of ancestors:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">LineItem</span>.ancestors
[<span style="color:#036;font-weight:bold">LineItem</span>, <span style="color:#777">#<AdderBuilder:0x...>, Object, ... ]</span>
</pre></div>
</div>
<p>This is a nice by-product of the encapsulation we have achieved by subclassing
<code>Module</code>.<a href="#note-8"><sup>8</sup></a> With an anonymous module, this is much harder
to debug:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">LineItem</span>.ancestors
[<span style="color:#036;font-weight:bold">LineItem</span>, <span style="color:#777">#<Module:0x...>, Object, ... ]</span>
</pre></div>
</div>
<p>Module Builders thus have the benefits of anonymous modules (their instances
can be defined “on the fly” and do not clutter the global constant namespace),
are easily traceable since their instances have a name (like “normal”
modules), and yet do not require defining (and calling) new methods on the
including class to bootstrap them.</p>
<p>We can go a step further. We previously added some logging code to the
including class. This logging code can also be rewritten as a module builder,
which we will call <code>LoggerBuilder</code>:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LoggerBuilder</span> < <span style="color:#036;font-weight:bold">Module</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(method_name, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#710">"</span></span>, <span style="color:#606">end_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#710">"</span></span>)
define_method method_name <span style="color:#080;font-weight:bold">do</span> |*args, &block|
print start_message
<span style="color:#080;font-weight:bold">super</span>(*args, &block).tap { print end_message }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we can log <em>any included or inherited method</em> by creating an instance of
<code>LoggerBuilder</code> with the name of the method to be logged:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
include <span style="color:#036;font-weight:bold">AdderBuilder</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
include <span style="color:#036;font-weight:bold">LoggerBuilder</span>.new(<span style="color:#A60">:+</span>, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Enter Adder...</span><span style="color:#b0b">\n</span><span style="color:#710">"</span></span>,
<span style="color:#606">end_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Exit Adder...</span><span style="color:#b0b">\n</span><span style="color:#710">"</span></span>)
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>And, presto, we’ve got a logged addable line item:</p>
<div class="CodeRay">
<div class="code"><pre>
l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>)
l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>)
l1 + l2
<span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>...
<span style="color:#777">#=> #<LineItem:... @amount=25.98, @tax=3.9></span>
</pre></div>
</div>
<p>But of course, we can now also log any other included or inherited method we
like:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">LineItem</span>.include(<span style="color:#036;font-weight:bold">LoggerBuilder</span>.new(<span style="color:#A60">:to_s</span>, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Stringifying...</span><span style="color:#b0b">\n</span><span style="color:#710">"</span></span>))
l1.to_s
<span style="color:#036;font-weight:bold">Stringifying</span>...
=> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">#<struct LineItem amount=9.99, tax=1.5></span><span style="color:#710">"</span></span>
</pre></div>
</div>
<p>There are many other directions you can take this; I’ve only scratched the
surface here. For a relatively simple (yet powerful) example of this “in the
wild”, take a look at <a href="https://github.com/dry-rb/dry-equalizer">Dry
Equalizer</a>, which dynamically defines
equality methods on a class using an instance of an <code>Equalizer</code> class that,
like <code>AdderBuilder</code>, subclasses <code>Module</code>.</p>
<h2>Module Builders and Composition</h2>
<p>You may have noticed that I have used the word “compose” quite a lot in this
post with respect to including modules into a class. Any time you include a
module which overrides a method and calls <code>super</code>, you are in effect using
function (method) composition, although it is not generally referred to as
such.</p>
<p>The same is true of our logging code in the last section. When we include an
instance of both <code>AdderBuilder</code> <em>and</em> <code>LoggerBuilder</code>, we are building <code>+</code> as
a composite function, which we could alternatively write as:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">add</span>(other)
<span style="color:#069">self</span>.class.new(amount + other.amount, tax + other.tax)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">log</span>(result)
print <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">"</span></span>
result.tap { print <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">"</span></span> }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other)
log(add(other))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>So <code>super</code> is essentially taking the output of the last method call and
incorporating it into the result of the current module’s method.</p>
<p>Composition happens to be a very interesting way to “combine” different
instances of a module builder, like we combined logging and adding modules in
the example above. With module builders, we can configure and combine many
modules to build more complex methods and class behaviours.</p>
<p>One way to do this is via a “branching” method flow. We define multiple
modules that each override the same method, but only trigger some special
processing if a condition on the method and its arguments matches the module’s
state; otherwise, the control flow continues up the class hierarchy to the
next module via <code>super</code>.</p>
<p>If you’ve worked with Ruby for any length of time, this “branching
composition” should make you think of one method:
<a href="http://ruby-doc.org/core-2.4.1/BasicObject.html#method-i-method_missing">method_missing</a>.
Rarely does anyone override <code>method_missing</code> without a call to <code>super</code>
somewhere, since doing so would catch <em>any</em> method not defined on the class, which is
generally not what you want.</p>
<p>Instead, a typical <code>method_missing</code> override looks something like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">method_missing</span>(method_name, *arguments, &block)
<span style="color:#080;font-weight:bold">if</span> method_name =~ <span style="color:#036;font-weight:bold">METHOD_NAME_REGEX</span>
<span style="color:#777"># ... do some special stuff ...</span>
<span style="color:#080;font-weight:bold">else</span>
<span style="color:#080;font-weight:bold">super</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>… where <code>METHOD_NAME_REGEX</code> is a regex used to determine if we should “do
some special stuff”.</p>
<p>I actually use exactly this branching flow in
<a href="https://github.com/shioyama/mobility">Mobility</a> in a module builder called
<a href="https://github.com/shioyama/mobility/blob/2c35c8aab08bbbbf43be41d05cbeab01a22ed3d6/lib/mobility/fallthrough_accessors.rb">FallthroughAccessors</a>.
Each instance of <code>FallthroughAccessors</code> is initialized with a set of
attributes (its “state”) and intercepts method calls where the method name
is made up of one of these attributes plus a locale suffix (e.g.
<code>title_fr</code> for the <code>title</code> attribute in French), which I refer to as an “<a href="https://github.com/shioyama/i18n_accessors">I18n
accessor</a>”. If the method name
matches, the attribute value in the suffix locale is returned,
otherwise control continues up the class hierarchy.</p>
<p>Composing <code>method_missing</code> in module builders this way results in an
sequence of nested regex conditionals spread across a series of modules, which
I call “interceptors”. The fact that these interceptors are entirely
encapsulated, with no external dependencies on the class itself, enables
Mobility to “plug in” i18n accessors for each translated attribute
independently, at any time and in complete isolation from each other.</p>
<p>Having found this pattern useful in Mobility, I extracted it into a gem called
<a href="https://github.com/shioyama/method_found">MethodFound</a>. MethodFound is
essentially one module builder,
<a href="https://github.com/shioyama/method_found/blob/bcbd5a9d61455fee7cce3ce882077868c1115d82/lib/method_found/interceptor.rb">MethodFound::Interceptor</a>,
which intercepts method calls that match a regex or proc (the “state” in this
case) and passes the
method name, any matches captured in the regex (or the return value of the
proc), and any method arguments to a block (the “do some special stuff” in the
conditional above). This block is evaluated in the context of the instance of
the class including the module.</p>
<p>Here’s an example:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Greeter</span> < <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:name</span>)
include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">Interceptor</span>.new(<span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#D20">\A</span><span style="color:#808">say_([a-zA-Z_]+)</span><span style="color:#D20">\Z</span><span style="color:#404">/</span></span>) { |method_name, matches|
<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>name<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20"> says: </span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>matches[<span style="color:#00D">1</span>].gsub(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20"> </span><span style="color:#710">'</span></span>).capitalize<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">.</span><span style="color:#710">"</span></span>
}
include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">Interceptor</span>.new(<span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#D20">\A</span><span style="color:#808">scream_([a-zA-Z_]+)</span><span style="color:#D20">\Z</span><span style="color:#404">/</span></span>) { |method_name, matches|
<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>name<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20"> screams: </span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>matches[<span style="color:#00D">1</span>].gsub(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20"> </span><span style="color:#710">'</span></span>).capitalize<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">!!!</span><span style="color:#710">"</span></span>
}
include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">Interceptor</span>.new(<span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#D20">\A</span><span style="color:#808">ask_([a-zA-Z_]+)</span><span style="color:#D20">\Z</span><span style="color:#404">/</span></span>) { |method_name, matches|
<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>name<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20"> asks: </span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>matches[<span style="color:#00D">1</span>].gsub(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20"> </span><span style="color:#710">'</span></span>).capitalize<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">?</span><span style="color:#710">"</span></span>
}
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Here we have composed three interceptors on <code>method_missing</code>. No other trace
is left in <code>Greeter</code>, but you can peek at its ancestors to see directly the three
module builders along with their regex matchers (interceptors override
<code>Module#inspect</code> to show this):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Greeter</span>.ancestors
=> [<span style="color:#036;font-weight:bold">Greeter</span>,
<span style="color:#777">#<MethodFound::Builder:0x...>,</span>
<span style="color:#777">#<MethodFound::Interceptor: /\Aask_([a-zA-Z_]+)\Z/>,</span>
<span style="color:#777">#<MethodFound::Interceptor: /\Ascream_([a-zA-Z_]+)\Z/>,</span>
<span style="color:#777">#<MethodFound::Interceptor: /\Asay_([a-zA-Z_]+)\Z/>,</span>
<span style="color:#036;font-weight:bold">Object</span>, ...]
</pre></div>
</div>
<p>So when a method is called that the class does not know, it will bubble up
first to the “ask” interceptor and see if it matches the regex. If it does,
it will return a value; if not, it continues up to the “scream” interceptor and
checks against the second regex, and so on.</p>
<p>Here is the result:</p>
<div class="CodeRay">
<div class="code"><pre>
greeter = <span style="color:#036;font-weight:bold">Greeter</span>.new(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Bob</span><span style="color:#710">"</span></span>)
greeter.say_hello_world
<span style="color:#777">#=> "Bob says: Hello world."</span>
greeter.scream_good_morning
<span style="color:#777">#=> "Bob screams: Good morning!!!"</span>
greeter.ask_how_do_you_do
<span style="color:#777">#=> "Bob asks: How do you do?"</span>
</pre></div>
</div>
<p>Given this example using module builders and <code>method_missing</code>, here’s a
problem for you: how would you implement what we have shown above <em>without</em>
module builders, but with the same level of flexibility?</p>
<p>The first thing that happens when you take away the module builder is
that you lose the home for your module state – “normal” modules have no
place to put it. So those regexes and blocks in the interceptor
will need to go somewhere else, and <em>the only other natural place to put them
is the including class itself</em>.</p>
<p>There are problems with this, which a real-world example should make clear.</p>
<h2>Module Builders and Encapsulation</h2>
<p>Let’s now take everything we’ve learned so far and apply it to some real
living code at the core of Ruby’s most popular application framework.</p>
<p>Continuing from our discussion in the last section, I want to focus here on
how modules are typically used to configure state, where that state is
stored, and the problems with this approach to module configuration.
I hope to convince you that module builders offer a much more natural way to
encapsulate this state in the place where it best belongs: <em>in the module
itself</em>.</p>
<h3>ActiveModel::AttributeMethods</h3>
<p>The module we will look at is ActiveModel’s
<a href="http://guides.rubyonrails.org/active_model_basics.html#attribute-methods">AttributeMethods</a>,
which adds support for prefixed and suffixed attribute
methods to Rails models<a href="#note-10"><sup>10</sup></a>. If you have worked with Rails for any length
of time, you are probably familiar with these prefix/suffix patterns through
their use in change-tracking methods like <code>name_changed?</code> in
<a href="http://api.rubyonrails.org/classes/ActiveModel/Dirty.html">ActiveModel::Dirty</a>.
The module implements both module bootstrapping (like
<code>AdderIncluder</code>) and <a href="(https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L427">message
interception using <code>method_missing</code></a>
(like <code>MethodFound::Interceptor</code>).</p>
<p>Let’s start with a simple class that implements a method
prefix and a method affix:<a href="#note-11"><sup>11</sup></a></p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Person</span>
include <span style="color:#036;font-weight:bold">ActiveModel</span>::<span style="color:#036;font-weight:bold">AttributeMethods</span>
attribute_method_affix <span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">reset_</span><span style="color:#710">'</span></span>, <span style="color:#606">suffix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_to_default!</span><span style="color:#710">'</span></span>
attribute_method_prefix <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">clear_</span><span style="color:#710">'</span></span>
define_attribute_methods <span style="color:#A60">:name</span>
attr_accessor <span style="color:#A60">:name</span>, <span style="color:#A60">:title</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">attributes</span>
{ <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">name</span><span style="color:#710">'</span></span> => <span style="color:#33B">@name</span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">title</span><span style="color:#710">'</span></span> => <span style="color:#33B">@title</span> }
<span style="color:#080;font-weight:bold">end</span>
private
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">clear_attribute</span>(attr)
send(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>attr<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">=</span><span style="color:#710">"</span></span>, <span style="color:#069">nil</span>)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">reset_attribute_to_default!</span>(attr)
send(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>attr<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">=</span><span style="color:#710">"</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Default </span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>attr.capitalize<span style="font-weight:bold;color:#666">}</span></span><span style="color:#710">"</span></span>)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>This is how we would use these attribute methods:</p>
<div class="CodeRay">
<div class="code"><pre>
person = <span style="color:#036;font-weight:bold">Person</span>.new
person.name = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">foo</span><span style="color:#710">"</span></span>
person.name
<span style="color:#777">#=> "foo"</span>
person.clear_name
person.name
<span style="color:#777">#=> nil</span>
person.reset_name_to_default!
person.name
<span style="color:#777">#=> "Default Name"</span>
</pre></div>
</div>
<p>Just looking at the code above, it is clear that the module is storing state
somewhere, since it needs to know what prefixes, suffixes and affixes have
been defined for a given class. We can see where this state is stored in
the code for <code>attribute_method_prefix</code>, which is called in <code>Person</code> to setup a
method prefix:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">attribute_method_prefix</span>(*prefixes)
<span style="color:#069">self</span>.attribute_method_matchers += prefixes.map! { |prefix| <span style="color:#036;font-weight:bold">AttributeMethodMatcher</span>.new <span style="color:#606">prefix</span>: prefix }
undefine_attribute_methods
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>So the array of prefix strings are mapped to instances of a class, <code>AttributeMethodMatcher</code>,
stored in an array <code>attribute_method_matchers</code> on the <code>Person</code> class.
(The affix and suffix methods are similar.)
These matchers are the module’s core configuration, and they are <em>stored
outside the module itself</em>.</p>
<p>What’s in one of these matchers? Let’s have a look at <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L394-L399">the <code>initialize</code>
method</a>
of the class to get an idea:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(options = {})
<span style="color:#33B">@prefix</span>, <span style="color:#33B">@suffix</span> = options.fetch(<span style="color:#A60">:prefix</span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#710">"</span></span>), options.fetch(<span style="color:#A60">:suffix</span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#710">"</span></span>)
<span style="color:#33B">@regex</span> = <span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#808">^(?:</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span><span style="color:#036;font-weight:bold">Regexp</span>.escape(<span style="color:#33B">@prefix</span>)<span style="font-weight:bold;color:#666">}</span></span><span style="color:#808">)(.*)(?:</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span><span style="color:#036;font-weight:bold">Regexp</span>.escape(<span style="color:#33B">@suffix</span>)<span style="font-weight:bold;color:#666">}</span></span><span style="color:#808">)$</span><span style="color:#404">/</span></span>
<span style="color:#33B">@method_missing_target</span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span><span style="color:#33B">@prefix</span><span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">attribute</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span><span style="color:#33B">@suffix</span><span style="font-weight:bold;color:#666">}</span></span><span style="color:#710">"</span></span>
<span style="color:#33B">@method_name</span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>prefix<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%s</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>suffix<span style="font-weight:bold;color:#666">}</span></span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>So essentially, a matcher is two things: a prefix and suffix pair, and a regex
generated from them (the rest is mostly for convenience). This internal state
is used for two different but related purposes.</p>
<p>The first of these two purposes is important but not well-documented at all; you can
find it explained inline in a comment
<a href="https://github.com/rails/rails/blob/46adb59b3474351e95ddc2357e67d169d6ffec51/activemodel/lib/active_model/attribute_methods.rb#L417-L422">here</a>.
The purpose is to provide a way to convert a hash of keys and
values returned from an <code>attributes</code> method into a set of first-class
attribute methods, supporting all defined prefix, suffix and affix method
patterns. In the example above, <code>attributes</code> returns a hash with keys <code>name</code>
and <code>title</code>, so these become attributes that dispatch to methods like
<code>clear_attribute</code> and <code>reset_attribute_to_default!</code>.</p>
<p>Like <a href="https://github.com/shioyama/method_found">MethodFound</a>, this is implemented using <code>method_missing</code>: any method call
which matches a matcher regex <em>and</em> whose target (captured from the
regex) is a key in <code>attributes</code> is <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L427">intercepted and dispatched to an attribute
handler</a>.
So <code>reset_title_to_default!</code> is dispatched to <code>reset_attribute_to_default!</code>,
with the name of the attribute (<code>"title"</code>) as its argument.</p>
<p>Method interception is great for an open-ended, changing hash of
attributes, but <code>method_missing</code> is slow as molasses. If you know what attributes
you want, it’s generally better to define the methods explicitly.</p>
<p>This is the second purpose of the method matchers, triggered
by calling <code>define_attribute_methods</code> with one or more attribute names <em>after</em>
setting up prefixes, suffixes and affixes: to define methods for each
combination of prefix and/or suffix on each attribute. <code>Person</code> defines
methods for <code>name</code>, so <code>clear_name</code> and <code>reset_name_to_default!</code> dispatch to
<code>clear_attribute</code> and <code>reset_attribute_to_default!</code>, respectively.</p>
<p>These methods are bound to an anonymous module <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L332">defined and
included</a>
in another method added to the <code>Person</code> class, named <code>generated_attribute_methods</code>. As instance methods, they take
precedence over method interception, so while both
<code>clear_title</code> and <code>clear_name</code> will return the same result, the former falls
through to <code>method_missing</code>, whereas the latter is handled (much more quickly)
by the attribute method.</p>
<div class="img-center" style="margin-top: 30px; margin-bottom: 30px">
<img style="margin-bottom: 20px" alt="ActiveModel AttributeMethods" src="/img/am-attributemethods.png"/>
<p style="text-align:center"><small>ActiveModel's AttributeMethods</small></p>
</div>
<p>You can see the defined attribute methods if you grep the methods on an
<code>Person</code> instance:</p>
<div class="CodeRay">
<div class="code"><pre>
person = <span style="color:#036;font-weight:bold">Person</span>.new
person.methods.grep <span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#808">name</span><span style="color:#404">/</span></span>
<span style="color:#777">#=> [:name, :name=, :reset_name_to_default!, :clear_name, ...]</span>
</pre></div>
</div>
<p>You can confirm these generated methods are defined on the module, and not somewhere else,
by grabbing the module and checking its instance methods:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Person</span>.ancestors
<span style="color:#777">#=> [Person, #<Module:Ox...>, ActiveModel::AttributeMethods, ...]</span>
<span style="color:#036;font-weight:bold">Person</span>.ancestors[<span style="color:#00D">1</span>].instance_methods
<span style="color:#777">#=> [:name, :name=, :reset_name_to_default!, :clear_name]</span>
</pre></div>
</div>
<p>You can also find the <code>method_missing</code> override described earlier
<a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L427">here</a>
on <code>ActiveModel::AttributeMethods</code>, just after the anonymous module in the class
hierarchy.</p>
<p>These two implementations of attribute methods, one using method interception
with <code>method_missing</code> and the other using method definition, complement each
other since they are suited to different access patterns (one for a large
open-ended set of attributes, the other for a smaller fixed set of
attributes). Mobility actually uses the same approach in defining i18n
accessors.<a href="#note-12"><sup>12</sup></a> However, whereas AttributeMethods adds a
dozen class methods and variables, Mobility adds none, instead hiding all
state in its modules.</p>
<p>Let’s now look at an alternative implementation of AttributeMethods which,
like Mobility, uses module builders to encapsulate state in the modules it builds.</p>
<h3>MethodFound::AttributeMethods</h3>
<p>This is where we bring all the ideas discussed so far together and apply them
to a real application. Before jumping into more code, let’s review the
elements implementing attribute methods on a class:</p>
<ol>
<li>a <strong>class variable</strong> (<code>attribute_method_matchers</code>) holding an array of
matchers for prefixes/suffixes/affixes</li>
<li>a <strong>method_missing override</strong> to dispatch method calls matching the
attribute matchers and the <code>attributes</code> hash to handlers accordingly</li>
<li>an included <strong>anonymous module</strong> (<code>generated_attribute_methods</code>) holding
defined attribute methods</li>
</ol>
<p>Now consider that <em>each of these core elements is located in a different
place</em>:</p>
<ul>
<li>the method matchers are defined on the <strong>including class</strong></li>
<li>the generated attribute methods are defined on an <strong>anonymous module</strong> included in
this class</li>
<li><code>method_missing</code> is defined on <strong>AttributeMethods itself</strong>, also included in
the class</li>
</ul>
<p>So we have the three elements of our implementation, coupled to each other
through the method matchers class variable, spread out across different parts
of the implementing system. Not only that, but the method matchers and
generated methods, which (as we will see below) are essentially functionally
independent, are stored together in the <em>same place</em> (an array and an
anonymous module, respectively).</p>
<p>Let’s consider an alternative implementation which distributes state
in a different way. In this implementation, we will group the following
elements together into a single module using a module builder:</p>
<ul>
<li>a single prefix/suffix pair, from which a matcher regex is generated</li>
<li>a single <code>method_missing</code> override which uses the regex to intercept methods</li>
<li>a method to define attribute methods for the given prefix/suffix pair</li>
</ul>
<p>The module builder is named
<a href="https://github.com/shioyama/method_found/blob/9af678d8ff0400eb33f275cddc66ec3fd6d16ee8/lib/method_found/attribute_interceptor.rb">AttributeInterceptor</a>.
It inherits from
<code>MethodFound::Interceptor</code>, which we saw in the last section, customizing the
initializer so that it accepts a prefix and/or suffix instead of the more
open-ended format of the default interceptor builder. Although it is not very
big, I won’t include the actual code here so we can instead focus on what the
class actually does.</p>
<div class="img-center" style="margin-top: 30px; margin-bottom: 30px">
<img style="margin-bottom: 20px" alt="MethodFound AttributeMethods" src="/img/mf-attributemethods.png"/>
<p style="text-align:center"><small>Implementing AttributeMethods with MethodFound interceptors</small></p>
</div>
<p>Let’s first take a look at intercepting. We can reproduce this by replacing
calls to <code>define_attribute_prefix</code> and <code>define_attribute_affix</code> in the
<code>Person</code> class above with attribute interceptors, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Person</span>
include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">AttributeInterceptor</span>.new(<span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">clear_</span><span style="color:#710">'</span></span>)
include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">AttributeInterceptor</span>.new(<span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">reset_</span><span style="color:#710">'</span></span>, <span style="color:#606">suffix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_to_default!</span><span style="color:#710">'</span></span>)
attr_accessor <span style="color:#A60">:name</span>, <span style="color:#A60">:title</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">attributes</span>
{ <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">name</span><span style="color:#710">'</span></span> => <span style="color:#33B">@name</span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">title</span><span style="color:#710">'</span></span> => <span style="color:#33B">@title</span> }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#777"># ...</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>We are instantiating two attribute interceptors and including each of them
into the class. There is no special case here for “affix”; we simply build an
interceptor with both a prefix and suffix.</p>
<p>With these modules included, <code>Person</code> behaves externally exactly the same as the earlier version
using the ActiveModel module, but the internal implementation is quite
different. This can be seen from its ancestors:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Person</span>.ancestors
<span style="color:#777">#=> [Person,</span>
<span style="color:#777">#<MethodFound::AttributeInterceptor: /\A(?:reset_)(.*)(?:_to_default!)\z/>,</span>
<span style="color:#777">#<MethodFound::AttributeInterceptor: /\A(?:clear_)(.*)(?:)\z/>,</span>
... ]
</pre></div>
</div>
<p>Like the MethodFound interceptors described earlier, these two modules
each override <code>method_missing</code> and branch the code path if the method name
matches their regex (stored on the module).</p>
<p>The resulting set of distributed
nested conditionals is equivalent to <a href="https://github.com/rails/rails/blob/23aa0a2bb551717f153ac75f24c017c43ab853f2/activemodel/lib/active_model/attribute_methods.rb#L360">this line from
ActiveModel</a>,
which iterates through attribute method matchers checking for matches:</p>
<div class="CodeRay">
<div class="code"><pre>
matchers.map { |method| method.match(method_name) }.compact
</pre></div>
</div>
<p>The key difference between these two implementations is that whereas this line
is executed in <code>ActiveModel::AttributeMethods</code> (a module), using a class variable
stored in <code>Person</code> (a class), the MethodFound version is executed <em>across</em>
independent modules through method composition, using <code>super</code>.</p>
<p>The implementation of generated attribute methods is similar.
<code>AttributeInterceptor</code> has a method <code>define_attribute_methods</code> which takes
one or more attribute names and defines attribute methods for each of them
with the module’s prefix and/or suffix, <em>on the module itself</em>. Again, because
the module already contains its own prefix and suffix, it has all the
information it needs to do this.</p>
<p>So functionality is truly encapsulated: a single module contains its own
prefix and suffix, the <code>method_missing</code> override to catch attribute method
calls, and a method to generate its own attribute method.<a href="#note-13"><sup>13</sup></a></p>
<p>Given this module builder, we can reproduce ActiveModel’s
implementation without any class methods or class variables, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Person</span>
[ <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">AttributeInterceptor</span>.new(<span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">clear_</span><span style="color:#710">'</span></span>),
<span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">AttributeInterceptor</span>.new(<span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">reset_</span><span style="color:#710">'</span></span>, <span style="color:#606">suffix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_to_default!</span><span style="color:#710">'</span></span>)
].each <span style="color:#080;font-weight:bold">do</span> |mod|
mod.define_attribute_methods(<span style="color:#A60">:name</span>)
include mod
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#777">#...</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>MethodFound includes another module, <code>MethodFound::AttributeMethods</code>,
which adds class methods do simplify the code above, so it can be used as a
drop-in replacement for <code>ActiveModel::AttributeMethods</code>. The
<a href="https://github.com/shioyama/method_found/blob/9532921d24d110e735d276c7e354c5dc0d0b40b1/lib/method_found/attribute_methods.rb#L65-L69">implementation</a> of
<code>define_attribute_methods</code> in this module is interesting:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_attribute_methods</span>(*attr_names)
ancestors.each <span style="color:#080;font-weight:bold">do</span> |ancestor|
ancestor.define_attribute_methods(*attr_names) <span style="color:#080;font-weight:bold">if</span> ancestor.is_a?(<span style="color:#036;font-weight:bold">AttributeInterceptor</span>)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Since the class no longer stores the matchers in its state, the module cannot
simply iterate through these matchers to define methods. Instead, it <em>iterates
through its own ancestors</em>, which are module instances each with a
prefix/suffix pair, and calls <code>define_attribute_methods</code> if the ancestor is an
attribute interceptor. This generates attribute methods on each interceptor
module, which can then be called by instances of the class.</p>
<p>The resulting implementation encapsulates related elements together into
separate modules without coupling via class variables or methods. This
encapsulation means that we could now design totally different types of
attribute interceptors and include them in the same way; as long as they have
the same interface (for generating attribute methods), the internals can be
entirely different and nothing else will need to change.</p>
<p>This, it seems to me, is what a “module” should really be: an
independent, interchangeable unit containing everything it needs to execute
only one aspect of some desired functionality. Not only that, but the
interface for building these modules falls directly out of Ruby’s own object
model, a model that has been around as long as the language itself.
Ruby has had this trick up its sleeve for years, <em>most of us just never
realized it</em>.</p>
<h2>What’s in a Name?</h2>
<p>You may argue that the “Module Builder Pattern” I’ve introduced is just a Ruby
subclass where the class happens to be <code>Module</code>, and what’s the big deal? And
technically, you would be right. I’m not the first to notice this idea, nor am
I the first to write about it. Just <code>Foo < Module</code>, and I could be done with it.</p>
<p>But programming languages are written and read by and for humans, and to
humans, names mean a lot. If you can’t find the module with the methods on it
buried in a list of ancestors, you won’t notice it, and if you can’t remember
that subclassing thing some blogger wrote about, you won’t use it. The fact
that this pattern has been around for this long without almost anybody
knowing about it is testament to this fact.</p>
<p>So I gave it a name. A catchy one.</p>
<p>Because as Rubyists, <strong>I think we need this pattern</strong>. Take a look at the code
of a large Ruby project and how modules are used (and abused) in practice. Our
modules trigger callbacks that bootstrap all kinds of extra state into our
classes, coupling them to the modules they include. A module should be
self-contained, but we use the excuse that we cannot configure modules at
runtime to justify storing configuration all over the place.</p>
<p>Ruby can do better than this, and we can do better than this. So take another
look at that bloated module, the one with the tentacles that seem to have
crept over every corner of your application, and give it a chance to be free.
I think you will find that it — and you — will be all the better for it.</p>
<hr />
<h4>Update (2017/10/20)</h4>
<p><small>This post has been translated into Japanese: see <a href="https://techracho.bpsinc.jp/hachi8833/2017_10_12/46276">part
1</a>, <a href="https://techracho.bpsinc.jp/hachi8833/2017_10_19/46619">part
2</a> and <a href="https://techracho.bpsinc.jp/hachi8833/2017_10_26/46623">part
3</a>.</small></p>
<h4>Update (2017/09/27)</h4>
<p><small>Since writing this blog post, I have also
<a href="https://www.youtube.com/watch?v=_E1yKPC-r1E">presented</a> the Module Builder Pattern at
<a href="http://rubykaigi.org/2017/presentations/shioyama.html">RubyKaigi
2017</a> in Hiroshima.
Slides from that talk can be found <a href="https://speakerdeck.com/shioyama/the-ruby-module-builder-pattern">on Speaker
Deck</a>, and a
video has also been posted to
<a href="https://www.youtube.com/watch?v=_E1yKPC-r1E">YouTube</a>.</p>
<h4>Notes</h4>
<p><small><a name="note-1"><sup>1</sup></a> The first reference I can find to the
idea of “subclassing Module” is in an <a href="http://solnic.codes/2012/08/13/subclassing-module-for-fun-and-profit">article in
2012</a>
by Piotr Solnica. <a href="https://medium.com/@eric.programmer/arguments-for-included-modules-in-ruby-8056b9fa2743">Eric
Anderson</a> and <a href="https://makandracards.com/makandra/43002-subclassing-module">Andreas
Robecke</a> have written about it more
recently, but hardly anything else even mentions it.</small><br />
<small><a name="note-2"><sup>2</sup></a> At the time of writing this post, it
is used
<a href="https://github.com/rails/rails/blob/0dd40fe72cfcf53b475a4084d61e9e931417c29e/activemodel/lib/active_model/validations/acceptance.rb#L25">here</a>
and
<a href="https://github.com/rails/rails/blob/0dd40fe72cfcf53b475a4084d61e9e931417c29e/activemodel/lib/active_model/type/helpers/accepts_multiparameter_time.rb#L4">here</a>.
</small><br />
<small><a name="note-3"><sup>3</sup></a> By this I mean you can include
modules on over another and use <code>super</code> to build complex composed method
chains. I do a lot of this in Mobility.</small><br />
<small><a name="note-4"><sup>4</sup></a> And, to be precise, which also has an
initializer which takes the values of these attributes as its
arguments.</small><br />
<small><a name="note-5"><sup>5</sup></a> I’ve used a trick here to add a
module’s methods as class methods to a class using <code>include</code>, by extending the
class when the module is included. This is a common trick, if it looks a bit
mysterious you might want to read up on the technique before
continuing.</small><br />
<small><a name="note-6"><sup>6</sup></a> This “no-clutter” property of
anonymous modules is also what makes them so handy in metaprogramming, since
it means you don’t need to worry about constant name collisions.</small><br />
<small><a name="note-7"><sup>7</sup></a> It is actually being defined from
these keys using a
<a href="https://en.wikipedia.org/wiki/Closure_(computer_programming">closure</a>.</small><br />
<small><a name="note-8"><sup>8</sup></a> Defining an <code>inspect</code> method in
the module builder class which includes the variables (e.g. <code>amount</code> and
<code>tax</code>) which define makes this ancestor chain even easier to
read.</small><br />
<small><a name="note-9"><sup>9</sup></a> MethodFound actually does a bit more
than this: it also “caches” the method name matched in the regex or proc and
defines it on the interceptor module, so that future calls will be much
faster. You can see this if you look at the methods on an interceptor after it
has matched a method using <code>method_missing</code>.</small><br />
<small><a name="note-10"><sup>10</sup></a> ActiveRecord models in fact include
<code>ActiveRecord::AttributeMethods</code>, which itself includes
<code>ActiveModel::AttributeMethods</code> and overrides some of the methods discussed
here, specifically <code>define_attribute_methods</code>. The relationship between these
two modules, and between how persisted and non-persisted attribute methods are
handled, is somewhat complex, but the ideas discussed here are relevant to
both.</small><br />
<small><a name="note-11"><sup>11</sup></a> Note that I have slightly modified
and simplified this example class from the one in the inline documentation to
highlight some things that the standard docs do not mention.</small><br />
<small><a name="note-12"><sup>12</sup></a> Mobility handles these two cases
with the module builders <code>FallthroughAccessors</code> and
<code>LocaleAccessors</code>. I also extracted these from Mobility into a gem called
<a href="https://github.com/shioyama/i18n_accessors">I18nAccessors</a>. </small><br />
<small><a name="note-13"><sup>13</sup></a> It actually also has a method
<code>alias_attribute</code> to alias attributes, like
<code>ActiveModel::AttributeMethods</code>.</small></br></p>
<h4>Figures</h4>
<p><small><sup>a</sup> “Drawing Hands” by M.C. Escher.
[<a href="https://en.wikipedia.org/wiki/Drawing_Hands">reference</a>]</small><br />
<small><sup>b</sup> “Automated space exploration and
industrialization using self-replicating systems.” From <em>Advanced Automation
for Space Missions: 5. REPLICATING SYSTEMS CONCEPTS: SELF-REPLICATING LUNAR
FACTORY AND DEMONSTRATION</em>.
[<a href="http://www.islandone.org/MMSG/aasm/chapter5.htm">reference</a>]</small><br />
<small><sup>c</sup> “Proposed demonstration of simple
robot self-replication.”
[<a href="https://en.wikipedia.org/wiki/Self-replicating_machine">reference</a>]</small></p>
Translating with Mobilityhttps://dejimata.com/2017/3/3/translating-with-mobility2017-03-032017-03-03Chris Salzberg<p>In the first real post on this blog, I described my experience <a href="/2013/10/11/hacking-globalize">hacking
globalize</a>, a
<a href="https://rubygems.org/gems/globalize">gem</a> that calls itself the “Rails
I18n de-facto standard library for ActiveRecord model/data translation.”</p>
<p>Globalize was the first gem that I contributed to, and the experience
was very important to me as a step toward becoming
a better and more knowledgeable programmer.</p>
<p>I am now one of the authors of that gem, having made <a href="https://github.com/globalize/globalize/commits?author=shioyama">many
commits</a> to
the repository. In the time since I first contributed to Globalize, I’ve
become keenly aware of the limitations of the gem, and of problems with
existing solutions for managing translations in Rails projects.</p>
<p>I’d like to look back in this post to the problem of translation and
to solutions devised so far, as motivation to introduce a
gem I’ve just released called <a href="https://rubygems.org/gems/mobility">Mobility</a>.
Mobility attempts to solve some core problems with the multitude
of Ruby-based translation gems out there right now.</p>
<h2>The Problem</h2>
<p>You have an application with user-generated content. You want to translate
that content into multiple languages. And – importantly – you want to do it
in such a way that you don’t need to recode all your existing presentation logic
(views, mailers, controllers, etc.) to handle the new languages.</p>
<p>In other words, things should <em>just work</em> – more or less the same as they do
with one language, except that switching
the language should tell the application to switch context to read or write
content stored in that language.</p>
<p>So this:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
post.title
</pre></div>
</div>
<p>… should return the title of the post, <em>in the language I am currently
viewing</em>.</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:en</span>
post.title
<span style="color:#777">#=> "Mobility (noun): quality of being changeable, adaptable or versatile"</span>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span>
post.title
<span style="color:#777">#=> "Mobility(名詞):動きやすさ、可動性"</span>
</pre></div>
</div>
<p>Likewise, I should be able to write translated values the same way:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:en</span>
post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>
post.save
</pre></div>
</div>
<p>Now the English title has changed, but the Japanese one has not:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
post.title
<span style="color:#777">#=> "Translating with Mobility"</span>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span>
post.title
<span style="color:#777">#=> "Mobility(名詞):動きやすさ、可動性"</span>
</pre></div>
</div>
<p>For bonus points, this solution should also work for querying attributes, so
something similar to this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>)
</pre></div>
</div>
<p>… should work even if <code>title</code> is a translated attribute.</p>
<p>For this to happen, something has to hook in to attribute accessors,
to detect changes in the locale and return language-specific content
accordingly. This may seem easy to do at first glance, but it gets tricky.</p>
<p>Notice that what we’re asking for is similar but different from the problem
solved by localization solutions such as the <a href="http://guides.rubyonrails.org/i18n.html">Rails I18n
API</a>, which
provide namespaced translation lookup (generally) for interface translation,
typically with static data.</p>
<p>With application content, we want translations at the class level, in the form
of persisted attributes of a model. Ultimately we’re talking about a
storage strategy.</p>
<p>As it turns out, there are a number of different strategies that different
people have come up with to solve this problem. Let’s take a look at them.</p>
<h2>Solutions</h2>
<h3><a name="strategy-1"></a>Strategy 1: Translatable Columns</h3>
<p>The most obvious solution to this problem is to create columns on your model
table in each locale you want to support, with some convention for identifying
the locale of the column.</p>
<p>Typically, this is done by appending a suffix to the column name made up of an
underscore plus the locale, with any dashes in the locale converted to underscores.</p>
<p>So <code>title</code> becomes <code>title_en</code>, <code>title_fr</code>, <code>title_ja</code>, and so on. Adding a new
locale to an application then requires that you add columns for that locale to
<em>each and every translated model table</em> (see figure below.)</p>
<div class="img-center">
<img alt="Column Strategy" src="/img/column_strategy.png"></a>
<p class="caption">Table structure of column strategy</p>
</div>
<p>For a small number of models, and a small number of locales, this is a
reasonable solution. It has the advantage that the column structure is easy to
understand, and querying by translated attribute is straightforward:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Mobility</span><span style="color:#710">"</span></span>)
<span style="color:#777">#=> #<Post:0x... id: 123 content_en: "Mobility" content_ja: "モビリティ"...></span>
<span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title_ja</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">モビリティ</span><span style="color:#710">"</span></span>)
<span style="color:#777">#=> #<Post:0x... id: 123 content_en: "Mobility" content_ja: "モビリティ"...></span>
</pre></div>
</div>
<p>Some key disadvantages of this approach:</p>
<ul>
<li>For many models and many locales, this approach requires many migrations,
each time a locale or model is added.</li>
<li>Space is not used very efficiently: every record has a column for every
locale, even if the locale is not used.</li>
</ul>
<p>Despite these disadvantages, this strategy is appropriate for applications
with a fixed number of locales. For a very nice implementation of this
approach, see <a href="https://github.com/barsoom/traco">Traco</a>.</p>
<p><em>Mobility supports this strategy with its <code>:column</code> backend.</em></p>
<h3><a name="strategy-2"></a>Strategy 2: Model Translation Tables</h3>
<p>Now consider that we have an application where we have many translated models,
with a potentially open-ended list of locales we want to support. A
service with user-generated content offered to users around the world
would be a good example of this pattern. In this case, the first strategy
becomes impractical as the number of locales grows; with, say, 50 locales, and
only a small number of actually translations, the vast majority of translated
columns would be null. On top of which, adding new locales to all the
translated models would be a pain, not to mention error-prone.</p>
<p>This brings us to the next strategy, shown in the database schema below. Here,
each model table is assigned its own translations table. Each row in the
translations table has:</p>
<ul>
<li>A column for the locale (<code>locale</code>)</li>
<li>A foreign key pointing to the model table (<code>post_id</code>)</li>
<li>One or more columns for translations (<code>title</code>, <code>content</code>)</li>
</ul>
<p>With this structure, we can now <em>join</em> the translation table (e.g.
<code>post_translations</code>) to the model table (e.g. <code>posts</code>) on the locale we want
to get the desired result: a model with translations in that locale.</p>
<div class="img-center">
<img alt="Model Translation Strategy" src="/img/globalize_strategy.png"></a>
<p class="caption">Table structure of model translation strategy</p>
</div>
<p>In ActiveRecord, we would have an association on the model <code>Post</code> to the model
<code>PostTranslation</code>, such that we can fetch translations with
<code>post.translations</code> and pick out the one we want:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
translation = post.translations.find <span style="color:#080;font-weight:bold">do</span> |translation|
translation.locale == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">en</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#777">#=> #<PostTranslation:0x... id: 123 locale: "en" ...></span>
translation.title
<span style="color:#777">#=> "Translating with Mobility"</span>
translation.content
<span style="color:#777">#=> "In the first real post on this blog..."</span>
</pre></div>
</div>
<p>With this approach, we’ve nicely solved some of the problems with the first
strategy:</p>
<ul>
<li>Since translations now have their own model and table, and we only create
rows in that table when we save a translation in that locale, space usage
is more efficient (no wasted columns/rows for locales that have no
translation).</li>
<li>We do not need any migrations to add locales: simply save the translation
for a new locale and it becomes available. So scalability on the “locale
axis” is very good.</li>
</ul>
<p>However, there are some clear disadvantages with this approach that should be
kept in mind:</p>
<ul>
<li>Having translations on a different table implies a lot of joining, which
(if not managed very carefully) can lead to performance degradation.</li>
<li>Joining can also add significant complexity to the code dealing with
translations. In particular, querying on translated attributes is
much more complicated than it was with the first strategy, where we
queried on the model column directly.</li>
</ul>
<p>Note also that although this strategy scales nicely along the “locale axis”,
it nonetheless requires a table for every translated model, and thus a
migration every time we add a new model or a new translated attribute to an
existing model. So this strategy may not scale well if an application has many
different forms of translated content, spread across a large and growing list
of models.</p>
<p>This strategy is implemented by
<a href="https://github.com/globalize/globalize">Globalize</a>, the gem mentioned at the
beginning of this post (it is also implemented by
<a href="https://github.com/francesc/rails-translate-models">many</a>
<a href="https://github.com/jo/puret">other</a>
<a href="https://github.com/janne/model_translations">gems</a>). For the reasons mentioned above,
Globalize has become the
preferred option for model translation in Rails. Like the first strategy, it
is a very useful approach for many (but not all) use cases.</p>
<p><em>Mobility supports this strategy with its <code>:table</code> backend.</em></p>
<h3>Strategy 3: Shared Translation Tables</h3>
<p>My experience working with Globalize, and in particular with scalability
issues mentioned above, prompted me to consider another strategy which would
scale better with model creation. I wanted a strategy which would be usable
<em>immediately</em> at the code level, without having to add a table or column every
time a new model or translated attribute is added.</p>
<p>This led me to the third strategy, which I’ve called the “key value” strategy.
The basic idea is simple: rather than using a dedicated table for each
translated model, instead create shared tables to store translations across
all models (as shown in the figure below). Two tables are required since we
will be storing two types of values: strings and text. (If you wanted to store
other column types, you could add more tables for each of those types, or cast
strings to some other type).</p>
<div class="img-center">
<img alt="KeyValue Strategy" src="/img/key_value_strategy.png"></a>
<p class="caption">Table structure of key value strategy</p>
</div>
<p>The key thing here is that rather than using a normal association as we did in
the last strategy, where <code>post_id</code> on the <code>post_translations</code> table pointed to
the <code>posts</code> table, we will instead use a <a href="http://guides.rubyonrails.org/association_basics.html#polymorphic-associations">polymorphic
association</a>.
By doing this, we can share the translations tables with as many models as we
like, migrating just once to create the shared tables.</p>
<p>The new tables have the following columns:</p>
<ul>
<li>A column for the locale (<code>locale</code>)</li>
<li>A foreign key pointing to the translated model id (<code>translatable_id</code>)</li>
<li>A column holding the model class (<code>translatable_type</code>)</li>
<li>A column holding the name of the attribute being translated (<code>key</code>)</li>
<li>A column holding the value of the translated attribute (<code>value</code>)</li>
</ul>
<p>With these tables in place, we define a polymorphic relationship on the model,
something like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
has_many <span style="color:#A60">:text_translations</span>,
<span style="color:#606">as</span>: <span style="color:#A60">:translatable</span>,
<span style="color:#606">class_name</span>: <span style="color:#036;font-weight:bold">TextTranslation</span>,
<span style="color:#606">inverse_of</span>: <span style="color:#A60">:translatable</span>
<span style="color:#777">#...</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>and a translation class:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">TextTranslation</span>
<span style="color:#069">self</span>.table_name = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">"</span></span>
belongs_to <span style="color:#A60">:translatable</span>, <span style="color:#606">polymorphic</span>: <span style="color:#069">true</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now that we have these models in place, we can easily fetch and set
translations of a model class:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
post.text_translations.find <span style="color:#080;font-weight:bold">do</span> |translation|
translation.key == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span> && translation.value == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#777">#=> #<TextTranslation:0x...</span>
<span style="color:#606">id</span>: <span style="color:#00D">123</span>
<span style="color:#606">locale</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">en</span><span style="color:#710">"</span></span>
key=<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span>
value=<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>
... >
</pre></div>
</div>
<p>To add translations to a new model, we simply add a polymorphic relationship
like the one above on <code>Post</code>, and we can immediately add translations to the
model.</p>
<p>In practice, there are some other issues that need to be handled to make this
work effectively, but done correctly this strategy has some notable benefits:</p>
<ul>
<li>No more migrations (after the first one), so an application can quickly add
new translated models.</li>
<li>As with Strategy 2, no extra effort to add new locales.</li>
<li>Relatively efficient in terms of storage, in the sense that we only create
records for translations in locales where they actually exist. However,
since each attribute translation gets a row in the table, for multiple
attributes on a single translation table, Strategy 2 may be more space
efficient since it stores translated attributes together in a single row.</li>
</ul>
<p>Now for the disadvantages:</p>
<ul>
<li>The join complexity described in the last section is even worse now, because
we have to join <em>every attribute separately</em> when searching for matches.
This is quite tricky but can be done. I’ll probably write about this in a
future post.</li>
<li>Since we have each attribute stored separately (as a key/value pair) it
becomes trickier to maintain our database in a consistent state. For
example, if we remove a model, its translations will remain in the
translations table(s) unless we explicitly remove them. Likewise, if we
decide to rename a translated attribute, we may end up with translations for
the previous attribute name leftover in the table.</li>
</ul>
<p>Despite these disadvantages, I consider this strategy to be applicable to
the widest variety of situations. It is the default strategy in Mobility,
implemented as the <code>:key_value</code> backend.</p>
<h3>Strategy 4: Serialized Data</h3>
<p>Yet another set of strategies take the approach of jamming all the
translations for an attribute into a single column. The first strategy
following this approach is one which serializes translations as a hash and
stores it on a text column.</p>
<p>At the model layer, this is pretty trivial in Rails:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
serialize <span style="color:#A60">:title_translations</span>, <span style="color:#036;font-weight:bold">Hash</span>
<span style="color:#080;font-weight:bold">end</span>
post = <span style="color:#036;font-weight:bold">Post</span>.new
post.title_translations = { <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">en</span><span style="color:#710">"</span></span> => <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span> }
post.save
</pre></div>
</div>
<p>We can now fetch translations simply by fetching the value for the key
matching a given locale:</p>
<div class="CodeRay">
<div class="code"><pre>
post.title_translations[<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">en</span><span style="color:#710">"</span></span>]
<span style="color:#777">#=> "Translating with Mobility"</span>
</pre></div>
</div>
<p>This strategy has the obvious advantage of simplicity: all our translations
are stored together on the attribute column. Like the model translations and
key-value strategies, locale scalability is not an issue since we can just add
new hash keys as we need them.</p>
<p>In addition, no additional migrations are necessary; simply adding a text
column to every model for each translated attribute is enough to support any
number of translations.</p>
<p>Despite these advantages, the serialized approach is not generally
recommended, for the key reason that <em>querying on serialized data is
not possible</em>. Indeed, Rails itself has discouraged the use of serialized
attributes in favour of database-specific storage solutions (described below).</p>
<p>Nonetheless, there are valid use cases for this strategy. One gem that
implements this approach is
<a href="https://github.com/artworklv/multilang">Multilang</a>.</p>
<p><em>Mobility supports this strategy with its <code>:serialized</code> backend.</em></p>
<h3>Strategy 5/6: PostgreSQL Hstore/Jsonb</h3>
<p>The last two strategies,
<a href="https://www.postgresql.org/docs/current/static/hstore.html">Hstore</a> and
<a href="https://www.postgresql.org/docs/current/static/datatype-json.html">jsonb</a>,
require the use of PostgreSQL as your database. Hstore columns store depth-1
hashes, where keys and values of the hash must be of string type. Jsonb
(“binary json”) columns store data in JSON format, which supports a much
broader range of primitives (strings, numbers, booleans and null) as well as
arrays, and unlimited depth. Jsonb is generally preferred over Hstore since it
is more flexible.</p>
<p>Further details on these data types can be found in the links above; for our
purposes, the main point is that they both store complex data types in a
single column, in a form that can be <em>easily and quickly queried</em>.</p>
<p>An example of such a query (on a Jsonb column) looks like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> (posts.title <span style="color:#F00;background-color:#FAA">@</span>> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">{"en":"Translating with Mobility"}</span><span style="color:#710">'</span></span>)::jsonb)
</pre></div>
</div>
<p>Querying on multiple locales at once is also simple given the flexibility of
the data format.</p>
<p>Gems that use Hstore as a storage format for translations include
<a href="https://github.com/yabawock/trasto">Trasto</a>, <a href="https://github.com/bithavoc/multilang-hstore">Multilang
Hstore</a> and <a href="https://github.com/Leadformance/hstore_translate">Hstore
Translate</a>. I only know of
one gem (as of yet) that uses Jsonb for storage, <a href="https://github.com/cfabianski/json_translate">Json
Translate</a>.</p>
<p><em>Mobility supports Hstore with its <code>:hstore</code> backend and Jsonb with its
<code>:jsonb</code> backend.</em></p>
<h2>Common Patterns</h2>
<p>Having worked extensively on Globalize and having delved into the code of many
of the other gems mentioned above, it became clear to me that each translation
solution ended up implementing many of the same things, over and over again.</p>
<p>The concept of a translation cache, for example, is
implemented for <a href="https://github.com/barsoom/traco/blob/ec0ea6e33a606032ddf0064bc86ca89626b0b357/lib/traco/class_methods.rb#L39">translatable columns</a>
in Traco, for
<a href="https://github.com/globalize/globalize/blob/5b0ff2c92e7536f908dc82a4c2a15b04be79e821/lib/globalize/active_record/adapter.rb#L4">model table
translations</a>
in Globalize,
and for <a href="https://github.com/bithavoc/multilang-hstore/blob/48ae91643a67941d8886325cdc67d357855ee546/lib/multilang-hstore/translation_keeper.rb#L3">hstore
translations</a>
in Multilang Hstore.</p>
<p>The concept of fallbacks (originally described by Sven Fuchs in <a href="http://svenfuchs.com/2009/7/19/experimental-ruby-i18n-extensions-pluralization-fallbacks-gettext-cache-and-chained-backend">this blog
post</a>)
is likewise implemented
<a href="https://github.com/barsoom/traco/blob/4c7039a0f786dd0528b2c070559d44ac901f6c5d/lib/traco/locale_fallbacks.rb#L3">again</a>,
and
<a href="https://github.com/globalize/globalize/blob/38443bcd07da78b7b8a9433e4c6c32dd51f964a3/lib/globalize.rb#L48">again</a>,
and
<a href="https://github.com/cfabianski/json_translate/blob/master/lib/json_translate/translates/instance_methods.rb#L16">again</a>.</p>
<p>Locale accessors, methods which fetch an attribute in a particular locale with
a dedicated method like <code>title_en</code>, is yet another repeated pattern found in
many gems, for example <a href="https://github.com/globalize/globalize-accessors">Globalize
Accessors</a> in the case of
Globalize.</p>
<p>Finally, tracking of changes (i.e. <a href="http://api.rubyonrails.org/classes/ActiveModel/Dirty.html">Active Model
Dirty</a> in Rails) is
another common pattern, found
<a href="https://github.com/globalize/globalize/blob/5b0ff2c92e7536f908dc82a4c2a15b04be79e821/lib/globalize/active_record/adapter_dirty.rb#L3">here</a>
in Globalize and
<a href="https://github.com/cfabianski/json_translate/blob/17419b9043ae2d69d5fb9c82e13a17c844ef7ecf/lib/json_translate/translates/instance_methods.rb#L34">here</a>
in Multilang Hstore, for example.</p>
<p>What is important to note here is that while each of these is implemented in
their own way, the repeated patterns (caching, fallbacks, locale accessors,
dirty tracking, etc.) are not in general dependent on the storage
implementation strategies described earlier.</p>
<p>In fact, the core pattern of fetching and updating
translations using getter and setter methods, which behind the scenes map to
some storage format, is itself similar across translation gems, independent of
the storage strategy.</p>
<p>This code from
<a href="https://github.com/yabawock/trasto/blob/develop/lib/trasto/translates.rb">Trasto</a>
captures this core pattern succinctly:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_localized_attribute</span>(column)
define_method(column) <span style="color:#080;font-weight:bold">do</span>
read_localized_value(column)
<span style="color:#080;font-weight:bold">end</span>
define_method(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>column<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">=</span><span style="color:#710">"</span></span>) <span style="color:#080;font-weight:bold">do</span> |value|
write_localized_value(column, value)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>(Here, <code>read_localized_value</code> fetches the value of the translated attribute
<code>column</code> in the current locale.)</p>
<p>This commonality between translation solutions motivated me to consider
whether it would be possible to separate the implementation of translation <em>as
a high-level pattern</em> from the lower-level implementation of <em>storing
translations</em> (an idea also discussed
<a href="https://github.com/globalize/globalize/issues/500">here</a>). This led me to
start developing Mobility.</p>
<h2>Introducing Mobility</h2>
<p>So what is Mobility? Well, I’ve spent most of this post explaining a bunch of
storage strategies and some things which are similar about them. Mobility
takes this whole idea to the next level and turns the problem of translating
attributes into the problem of mapping translated attribute accessors (readers
and writers) to an abstract pluggable “backend”, which encapsulates (and
isolates) all the storage logic.</p>
<p>Rather than jump into the details of how these backends work (which would
require a blog post of its own), let’s look at how we can use Mobility to
quickly implement any of the storage strategies above.</p>
<h3><a name="getting-started"></a>Getting Started</h3>
<p>First, add <code>mobility</code> to the Gemfile:</p>
<p><pre type="ruby">
gem ‘mobility’, '</p>
<p>In the first real post on this blog, I described my experience <a href="/2013/10/11/hacking-globalize">hacking
globalize</a>, a
<a href="https://rubygems.org/gems/globalize">gem</a> that calls itself the “Rails
I18n de-facto standard library for ActiveRecord model/data translation.”</p>
<p>Globalize was the first gem that I contributed to, and the experience
was very important to me as a step toward becoming
a better and more knowledgeable programmer.</p>
<p>I am now one of the authors of that gem, having made <a href="https://github.com/globalize/globalize/commits?author=shioyama">many
commits</a> to
the repository. In the time since I first contributed to Globalize, I’ve
become keenly aware of the limitations of the gem, and of problems with
existing solutions for managing translations in Rails projects.</p>
<p>I’d like to look back in this post to the problem of translation and
to solutions devised so far, as motivation to introduce a
gem I’ve just released called <a href="https://rubygems.org/gems/mobility">Mobility</a>.
Mobility attempts to solve some core problems with the multitude
of Ruby-based translation gems out there right now.</p>
<h2>The Problem</h2>
<p>You have an application with user-generated content. You want to translate
that content into multiple languages. And – importantly – you want to do it
in such a way that you don’t need to recode all your existing presentation logic
(views, mailers, controllers, etc.) to handle the new languages.</p>
<p>In other words, things should <em>just work</em> – more or less the same as they do
with one language, except that switching
the language should tell the application to switch context to read or write
content stored in that language.</p>
<p>So this:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
post.title
</pre></div>
</div>
<p>… should return the title of the post, <em>in the language I am currently
viewing</em>.</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:en</span>
post.title
<span style="color:#777">#=> "Mobility (noun): quality of being changeable, adaptable or versatile"</span>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span>
post.title
<span style="color:#777">#=> "Mobility(名詞):動きやすさ、可動性"</span>
</pre></div>
</div>
<p>Likewise, I should be able to write translated values the same way:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:en</span>
post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>
post.save
</pre></div>
</div>
<p>Now the English title has changed, but the Japanese one has not:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
post.title
<span style="color:#777">#=> "Translating with Mobility"</span>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span>
post.title
<span style="color:#777">#=> "Mobility(名詞):動きやすさ、可動性"</span>
</pre></div>
</div>
<p>For bonus points, this solution should also work for querying attributes, so
something similar to this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>)
</pre></div>
</div>
<p>… should work even if <code>title</code> is a translated attribute.</p>
<p>For this to happen, something has to hook in to attribute accessors,
to detect changes in the locale and return language-specific content
accordingly. This may seem easy to do at first glance, but it gets tricky.</p>
<p>Notice that what we’re asking for is similar but different from the problem
solved by localization solutions such as the <a href="http://guides.rubyonrails.org/i18n.html">Rails I18n
API</a>, which
provide namespaced translation lookup (generally) for interface translation,
typically with static data.</p>
<p>With application content, we want translations at the class level, in the form
of persisted attributes of a model. Ultimately we’re talking about a
storage strategy.</p>
<p>As it turns out, there are a number of different strategies that different
people have come up with to solve this problem. Let’s take a look at them.</p>
<h2>Solutions</h2>
<h3><a name="strategy-1"></a>Strategy 1: Translatable Columns</h3>
<p>The most obvious solution to this problem is to create columns on your model
table in each locale you want to support, with some convention for identifying
the locale of the column.</p>
<p>Typically, this is done by appending a suffix to the column name made up of an
underscore plus the locale, with any dashes in the locale converted to underscores.</p>
<p>So <code>title</code> becomes <code>title_en</code>, <code>title_fr</code>, <code>title_ja</code>, and so on. Adding a new
locale to an application then requires that you add columns for that locale to
<em>each and every translated model table</em> (see figure below.)</p>
<div class="img-center">
<img alt="Column Strategy" src="/img/column_strategy.png"></a>
<p class="caption">Table structure of column strategy</p>
</div>
<p>For a small number of models, and a small number of locales, this is a
reasonable solution. It has the advantage that the column structure is easy to
understand, and querying by translated attribute is straightforward:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Mobility</span><span style="color:#710">"</span></span>)
<span style="color:#777">#=> #<Post:0x... id: 123 content_en: "Mobility" content_ja: "モビリティ"...></span>
<span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title_ja</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">モビリティ</span><span style="color:#710">"</span></span>)
<span style="color:#777">#=> #<Post:0x... id: 123 content_en: "Mobility" content_ja: "モビリティ"...></span>
</pre></div>
</div>
<p>Some key disadvantages of this approach:</p>
<ul>
<li>For many models and many locales, this approach requires many migrations,
each time a locale or model is added.</li>
<li>Space is not used very efficiently: every record has a column for every
locale, even if the locale is not used.</li>
</ul>
<p>Despite these disadvantages, this strategy is appropriate for applications
with a fixed number of locales. For a very nice implementation of this
approach, see <a href="https://github.com/barsoom/traco">Traco</a>.</p>
<p><em>Mobility supports this strategy with its <code>:column</code> backend.</em></p>
<h3><a name="strategy-2"></a>Strategy 2: Model Translation Tables</h3>
<p>Now consider that we have an application where we have many translated models,
with a potentially open-ended list of locales we want to support. A
service with user-generated content offered to users around the world
would be a good example of this pattern. In this case, the first strategy
becomes impractical as the number of locales grows; with, say, 50 locales, and
only a small number of actually translations, the vast majority of translated
columns would be null. On top of which, adding new locales to all the
translated models would be a pain, not to mention error-prone.</p>
<p>This brings us to the next strategy, shown in the database schema below. Here,
each model table is assigned its own translations table. Each row in the
translations table has:</p>
<ul>
<li>A column for the locale (<code>locale</code>)</li>
<li>A foreign key pointing to the model table (<code>post_id</code>)</li>
<li>One or more columns for translations (<code>title</code>, <code>content</code>)</li>
</ul>
<p>With this structure, we can now <em>join</em> the translation table (e.g.
<code>post_translations</code>) to the model table (e.g. <code>posts</code>) on the locale we want
to get the desired result: a model with translations in that locale.</p>
<div class="img-center">
<img alt="Model Translation Strategy" src="/img/globalize_strategy.png"></a>
<p class="caption">Table structure of model translation strategy</p>
</div>
<p>In ActiveRecord, we would have an association on the model <code>Post</code> to the model
<code>PostTranslation</code>, such that we can fetch translations with
<code>post.translations</code> and pick out the one we want:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
translation = post.translations.find <span style="color:#080;font-weight:bold">do</span> |translation|
translation.locale == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">en</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#777">#=> #<PostTranslation:0x... id: 123 locale: "en" ...></span>
translation.title
<span style="color:#777">#=> "Translating with Mobility"</span>
translation.content
<span style="color:#777">#=> "In the first real post on this blog..."</span>
</pre></div>
</div>
<p>With this approach, we’ve nicely solved some of the problems with the first
strategy:</p>
<ul>
<li>Since translations now have their own model and table, and we only create
rows in that table when we save a translation in that locale, space usage
is more efficient (no wasted columns/rows for locales that have no
translation).</li>
<li>We do not need any migrations to add locales: simply save the translation
for a new locale and it becomes available. So scalability on the “locale
axis” is very good.</li>
</ul>
<p>However, there are some clear disadvantages with this approach that should be
kept in mind:</p>
<ul>
<li>Having translations on a different table implies a lot of joining, which
(if not managed very carefully) can lead to performance degradation.</li>
<li>Joining can also add significant complexity to the code dealing with
translations. In particular, querying on translated attributes is
much more complicated than it was with the first strategy, where we
queried on the model column directly.</li>
</ul>
<p>Note also that although this strategy scales nicely along the “locale axis”,
it nonetheless requires a table for every translated model, and thus a
migration every time we add a new model or a new translated attribute to an
existing model. So this strategy may not scale well if an application has many
different forms of translated content, spread across a large and growing list
of models.</p>
<p>This strategy is implemented by
<a href="https://github.com/globalize/globalize">Globalize</a>, the gem mentioned at the
beginning of this post (it is also implemented by
<a href="https://github.com/francesc/rails-translate-models">many</a>
<a href="https://github.com/jo/puret">other</a>
<a href="https://github.com/janne/model_translations">gems</a>). For the reasons mentioned above,
Globalize has become the
preferred option for model translation in Rails. Like the first strategy, it
is a very useful approach for many (but not all) use cases.</p>
<p><em>Mobility supports this strategy with its <code>:table</code> backend.</em></p>
<h3>Strategy 3: Shared Translation Tables</h3>
<p>My experience working with Globalize, and in particular with scalability
issues mentioned above, prompted me to consider another strategy which would
scale better with model creation. I wanted a strategy which would be usable
<em>immediately</em> at the code level, without having to add a table or column every
time a new model or translated attribute is added.</p>
<p>This led me to the third strategy, which I’ve called the “key value” strategy.
The basic idea is simple: rather than using a dedicated table for each
translated model, instead create shared tables to store translations across
all models (as shown in the figure below). Two tables are required since we
will be storing two types of values: strings and text. (If you wanted to store
other column types, you could add more tables for each of those types, or cast
strings to some other type).</p>
<div class="img-center">
<img alt="KeyValue Strategy" src="/img/key_value_strategy.png"></a>
<p class="caption">Table structure of key value strategy</p>
</div>
<p>The key thing here is that rather than using a normal association as we did in
the last strategy, where <code>post_id</code> on the <code>post_translations</code> table pointed to
the <code>posts</code> table, we will instead use a <a href="http://guides.rubyonrails.org/association_basics.html#polymorphic-associations">polymorphic
association</a>.
By doing this, we can share the translations tables with as many models as we
like, migrating just once to create the shared tables.</p>
<p>The new tables have the following columns:</p>
<ul>
<li>A column for the locale (<code>locale</code>)</li>
<li>A foreign key pointing to the translated model id (<code>translatable_id</code>)</li>
<li>A column holding the model class (<code>translatable_type</code>)</li>
<li>A column holding the name of the attribute being translated (<code>key</code>)</li>
<li>A column holding the value of the translated attribute (<code>value</code>)</li>
</ul>
<p>With these tables in place, we define a polymorphic relationship on the model,
something like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
has_many <span style="color:#A60">:text_translations</span>,
<span style="color:#606">as</span>: <span style="color:#A60">:translatable</span>,
<span style="color:#606">class_name</span>: <span style="color:#036;font-weight:bold">TextTranslation</span>,
<span style="color:#606">inverse_of</span>: <span style="color:#A60">:translatable</span>
<span style="color:#777">#...</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>and a translation class:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">TextTranslation</span>
<span style="color:#069">self</span>.table_name = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">"</span></span>
belongs_to <span style="color:#A60">:translatable</span>, <span style="color:#606">polymorphic</span>: <span style="color:#069">true</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now that we have these models in place, we can easily fetch and set
translations of a model class:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
post.text_translations.find <span style="color:#080;font-weight:bold">do</span> |translation|
translation.key == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span> && translation.value == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#777">#=> #<TextTranslation:0x...</span>
<span style="color:#606">id</span>: <span style="color:#00D">123</span>
<span style="color:#606">locale</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">en</span><span style="color:#710">"</span></span>
key=<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title</span><span style="color:#710">"</span></span>
value=<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>
... >
</pre></div>
</div>
<p>To add translations to a new model, we simply add a polymorphic relationship
like the one above on <code>Post</code>, and we can immediately add translations to the
model.</p>
<p>In practice, there are some other issues that need to be handled to make this
work effectively, but done correctly this strategy has some notable benefits:</p>
<ul>
<li>No more migrations (after the first one), so an application can quickly add
new translated models.</li>
<li>As with Strategy 2, no extra effort to add new locales.</li>
<li>Relatively efficient in terms of storage, in the sense that we only create
records for translations in locales where they actually exist. However,
since each attribute translation gets a row in the table, for multiple
attributes on a single translation table, Strategy 2 may be more space
efficient since it stores translated attributes together in a single row.</li>
</ul>
<p>Now for the disadvantages:</p>
<ul>
<li>The join complexity described in the last section is even worse now, because
we have to join <em>every attribute separately</em> when searching for matches.
This is quite tricky but can be done. I’ll probably write about this in a
future post.</li>
<li>Since we have each attribute stored separately (as a key/value pair) it
becomes trickier to maintain our database in a consistent state. For
example, if we remove a model, its translations will remain in the
translations table(s) unless we explicitly remove them. Likewise, if we
decide to rename a translated attribute, we may end up with translations for
the previous attribute name leftover in the table.</li>
</ul>
<p>Despite these disadvantages, I consider this strategy to be applicable to
the widest variety of situations. It is the default strategy in Mobility,
implemented as the <code>:key_value</code> backend.</p>
<h3>Strategy 4: Serialized Data</h3>
<p>Yet another set of strategies take the approach of jamming all the
translations for an attribute into a single column. The first strategy
following this approach is one which serializes translations as a hash and
stores it on a text column.</p>
<p>At the model layer, this is pretty trivial in Rails:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
serialize <span style="color:#A60">:title_translations</span>, <span style="color:#036;font-weight:bold">Hash</span>
<span style="color:#080;font-weight:bold">end</span>
post = <span style="color:#036;font-weight:bold">Post</span>.new
post.title_translations = { <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">en</span><span style="color:#710">"</span></span> => <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span> }
post.save
</pre></div>
</div>
<p>We can now fetch translations simply by fetching the value for the key
matching a given locale:</p>
<div class="CodeRay">
<div class="code"><pre>
post.title_translations[<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">en</span><span style="color:#710">"</span></span>]
<span style="color:#777">#=> "Translating with Mobility"</span>
</pre></div>
</div>
<p>This strategy has the obvious advantage of simplicity: all our translations
are stored together on the attribute column. Like the model translations and
key-value strategies, locale scalability is not an issue since we can just add
new hash keys as we need them.</p>
<p>In addition, no additional migrations are necessary; simply adding a text
column to every model for each translated attribute is enough to support any
number of translations.</p>
<p>Despite these advantages, the serialized approach is not generally
recommended, for the key reason that <em>querying on serialized data is
not possible</em>. Indeed, Rails itself has discouraged the use of serialized
attributes in favour of database-specific storage solutions (described below).</p>
<p>Nonetheless, there are valid use cases for this strategy. One gem that
implements this approach is
<a href="https://github.com/artworklv/multilang">Multilang</a>.</p>
<p><em>Mobility supports this strategy with its <code>:serialized</code> backend.</em></p>
<h3>Strategy 5/6: PostgreSQL Hstore/Jsonb</h3>
<p>The last two strategies,
<a href="https://www.postgresql.org/docs/current/static/hstore.html">Hstore</a> and
<a href="https://www.postgresql.org/docs/current/static/datatype-json.html">jsonb</a>,
require the use of PostgreSQL as your database. Hstore columns store depth-1
hashes, where keys and values of the hash must be of string type. Jsonb
(“binary json”) columns store data in JSON format, which supports a much
broader range of primitives (strings, numbers, booleans and null) as well as
arrays, and unlimited depth. Jsonb is generally preferred over Hstore since it
is more flexible.</p>
<p>Further details on these data types can be found in the links above; for our
purposes, the main point is that they both store complex data types in a
single column, in a form that can be <em>easily and quickly queried</em>.</p>
<p>An example of such a query (on a Jsonb column) looks like this:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> (posts.title <span style="color:#F00;background-color:#FAA">@</span>> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">{"en":"Translating with Mobility"}</span><span style="color:#710">'</span></span>)::jsonb)
</pre></div>
</div>
<p>Querying on multiple locales at once is also simple given the flexibility of
the data format.</p>
<p>Gems that use Hstore as a storage format for translations include
<a href="https://github.com/yabawock/trasto">Trasto</a>, <a href="https://github.com/bithavoc/multilang-hstore">Multilang
Hstore</a> and <a href="https://github.com/Leadformance/hstore_translate">Hstore
Translate</a>. I only know of
one gem (as of yet) that uses Jsonb for storage, <a href="https://github.com/cfabianski/json_translate">Json
Translate</a>.</p>
<p><em>Mobility supports Hstore with its <code>:hstore</code> backend and Jsonb with its
<code>:jsonb</code> backend.</em></p>
<h2>Common Patterns</h2>
<p>Having worked extensively on Globalize and having delved into the code of many
of the other gems mentioned above, it became clear to me that each translation
solution ended up implementing many of the same things, over and over again.</p>
<p>The concept of a translation cache, for example, is
implemented for <a href="https://github.com/barsoom/traco/blob/ec0ea6e33a606032ddf0064bc86ca89626b0b357/lib/traco/class_methods.rb#L39">translatable columns</a>
in Traco, for
<a href="https://github.com/globalize/globalize/blob/5b0ff2c92e7536f908dc82a4c2a15b04be79e821/lib/globalize/active_record/adapter.rb#L4">model table
translations</a>
in Globalize,
and for <a href="https://github.com/bithavoc/multilang-hstore/blob/48ae91643a67941d8886325cdc67d357855ee546/lib/multilang-hstore/translation_keeper.rb#L3">hstore
translations</a>
in Multilang Hstore.</p>
<p>The concept of fallbacks (originally described by Sven Fuchs in <a href="http://svenfuchs.com/2009/7/19/experimental-ruby-i18n-extensions-pluralization-fallbacks-gettext-cache-and-chained-backend">this blog
post</a>)
is likewise implemented
<a href="https://github.com/barsoom/traco/blob/4c7039a0f786dd0528b2c070559d44ac901f6c5d/lib/traco/locale_fallbacks.rb#L3">again</a>,
and
<a href="https://github.com/globalize/globalize/blob/38443bcd07da78b7b8a9433e4c6c32dd51f964a3/lib/globalize.rb#L48">again</a>,
and
<a href="https://github.com/cfabianski/json_translate/blob/master/lib/json_translate/translates/instance_methods.rb#L16">again</a>.</p>
<p>Locale accessors, methods which fetch an attribute in a particular locale with
a dedicated method like <code>title_en</code>, is yet another repeated pattern found in
many gems, for example <a href="https://github.com/globalize/globalize-accessors">Globalize
Accessors</a> in the case of
Globalize.</p>
<p>Finally, tracking of changes (i.e. <a href="http://api.rubyonrails.org/classes/ActiveModel/Dirty.html">Active Model
Dirty</a> in Rails) is
another common pattern, found
<a href="https://github.com/globalize/globalize/blob/5b0ff2c92e7536f908dc82a4c2a15b04be79e821/lib/globalize/active_record/adapter_dirty.rb#L3">here</a>
in Globalize and
<a href="https://github.com/cfabianski/json_translate/blob/17419b9043ae2d69d5fb9c82e13a17c844ef7ecf/lib/json_translate/translates/instance_methods.rb#L34">here</a>
in Multilang Hstore, for example.</p>
<p>What is important to note here is that while each of these is implemented in
their own way, the repeated patterns (caching, fallbacks, locale accessors,
dirty tracking, etc.) are not in general dependent on the storage
implementation strategies described earlier.</p>
<p>In fact, the core pattern of fetching and updating
translations using getter and setter methods, which behind the scenes map to
some storage format, is itself similar across translation gems, independent of
the storage strategy.</p>
<p>This code from
<a href="https://github.com/yabawock/trasto/blob/develop/lib/trasto/translates.rb">Trasto</a>
captures this core pattern succinctly:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_localized_attribute</span>(column)
define_method(column) <span style="color:#080;font-weight:bold">do</span>
read_localized_value(column)
<span style="color:#080;font-weight:bold">end</span>
define_method(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>column<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">=</span><span style="color:#710">"</span></span>) <span style="color:#080;font-weight:bold">do</span> |value|
write_localized_value(column, value)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>(Here, <code>read_localized_value</code> fetches the value of the translated attribute
<code>column</code> in the current locale.)</p>
<p>This commonality between translation solutions motivated me to consider
whether it would be possible to separate the implementation of translation <em>as
a high-level pattern</em> from the lower-level implementation of <em>storing
translations</em> (an idea also discussed
<a href="https://github.com/globalize/globalize/issues/500">here</a>). This led me to
start developing Mobility.</p>
<h2>Introducing Mobility</h2>
<p>So what is Mobility? Well, I’ve spent most of this post explaining a bunch of
storage strategies and some things which are similar about them. Mobility
takes this whole idea to the next level and turns the problem of translating
attributes into the problem of mapping translated attribute accessors (readers
and writers) to an abstract pluggable “backend”, which encapsulates (and
isolates) all the storage logic.</p>
<p>Rather than jump into the details of how these backends work (which would
require a blog post of its own), let’s look at how we can use Mobility to
quickly implement any of the storage strategies above.</p>
<h3><a name="getting-started"></a>Getting Started</h3>
<p>First, add <code>mobility</code> to the Gemfile:</p>
<div class="CodeRay">
<div class="code"><pre>
gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">mobility</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">> 0.1.3</span><span style="color:#710">'</span></span>
</pre></div>
</div>
<p>In Rails, you can use a generator to create an initializer file and migrations
for the shared translation tables for the key value strategy:</p>
<pre>
rails generate mobility:install
</pre>
<p>The initializer file has the line:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Mobility</span>.config.default_backend = <span style="color:#A60">:key_value</span>
</pre></div>
</div>
<p>With this configuration, the key value backend will be our default backend.
Run the two migrations to generate string and text translation tables with
<code>rake db:migrate</code>, and we’re ready to go.</p>
<p>Creating a translated attribute in a model is as easy as including (or
extending) the <code>Mobility</code> module and declaring translated attributes (similar
to Globalize and other gems):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
include <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">type</span>: <span style="color:#A60">:string</span>
translates <span style="color:#A60">:content</span>, <span style="color:#606">type</span>: <span style="color:#A60">:text</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we can use the attributes:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.new
<span style="color:#777">#=> #<Post id: nil></span>
post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Mobility</span><span style="color:#710">"</span></span>
post.title
<span style="color:#777">#=> "Mobility"</span>
<span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span>
post.title
<span style="color:#777">#=> nil</span>
post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">モビリティ</span><span style="color:#710">"</span></span>
post.title
<span style="color:#777">#=> "モビリティ"</span>
post.save
</pre></div>
</div>
<p>When you call <code>post.title</code>, you will see the SQL:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">translatable_id</span><span style="color:#710">"</span></span> = <span style="color:#00D">1</span>
<span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">translatable_type</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Post</span><span style="color:#710">'</span></span>
<span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">key</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">title</span><span style="color:#710">'</span></span>
</pre></div>
</div>
<p><code>post.content</code> looks similar, but instead fetches from the
<code>mobility_text_translations</code> table:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">translatable_id</span><span style="color:#710">"</span></span> = <span style="color:#00D">1</span>
<span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">translatable_type</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Post</span><span style="color:#710">'</span></span>
<span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">key</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">content</span><span style="color:#710">'</span></span>
</pre></div>
</div>
<p>It’s also possible to query for a post with finder methods using the <code>i18n</code>
scope:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.i18n.find_by(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Mobility</span><span style="color:#710">"</span></span>)
<span style="color:#777">#=> #<Post id: 1></span>
</pre></div>
</div>
<p>The SQL for this query is a somewhat tricky table join, which is necessary
since translations for this backend are stored on the shared translations
table. (I’ll leave the details of that for another blog post.)</p>
<p>What is important to note here is that although the format for declaring
translated attributes (with <code>translates</code>) is similar in form to other
translation gems, the implementation is totally different. In particular,
Mobility strives to keep the accessor and setup code for a given set of
attributes <em>totally isolated</em> from attributes declared separately. This is
what allows us to call <code>translates</code> twice in the model code above, one for
each type of attribute (string and text).</p>
<h3>Translation Options</h3>
<p>Behind the scenes, when <code>translates</code> is called, Mobility creates an anonymous
backend class which optionally includes modules for caching, fallbacks, and
dirty accessors. Which of these modules is included can be customized by
passing options:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
include <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">type</span>: <span style="color:#A60">:string</span>, <span style="color:#606">fallbacks</span>: { <span style="color:#606">en</span>: <span style="color:#A60">:ja</span> }
translates <span style="color:#A60">:content</span>, <span style="color:#606">type</span>: <span style="color:#A60">:text</span>, <span style="color:#606">fallbacks</span>: { <span style="color:#606">en</span>: <span style="color:#A60">:fr</span> }
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>I wouldn’t want to set up fallbacks like this in a real application, but
regardless, it actually works: if we call <code>post.title</code> and the title is not
set in English, it will fall back to Japanese. However, <code>post.content</code> will
fall back to French, since this is how we have configured the attributes.</p>
<p>The <code>fallbacks</code> option is an example of a shared option which can be included
for any backend. Others of this kind are <code>cache</code> (<code>true</code> by default), <code>dirty</code>
(limited to classes which include <code>ActiveModel::Dirty</code>) and <code>locale_accessors</code>.</p>
<p>Here is an example of adding locale accessors to attributes:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
include <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">type</span>: <span style="color:#A60">:string</span>, <span style="color:#606">locale_accessors</span>: [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>]
translates <span style="color:#A60">:content</span>, <span style="color:#606">type</span>: <span style="color:#A60">:text</span>, <span style="color:#606">locale_accessors</span>: [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>]
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now we can use the accessors to access attribute values in English or
Japanese:</p>
<div class="CodeRay">
<div class="code"><pre>
post = <span style="color:#036;font-weight:bold">Post</span>.first
post.title_en
<span style="color:#777">#=> "Mobility"</span>
post.title_ja
<span style="color:#777">#=> "モビリティ"</span>
</pre></div>
</div>
<p>For more details on how these options are set, see the API documentation on
the
<a href="http://www.rubydoc.info/gems/mobility/0.1.3/Mobility/Attributes">Mobility::Attributes</a>
class.</p>
<h3>Translation Backends</h3>
<p>As mentioned in <a href="#getting-started">Getting Started</a>, the default backend is
set in the initializer file to <code>:key_value</code>. But we can override it to use any
backend, for any attribute:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
include <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">backend</span>: <span style="color:#A60">:column</span>
translates <span style="color:#A60">:content</span>, <span style="color:#606">backend</span>: <span style="color:#A60">:table</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>In order for this to work, we will need to create the necessary columns for
the column backend (as described in <a href="#strategy-1">Strategy 1</a>, here for three
locales):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">AddColumnsToPosts</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Migration</span>[<span style="color:#60E">5.0</span>]
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">change</span>
add_column <span style="color:#A60">:posts</span>, <span style="color:#A60">:title_en</span>, <span style="color:#A60">:string</span>
add_column <span style="color:#A60">:posts</span>, <span style="color:#A60">:title_ja</span>, <span style="color:#A60">:string</span>
add_column <span style="color:#A60">:posts</span>, <span style="color:#A60">:title_fr</span>, <span style="color:#A60">:string</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>…and translations table for the table backend (as described in <a href="#strategy-2">Strategy
2</a>):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">CreatePostTranslations</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Migration</span>[<span style="color:#60E">5.0</span>]
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">change</span>
create_table <span style="color:#A60">:post_translations</span> <span style="color:#080;font-weight:bold">do</span> |t|
t.string <span style="color:#A60">:locale</span>
t.integer <span style="color:#A60">:post_id</span>
t.text <span style="color:#A60">:content</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>With these columns and table, we can use the attributes:</p>
<div class="CodeRay">
<div class="code"><pre>
post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>
=> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>
post.content = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">"</span></span>
=> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">"</span></span>
</pre></div>
</div>
<p>If we now save, this is the generated SQL:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">INSERT</span> <span style="color:#B06;font-weight:bold">INTO</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title_en</span><span style="color:#710">"</span></span>) <span style="color:#080;font-weight:bold">VALUES</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">'</span></span>)
<span style="color:#B06;font-weight:bold">INSERT</span> <span style="color:#B06;font-weight:bold">INTO</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">locale</span><span style="color:#710">"</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_id</span><span style="color:#710">"</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">content</span><span style="color:#710">"</span></span>) <span style="color:#080;font-weight:bold">VALUES</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>,<span style="color:#00D">5</span>,<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">'</span></span>)
</pre></div>
</div>
<p>So each attribute is treated entirely in isolation, and we can have multiple
backends operating on the same model without any conflicts.</p>
<p>This works as well for queries:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">Post</span>.i18n.find_by(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">"</span></span>, <span style="color:#606">content</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">"</span></span>)
<span style="color:#777">#=> #<Post id: 5, title_en: "Translating with Mobility", title_ja: nil, title_fr: nil></span>
</pre></div>
</div>
<p>Note the SQL:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">INNER</span> <span style="color:#080;font-weight:bold">JOIN</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">ON</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_id</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">id</span><span style="color:#710">"</span></span>
<span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">locale</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>
<span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">posts</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">title_en</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">'</span></span>
<span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">post_translations</span><span style="color:#710">"</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">content</span><span style="color:#710">"</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">'</span></span>
</pre></div>
</div>
<p>So components for each backend are combined to generate the final query. This
applies to any combination of backends.</p>
<h3>Mobility = Freedom</h3>
<p>Why go to the trouble to implement things this way? There are a few reasons,
but the driving motivation is to free the developer from having to constantly
think about the details of the underlying translation implementation when
working with translations. As its name implies, Mobility also strives to make
it easy to change these details (e.g. change strategy) without having to
rewrite an entire code base.</p>
<p>This in fact goes further than the examples described here. If you look at
Mobility’s
<a href="https://github.com/shioyama/mobility/blob/master/mobility.gemspec">gemspec</a>,
you will see that it does not depend on activerecord or on
activesupport. These are optional dependencies, and indeed the basic framework
for setting up translated attributes with a backend (including caching,
fallbacks and locale accessors) is totally independent of Rails. (Dirty
tracking can also be used on a non-persisted class as long as it inherits from
<code>ActiveModel::Dirty</code>.)</p>
<p>Thanks to this isolation of dependencies, Mobility can also support other ORM,
and currently does: all strategies described here are also implemented for
<a href="http://sequel.jeremyevans.net/">Sequel</a> models. So this will work (provided
translation tables have been created):</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> < ::<span style="color:#036;font-weight:bold">Sequel</span>::<span style="color:#036;font-weight:bold">Model</span>
include <span style="color:#036;font-weight:bold">Mobility</span>
translates <span style="color:#A60">:title</span>, <span style="color:#606">type</span>: <span style="color:#A60">:string</span>
translates <span style="color:#A60">:content</span>, <span style="color:#606">type</span>: <span style="color:#A60">:text</span>
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Mobility detects the parent class and generates a backend instance using a
class namespaced accordingly.</p>
<p>Aside from keeping with the “DRY” philosophy of not repeating yourself, having
a shared framework which works with many different configurations
(ActiveRecord models, Sequel models, PORO, etc.) creates an opportunity
to build something more interesting on <em>top</em> of the translation framework. I
think there are some really interesting possibilities in this space, but
current solutions being spread across more than a dozen different gems makes
it impossible to explore those possibilities.</p>
<h3>Try it!</h3>
<p>So, try it and let me know what you think! I’d be really eager to hear from
anyone interested in building interesting projects using Mobility for
translation. While this blog has been nearly dead for a long time, I plan to
follow up this post with more details on how Mobility works, other use cases,
and on further developments of the gem. Stay tuned!</p>
Debugging Spreehttps://dejimata.com/2014/4/17/debugging-spree2014-04-172014-04-17Chris Salzberg<p><a href="https://github.com/spree/spree">Spree</a> is the most popular e-commerce platform
built on Ruby on Rails. It’s made up of a set of different gems implementing
different functions of the platform: a
<a href="https://github.com/spree/spree/tree/master/frontend">front</a> and
<a href="https://github.com/spree/spree/tree/master/backend">back</a> end, an
<a href="https://github.com/spree/spree/tree/master/api">API</a>,
<a href="https://github.com/spree/spree/tree/master/cmd">command-line tools</a> and
a <a href="https://github.com/spree/spree/tree/master/core">core</a> powering a
cart and payment system…</p>
<p><a href="https://github.com/spree/spree">Spree</a> is the most popular e-commerce platform
built on Ruby on Rails. It’s made up of a set of different gems implementing
different functions of the platform: a
<a href="https://github.com/spree/spree/tree/master/frontend">front</a> and
<a href="https://github.com/spree/spree/tree/master/backend">back</a> end, an
<a href="https://github.com/spree/spree/tree/master/api">API</a>,
<a href="https://github.com/spree/spree/tree/master/cmd">command-line tools</a> and
a <a href="https://github.com/spree/spree/tree/master/core">core</a> powering a
cart and payment system.</p>
<div class="img-right">
<a href="http://spreecommerce.com">
<img src="/img/spree.png"></a>
<p class="caption">Spree: e-commerce solution for Ruby on Rails</p>
</div>
<p>Until recently although I had heard of it, I’d never had the chance to deploy an
application with Spree, let alone dig into its internals.</p>
<p>That all changed with my new job at <a href="http://www.degica.com/">Degica</a>, where
Spree is a core element of almost
everything we do. Not only do we use Spree to power our carts, we’ve also
extended it, built on it, transformed it and twisted it to make it do things it was
probably never intended to do.</p>
<p>Putting aside the question of whether this kind of “adaptation” is a
good thing (and you could fairly argue that in some cases it is not), as a
programmer it presents an interesting challenge: to ensure that the core
Spree engine and everything we’ve built on top of it are working smoothly together.
The more complex our adaptations become, the more difficult this can be.</p>
<p>One bug we hit on recently brought this challenge home to me, and got me
thinking about the whole process of programming and how it is affected by our
environment.</p>
<h2>A transaction in limbo</h2>
<p>The bug was brought to our attention by our customer support team, who
reported that some customers would make an order, get a confirmation email
with payment instructions, pay for the product, but then never receive the
expected license key(s) from us.</p>
<p>When we checked our system, we saw that the payment attached to the order was in
a “failed” state. Spree’s cart makes extensive use of the
<a href="https://github.com/pluginaweek/state_machine"><code>state_machine</code></a> gem to manage
<a href="http://guides.spreecommerce.com/developer/payments.html">transitions</a> between
the sequence of events in the cart checkout and payment process. The “failed”
state in this machine is a final state with no transitions – once payment
reached this state, the order was effectively killed on our side.</p>
<div class="img-left" style="max-width: 440px">
<a href="http://guides.spreecommerce.com/developer/payments.html">
<img src="/img/payment_flow.png"></a>
<p class="caption">Transitions between payment states in the Spree payment
flow</p>
</div>
<p>The mystery deepened when we checked with the gateway and learned
that, while the order had failed on our side, the payment had gone through on
theirs without any issues.
Customer support made a note of the bug on the order and manually
triggered delivery of
the keys as we began the process of combing our logs to figure out what had
happened.</p>
<p>Eventually we tracked the problem down to “double submission” by the
client: for whatever reason (perhaps an accidental double-click or javascript
bug), the customer sent two requests to our server with the same order number.
Since our servers run multiple workers
(<a href="http://unicorn.bogomips.org/">unicorns</a>), each request
triggered a parallel request to the payment gateway with the same information.</p>
<p>The gateway accepted the first request, returning a success response and
triggering our system to send out the
first confirmation email and mark payment as “pending”. But when the second
request hit the gateway, it returned an error complaining that the payment had
already been processed. This in turn caused the second
worker to <em>overwrite</em> the previous pending state with a “failed” state.</p>
<p>Through this sequence of events, the order was left in limbo:
its payment had been accepted by the gateway
but marked as failed on our system, so it was unable to transition to the “complete” state
and deliver the requested keys.</p>
<h2>The ghost in the (state) machine</h2>
<p>At this point, based on log data and a bit of guesswork, we had narrowed down
the <em>what</em> of the bug, but not the <em>why</em>. I found that
I could actually reproduce the bug running multiple unicorns on my laptop, but
the deep integration of the <code>state_machine</code> gem into Spree – not to mention the
commits we had added in <a href="https://github.com/freerunningtech/spree">our fork</a>
– made it hard to pinpoint where the bug was actually coming from.</p>
<p>The <a href="http://guides.spreecommerce.com/developer/payments.html">Spree
documentation</a> described
a “payment processing” state, “intended to prevent double submission” (see
diagram above). The
state is triggered by the <code>started_processing</code> method, which is
called in two key places: in the
<a href="https://github.com/spree/spree/blob/93c2081064bd386cc48d0e758f7afd2c1fe1525b/core/app/models/spree/payment/processing.rb#L25"><code>authorize!</code></a>
and
<a href="https://github.com/spree/spree/blob/93c2081064bd386cc48d0e758f7afd2c1fe1525b/core/app/models/spree/payment/processing.rb#L30"><code>purchase!</code></a>
methods of the <code>Spree::Payment::Processing</code> module, included in
<code>Spree::Payment</code>. Payment processing is wrapped in the <code>process!</code> method in a
<a href="https://github.com/spree/spree/blob/93c2081064bd386cc48d0e758f7afd2c1fe1525b/core/app/models/spree/payment/processing.rb#L4">conditional</a>
which checks that it is not already in that state – this is the mechanism to
prevent double submission mentioned in the documentation.</p>
<p>The conditional should block double submission, but when I dug deep enough, I found
that this was not happening.
By placing a <code>binding.pry</code> call in the <code>authorize!</code>
method, I could get two parallel processes simultaneously into the <code>processing</code>
state, which by all accounts should not be possible.</p>
<p>Running slower through the code with parallel pry sessions, I found that
although the first process would see the payment state as <code>processing</code>, the
second process would see it as <code>pending</code> until it too entered the processing
state. With both processes processing, I
then opened a mysql session and found that – contrary to what both pry
sessions were telling me – the state in the database was still <code>pending</code>.</p>
<p>The processes, in other words, were running in bubbles. Or, in database-speak,
“transactions”.</p>
<h2>Transactions in transactions</h2>
<p>A <a href="https://en.wikipedia.org/wiki/Database_transaction">transaction</a> is a unit
of work performed in a database which can be
rolled-back at any point prior to completion. Two processes running in
transactions are isolated until the point when the transaction ends and data
is written to the database.</p>
<p>A quick check from either pry session confirmed that the payment processing
code was indeed wrapped in a transaction:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>.connection.open_transactions
<span style="color:#777">#=> 1</span>
</pre></div>
</div>
<p>Before moving on, I wrote a <a href="https://github.com/shioyama/spree/commit/54f6df0b4aad720eb50a07ca9423532d29264287">quick failing
test</a>
isolate the problem:</p>
<div class="CodeRay">
<div class="code"><pre>
it <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">does not process payment within transaction</span><span style="color:#710">"</span></span> <span style="color:#080;font-weight:bold">do</span>
<span style="color:#777"># Make sure we are not already in a transaction</span>
<span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>.connection.open_transactions.should == <span style="color:#00D">0</span>
<span style="color:#036;font-weight:bold">Spree</span>::<span style="color:#036;font-weight:bold">Payment</span>.any_instance.should_receive(<span style="color:#A60">:authorize!</span>) <span style="color:#080;font-weight:bold">do</span>
<span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>.connection.open_transactions.should == <span style="color:#00D">0</span>
<span style="color:#080;font-weight:bold">end</span>
order.payments.create!({
<span style="color:#A60">:amount</span> => order.outstanding_balance,
<span style="color:#A60">:payment_method</span> => payment_method,
<span style="color:#A60">:source</span> => creditcard})
order.next!
<span style="color:#080;font-weight:bold">end</span>
</pre></div>
</div>
<p>Now all I had to do was get this test to pass. To do this, I had to dig into
<code>state_machine</code>, particularly the
<a href="https://github.com/pluginaweek/state_machine/commit/fb0259ea5aa17b2924a7007d6d1e2a12f743a301"><code>1.1.2</code></a> release which Spree <code>1-3-stable</code> uses.
As it turns out, there are some <a href="https://github.com/pluginaweek/state_machine/blob/fb0259ea5aa17b2924a7007d6d1e2a12f743a301/lib/state_machine/integrations/active_record.rb#L131">inline
comments</a>
that were a big help:</p>
<div class="CodeRay">
<div class="code"><pre>
<span style="color:#777"># To turn off transactions:</span>
<span style="color:#777">#</span>
<span style="color:#777"># class Vehicle < ActiveRecord::Base</span>
<span style="color:#777"># state_machine :initial => :parked, :use_transactions => false do</span>
<span style="color:#777"># ...</span>
<span style="color:#777"># end</span>
<span style="color:#777"># end</span>
<span style="color:#777">#</span>
<span style="color:#777"># If using the +save+ action for the machine, this option will be ignored as</span>
<span style="color:#777"># the transaction will be created by ActiveRecord within +save+. To avoid</span>
<span style="color:#777"># this, use a different action like so:</span>
<span style="color:#777">#</span>
<span style="color:#777"># class Vehicle < ActiveRecord::Base</span>
<span style="color:#777"># state_machine :initial => :parked, :use_transactions => false, :action => :save_state do</span>
<span style="color:#777"># ...</span>
<span style="color:#777"># end</span>
<span style="color:#777">#</span>
<span style="color:#777"># alias_method :save_state, :save</span>
<span style="color:#777"># end</span>
</pre></div>
</div>
<p>So here was my answer: add the <code>:use_transactions => false</code> and <code>:action =>
:save_state</code> options and alias <code>save</code> to <code>save_state</code>. (The
details of why we have to alias <code>save</code> to <code>save_state</code> are pretty involved,
but the basic point is that <code>state_machine</code> runs its transition callbacks
as ActiveRecord callbacks if the action is <code>save</code>, and these callbacks are
called <em>within</em> the transaction. State transition callbacks on any other
action are run <em>around</em> the action, so are not within the transaction.)</p>
<p>With these small changes, my test passed and the processing state actually
began to do its job. I’m happy to say that those changes have now been
incorporated into the master (<a href="https://github.com/spree/spree/pull/4542">PR</a>)
and <code>1-3-stable</code> (<a href="https://github.com/spree/spree/pull/4499">PR</a>) branches of
Spree, and I can count myself as one of the contributors to the project.
(Thanks in particular to <a href="https://github.com/huoxito">@huoxito</a> who read and
responded to my <a href="https://github.com/spree/spree/pull/4499#issuecomment-39213981">rambling
comments</a> on
the pull request.)</p>
<h2>Epilogue</h2>
<p>This little “bug hunting” expedition was quite an eye-opening experience for
me. The bug itself of course taught me a lot about the internals of Spree.
More than that, though, it highlighted to me how essential it is that programmers
have the space and freedom to dive deep into code – as deep as necessary –
to solve difficult, often unexpected problems.</p>
<p>Here is a case where a silently failing feature in the core of a widely-used library
emerges as an annoying double-click bug, one that could easily be ignored or
patched at the surface level. To actually <em>fix</em> the bug required the patience
to yank it out by its roots, delving through application code, to a dependency on a forked core
library (spree), to the interaction of that library with its own dependency
(state machine).</p>
<p>At any point, it would be very easy for something (an impatient manager,
impending deadline or unexpected event) to interrupt the bug hunt and leave
the core library broken. No disaster would have resulted had that happened in
this case, and I could have spared myself days of frustrating debugging.</p>
<p>But in the long run, this would leave us with a broken payment system at the
core of all our e-commerce applications. If anyone underestimated the
importance of having reliable open-source libraries, <a href="http://heartbleed.com/">recent
events</a> have provided ample evidence to convince them
otherwise.</p>
<p>Disregarding the bug would also have left
me, the programmer, with a gaping hole in my understanding. It would have
robbed me of a great chance to understand how the system works. And it would
have left me with the disempowering sense that the bug was beyond my understanding,
when in fact it was not.</p>
<p>I’m glad that didn’t happen to me in this case. But I have to wonder how often
it does, and how that type of failure – coming back empty-handed after hours
or even days of fruitless bug hunting – must affect programmers and the code
they write. The experience has given me a greater appreciation of how much a work
environment can contribute to code quality. And it has taught me to <em>slow
down</em>: to look at problems carefully before rushing in to solve them, to think
carefully before rushing to write code.</p>
<p>Here’s hoping those lessons will lead to more successful bug hunts, and less
bugs.</p>
Software, the Moving Targethttps://dejimata.com/2013/11/27/software-the-moving-target2013-11-272013-11-27Chris Salzberg<p>Yukihiro “Matz” Matsumoto, creator of the Ruby programming language, gave a fascinating
<a href="https://www.evernote.com/shard/s54/sh/9cd05cf6-6119-4baa-a1ae-31bd2f23ae34/16638f82bd08ac21c5ad4206d81630de">keynote talk</a> at this
year’s <a href="http://www.rubyworld-conf.org/en/">RubyWorld conference</a> the other day that
left me thinking. It was a talk about Ruby,
about the recent history of software, but also about much more than that…</p>
<p>Yukihiro “Matz” Matsumoto, creator of the Ruby programming language, gave a fascinating
<a href="https://www.evernote.com/shard/s54/sh/9cd05cf6-6119-4baa-a1ae-31bd2f23ae34/16638f82bd08ac21c5ad4206d81630de">keynote talk</a> at this
year’s <a href="http://www.rubyworld-conf.org/en/">RubyWorld conference</a> the other day that
left me thinking. It was a talk about Ruby,
about the recent history of software, but also about much more than that.</p>
<div class="img-right">
<a href="http://instagram.com/p/g9YcEsCqFa/">
<img alt="Yukihiro Matsumoto at RubyWorld 2013" src="/img/rubyworld-aiming-moving-target.png"></a>
<p class="caption">Yukihiro "Matz" Matsumoto at RubyWorld 2013<br />
<small>(Photo by Osamu Fukui)</small></p>
</div>
<p>In his trademark style, Matz recounted this history through the story of his
own life. That story
begins just before he first created Ruby, 20 years ago,
in an era where hardware was expensive, software took years to create,
and documentation was measured not in number of pages but in centimeters of thickness.</p>
<h2>“20 years ago, we should have admitted our ignorance.”</h2>
<p>Software was like nothing mankind had ever created before, a digital
embodiment of human thought. But too much money was
invested in hardware and infrastructure for companies to admit to clients
that they knew nothing about how to build it. Failure was just too costly.</p>
<p>Now, looking back from an era where hardware is cheap, software is commonly developed
in weeks or months, not
years, and documentation is measured in commits, not pages, some things become clear.</p>
<p>The first is that our basic assumptions were wrong, foremost among them the
assumption that “we know what we should make”. Were this true, software
development would simply be a matter of specifying requirements, setting out a
plan based on those requirements, and executing the plan.</p>
<p>Except, that doesn’t work. Or at least, it doesn’t work very well.</p>
<div class="img-left">
<a href="http://www.hsbt.org/diary/20131121.html"><img
alt="Moody skies at RubyWorld 2013" src="/img/rubyworld.png"></a>
<p class="caption">Moody skies at RubyWorld 2013<br /><small>(Photo by Hiroshi
Shibata)</small></p>
</div>
<p>Before creating Ruby, Matz himself worked at a company producing enterprise
software this way, with large projects executed over many years and code written
to specification, <a href="https://en.wikipedia.org/wiki/Waterfall_model">waterfall</a>-style. Clients said what they wanted,
specifications were drawn up, and programmers wrote code to satisfy those
specs.</p>
<p>And in end, something was produced. It may not have been very
satisfactory, or even satisfactory at all, but there was <em>something</em> to show for the
many years of time and resources invested, and that something was produced on
schedule. That was the minimum requirement.</p>
<p>But twenty years ago, Matz already felt that something was wrong. The big
projects at his company treated software just like automobiles, or cabinets, or toasters, something to be
designed, manufactured and delivered. But code is not physical, it’s digital,
and it follows a different set of laws.</p>
<p>The challenge of grappling with those laws was vastly underestimated then just
as it is now. The word “software” itself, coinded to describe the stuff being
coded, evoked the wrong image: software was not “soft” at all, but hard. Very hard.</p>
<div class="img-right">
<a href="https://commons.wikimedia.org/wiki/File:NASAComputerRoom7090.NARA.jpg"><img
alt="IBM 7090 computer" src="/img/ibm-7090.png"></a>
<p class="caption">Once upon a time, computing was expensive</p>
</div>
<h2>“Software may be the most complex thing ever created by human beings.”</h2>
<p>It is fascinating to me to think that software is at once one of the most complex
things we have ever created, and also something that defies our basic
instincts about what it is to “create”. Thousands of years of blood and sweat
have been spent learning to build things
in the physical world, so it’s not exactly surprising that we want to apply these lessons to
the digital world.</p>
<p>But they fail us, Matz said, and not in a good way – at high cost, without teaching us anything.
They fail us because
we are not developing
software in a predictable, cartesian world like the one we live in, but in an
abstract world that is unfamiliar and constantly changing.</p>
<p>In this world, where our real-world instincts are often wrong, we need to try
and fail often, in a repeatable way, and cheaply. Large projects of the kind
Matz had been working on 20 years ago do exactly the opposite.</p>
<p>The message was not exactly new, but Matz expressed it with such simplicity
that it seemed self-evident. He did not profess to have the solution to the
problem, or even suggest that there was one solution to the problem, but only
offered some hints drawn from experience.</p>
<p>“Keep moving,” he said, advising against the conservative “ostrich”
strategy of putting your head in the sand in the hope that things will return
to normal. (They won’t.)</p>
<p>“Work hard to code less,” he said. Collaborate, leverage others' work, use
existing tools. In other words, stand on the shoulder of giants. This made a lot of sense to me.</p>
<div class="img-left">
<a href="https://www.facebook.com/photo.php?fbid=470262379759385&set=pb.214817381970554.-2207520000.1385528208.&type=3&theater"><img
alt="Yukihiro Matsumoto at RubyWorld 2013: Write great code, change the world" src="/img/rubyworld-change-the-world.png"></a>
<p class="caption">Write great code, change the world</p>
</div>
<h2>“Write great code. Change the world.”</h2>
<p>Ultimately though, this was the most powerful message. Ruby was designed to make this possible, by <a href="http://www.artima.com/intv/rubyP.html">maximizing programmer
happiness</a>. But Matz, to his credit,
hardly mentioned Ruby. He was thinking more broadly.</p>
<p>And when thinking more broadly, something struck me as curious about the whole
gathering at RubyWorld.</p>
<p>Here we were, brought together by a programming
language named Ruby. Ruby was designed by one programmer in his free time at a large
corporation producing enterprise software to specification. Twenty years
later, this programming language is used around the world, forms the foundation for
one of the world’s most popular web frameworks,
and has arguably elevated Japan’s status and visibility in the global technology sphere.</p>
<p>The fact that this programmer had the free time to develop Ruby was pure luck.
Certainly nobody at his company ever allocated time to develop a quirky programming language to
build better software by increasing programmer happiness.</p>
<p>The situation today is not so different: industry and government invest
massive sums of money in
<a href="http://www.fujitsu.com/global/about/tech/k/">large, expensive projects</a>
destined to the dustbin of history, while largely ignoring a homegrown success story right at their doorstep.</p>
<p>Now here was the creator of Ruby telling us that the existing approach to building software is wrong.
Were Matz not such a gifted storyteller, this message might come off as
arrogant or paternalistic. But it didn’t.</p>
<p>And yet.</p>
<p>I watch everybody nodding their heads in agreement. But are they really
listening?</p>
<p>For a core group of hackers and entrepreneurs mostly in Tokyo, early adopters of
every new technology, what he’s saying is old news. They’ve heard the message
before – by now, many have internalized it. They’re looking into their
laptops as he speaks, acting on his advice without even hearing it.</p>
<p>Matz isn’t really speaking to them. He’s speaking to everyone else. And he’s
trying, in his very personable, self-effacing style, to get a message across
to them.</p>
<div class="img-right">
<a href="https://www.facebook.com/photo.php?fbid=470262379759385&set=pb.214817381970554.-2207520000.1385528208.&type=3&theate://commons.wikimedia.org/wiki/File:Alice_%26_Red_Queen.jpg"><img
alt="Red Queen and Alice in Alice in Wonderland" src="/img/red-queen.png"></a>
<p class="caption">The Red Queen from Alice in Wonderland</p>
</div>
<h2>“If you want to get somewhere else, you must run at least twice as fast as that! ”</h2>
<p>Software is not just something which affects software developers. Not anymore.
It’s not just in the computers, it’s in the automobiles and the toasters, too.
It won’t be long before it’s in almost everything.</p>
<p>And just as it is hard for software developers to build software, so it is
hard for everyone else to understand software, and the potential
of software. People look to lessons from
the past. When those lessons do not provide useful clues, they
stick their head in the sand, hoping things eventually return to normal.</p>
<p>But they won’t.</p>
<p>Software is fundamentally different. It’s a moving target. Simply to
keep up takes enormous effort – to actually push the boundaries, you need to
run twice as fast, think twice as hard, stretch your imagination twice as far
– so most people just stick to doing the same thing, and demanding more of the same
thing.</p>
<p>We need to change more than just how we create software. We need to change how people
understand software.</p>
<p>And in that sense, watching his talk, it seemed to me that Matz the
storyteller was at least as influential in today’s world as Matz the programmer
was 20 years ago. Ruby is a language that changed how programmers write code.
The lesson of Ruby is a story that could change how people understand code.</p>
<p>That’s a story that I want to read.</p>
Bundler, Meet Bowerhttps://dejimata.com/2013/11/4/bundler-meet-bower2013-11-042013-11-04Chris Salzberg<p><a href="http://bundler.io/">Bundler</a> is a package management tool for ruby gems…</p>
<p><a href="http://bundler.io/">Bundler</a> is a package management tool for ruby gems.
It takes in a ruby file with a list of gems and version numbers, like this:</p>
<div class="CodeRay">
<div class="code"><pre>
source <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">https://rubygems.org</span><span style="color:#710">'</span></span>
gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">rails</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">4.0.0</span><span style="color:#710">'</span></span>
gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">sqlite3</span><span style="color:#710">'</span></span>
gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">sass-rails</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">~> 4.0.0</span><span style="color:#710">'</span></span>
gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">uglifier</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">>= 1.3.0</span><span style="color:#710">'</span></span>
gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">coffee-rails</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">~> 4.0.0</span><span style="color:#710">'</span></span>
gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">jquery-rails</span><span style="color:#710">'</span></span>
</pre></div>
</div>
<p>and a single command:</p>
<div class="CodeRay">
<div class="code"><pre>
bundle install
</pre></div>
</div>
<p>and magically downloads all the dependencies for your project, so you don’t
have to hunt them down yourself or store them in your git repository.</p>
<p>That means that when you update a dependency, rather than see all the changes
to that dependency in your git history, you see this:</p>
<div class="CodeRay">
<div class="code"><pre>
-gem 'rails', '4.0.0'
+gem 'rails', '4.0.1'
</pre></div>
</div>
<p>Beautiful! Elegant and succinct – everything that I want to know expressed
in one line.</p>
<div style="text-align: center">
<img src="/img/bundler.png">
<p><small>Bundler: The best way to package your application's
dependencies.</small></p>
</div>
<p>But now wait, what about all those pesky assets? What do we do when we want to
update our version of <a href="http://backbonejs.org/">Backbone.js</a>? Well, without a package manager, we’ll end up
with an ugly commit like
<a href="https://github.com/netalab/cojiro/commit/19e9afc6e5865a0b5a7fb5ec87c4425a33eb23c8#diff-01501b24a7a876fc3ee38323019212d8">this</a>.</p>
<p>What? Did I make all those changes? No, but it sure looks like I did.</p>
<p>And how do I know that Backbone.js depends on jQuery and Underscore.js? I just
need to know. And if the dependencies change, I’ll have to go hunt down
the documentation to see what the new ones are.</p>
<p>Yuck. Not elegant at all.</p>
<p>It’s amazing to me that most of the Rails community seems satisfied with
this sorry state of affairs. I supect that this is one of many factors
contributing to the slow uptake of Javascript frameworks like Backbone.js by
Rails developers.</p>
<p>If assets are first-class citizens, then they deserve to be managed as such.</p>
<p>Enter <a href="http://bower.io/">Bower</a>, a package management tool for javascript libraries. It takes in a
JSON file with a list of javascript dependencies and version numbers, like
this:</p>
<div class="CodeRay">
<div class="code"><pre>
{
<span style="color:#606"><span style="color:#404">"</span><span>name</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">bower dependencies</span><span style="color:#710">"</span></span>,
<span style="color:#606"><span style="color:#404">"</span><span>dependencies</span><span style="color:#404">"</span></span>: {
<span style="color:#606"><span style="color:#404">"</span><span>backbone</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">1.0.0</span><span style="color:#710">"</span></span>,
<span style="color:#606"><span style="color:#404">"</span><span>backbone-relational</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">0.8.5</span><span style="color:#710">"</span></span>,
<span style="color:#606"><span style="color:#404">"</span><span>jquery</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">1.10.1</span><span style="color:#710">"</span></span>,
<span style="color:#606"><span style="color:#404">"</span><span>jquery-timeago</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">https://raw.github.com/rmm5t/jquery-timeago/b311508ee019650200f552015a48f69de18c5857/jquery.timeago.js</span><span style="color:#710">"</span></span>,
<span style="color:#606"><span style="color:#404">"</span><span>underscore</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">1.4.4</span><span style="color:#710">"</span></span>,
<span style="color:#606"><span style="color:#404">"</span><span>jquery-masonry</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">http://www.fishspotr.com/jquery.masonry.js</span><span style="color:#710">"</span></span>,
<span style="color:#606"><span style="color:#404">"</span><span>jquery-autosize</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">latest</span><span style="color:#710">"</span></span>
},
<span style="color:#606"><span style="color:#404">"</span><span>devDependencies</span><span style="color:#404">"</span></span>: {
<span style="color:#606"><span style="color:#404">"</span><span>sinon</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">http://sinonjs.org/releases/sinon-1.4.2.js</span><span style="color:#710">"</span></span>,
<span style="color:#606"><span style="color:#404">"</span><span>jasmine-sinon</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">0.1.0</span><span style="color:#710">"</span></span>
}
}
</pre></div>
</div>
<p>and a single command:</p>
<div class="CodeRay">
<div class="code"><pre>
bower install
</pre></div>
</div>
<p>and magically downloads all the dependencies for your project, so you don’t
have to hunt them down yourself or store them in your git repository.</p>
<p>That means that when you update a dependency, rather than see all the changes
to that dependency in your git history, you see this:</p>
<div class="CodeRay">
<div class="code"><pre>
- "backbone": "1.0.0",
+ "backbone": "1.1.0",
</pre></div>
</div>
<p>Beautiful! Elegant and succinct – everything that I want to know expressed
in one line.</p>
<div style="text-align: center">
<img src="/img/bower.png">
<p><small>Bower: A package manager for the web.</small></p>
</div>
<p>The amazing thing is that these two package managers work beautifully in
tandem: bundler for backend packages (gems) and bower for front-end assets.</p>
<p>I recently added bower to our pipeline and <a href="https://github.com/netalab/cojiro/commit/12bb5a00fa4eb95fde3023e8e5cefec63edc8371">zapped 8,935 lines in one
commit</a>.
Those were all lines of code from assets that should never have been in our
repository in the first place. All of that code is now expressed in a single
<code>bower.json</code> file, used at build-time to download all packages and
dependencies, just like you would with bundler.</p>
<p>Now that we’ve switched to bower, I can’t believe that anyone would manage
assets without it.</p>
<hr />
<p>For those interested in trying out bower with a Rails stack, here are a few
steps to get you going. First, assuming you have <a href="http://nodejs.org">node and
npm</a> installed
(they’re packaged together nowadays, so installing once will get you both), you’ll
can install bower with:</p>
<div class="CodeRay">
<div class="code"><pre>
npm install -g bower
</pre></div>
</div>
<p>Next, create a directory in <code>vendor/assets</code> called
<code>bower_components</code>. Then add a <code>.bowerrc</code> file to your root directory to
tell bower to install assets in that directory:</p>
<div class="CodeRay">
<div class="code"><pre>
{
<span style="color:#606"><span style="color:#404">"</span><span>directory</span><span style="color:#404">"</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">"</span><span style="color:#D20">vendor/assets/bower_components</span><span style="color:#710">"</span></span>
}
</pre></div>
</div>
<p>You’ll probably want to ignore that directory in your <code>.gitignore</code> file since
you the whole point is not to commit these dependencies to the git repository.</p>
<p>Now you’ll need to describe your dependencies in a JSON file named <code>bower.json</code>.
The key element of that file is the <code>dependencies</code> hash, which contains all
your apps dependencies and their version numbers (see the example above).</p>
<p>Once you have that file, just run: <code>bower install</code> to pull them all in with
whatever their dependencies.</p>
<p>There are a couple other caveats. You have to add <code>bower_components</code> to the
<code>assets.paths</code> config to get the asset pipeline to pick it up. In
<code>config/application.rb</code>, add the line:</p>
<div class="CodeRay">
<div class="code"><pre>
config.assets.paths << <span style="color:#036;font-weight:bold">Rails</span>.root.join(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">vendor</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">assets</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">bower_components</span><span style="color:#710">'</span></span>)
</pre></div>
</div>
<p>Also, since bower packages are each given their own directory, you will need
to require them with <code>jquery/jquery</code> rather than just <code>jquery</code> as you normally
would, e.g.:</p>
<div class="CodeRay">
<div class="code"><pre>
#= require jquery/jquery
</pre></div>
</div>
<p>Incidentally, if you dig into <a href="https://github.com/sstephenson/sprockets">Sprockets</a>, the Rack-based asset packaging system
used by the Rails asset pipeline, you’ll see that it has in fact recently
<a href="https://github.com/sstephenson/sprockets/commit/5dad425ed66eeb94223c266cbd2b1a1086d8252a">added support for <code>bower.json</code> configuration
files in
assets</a>
and if it finds one, will let you load the package
with its name rather than the full path. However, many bower packages continue to
name their configuration file <code>package.json</code> instead of <code>bower.json</code>, which
sprockets will not find, so this
setup will not work universally. (I submitted a <a href="https://github.com/sstephenson/sprockets/pull/484">pull
request</a> to change this
but it was rejected. Bah.)</p>
<p>Also, somewhat annoyingly, the bower assets will be given
a low priority in the asset search paths, so if you have gems that include
assets, those assets will take precedence. Better just to use the full path
(i.e. <code>jquery/jquery</code>) and avoid any potentially confusion bugs down the line.</p>
<p>That being said, using bower with a Rails stack is easy peasy and brings the full
ecosystem of asset packages to your fingertips. Take the plunge, you won’t
regret it!</p>
<p>Further reading: <a href="https://shellycloud.com/blog/2013/09/how-to-manage-front-end-packages-in-rails-with-bower">How to manage front-end packages in
Rails with Bower</a></p>