dejimata https://dejimata.com 2013-02-13 Chris Salzberg Transcript of my EuRuKo 2018 Talk https://dejimata.com/2018/8/23/transcript-of-my-euruko-2018-talk 2018-08-23 2018-08-23 Chris Salzberg <p>This is a very rough transcript of my talk, <a href="https://speakerdeck.com/shioyama/metaprogramming-for-generalists">Metaprogramming for Generalists</a>, given at this year&rsquo;s <a href="https://euruko2018.org/">EuRuKo</a> conference in Vienna,</p> <p>This is a very rough transcript of my talk, <a href="https://speakerdeck.com/shioyama/metaprogramming-for-generalists">Metaprogramming for Generalists</a>, given at this year&rsquo;s <a href="https://euruko2018.org/">EuRuKo</a> conference in Vienna, Austria. There is also a <a href="https://www.youtube.com/watch?v=1fIlcnrJHxs&amp;feature=youtu.be&amp;t=286">video of the talk on YouTube</a>.</p> <script async class="speakerdeck-embed" data-id="d451cf97ffa5413c9b79d2c73f3feb51" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script> <hr /> <p>Hi everybody, my name is Chris Salzberg and the title of this talk is &ldquo;Metaprogramming for Generalists”. I know I know, you&rsquo;re thinking, "He&rsquo;s starting the morning of the first day with metaprogramming?!&rdquo;</p> <p>But don&rsquo;t worry! This will be interesting.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="About Me" src="/img/201808/slide-1.png" /> </div> <h2>About Me</h2> <p>Ok, so first about me. My handle is @shioyama and I live in Tokyo, in Japan. But don&rsquo;t be fooled! I am <em>not</em> Japanese, in case that wasn&rsquo;t obvious. I&rsquo;m actually a Canadian originally from Montreal.</p> <p>I work at a company called <a href="https://degica.com">Degica</a> based in Tokyo. We have a booth so be sure to check us out.</p> <p>In the open source world, I&rsquo;m the author of a few gems, the most well-known one is a gem called <a href="https://github.com/shioyama/mobility">Mobility</a> for managing model translations.</p> <p>And I write about things like the <a href="/2017/5/20/the-ruby-module-builder-pattern">Module Builder Pattern</a> at my blog, dejimata.com.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Plan" src="/img/201808/slide-2.png" /> </div> <h2>Plan</h2> <p>This is going to be a two-part talk. In the first part, I&rsquo;m going to introduce a different, more meaningful way to think about metaprogramming that will be useful for solving generic problems.</p> <p>Then in the second part, we&rsquo;ll solve a generic problem together in order to build what I will refer to as &ldquo;generic software&rdquo;.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Generalist Metaprogramming" src="/img/201808/slide-3.png" /> </div> <h2>Generalist Metaprogramming</h2> <p>Ok, so let&rsquo;s start. &ldquo;Generalist Metaprogramming&rdquo;, what in the world is that?</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="What's Metaprogramming?" src="/img/201808/slide-4.png" /> </div> <h2>What&rsquo;s Metaprogramming?</h2> <p>Well, this year is Ruby&rsquo;s 25th birthday, right?</p> <p>And I think more than anything else, metaprogramming is an area where Ruby really stands out among programming languages. Because Ruby really embraces this concept. It underlies all these things we love, the DSLs, the human language-like syntax, all that stuff.</p> <p>But we have a darn hard time defining this thing.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Defining Metaprogramming" src="/img/201808/slide-9.png" /> </div> <h2>Defining Metaprogramming</h2> <p>Well ok, it&rsquo;s a noun. I think we can agree on that. Unfortunately, that&rsquo;s about all we can really agree on.</p> <p>If &ldquo;programming&rdquo; is &ldquo;writing code&rdquo;, then the first obvious definition is &ldquo;writing code that writes code&rdquo;. But this doesn&rsquo;t really cover everything we do with metaprogramming.</p> <p>So the second definition is a &ldquo;technique in which computer programs have the ability to treat programs as their data”. This one is from Wikipedia, it’s also something referred to as "reflection”.</p> <p>So now we&rsquo;re thinking more broadly. But if you look at the talk page on Wikipedia, you&rsquo;ll see that not everybody agrees with this one either.</p> <p>Another angle on metaprogramming is called &ldquo;introspection&rdquo;. A program is a subject, and it can find out things about other programs.</p> <p>And then, at some point, we just throw up our hands and call it &ldquo;a bag of tricks&rdquo;. This by the way is from <a href="https://pragprog.com/book/ppmetr/metaprogramming-ruby">Metaprogramming Ruby</a>, which by the way is a great book.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Defining Metaprogramming" src="/img/201808/slide-10.png" /> </div> <h2>Bag of Tricks</h2> <p>And what are those tricks? Well, here they are. Some of you will recognize many of these. I think everyone will recognize &ldquo;monkeypatching&rdquo;.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Working Definition" src="/img/201808/slide-11.png" /> </div> <h2>The &ldquo;What&rdquo; View of Metaprogramming</h2> <p>But the thing is that, in practice, we tend to use a working definition that is much more basic. We label a subset of Ruby as &ldquo;metaprogramming&rdquo;, methods like eval, class_eval, define_method, etc.</p> <p>And most talks about metaprogramming start there, with those methods, explaining how they work, what they do. They start with the &ldquo;what&rdquo; of metaprogramming.</p> <p>But I don&rsquo;t want to give that kind of talk today.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Metaprogramming as Fringe" src="/img/201808/slide-12.png" /> </div> <h2>Metaprogramming as Fringe</h2> <p>And this is the reason why. It&rsquo;s that for most programmers, this is what metaprogramming looks like. If they know about it at all, they just fence it off and label it as dangerous, or magical, or more likely just not useful to them.</p> <p>But another part of the community, myself included &mdash; and I suspect many people here today &mdash; finds these methods absolutely essential, actually the core of Ruby.</p> <p>And the reason for this divide is that most everything out there about metaprogramming with this &ldquo;what&rdquo; focus does not answer the obvious question: what on Earth is metaprogramming for?</p> <p>And that&rsquo;s the question I want to start with. To do that, I want to think about the structure of our ecosystem.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Metaprogramming as Fringe" src="/img/201808/slide-14.png" /> </div> <h2>Ruby</h2> <p>So let’s start with a bit of fire down here, and put Ruby into the fire. Matz is feeling the heat!</p> <p>So down here, this our programming language, where it is being forged out of fire and brimstone by Matz and his brave team of Ruby committers.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Applications" src="/img/201808/slide-15.png" /> </div> <h2>Applications</h2> <p>And then up here, we have where most of us spend our time: building applications. This is what pays our bills, this is what delivers value to our users.</p> <p>And as users of the language, application developers know enough of the language to use it, but most don&rsquo;t see or care much about its internals.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Applications" src="/img/201808/slide-16.png" /> </div> <h2>Libraries</h2> <p>Now, bridging these two worlds, we have the libraries. Or &ldquo;gems&rdquo;, but I&rsquo;ll use the term &ldquo;library&rdquo; here.</p> <p>Libraries fill the gaps in generic things we need - things that more than one application need - that are not supplied by the programming language.</p> <p>And in Ruby in particular, these libraries are almost seamless in how they integrate with the programming language and with applications.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Metaprogramming lives here" src="/img/201808/slide-17.png" /> </div> <h2>Metaprogramming Lives Here</h2> <p>And here&rsquo;s the thing, right? It&rsquo;s that metaprogramming overwhelmingly lives in this middle layer here, in the libraries.</p> <p>Just taking Rails as an example, in Rails, it’s metaprogramming that is used to define everything you care about: attribute methods, associations, all model-specific stuff, view rendering logic, whatever.</p> <p>But it&rsquo;s not only ActiveRecord. Most of the widely-used gems we use heavily leverage metaprogramming to do what they do.</p> <p>But applications don&rsquo;t use metaprogramming much at all.</p> <p>And that brings us to an obvious question, right?</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Why?" src="/img/201808/slide-18.png" /> </div> <h2>Why?</h2> <p><em>Why?</em> &mdash; Why is metaprogramming so much more prevalent in our dependencies than in our own application code?</p> <p>This is such an important question. It&rsquo;s telling us something very important about the nature of our programming language, and the ecosystem of components built out of it. But nobody is really asking this question, that I can see anyway.</p> <p>I think part of the reason is maybe because it seems kind of obvious maybe, but I think it’s actually quite subtle. So let’s think about it a bit together now.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Generalization" src="/img/201808/slide-20.png" /> </div> <h2>Generalization</h2> <p>Here&rsquo;s another word, &ldquo;generalization&rdquo;. It&rsquo;s the root of the second big word in the title of this talk.</p> <p>Generalization is what libraries do. They generalize problems that many different applications have into generic components.</p> <p>Generalization is the &ldquo;formulation of general concepts from specific instances by abstracting common properties.&rdquo; This is from Wikipedia again.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Abstracting Common Properties" src="/img/201808/slide-21.png" /> </div> <h2>Abstracting Common Properties</h2> <p>This part, &ldquo;abstracting common properties&rdquo; from specific instances, is what we are going to do in the second part of this talk.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="General Concepts" src="/img/201808/slide-22.png" /> </div> <h2>General Concepts</h2> <p>First, though, I want to focus on &ldquo;general concepts”, and try to figure out the relation of general concepts with metaprogramming.</p> <p>What&rsquo;s a general concept that is relevant to everyone here?</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Attribute" src="/img/201808/slide-23.png" /> </div> <h2>Attribute</h2> <p>Well, I&rsquo;m going to wager that this one is pretty relevant to almost everyone working with Ruby. The concept of &ldquo;attribute&rdquo;.</p> <p>We want to figure out what role metaprogramming is playing in this kind of general concept, because it&rsquo;s this kind of general concept that libraries implement for us.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Knockout Experiment" src="/img/201808/slide-24.png" /> </div> <h2>Knockout Experiment</h2> <p>So to do this we&rsquo;re going to take a cue from genetics. What does a geneticist do when they want to understand what a gene is doing?</p> <p>Well, they knock it out. They just knock it out. And the they look to see what&rsquo;s different about the mouse with the knocked out gene, compared to the mouse with the gene left in.</p> <p>So we&rsquo;re gong to do the same. We&rsquo;ll knock out metaprogramming from the concept of &ldquo;attribute&rdquo;, and see what happens.</p> <hr /> <div class="img-center"> <img alt="Control: Library View" src="/img/201808/slide-25.png" /> </div> <h2>Control: Library View</h2> <p>So let&rsquo;s start with the control. The control is the &ldquo;normal&rdquo; case, the one with metaprogramming.</p> <p>This is the library view on the control. In each case, control and knockout, there&rsquo;s a single class called &ldquo;Model&rdquo;, with a simple initializer that creates an empty hash to store attributes.</p> <p>For the control here, we then have a class method, define_attribute, which takes an attribute name and defines getter and setter methods for that attribute. The getter just grabs the value for the key matching the method name, and the setter sets the value for the key matching the method name.</p> <hr /> <div class="img-center"> <img alt="Control: Application View" src="/img/201808/slide-26.png" /> </div> <h2>Control: Application View</h2> <p>And now here we&rsquo;re switching to the application view, which uses this library.</p> <p>In the application we define a class <code>Talk</code> which subclasses <code>Model</code>. And we call define_attribute twice to define two attributes, <code>title</code> and <code>abstract</code>.</p> <p>And then here set and get the attributes like this, set the title and get the title, and set the abstract and get the abstract.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Abstraction is Transparent" src="/img/201808/slide-27.png" /> </div> <p>The point I want to make here is that the abstraction, remember the abstraction of &ldquo;attribute&rdquo;, is <em>transparent</em>, or you could say it&rsquo;s <em>invisible</em>. It&rsquo;s visible when we declare it of course, but after that it&rsquo;s entirely transparent when we use it.</p> <p>This is what we are used to and expect, but I want to highlight that.</p> <hr /> <div class="img-center"> <img alt="Knockout: Library View" src="/img/201808/slide-28.png" /> </div> <h2>Knockout: Library View</h2> <p>Ok now for the knockout. We start with the library view like we did for the control.</p> <p>In the knockout, we have no metaprogramming, so we can&rsquo;t use define_method like we did in the control. We can&rsquo;t define methods dynamically like that.</p> <p>So instead we define two instance methods, &ldquo;get_attribute&rdquo; and &ldquo;set_attribute&rdquo;, which is the only other way to really do it. Those methods do the same thing as the methods in the control, set and get attributes as keys on the @attributes hash.</p> <hr /> <div class="img-center"> <img alt="Knockout: Application View" src="/img/201808/slide-29.png" /> </div> <h2>Knockout: Application View</h2> <p>And here is the application view for the knockout.</p> <p>In this case we don&rsquo;t need to declare the attributes because our implementation will let you set and get anything you like (this is not really important).</p> <p>Now in this case, as I mentioned, when we want to get and set attributes, we can&rsquo;t just call &ldquo;title&rdquo; and &ldquo;abstract&rdquo; as methods because the library &ldquo;Model&rdquo; can&rsquo;t dynamically define those methods, since it has no metaprogramming to do that.</p> <p>So instead we call get_attribute and set_attribute to get and set attributes (keys on the @attributes hash).</p> <hr /> <div class="img-center"> <img alt="Abstraction is Visible" src="/img/201808/slide-30.png" /> </div> <h2>Abstraction is <em>Visible</em></h2> <p>So I want to highlight two things here. The first is that <em>the abstraction is now visible</em>. Remember that our abstraction is &ldquo;attribute&rdquo;, and in the control case once declared, we didn&rsquo;t need to think about it, we could just talk about the talk&rsquo;s title and abstract.</p> <p>But here, it&rsquo;s in our face. Every time you access an attribute, you need to actually write the word &ldquo;attribute&rdquo;.</p> <hr /> <div class="img-center"> <img alt="Names are Arguments" src="/img/201808/slide-31.png" /> </div> <h2>Names are Arguments</h2> <p>And the other thing is that names are now arguments, in this case symbols, that we pass to these methods. They are not methods now, and so we don&rsquo;t send them as messages to the objects but as arguments to methods on the object.</p> <hr /> <div class="img-center"> <img alt="Results: Application View" src="/img/201808/slide-33.png" /> </div> <h2>Results: Application View</h2> <p>Ok, so let&rsquo;s tabulate the results of our experiment! We have two rows here, the first for the view from the application, the second for the view from the library. It&rsquo;s really important to draw a distinction between these two.</p> <p>And we have two columns, one for the control and one for the knockout. Recall that the control is with metaprogramming, and the knockout is without metaprogramming.</p> <p>So from the application point of view, in the control the abstraction is transparent, or invisible, and we use the abstraction, i.e. we use attributes, by sending their names as messages to the object. And so we can say here that the library speaks in the language of the domain, i.e. you send the message &ldquo;title&rdquo; to the &ldquo;talk&rdquo; to get the title, nothing explicitly mentioning the abstraction itself there.</p> <p>In the knockout, on the other hand, the abstraction is very much visible, remember we had those methods get_attribute and set_attribute. Also different: you send the attributes as values to a method, remember they were the symbols &ldquo;title&rdquo; and &ldquo;abstract&rdquo;.</p> <p>So we can say here that <strong>the library speaks in the language of the abstraction</strong>. You really can&rsquo;t avoid it as an application developer.</p> <hr /> <div class="img-center"> <img alt="Results: Library View" src="/img/201808/slide-34.png" /> </div> <h2>Results: Library View</h2> <p>Now let&rsquo;s consider the library that implements this Model class. What’s going to be important here is the unknowns and what they refer to.</p> <p>In the control, unknowns refer to code, to the attribute methods that we define with &ldquo;define_attribute&rdquo;. They are unknown because while they could be &ldquo;title&rdquo; and &ldquo;abstract&rdquo;, they could also be anything else. This makes library code hard to understand. That&rsquo;s not obvious here because it&rsquo;s a trivial example, but in the next section I&rsquo;ll show how this can get more tricky.</p> <p>In the knockout, the unknowns are the names of the attributes and their values. Attribute names are not code, they’re just keys on a hash, and that makes it easier to reason about than code with data that references code (like the control).</p> <p>So if that&rsquo;s the case then, why as library authors don&rsquo;t we implement our code like the knockout?</p> <hr /> <div class="img-center"> <img alt="Knockout ORM" src="/img/201808/slide-35.png" /> </div> <h2>Knockout ORM</h2> <p>Well, here&rsquo;s the reason. Take the knockout example a bit further and imagine we implement an ORM like this, with associations. What would it look like?</p> <p>Well, like this. We now have two general concepts here, attribute and association, but we can&rsquo;t use metaprogramming for either of them, so the association abstraction is now also in your face.</p> <p>So although this type of explicit abstraction is easier to handle at the library level, now it&rsquo;s much more difficult to use at the application level.</p> <hr /> <div class="img-center"> <img alt="Would you use this?" src="/img/201808/slide-36.png" /> </div> <h2>Would you use this?</h2> <p>And so the question is: as an application developer, would you use this? Remember that at the application level, things like the name of the talk attributes <code>talk</code> and <code>abstract</code> are known to you.</p> <p>This means that if you wanted to, you could build AR like code even without metaprogramming by defining these methods explicitly, with <code>def</code>. It might take time, but it might also be better when it comes to actually using those methods, because now your model would speak your domain language.</p> <p>So the library would be at a disadvantage compared to the specific application. This is the really key point here.</p> <hr /> <div class="img-center"> <img alt="ORM with metaprogramming" src="/img/201808/slide-37.png" /> </div> <h2>ORM with metaprogramming</h2> <p>And now here&rsquo;s what it looks like with metaprogramming. This looks more like what we&rsquo;re used to.</p> <hr /> <div class="img-center"> <img alt="Metaprogramming is translating unknowns into code" src="/img/201808/slide-38.png" /> </div> <h2>Metaprogramming is Translating Unknowns into Code</h2> <p>So here&rsquo;s the takeaway: metaprogramming levels the playing field by enabling libraries to translate unknowns into code. It levels the playing field with the applications using the library.</p> <p>And seeing things this way &mdash; in terms of unknowns &mdash; really reveals a lot.</p> <p>Think about it for a second. What are the unknowns for the application? Well, the classic one is user input, right? Do you want to translate user input into code? Hell no!</p> <p>And so this explains the divide in perceptions of metaprogramming. From the library point of view, it&rsquo;s a leveler. From the application point of view it&rsquo;s a freaking menace!</p> <p>The other thing this new definition does is reveal some problems with our previous working definition, the &ldquo;what&rdquo; definition. Let&rsquo;s take a look at that now.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="eval('foo')" src="/img/201808/slide-40.png" /> </div> <h2>eval(&ldquo;foo&rdquo;)</h2> <p>Here we have <code>eval("foo")</code>, about the simplest kind of metaprogramming you can imagine, right?</p> <p>Now, our working definition says anything with a metaprogramming method is metaprogramming, so this is metaprogramming then, right? But something feels wrong about this.</p> <p>And the reason of course is that if we evaluate this, what do we get? We get <code>foo</code>, a call to the method foo or to a variable by that name. And that&rsquo;s not terribly magical is it? We certainly aren&rsquo;t &ldquo;translating unknowns into code&rdquo;.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="eval with string" src="/img/201808/slide-41.png" /> </div> <h2>eval(&ldquo;foo#{str}&rdquo;)</h2> <p>Ok, well what if we have a string, and we append &ldquo;foo&rdquo; to the string and eval that? Well, we still know the result here, which in this case would be calling foobar, whatever that is. So still not really interesting.</p> <p>So how can we make this string unknown?</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="fooval" src="/img/201808/slide-42.png" /> </div> <h2>fooval</h2> <p>Well, wrap it in a method, call it <code>fooval</code>. Now by definition, it&rsquo;s unknown, we can&rsquo;t know the value of the string here. So this is translating unknowns to code. And it&rsquo;s using <code>eval</code>, so in this case it lines up with both our expectation and our definition.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="attr_writer" src="/img/201808/slide-43.png" /> </div> <h2>attr_writer</h2> <p>Ok, let&rsquo;s take another case, <code>attr_writer</code>, with an argument &ldquo;foo&rdquo;.</p> <p>This is pretty straightforward, right? <code>attr_writer</code> is definitely not metaprogramming.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="attr_writer = def foo" src="/img/201808/slide-44.png" /> </div> <h2>attr_writer = def foo</h2> <p>And here&rsquo;s what this evaluates to, defining a writer method <code>foo</code>. And that&rsquo;s not translating unknowns to code, so ok this one lines up with our expectations.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="define_writer" src="/img/201808/slide-45.png" /> </div> <h2>define_writer</h2> <p>Ok, here&rsquo;s another one. This is almost the same as the second half of the <code>define_attribute</code> method I showed you earlier in the control (except that it sets an instance variable instead of a hash value). This one uses <code>define_method</code>, so in our working definition this is metaprogramming. It also translates unknowns to code, i.e. translates <code>name</code> to a method, so this lines up with our expectations too.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="define_writer aliasing attr_writer" src="/img/201808/slide-46.png" /> </div> <h2>define_writer aliasing attr_writer</h2> <p>But wait a second, that stuff in the method body can be re-written to look like this, right? It&rsquo;s just aliasing <code>attr_writer</code>. And we said that <code>attr_writer</code> is definitely not metaprogramming.</p> <p>But doing things like this, wrapping <code>attr_writer</code> in a method, or really just aliasing it, means that we&rsquo;re translating unknowns to code. Because that&rsquo;s really what <code>attr_writer</code>, and <code>attr_reader</code> and <code>attr_accessor</code>, do for us.</p> <p>We don&rsquo;t usually think about it this way because in 99% of cases we&rsquo;re passing known values to these methods, but they can be used this way too.</p> <p>So this is another case where our new definition does not line up with our working definition.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="what definition as Venn diagrams" src="/img/201808/slide-47.png" /> </div> <h2>Data Points in Venn Diagram</h2> <p>So let&rsquo;s visualize these results as Venn diagrams. Here we have our working definition, which I&rsquo;ll call the &ldquo;what&rdquo; definition. And we have the control and the knockout here, which seem fine.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="With additional data points" src="/img/201808/slide-48.png" /> </div> <h2>With additional data points</h2> <p>Now here are the other data points. Remember that <code>eval("foo")</code> is in this set, but it seemed like it didn&rsquo;t belong. The second <code>define_writer</code>, which basically aliased to <code>attr_writer</code>, seemed like it did belong there, but it&rsquo;s outside the set.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="New definition" src="/img/201808/slide-49.png" /> </div> <h2>New definition</h2> <p>So what we&rsquo;re going to do is to make a new definition, which will correspond to &ldquo;translating unknowns into code&rdquo;. And this definition is going to exclude the <code>eval("foo")</code> one, and include the second define_writer. It&rsquo;s going to do that because it uses a definition based on what things reduce to.</p> <p><code>eval("foo")</code> reduces to what we call &ldquo;normal programming&rdquo;, i.e. a simple method call. <code>attr_setter</code> wrapped in a method does not reduce to anything we&rsquo;d consider &ldquo;normal programming&rdquo;.</p> <p>The point is that this new definition is about behaviour. It&rsquo;s not about the what, it&rsquo;s about the <em>how</em>.</p> <p>So what’s normal programming then? Well, what if we invert this set on the right, and consider &ldquo;normal programming” as the niche.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Normal Programming" src="/img/201808/slide-50.png" /> </div> <h2>&ldquo;Normal&rdquo; Programming</h2> <p>So now what do we have. We have a set of methods and symbols and stuff that we consider &ldquo;normal&rdquo;. This is most of what you write as an application developer. Remember that this includes stuff that is normal and anything that reduces to stuff that&rsquo;s normal.</p> <p>In this world, code is code and data is data. Code can reference data, of course, but data never refers to code in this world. This is like the knockout example, where the attribute names did not refer to method names.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Normal Programming as Niche" src="/img/201808/slide-51.png" /> </div> <h2>Normal Programming as Niche</h2> <p>And if we now zoom out and look at the big picture, this is what we have. It is &ldquo;normal programming” in this picture, not "metaprogramming”, that is the niche. A very small niche in the much much larger space of computation, where we don’t draw any distinction between code and data.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Generic Software" src="/img/201808/slide-52.png" /> </div> <h2>Generic Software</h2> <p>And so where do the libraries that use metaprogramming live? Well, something like this, which I&rsquo;ll refer to as &ldquo;generic software&rdquo;. They venture beyond the bounds of &ldquo;normal programming&rdquo; to make generalizations. And to do that effectively, they break that code/data barrier.</p> <p>This way of understanding metaprogramming, as a means of generalizing from unknowns, is much more meaningful than any other definition out there that I have ever seen. And it really highlights what is so difficult, but also so powerful, about this concept.</p> <p>To show that in more detail now, let’s look at an example of this thing I’ve referred to as &ldquo;generic software”.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Building Generic Software" src="/img/201808/slide-53.png" /> </div> <h2>Building Generic Software</h2> <p>Ok, so &ldquo;generic software”, what in the world is that, you say?</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Generic Software: Definition" src="/img/201808/slide-54.png" /> </div> <h2>Generic Software: Definition</h2> <p>The term actually comes from a talk by Jeremy Evans, who is one of my heroes in Ruby. He gave a talk in 2012 about the development of Sequel, an ORM of which he’s the author, in which <a href="http://code.jeremyevans.net/presentations/heroku201205/index.html#68">he said</a>:</p> <blockquote><p>One of the best ways to write flexible software is to write generic software. Instead of designing a single API that completely handles a specific case, you write multiple APIs that handle smaller, more generic parts of that use case and then handling the entire case is just gluing those parts together.</p></blockquote> <p>And the key thing here is that the &ldquo;glue&rdquo;, in many cases and in particular in cases where the software needs to speak the domain language, will be metaprogramming, by the definition we came to in the last section.</p> <p>So I want to explore this idea of &ldquo;generic software&rdquo;, but we need a general concept. We&rsquo;ve already used &ldquo;attribute&rdquo;, so next I&rsquo;m going to use&hellip;</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Equality" src="/img/201808/slide-55.png" /> </div> <h2>Equality</h2> <p>Equality! Can&rsquo;t get much more general than that, right?</p> <hr /> <div class="img-center"> <img alt="Double Equals Equality" src="/img/201808/slide-56.png" /> </div> <h2>Double Equals Equality</h2> <p>So there are a few methods for equality in Ruby, we&rsquo;re going to be focusing here on &ldquo;double-equals&rdquo; equality. That&rsquo;s the kind of equality that compares object content, not object identity.</p> <p>So e.g., two strings which both have the characters <code>f</code> <code>o</code> <code>o</code> are double-equal equals even if they are different actual objects.</p> <hr /> <div class="img-center"> <img alt="Finding a Use Case" src="/img/201808/slide-59.png" /> </div> <h2>Finding a Use Case</h2> <p>Ok but we need some specific examples, like in the definition of generalization we saw earlier. So let&rsquo;s go where everyone goes for examples, to Rails! And grep for it.</p> <p>And we actually find a bunch of files that mention this type of equality.</p> <p>To start we&rsquo;ll use some classes defined in tests, just because they&rsquo;ll be a bit easier to visualize, but we&rsquo;ll come back afterward and apply it to models in other files here.</p> <hr /> <div class="img-center"> <img alt="Address" src="/img/201808/slide-60.png" /> </div> <h2>Address</h2> <p>So here&rsquo;s the <code>Address</code> class, just has a few <code>attr_reader</code>s for the attributes street, city and country, and an equality method that compares two addresses based on the values of those attributes.</p> <hr /> <div class="img-center"> <img alt="GpsLocation" src="/img/201808/slide-61.png" /> </div> <h2>Address</h2> <p>And we have this <code>GpsLocation</code> class, which is very similar. Here the methods are not defined using <code>attr_reader</code>, but that doesn&rsquo;t really matter for us. We just care that there is a double-equals method and that it&rsquo;s defined in terms of a list of methods.</p> <hr /> <div class="img-center"> <img alt="Address + GpsLocation (1)" src="/img/201808/slide-62.png" /> </div> <h2>Address + GpsLocation (1)</h2> <p>Ok, so let&rsquo;s line them up together. We notice a small difference, which is that <code>Address</code> has this <code>other.is_a?(self.class)</code> and <code>GpsLocation</code> does not, but it&rsquo;s not terribly important and could easily be added back, so we&rsquo;ll just remove that one to simplify things. It won&rsquo;t change anything about what I&rsquo;m going to show you.</p> <hr /> <div class="img-center"> <img alt="Address + GpsLocation (2)" src="/img/201808/slide-63.png" /> </div> <h2>Address + GpsLocation (2)</h2> <p>Ok so now we need to extract what is common about these two methods. We&rsquo;ll start by noticing that calling a method in Ruby is exactly the same thing as calling send with the method name as an argument. So let&rsquo;s do that instead.</p> <hr /> <div class="img-center"> <img alt="Extract with send" src="/img/201808/slide-64.png" /> </div> <h2>Extract with send</h2> <p>So you can see, in the first step here we&rsquo;ve extracted something that was code - the method calls - to something that is data, the method names we&rsquo;re passing to <code>send</code>. That&rsquo;s important.</p> <hr /> <div class="img-center"> <img alt="Use all?" src="/img/201808/slide-65.png" /> </div> <h2>Extract Array</h2> <p>Now we&rsquo;re going to refactor the sets of equality predicates using the enumerable method <code>all?</code>. This method takes a block and passes each element to the block, then takes the AND of the results.</p> <p>In this case, the block will be the comparison of sending the attribute name to self and to the other object, and this will give us the same result as before.</p> <p>(I&rsquo;ve dropped <code>GpsLocation</code> here by the way since everything we&rsquo;re doing now will be exactly the same for both classes, except the names will be different.)</p> <hr /> <div class="img-center"> <img alt="Abstraction: key" src="/img/201808/slide-66.png" /> </div> <h2>Abstraction: key</h2> <p>Notice now that we have the first indication of our abstraction, the argument to the block here, &ldquo;key&rdquo;. This is an abstraction for the method names that define equality on each of these classes.</p> <hr /> <div class="img-center"> <img alt="Keys" src="/img/201808/slide-67.png" /> </div> <h2>Keys</h2> <p>Ok now we&rsquo;re going to name the array of keys as a variable &ldquo;keys&rdquo; and pull it out. Again, nothing has changed here, we&rsquo;re just moving stuff around.</p> <hr /> <div class="img-center"> <img alt="Replace def with define_method" src="/img/201808/slide-68.png" /> </div> <h2>Replace def with define_method</h2> <p>Ok, so here&rsquo;s an important step. We&rsquo;re replacing <code>def</code> with <code>define_method</code>.</p> <p>Remember that in our original working definition, this would mean that this was now metaprogramming. But in fact, nothing has changed, not yet anyway.</p> <p>With our new definition, this is not metaprogramming because we are not translating any unknowns into code. We&rsquo;re only translating knowns into code, the array of keys in each case. And we know what those keys are.</p> <p>So you can see that our new definition is giving us more useful information about what is going on.</p> <hr /> <div class="img-center"> <img alt="Hoist Keys" src="/img/201808/slide-69.png" /> </div> <h2>Hoist Keys</h2> <p>Ok, now we &ldquo;hoist&rdquo; the <code>keys</code> out of the <code>define_method</code> block. We can do that because blocks have an open scope, so what&rsquo;s in the block can see the keys array, even if it&rsquo;s outside of the block itself. (This is now a closure, if that says anything to you.)</p> <hr /> <div class="img-center"> <img alt="Method Arguments" src="/img/201808/slide-70.png" /> </div> <h2>Method Arguments</h2> <p>Now I want to look at these things differently. These keys we&rsquo;re going to think of as arguments to a method. In <code>GpsLocation</code> the keys would be latitude and longitude, but for Address they are street, city, country.</p> <hr /> <div class="img-center"> <img alt="Method Body" src="/img/201808/slide-71.png" /> </div> <h2>Method Body</h2> <p>And now look at this code here. This is now independent of the keys, i.e. independent of what makes Address and GpsLocation different. So we can now extract it.</p> <p>But how? By wrapping it in a method, like we saw earlier.</p> <hr /> <div class="img-center"> <img alt="Equalize" src="/img/201808/slide-72.png" /> </div> <h2>equalize</h2> <p>So we create a class method <code>equalize</code>, which takes arguments <code>keys</code>. The splat just means we can pass arguments and they will be captured in an array.</p> <p>The code inside equalize is exactly the same as what we had in the last slide. But now since it&rsquo;s in a method, we need to call it, which we do, just below. We call it with street, city, country.</p> <hr /> <div class="img-center"> <img alt="Equalizer" src="/img/201808/slide-73.png" /> </div> <h2>Equalizer</h2> <p>Ok, and so now we can finally extract the module that will be our gem code. We extract that class method into a module, <code>Equalizer</code>, and we extend <code>Address</code> with it and call <code>equalize</code> with the keys.</p> <p>That module is now our gem. And notice our unknowns? They are the keys to the equalize method.</p> <hr /> <div class="img-center"> <img alt="require 'equalizer'" src="/img/201808/slide-74.png" /> </div> <h2>require &ldquo;equalizer&rdquo;</h2> <p>And now that it&rsquo;s a gem, a library, we can just require it, like this.</p> <p>And now things are starting to look pretty nice and clean, right? The Address class is simpler than it was originally, but it&rsquo;s very clear what is defining equality on each of them.</p> <p>This is the sign of a good abstraction: it captures the essence of a general concept.</p> <hr /> <div class="img-center"> <img alt="Equalizer in Big Picture" src="/img/201808/slide-75.png" /> </div> <h2>Equalizer in Big Picture</h2> <p>And now in that diagram I showed you earlier, here is where we are. We&rsquo;ve extracted the generic component from <code>Address</code> (and <code>GpsLocation</code> would be the same), and this generic component, this generic software, now lives in this larger space outside of what we call &ldquo;normal programming&rdquo;.</p> <p><code>Equalizer</code> has data that refers to code, the keys that refer to method names defining equality.</p> <hr /> <div class="img-center"> <img alt="Refactor other Classes with Equalizer" src="/img/201808/slide-76.png" /> </div> <h2>Refactor other Classes with Equalizer</h2> <p>And now that we&rsquo;ve done this, we can apply <code>Equalizer</code> to other classes. Here we have <code>GpsLocation</code>, but also a bunch of other classes from that list I showed you earlier. You can see we&rsquo;ve hit on a very general pattern.</p> <hr /> <div class="img-center"> <img alt="Equalizer Again" src="/img/201808/slide-77.png" /> </div> <h2>Equalizer Again</h2> <p>So here’s our <code>Equalizer</code> module again. This is a very compact little module, but there are a few important things going on.</p> <p>Look at how define_method captures the keys in a block. Note that you couldn’t do this without <code>define_method</code>, because <code>def</code> has a closed scope, it doesn’t accept a block. Likewise, we could not build this method without calling the equalizer methods using <code>send</code>, because the <code>keys</code> here are unknown values.</p> <p>Ruby provides methods like <code>define_method</code> and <code>send</code> for a very specific purpose, but that purpose is not clear unless you are solving a generic problem like this. Of course you could also use <code>eval</code> – <code>eval</code> is the &ldquo;universal solvent&rdquo; that dissolves anything, but it is an ultimate fallback and rarely used in practice.</p> <hr /> <div class="img-center"> <img alt="Add inspect method" src="/img/201808/slide-78.png" /> </div> <h2>Add inspect method</h2> <p>And the beauty of this is that once we have these keys, and these methods for dynamically defining methods and sending messages that Ruby provides, we can take this further.</p> <p>We can observe that whatever defines equality basically defines identity. So we can use that fact to define an inspect method which shows the values of the key methods.</p> <p>And you can see that this is useful in the example here.</p> <hr /> <div class="img-center"> <img alt="Add hash method" src="/img/201808/slide-79.png" /> </div> <h2>Add hash method</h2> <p>And we can go further building on the abstraction, and add a <code>hash</code> method. The <code>hash</code> method is used to determine if two keys on a hash are the same or not, so by defining this method we allow instances of classes that extend <code>Equalizer</code> to be used as hash keys.</p> <p>Here’s an example where <code>GpsLocation</code>s are used as a hash of number of times a given location has been visited. We have two points, but they are the same latitude/longitude, so only one key in the hash is updated.</p> <hr /> <div class="img-center"> <img alt="Dry Equalizer" src="/img/201808/slide-80.png" /> </div> <h2>Dry Equalizer</h2> <p>Now, if all this seems useful to you, I have some good news! There’s a gem for it, called <a href="https://github.com/dry-rb/dry-equalizer">Dry Equalizer</a>.</p> <p>It&rsquo;s one of the dry-rb gems, which are fantastic examples of generic software by the way. It&rsquo;s a very very small gem, just a bit more than what we have here, done a little bit more cleverly, but basically the same idea.</p> <hr /> <div class="img-center"> <img alt="Dry Equalizer in dry-rb gem network" src="/img/201808/slide-80.png" /> </div> <h2>DryEqualizer in dry-rb gem network</h2> <p>DryEqualizer may seem quite simple, but it plays a very important role in the network of dry-rb gems, which are being more and more heavily used in many applications. dry-equalizer is actually right at the core of this network, because it is so simple and so generic.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Being a Generalist (1)" src="/img/201808/slide-83.png" /> </div> <h2>Being a Generalist (1)</h2> <p>Ok, so just before I close here, I want to come back to the title of this talk.</p> <p>The title was &ldquo;Metaprogramming for Generalists&rdquo;, not &ldquo;Metaprogramming and Generalization&rdquo;, or &ldquo;Metaprogramming and Generic Software”.</p> <p>And there&rsquo;s a reason for that.</p> <p>&ldquo;Generalist&rdquo; usually means someone who has broad knowledge across different fields, and the ability to bridge ideas from various different areas.</p> <p>And you can do that by jumping from one field to the other and learning high-level things about them, like this.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Being a Generalist (2)" src="/img/201808/slide-84.png" /> </div> <h2>Being a Generalist (2)</h2> <p>But there&rsquo;s another way to think of the &ldquo;generalist&rdquo;, as someone who digs deep down, to the general concepts that underlie all sorts of different problems. Concepts like &ldquo;attribute&rdquo; and &ldquo;equality&rdquo;.</p> <p>Once you find these core concepts, and capture them in an abstraction, like the Equalizer module you saw, then you really grasp something very fundamental about a whole class of problems at once.</p> <p>And then you can build on the abstraction, and everything you build will broaden the reach of the software.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Being a Generalist (3)" src="/img/201808/slide-85.png" /> </div> <h2>Being a Generalist (3)</h2> <p>And as a community, we really need more of you to do this, to delve down to this world.</p> <p>Solving generic problems, and solving them well, benefits everyone at all levels, by making every specific instance of that general problem easier to solve.</p> <p>This is the metaprogramming magic that makes our ecosystem so powerful. It’s what makes Ruby the language it is.</p> <hr /> <div class="img-center"> <img style="padding-bottom: 20px" alt="Thanks" src="/img/201808/slide-86.png" /> </div> <h2>Thanks!</h2> Arel with Wharel https://dejimata.com/2018/5/30/arel-with-wharel 2018-05-30 2018-05-30 Chris Salzberg <p>Object Relational Mappers (ORM) are complex things. As the term implies, an ORM provides a mapping from objects in an object-oriented language like Ruby</p> <p>Object Relational Mappers (ORM) are complex things. As the term implies, an ORM provides a mapping from objects in an object-oriented language like Ruby to a relational persistence layer. In theory, this makes it possible to abstract away details of this layer behind objects; in practice, of course, things are not so simple.</p> <p>In this post, I&rsquo;ll show you how I&rsquo;ve leveraged a couple very simple features of Ruby&rsquo;s object model to simplify building queries in ActiveRecord, in the form of a teeny tiny gem called <a href="https://github.com/shioyama/wharel">Wharel</a>. Wharel allows you to create Arel predicates in blocks passed to ActiveRecord query methods, without all the standard boilerplate.</p> <p>Before I explain Wharel, though, let&rsquo;s take a few steps back and consider the problem this gem tries to solve.</p> <h2>Querying with ActiveRecord</h2> <p>Part of the challenge of building and extending an ORM is striking a good balance between convenience and control. ActiveRecord offers you by default a fairly minimal interface for making queries which (I suppose) satisfies about 90% of users' basic needs. That interface is made up of the handful of methods like <code>where</code> and <code>not</code>, which allow you to create queries built up of simple equality predicates, optionally negated:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.where(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span>).where.not(<span style="color:#606">content</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">bar</span><span style="color:#710">&quot;</span></span>) <span style="color:#777">#=&gt; SELECT &quot;posts&quot;.* FROM &quot;posts&quot; WHERE &quot;posts&quot;.&quot;title&quot; = 'foo' AND &quot;posts&quot;.&quot;content&quot; != 'bar'</span> </pre></div> </div> <p>ActiveRecord also supports <code>SELECT</code>, <code>HAVING</code>, <code>GROUP BY</code>, along with the combinators <code>AND</code> and (recently) <code>OR</code>, as well as inner and outer <code>JOIN</code>, and a few others I forget just now.</p> <p>Which is all fine and good, and may be adequate to power queries in your little CRUD app. But very quickly you will find that <code>where</code> and friends are not enough. Suppose, for example, you want to search for partial matches on string-valued data. A Rails developer would typically drop down to SQL to do this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.where(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title LIKE ?</span><span style="color:#710">&quot;</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">%</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>params[<span style="color:#A60">:title</span>]<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%</span><span style="color:#710">&quot;</span></span>) <span style="color:#777">#=&gt; SELECT `posts`.* FROM `posts` WHERE (title LIKE '%foo%')</span> </pre></div> </div> <p>&hellip; where <code>params[:title]</code> is a title from user input, and the application is using MySQL as its database.</p> <p>In this case, we&rsquo;re not really using the ORM much at all, other than to sanitize the data (<code>params[:title]</code>) interpolated into the SQL string. We can&rsquo;t extend or re-use this query in other contexts. We can&rsquo;t take it apart, build it up, or compose it. We can&rsquo;t use it with a different database which uses a different name for <code>LIKE</code> (like PostgreSQL, which uses <code>ILIKE</code> for case-insensitive matching).</p> <p>In short, we can either use this SQL snippet as-is, or re-write it to be something different.</p> <h2>Enter Arel</h2> <p>For one-off situations, this may well be enough. But for anything else, ActiveRecord offers another layer in the form of Arel, a <a href="https://en.wikipedia.org/wiki/Relational_algebra">relational algebra</a> for building queries abstracted from the particulars of any particular database. If ActiveRecord is a mapping between Ruby model objects and a relational database, Arel is the fine-grained mapping between Ruby query objects and SQL query fragments.</p> <p>Let&rsquo;s get concrete. Convert the <code>LIKE</code> query above to Arel and we get:</p> <div class="CodeRay"> <div class="code"><pre> posts = <span style="color:#036;font-weight:bold">Post</span>.arel_table <span style="color:#036;font-weight:bold">Post</span>.where(posts[<span style="color:#A60">:title</span>].matches(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">%</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>params[<span style="color:#A60">:title</span>]<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%</span><span style="color:#710">&quot;</span></span>)) <span style="color:#777">#=&gt; SELECT `posts`.* FROM `posts` WHERE (`posts`.`title` LIKE '%foo%')</span> </pre></div> </div> <p>Here, we needed to first call <code>Post.arel_table</code> to get an instance of <code>Arel::Table</code> corresponding to the <code>posts</code> table for the model. We can then call <code>posts[:title]</code> to get an arel node (actually, an <code>Arel::Attribute</code>) for the <code>title</code> column.</p> <p>We can then use this <code>title</code> node to build a query using the <code>matches</code> method, which corresponds to <code>LIKE</code> in MySQL.</p> <p>This may not seem like such a big win, since anyway you could have written the query in raw SQL with less lines of code. But what if you wanted to re-use that snippet in combination with another condition?</p> <p>You could certainly do this with a new inline SQL, but a more scalable way is to use Arel. For example:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ApplicationRecord</span> scope <span style="color:#A60">:with_title</span>, -&gt;(str) { where(matches_title(str)) } scope <span style="color:#A60">:with_title_or_subtitle</span>, -&gt;(str) { where(matches_title(str).or(matches_subtitle(str))) } <span style="color:#080;font-weight:bold">class</span> &lt;&lt; <span style="color:#B06;font-weight:bold">self</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">matches_title</span>(title) arel_table[<span style="color:#A60">:title</span>].matches(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">%</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>title<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%</span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">matches_subtitle</span>(subtitle) arel_table[<span style="color:#A60">:subtitle</span>].matches(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">%</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>subtitle<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%</span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>In this case, we&rsquo;ve used the arel predicates returned by <code>matches_title</code> and <code>matches_subtitle</code> to build a combined predicate using the pattern <code>a.or(b)</code>, which generates SQL like:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#777"># Post.with_title_or_subtitle(&quot;foo&quot;)</span> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">posts</span><span style="color:#710">`</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">posts</span><span style="color:#710">`</span></span> <span style="color:#080;font-weight:bold">WHERE</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">posts</span><span style="color:#710">`</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">title</span><span style="color:#710">`</span></span> <span style="color:#080;font-weight:bold">LIKE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">%foo%</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">OR</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">posts</span><span style="color:#710">`</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">`</span><span style="color:#D20">subtitle</span><span style="color:#710">`</span></span> <span style="color:#080;font-weight:bold">LIKE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">%foo%</span><span style="color:#710">'</span></span>) </pre></div> </div> <p>As you can imagine, with more complex queries, using Arel building blocks can be quite useful. But it should also be apparent that Arel&rsquo;s &ldquo;wordiness&rdquo; can be a real turn-off, and partly explains why Rails developers tend to prefer one-off SQL fragments over Arel predicates.</p> <h2>Where + Arel = Wharel</h2> <p>Recently I&rsquo;ve been looking a lot at the internals of <a href="http://sequel.jeremyevans.net/">Sequel</a>, a Ruby ORM developed by <a href="https://github.com/jeremyevans">Jeremy Evans</a> that is superficially similar in many ways to ActiveRecord, but which has a vastly different internal implementation and much broader feature set.</p> <p>One of the things that fascinated me about Sequel is its interface for building queries, which allows you to use blocks in the form of so-called &ldquo;<a href="http://sequel.jeremyevans.net/rdoc/files/doc/querying_rdoc.html#label-Virtual+Row+Blocks">virtual rows</a>&rdquo;, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Artist</span>.where{id &gt; <span style="color:#00D">5</span>} </pre></div> </div> <p>Here, we are querying on a column <code>id</code>. But notice that rather than passing a hash of keys and values to <code>where</code>, we are instead passing a <em>block</em> which references the <em>method</em> <code>id</code> and builds it into a predicate with <code>id &gt; 5</code>.</p> <p>This type of query would not be possible passing a hash to ActiveRecord&rsquo;s <code>where</code>, since a hash can only generate equality predicates. Instead, we would need to build the appropriate predicate with Arel, like this:</p> <div class="CodeRay"> <div class="code"><pre> artists = <span style="color:#036;font-weight:bold">Artist</span>.arel_table <span style="color:#036;font-weight:bold">Artist</span>.where(artists[<span style="color:#A60">:id</span>].gt(<span style="color:#00D">5</span>)) </pre></div> </div> <p>For such a simple query, I find the ActiveRecord/Arel code to be much harder to read. Unsurprisingly, others have felt the same way.</p> <p>Many years ago, <a href="https://github.com/ernie">Ernie Miller</a> set out to improve ActiveRecord with an interface similar to Sequel, in the form of a gem called <a href="https://github.com/activerecord-hackery/squeel">Squeel</a>. Squeel was popular, but ultimately the monkey-patching magic it required to bend ActiveRecord into shape burnt out its maintainer, and the gem is now abandoned. More recently, a gem named <a href="https://github.com/rzane/baby_squeel">BabySqueel</a> has attempted to recreate some of this interface without Squeel&rsquo;s monkey-patching baggage.</p> <p>While working on <a href="https://github.com/shioyama/mobility">Mobility</a>, I recently had a need to incorporate Arel predicates with ActiveRecord relations in a way that went beyond AR&rsquo;s standard query method interface. I decided to implement something like the block-based query interface from Sequel, but vastly simplified.</p> <p>In the process, it became apparent to me how <em>simple</em> it is to do this, so I extracted the core logic into a separate gem I&rsquo;ve called <a href="https://github.com/shioyama/wharel">Wharel</a>, a portmanteau of &ldquo;Where&rdquo; and &ldquo;Arel&rdquo;.</p> <p>What does Wharel do? Well, add it to your Gemfile and you can simplify queries like the one above to:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Artist</span>.where { id.gt(<span style="color:#00D">5</span>) } </pre></div> </div> <p>Wharel also supports combining nodes from different models by optionally using a block argument, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Artist</span>.joins(<span style="color:#A60">:posts</span>).where { |a| <span style="color:#036;font-weight:bold">Post</span>.where { |p| p.title.matches(a.name) } } </pre></div> </div> <p>(This query matches artists who have at least one post with a title that matches their name.)</p> <p>As you can see, Wharel adds an optional block format like Sequel, within which any attributes called as methods are converted to their corresponding Arel node. In this case, <code>id</code> becomes the arel node <code>Artist.arel_table[:id]</code>.</p> <p>That&rsquo;s it. There&rsquo;s no additional magic to support predicates like <code>id &gt; 5</code>, which Sequel, Squeel and BabySqueel offer, because I wanted to keep Wharel to an absolute minimum level of complexity.</p> <p>And Wharel is indeed <em>very</em> simple: its core is a mere <em>31 lines</em> of code. How can that be? Let&rsquo;s have a look inside.</p> <h2>Virtual Rows and BasicObjects</h2> <p>Here is the <code>Wharel</code> module:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Wharel</span> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">QueryMethods</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">where</span>(opts = <span style="color:#A60">:chain</span>, *rest, &amp;block) block_given? ? <span style="color:#080;font-weight:bold">super</span>(<span style="color:#036;font-weight:bold">VirtualRow</span>.build_query(<span style="color:#069">self</span>, &amp;block)) : <span style="color:#080;font-weight:bold">super</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">WhereChain</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">not</span>(*args, &amp;block) block_given? ? <span style="color:#080;font-weight:bold">super</span>(<span style="color:#036;font-weight:bold">VirtualRow</span>.build_query(<span style="color:#33B">@scope</span>, &amp;block)) : <span style="color:#080;font-weight:bold">super</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">VirtualRow</span> &lt; <span style="color:#036;font-weight:bold">BasicObject</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(klass) <span style="color:#33B">@klass</span> = klass <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">method_missing</span>(m, *) <span style="color:#33B">@klass</span>.column_names.include?(m.to_s) ? <span style="color:#33B">@klass</span>.arel_table[m] : <span style="color:#080;font-weight:bold">super</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">class</span> &lt;&lt; <span style="color:#B06;font-weight:bold">self</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">build_query</span>(klass, &amp;block) row = new(klass) query = block.arity.zero? ? row.instance_eval(&amp;block) : block.call(row) ::<span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Relation</span> === query ? query.arel.constraints.inject(&amp;<span style="color:#A60">:and</span>) : query <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>There is really not much here. The <code>QueryMethods</code> module is required to patch ActiveRecord to handle blocks; nothing special there. What <em>is</em> special is the <code>VirtualRow</code> class, which subclasses <code>BasicObject</code>.</p> <p>Why <code>BasicObject</code>? You may have seen this esoteric, minimalist object somewhere in a diagram of Ruby&rsquo;s object model, and recall that every <code>Object</code> inherits from it. Indeed, <code>BasicObject</code> is a <em>very</em> simple object with almost no methods at all, and for most contexts not something you&rsquo;d really want to lean on too much.</p> <p>But in this case, the fact that <code>BasicObject</code> is so minimal happens to be very useful, because it allows us to create what&rsquo;s called a <a href="https://www.sethvargo.com/the-cleanroom-pattern/">Clean Room</a>. Take a look at <code>build_query</code>, which you&rsquo;ll see we call from <code>where</code> or <code>not</code> to convert a block to an Arel predicate:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">VirtualRow</span>.build_query(<span style="color:#069">self</span>, &amp;block) </pre></div> </div> <p>Here, <code>self</code> is the class or relation (for <code>not</code> it is <code>@scope</code>, which is basically the same), and <code>block</code> is the block passed to <code>where</code> or <code>not</code>.</p> <p>In <code>build_query</code>, we first create an instance of <code>VirtualRow</code> with a reference to the class (or relation) (this is the <code>new(klass)</code> line). We then check the <code>arity</code> of the block (don&rsquo;t be scared, &ldquo;arity&rdquo; is just another word for &ldquo;number of arguments&rdquo;).</p> <p>If the block has no arguments (<code>block.arity.zero?</code>), like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.where { title.eq(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span>) } </pre></div> </div> <p>&hellip; then we call <code>instance_eval</code> on the virtual row and pass it the block.</p> <p>Now what happens?</p> <p>Well, <code>instance_eval</code> is a bit like that movie, <a href="https://en.wikipedia.org/wiki/Being_John_Malkovich">Being John Malkovich</a>, when you go through the door and enter the mind of actor John Malkovich. Except in this case, the door takes you to the mind of whatever object you&rsquo;re calling <code>instance_eval</code> on. Any method you call from this context is received by that object, just like everything you see when you go though the door is what John Malkovich is seeing.</p> <p>Here, the object is the virtual row. And this is where the fact that <code>VirtualRow</code> subclasses <code>BasicObject</code> comes in useful. Since <code>VirtualRow</code> has virtually no methods (no pun intended), anything you call on it will fall through to <code>method_missing</code>. So if you call <code>title</code>, <code>method_missing</code> will be called with <code>:title</code> as its first argument.</p> <p>What do we do then? Well, just define a <code>method_missing</code> to catch these method calls and &ldquo;route&rdquo; them to the model&rsquo;s Arel table, which Wharel does in three lines of code:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">method_missing</span>(m, *) <span style="color:#33B">@klass</span>.column_names.include?(m.to_s) ? <span style="color:#33B">@klass</span>.arel_table[m] : <span style="color:#080;font-weight:bold">super</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Here, <code>@klass</code> would be <code>Artist</code>, so we&rsquo;re calling <code>Artist.arel_table[:title]</code>, which will get us the Arel node for <code>title</code>. Perfect! That&rsquo;s exactly what we wanted.</p> <p>Since this works for any attribute (as long as the attribute name does not overlap with a method on <code>BasicObject</code>, which is exceedingly unlikely), we can build up any arbitrary Arel predicate in this block without ever having to explicitly call <code>arel_table</code> on any model.</p> <h2>That Object Model Thing</h2> <p>Although this post has Arel in its title, the real point I&rsquo;m trying to make with Wharel is to show how some of Ruby&rsquo;s seemingly esoteric features, like <code>BasicObject</code> and <code>instance_eval</code>, can actually be very powerful. With only thirty lines of code, we&rsquo;ve created an interface that eliminates a huge amount of Arel boilerplate code in any Rails app. I find this really remarkable.</p> <p>Although I don&rsquo;t think many Rubyists are very familiar with it, Ruby&rsquo;s Object Model is actually very powerful and I highly recommend digging deeper into its details. Books like <a href="https://pragprog.com/book/ppmetr/metaprogramming-ruby">Metaprogramming Ruby</a> by Paolo Perrotta are a great place to start, but it&rsquo;s also a good idea to just pry open your favourite gem (like I did with Sequel) and see what it&rsquo;s doing under the hood. If it&rsquo;s doing something powerful, it&rsquo;s probably leveraging this model to do it.</p> JSONify your Ruby Translations https://dejimata.com/2018/3/20/jsonify-your-ruby-translations 2018-03-20 2018-03-20 Chris Salzberg <p><a href="https://en.wikipedia.org/wiki/JSON">JSON</a> is a ubiquitous format for storing and transmitting complex nested data, originally in Javascript but today</p> <p><a href="https://en.wikipedia.org/wiki/JSON">JSON</a> is a ubiquitous format for storing and transmitting complex nested data, originally in Javascript but today in every major programming language. JSON is also increasingly used as a persisted data type, notably by <a href="https://en.wikipedia.org/wiki/PostgreSQL">PostgreSQL</a> with its <a href="https://www.postgresql.org/docs/9.4/static/datatype-json.html">json and jsonb types</a>, and more recently <a href="https://dev.mysql.com/doc/refman/5.7/en/json.html">MySQL also supporting JSON natively</a>. Ruby ORM like Sequel and ActiveRecord<a href="#note-1"><sup>1</sup></a> offer interfaces for querying on this kind of JSON-formatted data.</p> <div class="img-center" style="max-width: 100%; margin: 40px 0 40px 0"> <img alt="JSON Translation Data" src="/img/json_locales.png"></a> </div> <p>Given its usefulness, I&rsquo;ve naturally been interested in applying JSON to the storage of i18n data, a topic which I have <a href="/2017/3/3/translating-with-mobility">written about before</a>. As an arbitrary-depth hash format, JSON is a good fit for this kind of data, which tends to assume a similar nested key/value structure. With databases now providing ways to search JSON data quickly, we&rsquo;ve got everything we need for a powerful translation storage and querying engine.</p> <p>In this post I&rsquo;ll introduce a new strategy for leveraging JSON data formats to store translations in a single table column. The strategy is implemented in <a href="https://github.com/shioyama/mobility">Mobility</a>, the i18n gem I&rsquo;ve written about here before, and is inspired by (among other things) a library named <a href="https://github.com/belaustegui/trans">Trans</a> for Elixir.</p> <p>Before jumping into this new JSON-based storage strategy, however, I&rsquo;d like to first take a step back and think about the broader &ldquo;translation mapping&rdquo; problem, and how JSON fits into it.</p> <h2>Translation Mapping</h2> <p>When you think of translation as a Ruby developer, you may think of a YAML file with lots of nested translation keys and their corresponding translated values, popularized in Ruby by the <a href="https://github.com/svenfuchs/i18n">I18n</a> gem. Or you may think of <a href="https://en.wikipedia.org/wiki/Gettext">gettext</a> and .po (portable object) files. Or you may think of some other similar file-based key/value storage.</p> <p>A key property of these kinds of internationalization and localization systems is that the context in which translations are <em>written</em> is fundamentally different from the context in which translations are <em>used</em>. Your users, for example, can&rsquo;t edit the YAML i18n files for your application directly through the application itself.</p> <div class="img-center" style="max-width: 100%; margin: 40px 0 20px 0"> <img alt="JSON Translation Data" src="/img/yamlfile.png"></a> <p class="caption">Some YAML translations</p> </div> <p>That may be obvious to you, but it&rsquo;s important because I want to contrast this role of i18n strings with another, fundamentally different role translation can play in a running application. The key characteristic of translation in this second role is that <em>reading and writing happen in the same context</em>.</p> <p>An example of this second category would be a blog platform which allows authors to translate their articles into different languages, and query for articles in any language. Another example would be an <a href="https://www.doorkeeper.jp/">event site</a> that allows its users to translate their event listings. But it could also include any of <a href="https://github.com/shioyama/mobility#companies-using-mobility">many other contexts</a> in which translation data is dynamically added, updated, deleted, and queried through the application itself.</p> <div class="img-right" style="margin: 20px 0 20px 20px; overflow: hidden"> <img alt="Event Traslations on Doorkeeper.jp" src="/img/doorkeeper-translation.png"></a> <p class="caption">Translation Interface on <a href="https://doorkeeper.jp/" target="_blank">Doorkeeper.jp</a></p> </div> <p>If ActiveRecord and Sequel are &ldquo;Object Relational Mappers&rdquo; (ORM), I like to think of the libraries that help with this type of translation mapping as <strong>Translation Relational Mappers</strong>. They act as a layer between the ORM (data) and the interface for writing, reading and querying translations.</p> <p>Mobility is unique among libraries I have seen (in Ruby and any other language) in that it takes the &ldquo;mapping&rdquo; part of this problem very seriously. While other libraries tend to support one storage strategy or another, Mobility supports many storage strategies through one common interface.</p> <p>Which brings us to the question: what <em>is</em> a &ldquo;storage strategy&rdquo;? An example will help make things more concrete. In Ruby at least, there is one approach that is more popular than any other, and it does not use JSON.</p> <h2>Translations in Tables</h2> <p><a href="/2013/10/11/hacking-globalize">Globalize</a> is the most well-known gem for translating ActiveRecord i18n model data and has popularized a particular approach to storing persisted translation data.</p> <p>In this strategy, translations for a model are stored in a model-specific table. If your model <code>Post</code> uses a table named <code>posts</code>, you would create a translation table named <code>post_translations</code> with a <code>locale</code> string column and other columns for each translated attribute. Then when you access <code>post.title</code>, the gem fetches the translation for the model in the current locale and returns the value of the <code>title</code> column on the translation.</p> <p>Keeping all translations for a given model in the same place has some advantages. It means, among other things, that you can call <code>post.translations.each</code> and cycle through each of your translations as a separate record; <a href="http://www.spinacms.com/">SpinaCMS</a> does this <a href="https://github.com/SpinaCMS/Spina/blob/376e659a2fa76331821dfbb90814fdec79f07966/app/views/spina/sitemaps/show.xml.builder#L9-L15">when generating its sitemap</a>, for example. Mobility supports this type of translation strategy with its <a href="https://github.com/shioyama/mobility/wiki/Table-Backend">Table backend</a>, which is what Spina uses.</p> <p>Although it&rsquo;s nice having translations in a separate table, there are also some significant downsides. Querying in particular becomes much more complicated since you need to constantly <code>JOIN</code> your translation table(s).</p> <p>Here&rsquo;s the kind of query you get with this strategy (taken from the <a href="https://github.com/shioyama/mobility/wiki/Table-Backend">Mobility Wiki</a>):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">INNER</span> <span style="color:#080;font-weight:bold">JOIN</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">ON</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_id</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">id</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">locale</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">'</span></span> </pre></div> </div> <p>That&rsquo;s not too bad, but you can imagine that once you start adding complexity you quickly hit edge cases which can push the limits of your ORM. This has prompted many people, myself included, to consider other options for storing translations. One of these is to store translations in JSON format.</p> <h2>Enter the B</h2> <p>A few years back, with the release of its version 9.4, PostgreSQL <a href="https://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.4">introduced a new datatype called JSONB</a>. The &ldquo;B&rdquo; in &ldquo;JSONB&rdquo; signifies that JSON data are stored in this column format not as a string, but as binary data. This allows for fast lookups and more powerful search queries, through new operators like &ldquo;contains&rdquo; (<code>@</code>) and &ldquo;exists&rdquo; (<code>?</code>) which apply to JSON objects and keys, respectively. It also allows you to store a mix of integers, strings, arrays and hashes in the same column. This is a considerable step up from PostgreSQL&rsquo;s <a href="https://www.postgresql.org/docs/current/static/hstore.html">Hstore format</a> which, although also a key/value store, is limited to depth-1 string-valued data.</p> <p>One of my original goals with Mobility was to support this type of innovative format <em>alongside</em> other strategies like the table-based one described in the last section. This, indeed, is Mobility&rsquo;s killer feature: just like how you can swap MySQL for PostgreSQL as your database and keep your ActiveRecord code mostly unchanged, Mobility allows you to swap storage backends while retaining the same interface for accessing and querying translations.</p> <p>The first implementation of JSON in Mobility was a backend called <code>Jsonb</code>. Using this backend looks like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ApplicationRecord</span> extend <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">backend</span>: <span style="color:#A60">:jsonb</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Given that you have a <code>title</code> column of <code>jsonb</code> type, Mobility will allow you to call a virtual attribute named <code>title</code> and (if we&rsquo;re in the English locale) will return to you the value on the <code>en</code> key of the JSON hash.</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.create(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">&quot;</span></span>) post.title <span style="color:#777">#=&gt; &quot;JSONify your Ruby Translations&quot;</span> <span style="color:#036;font-weight:bold">Mobility</span>.with_locale(<span style="color:#A60">:ja</span>) { post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Rubyの翻訳をJSON化しよう</span><span style="color:#710">&quot;</span></span> } post.save post = <span style="color:#036;font-weight:bold">Post</span>.first post[<span style="color:#A60">:title</span>] <span style="color:#777">#=&gt; {&quot;en&quot;=&gt;&quot;JSONify your Ruby Translations&quot;, &quot;ja&quot;=&gt;&quot;Rubyの翻訳をJSON化しよう&quot;}</span> </pre></div> </div> <p>It will also allow you to query on this attribute just like it was a table column (via a scope named <code>i18n</code>):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.i18n.where(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">&quot;</span></span>).to_sql <span style="color:#777">#=&gt; SELECT &quot;posts&quot;.* FROM &quot;posts&quot; WHERE ((&quot;posts&quot;.&quot;title&quot; -&gt; 'en') = &quot;JSONify your Ruby Translations&quot;)</span> </pre></div> </div> <p>So Mobility is mapping the query on <code>title</code> to a query format specific to <code>jsonb</code>, using the <code>-&gt;</code> operator.<sup><a href="#note-2">2</a></sup></p> <p>Mobility actually does some more subtle things here. If you pass an array as the query value, for example, Mobility will treat this as a set of values to match, just like you would with a normal table column. So the query:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.i18n.find_by(<span style="color:#606">title</span>: [<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">bar</span><span style="color:#710">&quot;</span></span>]) </pre></div> </div> <p>would generate the SQL:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">WHERE</span> ((<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span> -&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>) = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">&quot;foo&quot;</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">OR</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span> -&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>) = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">&quot;bar&quot;</span><span style="color:#710">'</span></span>) </pre></div> </div> <p>With the Table backend mentioned earlier, the same code would generate a very different query:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">INNER</span> <span style="color:#080;font-weight:bold">JOIN</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">ON</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_id</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">id</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">locale</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">content</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">IN</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">foo</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">bar</span><span style="color:#710">'</span></span>) </pre></div> </div> <p>Whereas the Jsonb backend uses <code>OR</code>, the Table backend uses <code>IN</code> to match translations that have a column value in <code>('foo', 'bar')</code>. As a Mobility user you don&rsquo;t need to really worry about these details; just write your ActiveRecord (or Sequel) code as you would for an untranslated attribute and the query should &ldquo;just work&rdquo;<sup>TM</sup>.</p> <h2>Keeping &lsquo;em together</h2> <p>One of the nice things about supporting many storage strategies in a single gem is that common implementation patterns can be leveraged to re-use code and minimize duplication. While there are <a href="https://github.com/cfabianski/hstore_translate">gems offering Hstore-based model translation</a> and <a href="https://github.com/olegantonyan/translateable">gems offering JSON-based model translation</a>, the implementation for each one is different in arbitrary ways (<code>translates</code>/<code>translatable</code> etc). Rather than re-invent the wheel, Mobility uses shared modules for <a href="https://github.com/shioyama/mobility/blob/aa52e89e50ffc0cb817db4ca9e4623ed69b6bb84/lib/mobility/backends/active_record/pg_hash.rb">accessing</a> and <a href="https://github.com/shioyama/mobility/blob/aa52e89e50ffc0cb817db4ca9e4623ed69b6bb84/lib/mobility/backends/active_record/pg_query_methods.rb">querying</a> PostgreSQL data types to eliminate duplication.</p> <p>Which brings us to the new storage strategy mentioned at the top of this article. While playing around with <a href="https://elixir-lang.org/">Elixir</a> recently, I came upon a package named <a href="https://hex.pm/packages/trans">Trans</a> which uses a translation strategy I hadn&rsquo;t previously considered. Like the Jsonb backend mentioned in the last section, this strategy uses a <code>json</code> or <code>jsonb</code> column in a PostgreSQL database to store translations. However, rather than store each set of attribute translations on its own column, Trans uses the fact that json columns can be deeply nested to store all translation on a single column, named <code>translations</code>. This would not be possible using Hstore since it is a depth-1 data type.</p> <p>An example of this is shown in the figure below.</p> <div class="img-center" style="margin: 20px 0 40px 0; max-width: 100%; overflow: hidden"> <img alt="JSON Translation Data" src="/img/json_translations.png" style="min-width: 650px"></a> <p class="caption">Model Translations as a depth-2 JSON hash</p> </div> <p>This is an interesting storage format with some notable advantages:</p> <ul> <li>It <strong>minimizes the number of migrations</strong> needed to just one per translated model. Translated attributes can be added and removed from a model with only minimal code changes, since the database schema does not change when you add new keys to a json/jsonb hash.</li> <li>It <strong>requires no complex JOIN queries</strong> since translations are all on the same table.</li> <li>It allows <strong>cycling through model translations</strong> like with table translations (with <code>post.translations.each</code>), except without a <code>JOIN</code>.</li> </ul> <p>Borrowing the name from Trans, I added this strategy as a new backend named &ldquo;<a href="https://github.com/shioyama/mobility/wiki/Container-Backend">Container</a>&rdquo; in Mobility 0.4. In Mobility 0.5., I also added support for <code>json</code>-type columns in addition to <code>jsonb</code> (there are subtle differences between the two that are out of the scope of this post, but in general you probably want to use <code>jsonb</code>).</p> <p>Just like the examples with other backends mentioned earlier, Mobility thus now supports both reading/writing <em>and</em> querying on translations stored in a single <code>json</code> or <code>jsonb</code> column<sup><a href="#note-3">3</a></sup>. Just specify the <code>container</code> backend in your model (or as a default in your Mobility configuration):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ApplicationRecord</span> extend <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">backend</span>: <span style="color:#A60">:container</span> <span style="color:#777"># Or if you want to set a different column name:</span> <span style="color:#777"># translates :title, backend: :container, column_name: :my_translations</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Mobility will generate the appropriate query to find translations on the shared <code>translations</code> column. So:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.i18n.where(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">&quot;</span></span>) </pre></div> </div> <p>will generate the SQL:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">WHERE</span> (((<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">translations</span><span style="color:#710">&quot;</span></span> -&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>) -&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">title</span><span style="color:#710">'</span></span>) = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">JSONify your Ruby Translations</span><span style="color:#710">&quot;</span></span>) </pre></div> </div> <p>You will notice that this query is very similar to the earlier one for querying on a JSON/JSONB column for a single attribute. The difference is that we have changed the clause:</p> <pre> "posts"."title" -> 'en' </pre> <p>to the clause:</p> <pre> ("post"."translations" -> 'en') -> 'title' </pre> <p>Mobility leverages this similarity such that only small tweaks are necessary to support querying on the Container backend. Indeed, although as of version 0.5 there are now <em>three</em> JSON-based backends in Mobility (<code>Json</code>, <code>Jsonb</code> and <code>Container</code>), the actual amount of code is relatively small given how many features are supported. This is exactly the benefits I was after when I designed Mobility in the first place.</p> <h2>JSONify it, even just a little bit</h2> <p>The JSON format offers many advantages in storing translations, but of course it&rsquo;s not a silver bullet. Some downsides include:</p> <ul> <li>Storing all translations on the same column can stress the database, triggering space reallocation and possibly slow down performance.</li> <li>Parsing translation data may be more complex that simply grabbing the value on a string or text column.</li> <li>The range of query options available is limited by database support for the column type (<code>json</code> or <code>jsonb</code>), which will generally lag query options for normal columns.</li> <li>Only works on PostgreSQL currently (no Ruby ORM support yet for MySQL json columns).</li> </ul> <p>If you look around the nets, you can find <a href="https://elixirforum.com/t/trans-embedded-translations-for-elixir/840/18">a lot of debate</a> about the advantages and disadvantages of different storage strategies for i18n data. Some argue for storing <a href="https://elixirforum.com/t/trans-embedded-translations-for-elixir/840/13">each translation on its own column</a>. Others argue for <a href="https://groups.google.com/forum/#!topic/sequel-talk/qPQTxFcikeI">storing translations on a separate table</a>. Others <a href="https://www.bounga.org/ruby/2017/11/15/translating-activerecord-data-using-json-datatype/">argue for the JSON data type strategies described here</a>.</p> <p>Well, Mobility supports <em>all these strategies</em> with the minimum code required to do so. It does so, moreover, in such a way that you can mix or match however you like. Want to give JSON a whirl on one attribute, but stick with another gem for the other attributes? You can do that. Want to use one backend for most attributes, but test a JSON backend on some other attributes? You can do that too.</p> <p>The gem&rsquo;s flexibility is why I gave Mobility its name. So if you&rsquo;re considering storing translations in your application, I&rsquo;d highly recommend giving JSON a try, alongside other options. No single solution will work best for everybody, but there are enough options to mix and match that something out there should fit your needs.</p> <p>And of course, if you find JSON doesn&rsquo;t work for you, translation tables don&rsquo;t work for you, <a href="https://github.com/shioyama/mobility/wiki/Column-Backend">translatable columns</a> don&rsquo;t work for you, a <a href="https://github.com/shioyama/mobility/wiki/KeyValue-Backend">polymorphic key/value store</a> doesn&rsquo;t work for you&hellip; well, you can always <a href="https://github.com/shioyama/mobility/wiki/Introduction-to-Mobility-Backends">roll your own</a>.</p> <p>If you do, let me know what you find! &#x1f609;</p> <h4>Notes</h4> <p><small><a name="note-1"><sup>1</sup></a> Actually, ActiveRecord&rsquo;s support is pretty dismal compared to Sequel.</small><br /> <small><a name="note-2"><sup>2</sup></a> Incidentally, this operator accepts any JSON object, so you can translate and query translated hash data in the same way, for example. See the <a href="https://github.com/shioyama/mobility/wiki/Postgres-Backends-(Column-Attribute)" target="_blank">Mobility Wiki</a> for more details.</small><br /> <small><a name="note-3"><sup>3</sup></a> You don&rsquo;t need to specify whether you are using a JSON column or a JSONB column, Mobility will check the db schema and figure it out based on the attribute name.<small></p> Mobility 0.3: Ready for Prime Time https://dejimata.com/2017/12/5/mobility-0-3-ready-for-prime-time 2017-12-05 2017-12-05 Chris Salzberg <p>Just a little over six months ago, I released the first version of the Ruby translation framework I&rsquo;ve called <a href="https://github.com/shioyama/mobility">Mobility</a>. With the third release of</p> <p>Just a little over six months ago, I released the first version of the Ruby translation framework I&rsquo;ve called <a href="https://github.com/shioyama/mobility">Mobility</a>. With the third release of the gem &ndash; and a <a href="https://github.com/shioyama/mobility#companies-using-mobility">growing list of companies</a> using it to power their production applications &ndash; I&rsquo;m happy to say that <strong>Mobility is today more stable and reliable than ever before</strong>.</p> <div class="img-center" style="margin: 60px 0 40px 0"> <img alt="Column Strategy" src="/img/rocket.png"></a> <p class="caption"> And we have a liftoff<a href="#note-1"><sup>1</sup></a> </p> </div> <p>As its name implies, <strong>Mobility is all about <a href="/2017/3/3/translating-with-mobility">giving you choices</a></strong>. Whether you store your model translations in a <a href="https://github.com/shioyama/mobility/wiki/Table-Backend">set of model-specific tables</a>, in a <a href="https://github.com/shioyama/mobility/wiki/KeyValue-Backend">single shared table</a>, as <a href="https://github.com/shioyama/mobility/wiki/Column-Backend">columns on each model table</a>, or as values in a Postgres <a href="https://github.com/shioyama/mobility/wiki/Postgres-Backends">jsonb or hstore column</a>, Mobility has you covered. Whether you use <a href="https://github.com/shioyama/mobility#fallbacks">fallbacks</a>, <a href="https://github.com/shioyama/mobility#dirty-tracking">dirty change tracking</a>, <a href="https://github.com/shioyama/mobility#querying">query support</a>, or any of its other features, Mobility will do what you need it to do without getting in the way.</p> <p>This week, I&rsquo;ve released <a href="https://rubygems.org/gems/mobility">Mobility 0.3</a>, and in the spirit of its name, this release (among other things) will keep you up to speed with your gem dependencies. Version 0.3 includes full support for the latest versions of its ORM dependencies: Rails/ActiveRecord 5.2 (literally just released in beta) as well as <a href="http://sequel.jeremyevans.net/">Sequel</a> 5.0.</p> <p>This release also comes with some <a href="https://github.com/shioyama/mobility/commits?author=pwim">important changes</a> by a new contributor: Paul McMahon (<a href="https://github.com/pwim">@pwim</a>), who has been also helping out commenting on and reviewing PRs. Having someone with Paul&rsquo;s skill and experience helping out has been a huge boost; the fact that Paul is using Mobility in production on his own service, <a href="https://www.doorkeeper.jp/">Doorkeeper</a>, has also been great feedback for identifying important issues and fixing them quickly.</p> <p>Paul and I have put together a short wiki page on <a href="https://github.com/shioyama/mobility/wiki/Migrating-from-Globalize">migrating from Globalize to Mobility</a>, which is what he did with Doorkeeper; be sure to check it out if you are using Globalize and considering switching to Mobility. For most users, Mobility will be a nearly 100% drop-in replacement for Globalize, and the migration process should be quite simple (as it was for Paul). However, for anyone with questions, I have setup a <a href="https://gitter.im/mobility-ruby/mobility">gitter community</a> for discussions, and I also am watching for StackOverflow questions with the <a href="https://stackoverflow.com/questions/tagged/mobility">mobility tag</a>.</p> <p>So, without further ado, here are the changes&hellip;</p> <h2>Support for Rails 5.2 and Sequel 5</h2> <p>Yes, Rails 5.2.beta.2 was <a href="https://weblog.rubyonrails.org/2017/11/27/Rails-5-2-Active-Storage-Redis-Cache-Store-HTTP2-Early-Hints-Credentials/">just released this week</a>, and we&rsquo;re <a href="https://github.com/shioyama/mobility/pull/114">already running tests against it</a>. After a few small <a href="https://github.com/shioyama/mobility/pull/116">compatibility changes</a>, those specs are now all passing and we are ready for ActiveRecord 5.2!</p> <p>Sequel 5.0 was <a href="https://github.com/jeremyevans/sequel/blob/master/doc/release_notes/5.0.0.txt">also recently released</a>, and with a <a href="https://github.com/shioyama/mobility/pull/121">small update to Mobility</a> released in <a href="https://rubygems.org/gems/mobility/versions/0.3.2">0.3.2</a>, all features of the Sequel backends are now working with this version as well.</p> <h2>Support new ActiveRecord Dirty methods</h2> <p>In Rails 5.1, <a href="https://github.com/rails/rails/pull/25337">new dirty tracking methods were added</a> to differentiate between changes that have been persisted (saved) to the database and those that have only been changed in-memory (<em>set</em> but not yet <em>saved</em>).</p> <p>There is a new method <code>saved_changes</code> to return changes to attributes that have been saved, as well as attribute-specific methods (for an attribute <code>title</code>):</p> <ul> <li><code>saved_change_to_title?</code></li> <li><code>saved_change_to_title</code></li> <li><code>title_before_last_save</code></li> <li><code>will_save_change_to_title?</code></li> <li><code>title_change_to_be_saved</code></li> <li><code>title_in_database</code></li> </ul> <p>Support for these methods <a href="https://github.com/shioyama/mobility/pull/111">has now been added</a>. Just enable the <code>dirty</code> plugin and the methods will be available for translated attributes on your model.</p> <h2>Fallbacks when Locale is Specified</h2> <p>Paul <a href="https://github.com/shioyama/mobility/issues/85">noticed</a> that the way locale accessors and fallbacks interacted was not very intuitive in version 0.2, and did not correspond to how Globalize works (and to users' typical expectations).</p> <p>Here&rsquo;s an example where we are setting the title in English, then calling <code>title_ja</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Event</span> &lt; <span style="color:#036;font-weight:bold">ApplicationRecord</span> extend <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">fallbacks</span>: { <span style="color:#606">en</span>: <span style="color:#A60">:ja</span>, <span style="color:#606">ja</span>: <span style="color:#A60">:en</span> } <span style="color:#080;font-weight:bold">end</span> <span style="color:#036;font-weight:bold">Event</span>.new(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span>).title_ja <span style="color:#777">#=&gt; &quot;foo&quot; instead of nil</span> </pre></div> </div> <p>Although fallbacks are enabled from Japanese to English, since we are explicitly specifying the locale we want (Japanese) in <code>title_ja</code>, one would reasonably expect the fallbacks to not take effect, but in fact they do.</p> <p>This has been fixed in 0.3, so that you get the value in the locale you specified, without falling back to other locales:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Event</span>.new(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span>).title_ja <span style="color:#777">#=&gt; nil</span> </pre></div> </div> <p>This also applies when you fetch an attribute in a locale using the getter options hash:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Event</span>.new(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span>).title(<span style="color:#606">locale</span>: <span style="color:#A60">:ja</span>) <span style="color:#777">#=&gt; nil</span> </pre></div> </div> <p>This is a really nice fix since as a byproduct it <a href="https://github.com/shioyama/mobility/pull/86/files#r137799779">also makes the dirty plugin work more naturally with fallbacks</a> (showing changes from the previous value <em>without</em> fallbacks).</p> <h2>Duplicate Translations when Duplicating Model</h2> <p>Paul noticed that the ActiveRecord Table backend does not duplicate translations when <code>dup</code> is called on the model. This is different from Globalize, which duplicates the translations <a href="https://github.com/globalize/globalize/blob/ef1e362f3144b2276d89f47407c67ffa3fc1f9f2/lib/globalize/active_record/instance_methods.rb#L98-L105">as well</a>.</p> <p>After some discussion, Paul went ahead and <a href="https://github.com/shioyama/mobility/pull/84">implemented this feature</a> by patching <code>initialize_dup</code>. So as of version 0.3, if you <code>dup</code> your translated model instance, you&rsquo;ll get a copy both of the model and any translations.</p> <p>Most other backends work without any such patch, since they map to columns on the model itself. However, Paul noticed that the KeyValue backends (which like the Table backend store translations in associations) were affected by the same issue. I have since followed-up and <a href="https://github.com/shioyama/mobility/pull/126">also fixed this issue for the AR KeyValue backend</a>.</p> <p>I have just released this fix in <a href="https://rubygems.org/gems/mobility/versions/0.3.3">0.3.3</a>, so if you update to the latest release, you should have no dup-ing issues with ActiveRecord models for any backend. Sequel handles duplicating slightly differently, so will require more attention; a fix should be out in the next minor release.</p> <h2>Invalidate Cache Key</h2> <p>This is <a href="https://github.com/shioyama/mobility/pull/104">another one by Paul</a>. Up to version 0.2, Mobility would not &ldquo;touch&rdquo; the associated translations for the Table and KeyValue backends. This results in problems with caching, since ActiveRecord doesn&rsquo;t know that the attributes have changed.</p> <p>The fix we decided on for this issue is simply to add a <code>touch: true</code> to the association defined for translations on the model. With this change, updating a translation will update the <code>updated_at</code> timestamp on the model. Unlike Globalize, which includes this setting as an option, as of version 0.3, Mobility sets <code>touch: true</code> by default, since there is not conceivable reason we could see that you would <em>not</em> want this enabled.</p> <h2>Convert AttributeMethods to a Plugin</h2> <p>One of the tricky parts of building a gem like Mobility is deciding how far to go in overriding core methods of the ORM.</p> <p>There is a fine line you need to draw. Naturally, you want to make translated attributes behave just like normal attributes, since this is natural for the user and means they don&rsquo;t need to think too much about them.</p> <p>However, if you go to far with this, then the ORM (typically ActiveRecord) gets confused and thinks they are actually table columns, with unpredictable consequences. This has happened <a href="https://github.com/shioyama/mobility/issues/97">with Mobility</a> and happens frequently with Globalize, which does much more patching. The results can be very hard to debug.</p> <p>One of the ways that Mobility patched ActiveRecord previous to version 0.3 was to add translated attributes to the <code>attributes</code> hash:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#777"># Version &lt; 0.3</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ApplicationRecord</span> extend <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span> <span style="color:#080;font-weight:bold">end</span> post = <span style="color:#036;font-weight:bold">Post</span>.new post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span> post.attributes <span style="color:#777">#=&gt; {&quot;id&quot;=&gt;nil, &quot;created_at&quot;=&gt;nil, &quot;updated_at&quot;=&gt;nil, &quot;title&quot;=&gt;&quot;foo&quot;}</span> </pre></div> </div> <p>Notice that the last key in the hash returned by <code>post.attributes</code> is <code>title</code>, although <code>title</code> is not actually a table column.</p> <p>This may seem nice, but in practice it can cause problems both with ActiveRecord itself as well as with other gems. Since <code>title</code> is a key in this hash, a gem might reasonably assume that you can call <code>read_attirbute("title")</code> and get the value of the title, but Mobility (unlike Globalize) does not patch <code>read_attribute</code> to do this, so this fails.</p> <p>To avoid compatibility issues, this feature of overriding <code>attributes</code> has now been extracted into a plugin, which by default is <em>off</em>, so the instance and model above would not by include <code>title</code> as a key in <code>attributes</code>:</p> <div class="CodeRay"> <div class="code"><pre> post.attributes <span style="color:#777">#=&gt; {&quot;id&quot;=&gt;nil, &quot;created_at&quot;=&gt;nil, &quot;updated_at&quot;=&gt;nil}</span> </pre></div> </div> <p>You can enable it by passing <code>attribute_methods: true</code> to <code>translates</code> (or enabling it by default in the <code>default_options</code> configuration (see below).</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#777"># Version 0.3, with attribute_methods plugin enabled</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ApplicationRecord</span> extend <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">attribute_methods</span>: <span style="color:#069">true</span> <span style="color:#080;font-weight:bold">end</span> post = <span style="color:#036;font-weight:bold">Post</span>.new post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span> post.attributes <span style="color:#777">#=&gt; {&quot;id&quot;=&gt;nil, &quot;created_at&quot;=&gt;nil, &quot;updated_at&quot;=&gt;nil, &quot;title&quot;=&gt;&quot;foo&quot;}</span> </pre></div> </div> <p>Although you can do this with the plugin, in general unless you know you need it, my recommendation would be not to enable attribute methods since it can cause conflicts with other gems. For example, see <a href="https://github.com/shioyama/mobility/issues/97">this issue</a> about a compatibility conflict with the <a href="https://github.com/rmm5t/strip_attributes">strip_attributes gem</a>.</p> <h2>Deprecate Default Options Setter</h2> <p>The last change is a deprecation, not always the most exciting thing for users of a gem. However, this is an important one. Reasons for the change are described in <a href="https://github.com/shioyama/mobility/issues/92">this issue</a>.</p> <p>In a nutshell, versions of Mobility prior to 0.3 would allow you to set a hash of default options, which would be merged with whatever options you pass to <code>translates</code> from any model:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.configure <span style="color:#080;font-weight:bold">do</span> |config| config.default_options = { <span style="color:#606">dirty</span>: <span style="color:#069">true</span>, <span style="color:#606">fallbacks</span>: <span style="color:#069">true</span>, <span style="color:#777"># ...</span> } <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>The problem with this is that, by directly setting the defaults, you <em>override</em> the &ldquo;default defaults&rdquo;, as it were. These defaults can be found <a href="https://github.com/shioyama/mobility/blob/6e9c8168b324c4ca12bc63b8a8af53c17c31e02c/lib/mobility/configuration.rb#L76-L83">here</a>, and you can see some plugin settings that might not be familiar.</p> <p>One of these is for a <a href="http://www.rubydoc.info/gems/mobility/Mobility/Plugins/Presence">&ldquo;presence&rdquo; plugin</a> which is on by default. This plugin simply filters any values set or fetched from the backend to filter out blank strings. I noticed that some users had blanks in their data, which should not happen if this plugin is enabled; it turns out, the reason is because they had set the <code>default_options</code> hash directly, without passing any value for the <code>presence</code> key, thus implicitly turning it off.</p> <p>To avoid this from happening, <code>default_options=</code> (the setter) has been deprecated in version 0.3. If you are currently setting it, you will see a warning like this:</p> <pre> WARNING: The default_options= setter has been deprecated. Set each option on the default_options hash instead, like this: config.default_options[:fallbacks] = { ... } config.default_options[:dirty] = true </pre> <p>This setter method will be removed in the next minor release. To get rid of the deprecation warning, set defaults for each key on the <code>default_options</code> hash, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.configure <span style="color:#080;font-weight:bold">do</span> |config| config.default_options[<span style="color:#A60">:dirty</span>] = <span style="color:#069">true</span> config.default_options[<span style="color:#A60">:fallbacks</span>] = <span style="color:#069">true</span> <span style="color:#777"># ...</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Although this requires a few more keystrokes, it avoids the problem of overriding the &ldquo;default defaults&rdquo; stored initially in <code>config.default_options</code>. I think this is a saner way to set defaults, ensuring that you only opt-out of important plugins if you explicitly set the default.</p> <h2>Looking Ahead: A Christmas Release</h2> <p>The next release of Mobility will most likely be for the first major version (1.0.0), which I&rsquo;m aiming to finish in time for Christmas. There will be a couple of breaking changes in this release, mostly just method renaming which should only affect some users and will be easy to fix.</p> <p>As mentioned above, I&rsquo;m eager to get any feedback and suggestions. Please join the <a href="https://gitter.im/mobility-ruby/mobility">gitter</a> community or post your questions to StackOverflow. If you&rsquo;re building a gem which depends on translations and are considering using Mobility, please contact me since this is a use case I&rsquo;m particularly interested in.</p> <p>Thanks for everyone&rsquo;s support, and stay tuned for the next release!</p> <h3>Notes</h3> <p><small><a name="note-1"><sup>1</sup></a> NASA Orbital ATK CRS-8 Launch (<a href="https://www.flickr.com/photos/nasahqphoto/37648937724">ref</a>)</small><br></p> Mobility 0.2: Now with Plugins https://dejimata.com/2017/8/13/mobility-0-2-now-with-plugins 2017-08-13 2017-08-13 Chris Salzberg <p>It&rsquo;s been a little while since last I <a href="/2017/3/3/translating-with-mobility">posted</a> about <a href="https://github.com/shioyama/mobility">Mobility</a>, the pluggable translation framework for Ruby that I&rsquo;ve been working on recently. I thought I&rsquo;d take this opportunity with the release of the second version of the gem to highlight some of the important changes and new features, and to recap how far things have come.</p> <h2>Mobility and the Module Builder Pattern</h2> <p>In the time since I posted <a href="/2017/3/3/translating-with-mobility">Translating with Mobility</a>, one of the techniques I have used heavily in building the gem &ndash; the <a href="/2017/5/20/the-ruby-module-builder-pattern">Module Builder Pattern</a>, as I&rsquo;ve termed it &ndash; has become a <a href="https://news.ycombinator.com/item?id=14386723">hot topic of its own</a>. (I&rsquo;ll actually be <a href="http://rubykaigi.org/2017/presentations/shioyama.html">talking about module builders</a> at the coming RubyKaigi in Hiroshima, check it out if you&rsquo;ll be attending.)</p> <p>In a nutshell, the Module Builder Pattern is a simple way to create customizable modules to mix into classes, one which falls naturally out of Ruby&rsquo;s object model. It amounts to simply subclassing the <code>Module</code> class, defining module methods in an initializer or <code>included</code> hook, and including instances of the subclass in other classes. See the slide below from my recent talk at the <a href="https://trbmeetup.doorkeeper.jp/events/63110">Tokyo Rubyist Meetup</a>, where I explain the basic elements of a module builder and how I use it in a core Mobility class, <code>Mobility::Attributes</code>.</p> <div class="img-center" style="margin: 40px"> <img alt="Elements of a Module Builder" src="/img/tokyorubyist-modulebuilder.png"/> </div> <p>From its first release, Mobility has used this pattern all over the place; I could never have achieved the flexibility the gem needed without it. However, in the latest release, I&rsquo;ve pushed this yet further in order to achieve something just as important: <em>extensibility</em>.</p> <h2>Now with Plugins</h2> <p>Let&rsquo;s recall that in Mobility, you define a translated attribute on a model by calling <code>translates</code> with one or more attribute names, and a hash to configure which options are applied: <a href="https://github.com/shioyama/mobility#dirty-tracking">dirty-tracking</a>, <a href="https://github.com/shioyama/mobility">fallbacks</a>, <a href="https://github.com/shioyama/mobility#getting-and-setting-translations">locale accessors</a>, and so on. Under the hood, Mobility creates an instance of the <a href="https://github.com/shioyama/mobility/blob/93af67072bb69b10818923a03ca73aa7b8eb0cc2/lib/mobility/attributes.rb#L100">Mobility::Attributes</a> module builder and includes it in the class. I&rsquo;ll use this form directly here to emphasize what is actually going on:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> extend <span style="color:#036;font-weight:bold">Mobility</span> include <span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Attributes</span>.new(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">content</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">locale_accessors</span>: [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>], <span style="color:#606">fallbacks</span>: <span style="color:#069">true</span>) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Here, we are building a module instance of <code>Mobility::Attributes</code> for the attributes <code>title</code> and <code>content</code>, with locale accessors for English and Japanese and fallbacks enabled. (The format of arguments to this initializer have changed slightly since version 0.1, see this <a href="https://github.com/shioyama/mobility/pull/72">pull request</a> for details).</p> <p>What happens when we include the instance of this class into <code>Post</code>? Well, here&rsquo;s the first change with version 0.2: I&rsquo;ve made the magic here a whole lot clearer, and more extensible as a result (see PRs <a href="https://github.com/shioyama/mobility/pull/50">#50</a> and <a href="https://github.com/shioyama/mobility/pull/64">#62</a>).</p> <p>Previously there was a lot of hard-coded references to options directly in <code>Attributes</code>, but the code is now entirely free of such references. In fact, there are just a few lines in the builder which deal with options:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.plugins.each <span style="color:#080;font-weight:bold">do</span> |name| plugin = get_plugin_class(name) plugin.apply(<span style="color:#069">self</span>, options[name]) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Here, <code>Mobility.plugins</code> is an alias for <code>Mobility.config.plugins</code>, a new configuration setting which defaults to <a href="https://github.com/shioyama/mobility/blob/e1f2987d67dd113b8c69dc82d80fb55ec32807f4/lib/mobility/configuration.rb#L73-L81">this array</a> (order of elements is important here<a href="#note-1"><sup>1</sup></a>):</p> <div class="CodeRay"> <div class="code"><pre> [<span style="color:#A60">:cache</span>, <span style="color:#A60">:dirty</span>, <span style="color:#A60">:fallbacks</span>, <span style="color:#A60">:presence</span>, <span style="color:#A60">:default</span>, <span style="color:#A60">:fallthrough_accessors</span>, <span style="color:#A60">:locale_accessors</span>] </pre></div> </div> <p>The private <code>get_plugin_class</code> method fetches constants under <code>Mobility::Plugins</code> matching CamelCase-ed values of the plugin names from this array, then applies them with the attributes instance and option values as its arguments.</p> <p>So <code>locale_accessors: [:en, :ja]</code> will fetch <code>Mobility::Plugins::LocaleAccesors</code> and call it like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Plugins</span>::<span style="color:#036;font-weight:bold">LocaleAccessors</span>.apply(<span style="color:#069">self</span>, [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>]) </pre></div> </div> <p>&hellip;where <code>self</code> is the instance of <code>Mobility::Attributes</code> being included into the model class, <code>Post</code>.</p> <p>Notice that since we have no hard-coded references to option keys in this module now, <em>any</em> option key included in <code>Mobility.plugins</code> will apply a plugin with that name (as long as that plugin exists), with the attributes instance and option value passed into the plugin&rsquo;s <code>apply</code> class method.</p> <p>This makes is very easy to design new extensions: all they need is to have a class method, <code>apply</code>, which takes this <code>Attributes</code> instance (which itself has references to the backend class and attribute names), and an option value to configure how the plugin should be applied. (Incidentally, I borrow the term &ldquo;plugins&rdquo; here, and the method name <code>apply</code>, from <a href="http://sequel.jeremyevans.net/">Sequel</a>, where plugins are defined in a somewhat similar way.)</p> <p>Below is another slide from my recent talk, illustrating how these options trigger inclusion of yet more module builder instances into the backend class and into the Attributes instance itself.</p> <div class="img-center" style="margin: 40px"> <img alt="Mobility Plugins" src="/img/tokyorubyist-plugins.png" /> </div> <p>Along with this introduction of plugins and the <code>Mobility::Plugins</code> namespace, I&rsquo;ve also reorganized the backend namespace such that <code>Mobility::Backends</code> (plural) is now exclusively used for backend classes.</p> <p>The result is that it becomes very straightforward to build custom backends and plugins: define a class in the appropriate namespace with the required structure, and call <code>translates</code> (or include an instance of <code>Mobility::Attributes</code>) in the model class, with any backend or option keys, and that&rsquo;s it! You ready to go.</p> <h2>Default Options</h2> <p>Another minor annoyance some users of Mobility may have encountered in 0.1 is that while it was possible to set a default backend to use across all models (with the <code>default_backend</code> configuration option), there was no way to define a default set of options (to configure plugins and the backend). Along with the changes to enable plugins above, I also <a href="https://github.com/shioyama/mobility/pull/50">added this as a configuration setting</a>, with a value that defaults to:</p> <div class="CodeRay"> <div class="code"><pre> { <span style="color:#606">cache</span>: <span style="color:#069">true</span>, <span style="color:#606">dirty</span>: <span style="color:#069">false</span>, <span style="color:#606">fallbacks</span>: <span style="color:#069">nil</span>, <span style="color:#606">presence</span>: <span style="color:#069">true</span>, <span style="color:#606">default</span>: <span style="color:#069">nil</span> } </pre></div> </div> <p>These default option values were previously hard-coded, but are now explicitly set and isolated in the configuration instance, which makes them much easier to reason about and change.</p> <p>Want to enable fallbacks by default on all your models? Just override the default for that key:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.default_options[<span style="color:#A60">:fallbacks</span>] = <span style="color:#069">true</span> </pre></div> </div> <p>You can also set backend options here, so for example if you want to set the default <code>type</code> for the KeyValue backend to be &ldquo;string&rdquo; instead of &ldquo;text&rdquo;, you could do this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.default_options[<span style="color:#A60">:type</span>] = <span style="color:#A60">:string</span> </pre></div> </div> <p>Note that <em>how</em> these option values are interpreted by actual plugins and backends depends on the plugin or backend, and there is some unavoidable variation there. But the values passed in to <code>apply</code> (plugin) and <code>new</code> (backend) are entirely determined from this hash.</p> <h2>Enumerable Backends</h2> <p>Backends as implemented prior to version 0.2 could only do two things: read and write values for a given locale. If you wanted to know <em>what</em> locales were available for a given attribute, you would be stuck.</p> <p>This was raised as an <a href="https://github.com/shioyama/mobility/issues/39">issue</a>, and I decided that rather than just add a method returning locales, I&rsquo;d go a bit further and try to solve a more general problem: iterating through translations. I implemented this in <a href="https://github.com/shioyama/mobility/pull/71">PR #71</a> by adding a new <code>each_locale</code> method to each existing backend returning successive locales to a block.</p> <p>From these backend-specific <code>each_locale</code> methods, an <code>each</code> method is defined in <code>Mobility::Backend</code> yielding instances of a <code>Translation</code> struct wrapping the locale and backend. We then include Ruby&rsquo;s powerful <a href="https://ruby-doc.org/core-2.4.1/Enumerable.html">Enumerable</a> module, giving us with it a whole slew of traversal and search methods. A method <code>locales</code> is also added to <code>Mobility::Backend</code> which returns the available locales for the attribute (the original request).</p> <p>With this change, you can now do things like select translations that match a condition:</p> <p><pre type="ruby"> post = Post.create(title_en: &ldquo;English foo&rdquo;, title_de: &ldquo;German foo&rdquo;, title_ja: &ldquo;Something else&rdquo;) post.title_backend.select { |t| t.read =</p> <p>It&rsquo;s been a little while since last I <a href="/2017/3/3/translating-with-mobility">posted</a> about <a href="https://github.com/shioyama/mobility">Mobility</a>, the pluggable translation framework for Ruby that I&rsquo;ve been working on recently. I thought I&rsquo;d take this opportunity with the release of the second version of the gem to highlight some of the important changes and new features, and to recap how far things have come.</p> <h2>Mobility and the Module Builder Pattern</h2> <p>In the time since I posted <a href="/2017/3/3/translating-with-mobility">Translating with Mobility</a>, one of the techniques I have used heavily in building the gem &ndash; the <a href="/2017/5/20/the-ruby-module-builder-pattern">Module Builder Pattern</a>, as I&rsquo;ve termed it &ndash; has become a <a href="https://news.ycombinator.com/item?id=14386723">hot topic of its own</a>. (I&rsquo;ll actually be <a href="http://rubykaigi.org/2017/presentations/shioyama.html">talking about module builders</a> at the coming RubyKaigi in Hiroshima, check it out if you&rsquo;ll be attending.)</p> <p>In a nutshell, the Module Builder Pattern is a simple way to create customizable modules to mix into classes, one which falls naturally out of Ruby&rsquo;s object model. It amounts to simply subclassing the <code>Module</code> class, defining module methods in an initializer or <code>included</code> hook, and including instances of the subclass in other classes. See the slide below from my recent talk at the <a href="https://trbmeetup.doorkeeper.jp/events/63110">Tokyo Rubyist Meetup</a>, where I explain the basic elements of a module builder and how I use it in a core Mobility class, <code>Mobility::Attributes</code>.</p> <div class="img-center" style="margin: 40px"> <img alt="Elements of a Module Builder" src="/img/tokyorubyist-modulebuilder.png"/> </div> <p>From its first release, Mobility has used this pattern all over the place; I could never have achieved the flexibility the gem needed without it. However, in the latest release, I&rsquo;ve pushed this yet further in order to achieve something just as important: <em>extensibility</em>.</p> <h2>Now with Plugins</h2> <p>Let&rsquo;s recall that in Mobility, you define a translated attribute on a model by calling <code>translates</code> with one or more attribute names, and a hash to configure which options are applied: <a href="https://github.com/shioyama/mobility#dirty-tracking">dirty-tracking</a>, <a href="https://github.com/shioyama/mobility">fallbacks</a>, <a href="https://github.com/shioyama/mobility#getting-and-setting-translations">locale accessors</a>, and so on. Under the hood, Mobility creates an instance of the <a href="https://github.com/shioyama/mobility/blob/93af67072bb69b10818923a03ca73aa7b8eb0cc2/lib/mobility/attributes.rb#L100">Mobility::Attributes</a> module builder and includes it in the class. I&rsquo;ll use this form directly here to emphasize what is actually going on:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> extend <span style="color:#036;font-weight:bold">Mobility</span> include <span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Attributes</span>.new(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">content</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">locale_accessors</span>: [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>], <span style="color:#606">fallbacks</span>: <span style="color:#069">true</span>) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Here, we are building a module instance of <code>Mobility::Attributes</code> for the attributes <code>title</code> and <code>content</code>, with locale accessors for English and Japanese and fallbacks enabled. (The format of arguments to this initializer have changed slightly since version 0.1, see this <a href="https://github.com/shioyama/mobility/pull/72">pull request</a> for details).</p> <p>What happens when we include the instance of this class into <code>Post</code>? Well, here&rsquo;s the first change with version 0.2: I&rsquo;ve made the magic here a whole lot clearer, and more extensible as a result (see PRs <a href="https://github.com/shioyama/mobility/pull/50">#50</a> and <a href="https://github.com/shioyama/mobility/pull/64">#62</a>).</p> <p>Previously there was a lot of hard-coded references to options directly in <code>Attributes</code>, but the code is now entirely free of such references. In fact, there are just a few lines in the builder which deal with options:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.plugins.each <span style="color:#080;font-weight:bold">do</span> |name| plugin = get_plugin_class(name) plugin.apply(<span style="color:#069">self</span>, options[name]) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Here, <code>Mobility.plugins</code> is an alias for <code>Mobility.config.plugins</code>, a new configuration setting which defaults to <a href="https://github.com/shioyama/mobility/blob/e1f2987d67dd113b8c69dc82d80fb55ec32807f4/lib/mobility/configuration.rb#L73-L81">this array</a> (order of elements is important here<a href="#note-1"><sup>1</sup></a>):</p> <div class="CodeRay"> <div class="code"><pre> [<span style="color:#A60">:cache</span>, <span style="color:#A60">:dirty</span>, <span style="color:#A60">:fallbacks</span>, <span style="color:#A60">:presence</span>, <span style="color:#A60">:default</span>, <span style="color:#A60">:fallthrough_accessors</span>, <span style="color:#A60">:locale_accessors</span>] </pre></div> </div> <p>The private <code>get_plugin_class</code> method fetches constants under <code>Mobility::Plugins</code> matching CamelCase-ed values of the plugin names from this array, then applies them with the attributes instance and option values as its arguments.</p> <p>So <code>locale_accessors: [:en, :ja]</code> will fetch <code>Mobility::Plugins::LocaleAccesors</code> and call it like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Plugins</span>::<span style="color:#036;font-weight:bold">LocaleAccessors</span>.apply(<span style="color:#069">self</span>, [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>]) </pre></div> </div> <p>&hellip;where <code>self</code> is the instance of <code>Mobility::Attributes</code> being included into the model class, <code>Post</code>.</p> <p>Notice that since we have no hard-coded references to option keys in this module now, <em>any</em> option key included in <code>Mobility.plugins</code> will apply a plugin with that name (as long as that plugin exists), with the attributes instance and option value passed into the plugin&rsquo;s <code>apply</code> class method.</p> <p>This makes is very easy to design new extensions: all they need is to have a class method, <code>apply</code>, which takes this <code>Attributes</code> instance (which itself has references to the backend class and attribute names), and an option value to configure how the plugin should be applied. (Incidentally, I borrow the term &ldquo;plugins&rdquo; here, and the method name <code>apply</code>, from <a href="http://sequel.jeremyevans.net/">Sequel</a>, where plugins are defined in a somewhat similar way.)</p> <p>Below is another slide from my recent talk, illustrating how these options trigger inclusion of yet more module builder instances into the backend class and into the Attributes instance itself.</p> <div class="img-center" style="margin: 40px"> <img alt="Mobility Plugins" src="/img/tokyorubyist-plugins.png" /> </div> <p>Along with this introduction of plugins and the <code>Mobility::Plugins</code> namespace, I&rsquo;ve also reorganized the backend namespace such that <code>Mobility::Backends</code> (plural) is now exclusively used for backend classes.</p> <p>The result is that it becomes very straightforward to build custom backends and plugins: define a class in the appropriate namespace with the required structure, and call <code>translates</code> (or include an instance of <code>Mobility::Attributes</code>) in the model class, with any backend or option keys, and that&rsquo;s it! You ready to go.</p> <h2>Default Options</h2> <p>Another minor annoyance some users of Mobility may have encountered in 0.1 is that while it was possible to set a default backend to use across all models (with the <code>default_backend</code> configuration option), there was no way to define a default set of options (to configure plugins and the backend). Along with the changes to enable plugins above, I also <a href="https://github.com/shioyama/mobility/pull/50">added this as a configuration setting</a>, with a value that defaults to:</p> <div class="CodeRay"> <div class="code"><pre> { <span style="color:#606">cache</span>: <span style="color:#069">true</span>, <span style="color:#606">dirty</span>: <span style="color:#069">false</span>, <span style="color:#606">fallbacks</span>: <span style="color:#069">nil</span>, <span style="color:#606">presence</span>: <span style="color:#069">true</span>, <span style="color:#606">default</span>: <span style="color:#069">nil</span> } </pre></div> </div> <p>These default option values were previously hard-coded, but are now explicitly set and isolated in the configuration instance, which makes them much easier to reason about and change.</p> <p>Want to enable fallbacks by default on all your models? Just override the default for that key:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.default_options[<span style="color:#A60">:fallbacks</span>] = <span style="color:#069">true</span> </pre></div> </div> <p>You can also set backend options here, so for example if you want to set the default <code>type</code> for the KeyValue backend to be &ldquo;string&rdquo; instead of &ldquo;text&rdquo;, you could do this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.default_options[<span style="color:#A60">:type</span>] = <span style="color:#A60">:string</span> </pre></div> </div> <p>Note that <em>how</em> these option values are interpreted by actual plugins and backends depends on the plugin or backend, and there is some unavoidable variation there. But the values passed in to <code>apply</code> (plugin) and <code>new</code> (backend) are entirely determined from this hash.</p> <h2>Enumerable Backends</h2> <p>Backends as implemented prior to version 0.2 could only do two things: read and write values for a given locale. If you wanted to know <em>what</em> locales were available for a given attribute, you would be stuck.</p> <p>This was raised as an <a href="https://github.com/shioyama/mobility/issues/39">issue</a>, and I decided that rather than just add a method returning locales, I&rsquo;d go a bit further and try to solve a more general problem: iterating through translations. I implemented this in <a href="https://github.com/shioyama/mobility/pull/71">PR #71</a> by adding a new <code>each_locale</code> method to each existing backend returning successive locales to a block.</p> <p>From these backend-specific <code>each_locale</code> methods, an <code>each</code> method is defined in <code>Mobility::Backend</code> yielding instances of a <code>Translation</code> struct wrapping the locale and backend. We then include Ruby&rsquo;s powerful <a href="https://ruby-doc.org/core-2.4.1/Enumerable.html">Enumerable</a> module, giving us with it a whole slew of traversal and search methods. A method <code>locales</code> is also added to <code>Mobility::Backend</code> which returns the available locales for the attribute (the original request).</p> <p>With this change, you can now do things like select translations that match a condition:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.create(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">English foo</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">title_de</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">German foo</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">title_ja</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Something else</span><span style="color:#710">&quot;</span></span>) post.title_backend.select { |t| t.read =~ <span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#808">foo</span><span style="color:#404">/</span></span> }.map &amp;<span style="color:#A60">:locale</span> <span style="color:#777">#=&gt; [:en, :de]</span> </pre></div> </div> <p>Although this is a somewhat contrived example, it&rsquo;s not hard to see that having the full arsenal of Enumerable methods at your disposal could be quite useful in a variety of contexts.</p> <h2><del>Autoload</del> &rarr; Require</h2> <p>Mobility 0.1.x uses <a href="https://ruby-doc.org/core-2.4.1/Kernel.html#method-i-autoload">autoload</a> to load nearly all its constants, but as Rubyists no doubt know, autoload <a href="https://bugs.ruby-lang.org/issues/921">has its issues</a>, and <a href="http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/41149">may even be deprecated</a> in a future release of Ruby. So I decided to bite the bullet and convert all <code>autoload</code> calls to <code>require</code> (<a href="https://github.com/shioyama/mobility/pull/65">here&rsquo;s the PR</a>). For this I followed the pattern (again) of Sequel, which only requires plugins when they are called with the <code>plugin</code> method.</p> <p>So Mobility now does a similar thing for backends and plugins. Here is the code loading a backend when it is triggered from a call to <code>translates</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">get_backend_class</span>(backend) <span style="color:#080;font-weight:bold">return</span> backend <span style="color:#080;font-weight:bold">if</span> <span style="color:#036;font-weight:bold">Module</span> === backend require <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility/backends/</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>backend<span style="font-weight:bold;color:#666">}</span></span><span style="color:#710">&quot;</span></span> get_class_from_key(<span style="color:#036;font-weight:bold">Mobility</span>::<span style="color:#036;font-weight:bold">Backends</span>, backend) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Rather than simply finding the constant, we first require the backend at <code>"mobility/backends/#{backend}"</code>, then once we have loaded the file we make a <code>const_get</code> call to actually fetch the constant (which previously would have triggered <code>autoload</code> to find the backend file in the same location).</p> <p>Admittedly, this now requires the plugin/backend creator to put the backend/plugin in the requisite file location or else Mobility will not find it. But that seems to me a quite reasonable thing to do anyway.</p> <p>It also means that you can&rsquo;t just <code>require "mobility"</code>, and then expect <code>Mobility::Backends::KeyValue</code> (or any backend class) to be loaded, because it will not be required until you actually <em>use it</em>. You need to either call <code>translates</code>, or require the class explicitly, to load the backend.</p> <h2>Mobility::Util</h2> <p>Continuing with the autoload clean-up, I also decided to fix something else that had been bothering me for a while: <code>mobility/core_ext</code>. When I originally decided to make Mobility support Sequel in addition to ActiveRecord, I found it hard to entirely give up ActiveSupport since (like many Rubyists) I had become so accustomed to using methods like <code>present?</code>, <code>camelize</code>, <code>singularize</code>, etc.</p> <p>So by default, if ActiveSupport could not be loaded, I included a bare minimum implementation of just a few of these methods, filed under <code>mobility/core_ext</code>. In addition, for Sequel backends, I also just assumed that the <a href="http://www.rubydoc.info/gems/sequel/4.26.0/Sequel/Inflections">Inflections</a> module was loaded, otherwise the backends would not work (I required it in Mobility&rsquo;s Sequel specs).</p> <p>In <a href="https://github.com/shioyama/mobility/pull/60">this PR</a>, I replaced <code>mobility/core_ext</code>, which (like ActiveSupport) monkeypatched core classes like <code>Object</code> and <code>String</code>, with a module <code>Mobility::Util</code> which contains all the methods needed. <code>Util</code> can be either included in a class, or its methods can be called as class methods: e.g. <code>Util.present?(string)</code> will call <code>string.present?</code> if the object responds to <code>present?</code>, otherwise it will do the work itself to get the result.</p> <p>Now Sequel does not need to include the Inflections module to use Mobility, and the core parts of Mobility (e.g. <code>Mobility::Attributes</code>) do not require any special monkeypatching to work. This is important for anyone wanting to use Mobility for an application where they do <em>not</em> want to patch core Ruby classes (a good thing).</p> <h2>Super as Option</h2> <p>In some cases, Mobility overrides a method to redefine it as a translation accessor. This is the case, for example, with backends that store translations on a single column: the PostgreSQL (Jsonb + Hstore) backends, as well as the Serialized backend (which stores translations serialized as a json or yaml hash).</p> <p>With these backends, the original attribute method, say <code>title</code>, returns a hash containing all translations as key/value pairs (where locales are the keys). Mobility overrides this column accessor to return value of the hash at the current locale, using (for AR models) <code>read_attribute</code> to avoid infinite recursion (Sequel is similar).</p> <p>A few people had asked about how they could access the original (hash) column, so in <a href="https://github.com/shioyama/mobility/pull/62">this PR</a> I added a new <code>super</code> option which bypasses the Mobility accessors (reader and writer) and continues up to the original method, in this case the column method.</p> <p>So something like this:</p> <div class="CodeRay"> <div class="code"><pre> post.title <span style="color:#777">#=&gt; &quot;Translating with Mobility&quot;</span> post.title(<span style="color:#606">super</span>: <span style="color:#069">true</span>) <span style="color:#777">#=&gt; { &quot;en&quot; =&gt; &quot;Translating with Mobility&quot;, &quot;ja&quot; =&gt; &quot;Traduction avec Mobilité&quot; }</span> </pre></div> </div> <p>The writer can also be &ldquo;super-ed&rdquo;, although this is somewhat trickier to do:</p> <div class="CodeRay"> <div class="code"><pre> post.send(<span style="color:#A60">:title=</span>, {}, <span style="color:#606">super</span>: <span style="color:#069">true</span>) <span style="color:#777">#=&gt; {}</span> post.title <span style="color:#069">nil</span> </pre></div> </div> <h2>Other</h2> <ul> <li>The default name for associaitons in the KeyValue and Table backends has been simplified. KeyValue associations named <code>mobility_string_translations</code> and <code>mobility_text_translations</code> are now simply <code>string_translations</code> and <code>text_translations</code>, respectively. For the Table backend, <code>mobility_model_translations</code> has been simplified to simply <code>translations</code> (like Globalize).</li> <li>In <a href="https://github.com/shioyama/mobility/pull/58">PR #58</a>, I refactored a somewhat ugly part of the code dealing with caching. I&rsquo;m still not completely happy with how this is done, but it&rsquo;s at least an improvement over the earlier technique.</li> <li>I now have some minimal object allocation specs, added in <a href="https://github.com/shioyama/mobility/pull/57">PR #57</a>. I&rsquo;d like to extend this to benchmark testing as well, but not quite sure how to proceed yet on that one (any suggestions welcome!)</li> <li>The gem is now <em>signed</em>. You of course do not need to check this signature, but if you want to, there are instructions in the README on how to do this.</li> <li><code>extend Mobility</code> is now the preferred way to use Mobility in models (see <a href="https://github.com/shioyama/mobility/issues/69">this issue</a>).</li> <li>I&rsquo;ve bumped Ruby support to minimum 2.2.7. This is pretty standard.</li> </ul> <h2>Finally, a few words</h2> <p>Mobility is a <strong>labour of love</strong>, and I think it&rsquo;s worthwhile to mention that here. I&rsquo;ve spent countless hours writing and re-writing Mobility&rsquo;s code, and many more thinking and rethinking how I can refactor and improve that code.</p> <p>I have both selfish and unselfish motivations for doing this: on the one hand, I want to show off the best that I can do with a clean slate and a clear problem (translation) that I happen to know reasonably well. On the other, I genuinely want to help people who are trying to share content across language barriers, a problem which has been an interest of mine for a <a href="http://translationjournal.net/journal/45global.htm">long while</a>.</p> <p>So it&rsquo;s been enormously gratifying for me to see the positive response Mobility has received so far in the form of encouraging comments and reactions, a growing list of <a href="https://github.com/shioyama/mobility/stargazers">stargazers</a>, and as actual <a href="https://github.com/shioyama/mobility/graphs/contributors">contributions</a> to the gem code. I&rsquo;ve had very thoughtful input on <a href="https://github.com/shioyama/mobility/issues/69">edge cases</a> and <a href="https://github.com/shioyama/mobility/pull/71#issuecomment-321836122">subtle points</a> that I would never have considered, as well as some great feedback at recent talks in <a href="https://www.montrealrb.com/events/19-june-20th-meetup">Montreal</a> and <a href="https://trbmeetup.doorkeeper.jp/events/63110">Tokyo</a> (see slides from the latter of those talks below).</p> <div style="width: 100%; max-width: 500px; margin: auto; margin-top: 20px; margin-bottom: 20px"> <script async class="speakerdeck-embed" data-id="38720207c49c4d8da954e6a26499ed0a" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script> <p class="caption">Slides from my recent talk at the Tokyo Rubyist Meetup</p> </div> <p>There are many <a href="https://groups.google.com/forum/#!topic/capistrano/nmMaqWR1z84">horror stories</a> about open source software maintainers getting exhausted from the demands that come with success. Mobility hasn&rsquo;t reached anywhere near that level of success yet, but so far at least I don&rsquo;t find myself exhausted at all. On the contrary, it&rsquo;s been an extremely rewarding learning experience!</p> <p>If you&rsquo;re using Mobility for a project or application, please let me know. I&rsquo;m eager to here about the kind of challenges people are tackling and how Mobility might be better able to help solve them.</p> <h4>Notes</h4> <p><small><a name="note-1"><sup>1</sup></a> The order of plugins in <code>Mobility.plugins</code> is important: many of the plugin classes include modules into the backend overriding <code>read</code> and <code>write</code>, inclusion order determining the order of method composition. In the list, for example, <code>cache</code> comes before <code>default</code> (and included in that order) because we want to cache actual backend values, not default values.</small></p> The Ruby Module Builder Pattern https://dejimata.com/2017/5/20/the-ruby-module-builder-pattern 2017-05-20 2017-05-20 Chris Salzberg <p>There&rsquo;s an interesting pattern I&rsquo;ve discovered recently in Ruby that is very powerful, yet apparently not widely known or appreciated.<a href="#note-1"><sup>1</sup></a> I call this pattern the <em>Module Builder Pattern</em>. I&rsquo;ve used it heavily in designing <a href="https://github.com/shioyama/mobility">Mobility</a>, a pluggable translation framework I released a couple months ago, and it served me so well I thought I should share what I&rsquo;ve learned.</p> <div class="img-right"> <img alt="Drawing Hands" src="/img/DrawingHands.jpg" /> </div> <p>At its core, the Module Builder is an extremely simple, elegant, and versatile way of dynamically defining named, customizable modules to mix into classes. This is done by <a href="http://solnic.codes/2012/08/13/subclassing-module-for-fun-and-profit">subclassing the <code>Module</code> class</a>, initializing these subclasses at runtime, and including the resulting modules into other classes.</p> <p>If the idea of <em>subclassing</em> <code>Module</code> makes your head spin, read on. Despite its seemingly esoteric nature, the Module Builder does some very useful things. It is used extensively in projects such as <a href="http://dry-rb.org/">dry-rb</a> to great effect, but buried out of view among advanced coding styles that conceal its essential simplicity. It is also used sporadically in Rails<a href="#note-2"><sup>2</sup></a>, but only to a very limited extent. I think it&rsquo;s fair to say that most Rubyists have never seen the pattern, let alone ever used it in their daily work.</p> <p>To explain Module Builder, I will work through a series of progressively more complex examples, ending with some code extracted from my work on <a href="https://github.com/shioyama/mobility">Mobility</a> into a gem called <a href="https://github.com/shioyama/method_found">MethodFound</a>. The examples begin with some simple core concepts I think most readers will understand, then advance to a slightly more in-depth discussion exposing the pattern&rsquo;s tangible benefits in real-life applications.</p> <h2>Modules in Ruby</h2> <p>Let&rsquo;s first recall that a module in Ruby is basically a collection of methods and constants which you can include in any class, reducing duplication and encapsulating functionality in reusable, composable<a href="#note-3"><sup>3</sup></a> units. Callbacks like <a href="http://ruby-doc.org/core-2.4.1/Module.html#method-i-included"><code>included</code></a> and <a href="http://ruby-doc.org/core-2.4.1/Module.html#method-i-extended"><code>extended</code></a> allow you to execute some custom code every time your module is included (or extended, etc) in a class or module.</p> <p>So suppose we have a module <code>MyModule</code> with a method <code>foo</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">MyModule</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">foo</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we can add this method to any class by including <code>MyModule</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">MyClass</span> include <span style="color:#036;font-weight:bold">MyModule</span> <span style="color:#080;font-weight:bold">end</span> a = <span style="color:#036;font-weight:bold">MyClass</span>.new a.foo <span style="color:#777">#=&gt; &quot;foo&quot;</span> </pre></div> </div> <p>That&rsquo;s pretty straightforward, but also very limited: <code>foo</code> is pre-defined with a hard-coded return value.</p> <p>Let&rsquo;s take a slightly more &ldquo;real-world&rdquo; case. Say we&rsquo;ve got a class, <code>Point</code>, with two attributes <code>x</code> and <code>y</code>, which for simplicity&rsquo;s sake we&rsquo;ll define using a <a href="http://ruby-doc.org/core-2.4.1/Struct.html">Struct</a>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Point</span> = <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:x</span>, <span style="color:#A60">:y</span>) </pre></div> </div> <p>Now let&rsquo;s create a module which can add two points together at their respective <code>x</code> and <code>y</code> values. Call it <code>Addable</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Addable</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(point) <span style="color:#036;font-weight:bold">Point</span>.new(point.x + x, point.y + y) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#036;font-weight:bold">Point</span>.include <span style="color:#036;font-weight:bold">Addable</span> p1 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">1</span>, <span style="color:#00D">2</span>) p2 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">2</span>, <span style="color:#00D">3</span>) p1 + p2 <span style="color:#777">#=&gt; #&lt;Point:.. @x=3, @y=5&gt;</span> </pre></div> </div> <p>The module is slightly more reusable now. But it still has rigid constraints: it will only work with a class called <code>Point</code> which has attributes <code>x</code> and <code>y</code>.</p> <p>We can improve this a bit by removing the constant <code>Point</code> in the module, and replacing it with <code>self.class</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Addable</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) <span style="color:#069">self</span>.class.new(x + other.x, y + other.y) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now our <code>Addable</code> module can be used with <em>any class</em> that has accessors <code>x</code> and <code>y</code>,<a href="#note-4"><sup>4</sup></a> which makes it much more flexible. Also, thanks to Ruby&rsquo;s very dynamic treatment of types, the module can be used to &ldquo;add&rdquo; a potentially wide variety of pairwise data types, as long as their elements have a <code>+</code> method.</p> <p>In addition, note that modules in Ruby have the property that their methods can be composed through inclusion or inheritance by using the <code>super</code> keyword in the including or parent class. So if we wanted to log calls to our adder method, we could do this by defining <code>+</code> inside of the <code>Point</code> class, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Point</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:x</span>, <span style="color:#A60">:y</span>) include <span style="color:#036;font-weight:bold">Addable</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">super</span>.tap { puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">&quot;</span></span> } <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>And we can see that our logging indeed works as expected, calling the method on the class first, and then through <code>super</code> the method defined on the module:</p> <div class="CodeRay"> <div class="code"><pre> p1 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">1</span>, <span style="color:#00D">2</span>) p2 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">2</span>, <span style="color:#00D">3</span>) p1 + p2 <span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#777">#=&gt; #&lt;Point:.. @x=3, @y=5&gt;</span> </pre></div> </div> <p>There are many other things to say about modules; we&rsquo;ve really only touched the tip of the iceberg here. However, the main topic of this post is module <em>builders</em>, not modules, so let&rsquo;s take a look at that.</p> <div class="img-center" style="margin: 60px 0 70px 0"> <img alt="Automated space exploration and industrialization using self-replicating systems" src="/img/selfreplicatingsystem.png" /> <p class="caption">Building a Module that Builds Modules</p> </div> <h2>Modules that Build</h2> <p>Modules are powerful because (when used effectively) they encapsulate certain shared patterns of functionality. In the example above, this functionality is the addition of variable pairs (points, or any other class with pairs of variables <code>x</code> and <code>y</code>). Identifying shared patterns like this and expressing them in the form of a module or set of modules makes it possible to tackle very complex problems with simple, modular solutions.</p> <p>That said, the module above is still quite specific, constrained to classes with variables <code>x</code> and <code>y</code>. What if we wanted to loosen this constraint and have the module work for any set of variables? How would we do that?</p> <p>Well, a module can&rsquo;t be configured once defined, so we&rsquo;ll need some other way to loosen this constraint. One way is to define a method on the module that <em>itself</em> adds the desired functionality to the class. Let&rsquo;s call this method <code>define_adder</code><a href="#note-5"><sup>5</sup></a>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">AdderDefiner</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_adder</span>(*keys) define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other| <span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) })) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Instead of defining a <code>+</code> method like we did in the <code>Addable</code> module, we instead define <em>a method which defines the method</em>. We&rsquo;ll call this &ldquo;method-defining method&rdquo; the <em>bootstrap method</em> here, since it &ldquo;bootstraps&rdquo; the module&rsquo;s instance methods dynamically into the class.</p> <p>This bootstrap method takes an arbitrary number of arguments, mapped to an array <code>keys</code> using the <a href="https://renugasaraswathy.wordpress.com/2014/09/03/splat-operator-in-ruby-2/">splat operator</a>. These &ldquo;keys&rdquo; are the <em>names</em> of the variables to add (or, more precisely, the names of the methods which return the variable values). It then defines a method, called <code>+</code>, which adds the values of these variables at each key and creates an instance from the results using <code>self.class.new</code>.</p> <p>Let&rsquo;s see if it works. We&rsquo;ll create a new class, <code>LineItem</code>, with different reader names, and <code>extend</code> the module in this case rather than include it (since <code>define_adder</code> needs to be a class method):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) extend <span style="color:#036;font-weight:bold">AdderDefiner</span> define_adder(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we can add line item pairs, just like we did point pairs earlier:</p> <div class="CodeRay"> <div class="code"><pre> l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>) l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>) l1 + l2 <span style="color:#777">#=&gt; #&lt;LineItem:... @amount=25.98, @tax=3.9&gt;</span> </pre></div> </div> <p>So it works! And we now have no dependency on variable names (or even on number of variables), so our module is more flexible.</p> <p>But there&rsquo;s a problem. When we copy over our earlier logging code from the <code>Point</code> class to log calls to <code>+</code>, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) extend <span style="color:#036;font-weight:bold">AdderDefiner</span> define_adder(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">super</span>.tap { puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">&quot;</span></span> } <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>&hellip; and then add two line items again, instead of getting the expected logs, things blow up with an exception:</p> <div class="CodeRay"> <div class="code"><pre> NoMethodError: super: no superclass method `+' for #&lt;struct LineItem amount=9.99, tax=1.5&gt; </pre></div> </div> <p>What&rsquo;s going on here?</p> <p>Take a good look at how <code>define_adder</code> bootstraps the <code>+</code> method: it uses <code>define_method</code> to add the method <em>directly to the class</em>, in this case <code>LineItem</code>. A class can only have one definition of any given method, so when, later in the class, we again define <code>+</code>, we clobber the earlier definition. There is no &ldquo;superclass&rdquo; here since we have not included or inherited anything when defining the method.</p> <p>Solving this problem will require some &ldquo;module building&rdquo; magic. Let&rsquo;s look at the solution first, and then see how it solves this issue:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">AdderIncluder</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_adder</span>(*keys) adder = <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span> define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other| <span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) })) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> include adder <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we replace <code>AdderDefiner</code> in the logged <code>LineItem</code> code above with <code>AdderIncluder</code> and we find that it works:</p> <div class="CodeRay"> <div class="code"><pre> l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>) l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>) l1 + l2 <span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#777">#=&gt; #&lt;LineItem:... @amount=25.98, @tax=3.9&gt;</span> </pre></div> </div> <p>The trick that makes this work is the use of an <em>anonymous module</em> in <code>define_adder</code> to define the <code>+</code> method, which is different from the previous technique of defining the method directly on the calling class.</p> <p>Recall that in Ruby, any module (small &ldquo;m&rdquo;) is simply an instance of a class, and this class is <a href="http://ruby-doc.org/core-2.4.1/Module.html">Module</a> (big &ldquo;M&rdquo;). Like other classes, <code>Module</code> has a <code>new</code> method, in this case one which <a href="http://ruby-doc.org/core-2.4.1/Module.html#method-c-new">can take a block argument</a> and evaluate the block in the context of the newly-created module.</p> <p>So to generate a &ldquo;module on the fly&rdquo;, you can simply call <code>Module.new</code>, pass a block with some method definitions, and use it just as you would any other (named) module:</p> <div class="CodeRay"> <div class="code"><pre> mod = <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">with_foo</span> <span style="color:#069">self</span> + <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20"> with Foo</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">The Ruby Module Builder Pattern</span><span style="color:#710">&quot;</span></span> title.extend mod title.with_foo <span style="color:#777">#=&gt; &quot;The Ruby Module Builder Pattern with Foo&quot;</span> </pre></div> </div> <p>This is a handy trick, often used in testing when you don&rsquo;t want to clutter the global namespace with a one-off module used only in a single test (works for <code>Class</code> too).<a href="#note-6"><sup>6</sup></a></p> <p>The method defined in the block in <code>AdderIncluder</code> is the same <code>+</code> method defined earlier in <code>AdderDefiner</code>, and in this sense, the two modules perform a very similar function. The key difference is that whereas <code>AdderDefiner</code> defines the method <em>on</em> the class, <code>AdderIncluder</code> <em>includes a module</em> with the method <em>into</em> the class.</p> <p>If you look up the chain of ancestors, you will see this module just after the <code>LineItem</code> class itself:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">LineItem</span>.ancestors <span style="color:#777">#=&gt; [LineItem,</span> <span style="color:#777">#&lt;Module:0x...&gt;,</span> ... </pre></div> </div> <p>Given this ancestor chain, it is clear why <code>super</code> works here: when you call <code>+</code> on <code>LineItem</code>, the method calls <code>super</code>, which continues up the class hierarchy to the anonymous module where our <code>+</code> method is defined. It then calculates and returns the result, which gets composed in the <code>LineItem</code> adder with the logging code.</p> <p>Although at first glance it might seem somewhat strange and unfamiliar, this technique of defining a bootstrap method to include a dynamically-generated anonymous module is in fact extremely useful, and widely used for this reason. This is particularly true for a flexible framework like Rails, where simply <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activerecord/lib/active_record/attribute_methods.rb#L13">defining a model</a> or <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/actionpack/lib/action_dispatch/routing/mapper.rb#L652">setting up routes</a> triggers calls to many bootstrap methods like <code>define_adder</code>. The technique can also be found in other gems that require a similar level of flexiblity (including <a href="https://github.com/shioyama/mobility">Mobility</a>).</p> <p>Bootstrapped module building can thus be considered a very important metaprogramming technique in Ruby. Indeed, it underpins the very dynamic nature that enables frameworks like Rails to work the way they do.</p> <p>That said, I hope to convince you here that the potential of module building goes well beyond the bootstrapping technique shown above, and as such that it is a pattern that deserves more attention. To do this, I&rsquo;m going to return to the Adder module one last time, and define it in a slightly different way.</p> <h2>The Module Builder</h2> <p>So far we&rsquo;ve looked at three &ldquo;adder&rdquo; modules:</p> <ol> <li><code>Addable</code>, which adds a <code>+</code> method on two variables <em>x</em> and <em>y</em>,</li> <li><code>AdderDefiner</code>, which adds a class method <code>define_adder</code> that defines a <code>+</code> method on an arbitrary set of variables, and</li> <li><code>AdderIncluder</code>, which also adds a class method <code>define_adder</code> that defines the <code>+</code> method, but in such a way that the method can be composed using <code>super</code>.</li> </ol> <p>Although not very flexible, note that in hindsight, the first <code>Addable</code> module had some good things going for it:</p> <ul> <li>It only included the <code>+</code> method, and left no other bootstrapping &ldquo;artifacts&rdquo; like <code>define_adder</code> (unlike both <code>AdderDefiner</code> and <code>AdderIncluder</code>)</li> <li>It allowed the parent class to access its methods using <code>super</code> (like <code>AdderIncluder</code> but not <code>AdderDefiner</code>)</li> <li>It had a <em>name</em>, and was therefore very easy to recognize in the chain of ancestors. While <code>AdderDefiner</code> and <code>AdderIncluder</code> <em>themselves</em> appear in the chain of ancestors, the fact that the bootstrap method was called, and <em>how</em> it was called, is hard or impossible to discern.</li> </ul> <p>In this section, I will show how we can solve the adder problem in a way that retains these benefits, while also offering the flexibility we sought in the last section.</p> <div class="img-center" style="margin: 70px 0 40px 0"> <img style="padding-bottom: 20px" alt="A Self-Replicating Machine" src="/img/selfreplicatingmachine-medium.png" /> <p class="caption">Module Builders</p> </div> <p>Let&rsquo;s start by defining an <code>AdderBuilder</code> class which inherits from <code>Module</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">AdderBuilder</span> &lt; <span style="color:#036;font-weight:bold">Module</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(*keys) define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other| <span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) })) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>This is a short but very dense little class. Before going into the details of what it&rsquo;s doing here, let&rsquo;s first check that it works. We&rsquo;ll include (not extend) an <em>instance</em> of this class (a module) into <code>LineItem</code> this time:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) include <span style="color:#036;font-weight:bold">AdderBuilder</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) <span style="color:#080;font-weight:bold">end</span> l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>) l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>) l1 + l2 <span style="color:#777">#=&gt; #&lt;LineItem:... @amount=25.98, @tax=3.9&gt;</span> </pre></div> </div> <p>So, as we can see, this little class does the same thing that we achieved with the <code>define_adder</code> bootstrap method in <code>AdderDefiner</code>, except <em>without ever defining a class method to do it</em>.</p> <p>How does it do this? First, it subclasses <code>Module</code>. Since we can call <code>new</code> on <code>Module</code>, it&rsquo;s not much of a stretch to imagine that we can also subclass it and define our own methods such as <code>initialize</code>. And indeed, as shown above, we <em>can</em> do this.</p> <p>Here&rsquo;s that initializer again:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(*keys) define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other| <span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) })) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>There are two levels at play here, and it&rsquo;s important to recognize both of them. On one level, we are initializing a new instance of a class, <code>Module</code>. This instance is itself a module, so it&rsquo;s a collection of methods we will mix into other classes. We are then, as we initialize this module, defining one of these methods, a method called <code>+</code>.</p> <p>The method <code>+</code> itself is being defined <em>in terms of</em> the method names we have passed as arguments to <code>Module.new</code> (the &ldquo;keys&rdquo;, which were <code>:amount</code> and <code>:tax</code> in the example above).<a href="#note-7"><sup>7</sup></a> When we call <code>self.class.new</code> <em>in</em> the method, <em>this</em> <code>new</code> will be evaluated in the context of the class including this new module, and thus would evaluate to <code>LineItem</code> (or <code>Point</code>, or whatever class the module was included into).</p> <p>So <code>AdderBuilder.new(:amount, :tax)</code> evaluates to a module functionally equivalent to this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) <span style="color:#069">self</span>.class.new(amount + other.amount, tax + other.tax) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>&hellip; which is what we had in <code>Addable</code>, with <code>x</code> and <code>y</code> swapped for <code>amount</code> and <code>tax</code>. But the power here comes from the fact that we can define our dynamic module in terms of <em>whatever variable names we want.</em></p> <p>And herein lies the essential ingenuity of this pattern: that <strong>it allows you to define modules not as fixed collections of methods and constants, but as configurable prototypes for building such collections</strong>. There is no need for class methods or other metaprogramming tricks to bootstrap the module-inclusion step; the builder builds modules directly, so they can be included in one step.</p> <p>Not only that, but the modules created in this process, unlike the anonymous modules generated by techniques described in the last section, actually have a name, so you can clearly see them in the chain of ancestors:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">LineItem</span>.ancestors [<span style="color:#036;font-weight:bold">LineItem</span>, <span style="color:#777">#&lt;AdderBuilder:0x...&gt;, Object, ... ]</span> </pre></div> </div> <p>This is a nice by-product of the encapsulation we have achieved by subclassing <code>Module</code>.<a href="#note-8"><sup>8</sup></a> With an anonymous module, this is much harder to debug:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">LineItem</span>.ancestors [<span style="color:#036;font-weight:bold">LineItem</span>, <span style="color:#777">#&lt;Module:0x...&gt;, Object, ... ]</span> </pre></div> </div> <p>Module Builders thus have the benefits of anonymous modules (their instances can be defined &ldquo;on the fly&rdquo; and do not clutter the global constant namespace), are easily traceable since their instances have a name (like &ldquo;normal&rdquo; modules), and yet do not require defining (and calling) new methods on the including class to bootstrap them.</p> <p>We can go a step further. We previously added some logging code to the including class. This logging code can also be rewritten as a module builder, which we will call <code>LoggerBuilder</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LoggerBuilder</span> &lt; <span style="color:#036;font-weight:bold">Module</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(method_name, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">end_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#710">&quot;</span></span>) define_method method_name <span style="color:#080;font-weight:bold">do</span> |*args, &amp;block| print start_message <span style="color:#080;font-weight:bold">super</span>(*args, &amp;block).tap { print end_message } <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we can log <em>any included or inherited method</em> by creating an instance of <code>LoggerBuilder</code> with the name of the method to be logged:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) include <span style="color:#036;font-weight:bold">AdderBuilder</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) include <span style="color:#036;font-weight:bold">LoggerBuilder</span>.new(<span style="color:#A60">:+</span>, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Enter Adder...</span><span style="color:#b0b">\n</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">end_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Exit Adder...</span><span style="color:#b0b">\n</span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>And, presto, we&rsquo;ve got a logged addable line item:</p> <div class="CodeRay"> <div class="code"><pre> l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>) l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>) l1 + l2 <span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#777">#=&gt; #&lt;LineItem:... @amount=25.98, @tax=3.9&gt;</span> </pre></div> </div> <p>But of course, we can now also log any other included or inherited method we like:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">LineItem</span>.include(<span style="color:#036;font-weight:bold">LoggerBuilder</span>.new(<span style="color:#A60">:to_s</span>, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Stringifying...</span><span style="color:#b0b">\n</span><span style="color:#710">&quot;</span></span>)) l1.to_s <span style="color:#036;font-weight:bold">Stringifying</span>... =&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">#&lt;struct LineItem amount=9.99, tax=1.5&gt;</span><span style="color:#710">&quot;</span></span> </pre></div> </div> <p>There are many other directions you can take this; I&rsquo;ve only scratched the surface here. For a relatively simple (yet powerful) example of this &ldquo;in the wild&rdquo;, take a look at <a href="https://github.com/dry-rb/dry-equalizer">Dry Equalizer</a>, which dynamically defines equality methods on a class using an instance of an <code>Equalizer</code> class that, like <code>AdderBuilder</code>, subclasses <code>Module</code>.</p> <h2>Module Builders and Composition</h2> <p>You may have noticed that I have used the word &ldquo;compose&rdquo; quite a lot in this post with respect to including modules into a class. Any time you include a module which overrides a method and calls <code>super</code>, you are in effect using function (method) composition, although it is not generally referred to as such.</p> <p>The same is true of our logging code in the last section. When we include an instance of both <code>AdderBuilder</code> <em>and</em> <code>LoggerBuilder</code>, we are building <code>+</code> as a composite function, which we could alternatively write as:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">add</span>(other) <span style="color:#069">self</span>.class.new(amount + other.amount, tax + other.tax) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">log</span>(result) print <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">&quot;</span></span> result.tap { print <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">&quot;</span></span> } <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) log(add(other)) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>So <code>super</code> is essentially taking the output of the last method call and incorporating it into the result of the current module&rsquo;s method.</p> <p>Composition happens to be a very interesting way to &ldquo;combine&rdquo; different instances of a module builder, like we combined logging and adding modules in the example above. With module builders, we can configure and combine many modules to build more complex methods and class behaviours.</p> <p>One way to do this is via a &ldquo;branching&rdquo; method flow. We define multiple modules that each override the same method, but only trigger some special processing if a condition on the method and its arguments matches the module&rsquo;s state; otherwise, the control flow continues up the class hierarchy to the next module via <code>super</code>.</p> <p>If you&rsquo;ve worked with Ruby for any length of time, this &ldquo;branching composition&rdquo; should make you think of one method: <a href="http://ruby-doc.org/core-2.4.1/BasicObject.html#method-i-method_missing">method_missing</a>. Rarely does anyone override <code>method_missing</code> without a call to <code>super</code> somewhere, since doing so would catch <em>any</em> method not defined on the class, which is generally not what you want.</p> <p>Instead, a typical <code>method_missing</code> override looks something like this:</p> <p><pre type="ruby"> def method_missing(method_name, *arguments, &amp;block) if method_name =</p> <p>There&rsquo;s an interesting pattern I&rsquo;ve discovered recently in Ruby that is very powerful, yet apparently not widely known or appreciated.<a href="#note-1"><sup>1</sup></a> I call this pattern the <em>Module Builder Pattern</em>. I&rsquo;ve used it heavily in designing <a href="https://github.com/shioyama/mobility">Mobility</a>, a pluggable translation framework I released a couple months ago, and it served me so well I thought I should share what I&rsquo;ve learned.</p> <div class="img-right"> <img alt="Drawing Hands" src="/img/DrawingHands.jpg" /> </div> <p>At its core, the Module Builder is an extremely simple, elegant, and versatile way of dynamically defining named, customizable modules to mix into classes. This is done by <a href="http://solnic.codes/2012/08/13/subclassing-module-for-fun-and-profit">subclassing the <code>Module</code> class</a>, initializing these subclasses at runtime, and including the resulting modules into other classes.</p> <p>If the idea of <em>subclassing</em> <code>Module</code> makes your head spin, read on. Despite its seemingly esoteric nature, the Module Builder does some very useful things. It is used extensively in projects such as <a href="http://dry-rb.org/">dry-rb</a> to great effect, but buried out of view among advanced coding styles that conceal its essential simplicity. It is also used sporadically in Rails<a href="#note-2"><sup>2</sup></a>, but only to a very limited extent. I think it&rsquo;s fair to say that most Rubyists have never seen the pattern, let alone ever used it in their daily work.</p> <p>To explain Module Builder, I will work through a series of progressively more complex examples, ending with some code extracted from my work on <a href="https://github.com/shioyama/mobility">Mobility</a> into a gem called <a href="https://github.com/shioyama/method_found">MethodFound</a>. The examples begin with some simple core concepts I think most readers will understand, then advance to a slightly more in-depth discussion exposing the pattern&rsquo;s tangible benefits in real-life applications.</p> <h2>Modules in Ruby</h2> <p>Let&rsquo;s first recall that a module in Ruby is basically a collection of methods and constants which you can include in any class, reducing duplication and encapsulating functionality in reusable, composable<a href="#note-3"><sup>3</sup></a> units. Callbacks like <a href="http://ruby-doc.org/core-2.4.1/Module.html#method-i-included"><code>included</code></a> and <a href="http://ruby-doc.org/core-2.4.1/Module.html#method-i-extended"><code>extended</code></a> allow you to execute some custom code every time your module is included (or extended, etc) in a class or module.</p> <p>So suppose we have a module <code>MyModule</code> with a method <code>foo</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">MyModule</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">foo</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we can add this method to any class by including <code>MyModule</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">MyClass</span> include <span style="color:#036;font-weight:bold">MyModule</span> <span style="color:#080;font-weight:bold">end</span> a = <span style="color:#036;font-weight:bold">MyClass</span>.new a.foo <span style="color:#777">#=&gt; &quot;foo&quot;</span> </pre></div> </div> <p>That&rsquo;s pretty straightforward, but also very limited: <code>foo</code> is pre-defined with a hard-coded return value.</p> <p>Let&rsquo;s take a slightly more &ldquo;real-world&rdquo; case. Say we&rsquo;ve got a class, <code>Point</code>, with two attributes <code>x</code> and <code>y</code>, which for simplicity&rsquo;s sake we&rsquo;ll define using a <a href="http://ruby-doc.org/core-2.4.1/Struct.html">Struct</a>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Point</span> = <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:x</span>, <span style="color:#A60">:y</span>) </pre></div> </div> <p>Now let&rsquo;s create a module which can add two points together at their respective <code>x</code> and <code>y</code> values. Call it <code>Addable</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Addable</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(point) <span style="color:#036;font-weight:bold">Point</span>.new(point.x + x, point.y + y) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#036;font-weight:bold">Point</span>.include <span style="color:#036;font-weight:bold">Addable</span> p1 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">1</span>, <span style="color:#00D">2</span>) p2 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">2</span>, <span style="color:#00D">3</span>) p1 + p2 <span style="color:#777">#=&gt; #&lt;Point:.. @x=3, @y=5&gt;</span> </pre></div> </div> <p>The module is slightly more reusable now. But it still has rigid constraints: it will only work with a class called <code>Point</code> which has attributes <code>x</code> and <code>y</code>.</p> <p>We can improve this a bit by removing the constant <code>Point</code> in the module, and replacing it with <code>self.class</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">Addable</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) <span style="color:#069">self</span>.class.new(x + other.x, y + other.y) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now our <code>Addable</code> module can be used with <em>any class</em> that has accessors <code>x</code> and <code>y</code>,<a href="#note-4"><sup>4</sup></a> which makes it much more flexible. Also, thanks to Ruby&rsquo;s very dynamic treatment of types, the module can be used to &ldquo;add&rdquo; a potentially wide variety of pairwise data types, as long as their elements have a <code>+</code> method.</p> <p>In addition, note that modules in Ruby have the property that their methods can be composed through inclusion or inheritance by using the <code>super</code> keyword in the including or parent class. So if we wanted to log calls to our adder method, we could do this by defining <code>+</code> inside of the <code>Point</code> class, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Point</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:x</span>, <span style="color:#A60">:y</span>) include <span style="color:#036;font-weight:bold">Addable</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">super</span>.tap { puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">&quot;</span></span> } <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>And we can see that our logging indeed works as expected, calling the method on the class first, and then through <code>super</code> the method defined on the module:</p> <div class="CodeRay"> <div class="code"><pre> p1 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">1</span>, <span style="color:#00D">2</span>) p2 = <span style="color:#036;font-weight:bold">Point</span>.new(<span style="color:#00D">2</span>, <span style="color:#00D">3</span>) p1 + p2 <span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#777">#=&gt; #&lt;Point:.. @x=3, @y=5&gt;</span> </pre></div> </div> <p>There are many other things to say about modules; we&rsquo;ve really only touched the tip of the iceberg here. However, the main topic of this post is module <em>builders</em>, not modules, so let&rsquo;s take a look at that.</p> <div class="img-center" style="margin: 60px 0 70px 0"> <img alt="Automated space exploration and industrialization using self-replicating systems" src="/img/selfreplicatingsystem.png" /> <p class="caption">Building a Module that Builds Modules</p> </div> <h2>Modules that Build</h2> <p>Modules are powerful because (when used effectively) they encapsulate certain shared patterns of functionality. In the example above, this functionality is the addition of variable pairs (points, or any other class with pairs of variables <code>x</code> and <code>y</code>). Identifying shared patterns like this and expressing them in the form of a module or set of modules makes it possible to tackle very complex problems with simple, modular solutions.</p> <p>That said, the module above is still quite specific, constrained to classes with variables <code>x</code> and <code>y</code>. What if we wanted to loosen this constraint and have the module work for any set of variables? How would we do that?</p> <p>Well, a module can&rsquo;t be configured once defined, so we&rsquo;ll need some other way to loosen this constraint. One way is to define a method on the module that <em>itself</em> adds the desired functionality to the class. Let&rsquo;s call this method <code>define_adder</code><a href="#note-5"><sup>5</sup></a>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">AdderDefiner</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_adder</span>(*keys) define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other| <span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) })) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Instead of defining a <code>+</code> method like we did in the <code>Addable</code> module, we instead define <em>a method which defines the method</em>. We&rsquo;ll call this &ldquo;method-defining method&rdquo; the <em>bootstrap method</em> here, since it &ldquo;bootstraps&rdquo; the module&rsquo;s instance methods dynamically into the class.</p> <p>This bootstrap method takes an arbitrary number of arguments, mapped to an array <code>keys</code> using the <a href="https://renugasaraswathy.wordpress.com/2014/09/03/splat-operator-in-ruby-2/">splat operator</a>. These &ldquo;keys&rdquo; are the <em>names</em> of the variables to add (or, more precisely, the names of the methods which return the variable values). It then defines a method, called <code>+</code>, which adds the values of these variables at each key and creates an instance from the results using <code>self.class.new</code>.</p> <p>Let&rsquo;s see if it works. We&rsquo;ll create a new class, <code>LineItem</code>, with different reader names, and <code>extend</code> the module in this case rather than include it (since <code>define_adder</code> needs to be a class method):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) extend <span style="color:#036;font-weight:bold">AdderDefiner</span> define_adder(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we can add line item pairs, just like we did point pairs earlier:</p> <div class="CodeRay"> <div class="code"><pre> l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>) l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>) l1 + l2 <span style="color:#777">#=&gt; #&lt;LineItem:... @amount=25.98, @tax=3.9&gt;</span> </pre></div> </div> <p>So it works! And we now have no dependency on variable names (or even on number of variables), so our module is more flexible.</p> <p>But there&rsquo;s a problem. When we copy over our earlier logging code from the <code>Point</code> class to log calls to <code>+</code>, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) extend <span style="color:#036;font-weight:bold">AdderDefiner</span> define_adder(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">super</span>.tap { puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">&quot;</span></span> } <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>&hellip; and then add two line items again, instead of getting the expected logs, things blow up with an exception:</p> <div class="CodeRay"> <div class="code"><pre> NoMethodError: super: no superclass method `+' for #&lt;struct LineItem amount=9.99, tax=1.5&gt; </pre></div> </div> <p>What&rsquo;s going on here?</p> <p>Take a good look at how <code>define_adder</code> bootstraps the <code>+</code> method: it uses <code>define_method</code> to add the method <em>directly to the class</em>, in this case <code>LineItem</code>. A class can only have one definition of any given method, so when, later in the class, we again define <code>+</code>, we clobber the earlier definition. There is no &ldquo;superclass&rdquo; here since we have not included or inherited anything when defining the method.</p> <p>Solving this problem will require some &ldquo;module building&rdquo; magic. Let&rsquo;s look at the solution first, and then see how it solves this issue:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">module</span> <span style="color:#B06;font-weight:bold">AdderIncluder</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_adder</span>(*keys) adder = <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span> define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other| <span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) })) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> include adder <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we replace <code>AdderDefiner</code> in the logged <code>LineItem</code> code above with <code>AdderIncluder</code> and we find that it works:</p> <div class="CodeRay"> <div class="code"><pre> l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>) l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>) l1 + l2 <span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#777">#=&gt; #&lt;LineItem:... @amount=25.98, @tax=3.9&gt;</span> </pre></div> </div> <p>The trick that makes this work is the use of an <em>anonymous module</em> in <code>define_adder</code> to define the <code>+</code> method, which is different from the previous technique of defining the method directly on the calling class.</p> <p>Recall that in Ruby, any module (small &ldquo;m&rdquo;) is simply an instance of a class, and this class is <a href="http://ruby-doc.org/core-2.4.1/Module.html">Module</a> (big &ldquo;M&rdquo;). Like other classes, <code>Module</code> has a <code>new</code> method, in this case one which <a href="http://ruby-doc.org/core-2.4.1/Module.html#method-c-new">can take a block argument</a> and evaluate the block in the context of the newly-created module.</p> <p>So to generate a &ldquo;module on the fly&rdquo;, you can simply call <code>Module.new</code>, pass a block with some method definitions, and use it just as you would any other (named) module:</p> <div class="CodeRay"> <div class="code"><pre> mod = <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">with_foo</span> <span style="color:#069">self</span> + <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20"> with Foo</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">The Ruby Module Builder Pattern</span><span style="color:#710">&quot;</span></span> title.extend mod title.with_foo <span style="color:#777">#=&gt; &quot;The Ruby Module Builder Pattern with Foo&quot;</span> </pre></div> </div> <p>This is a handy trick, often used in testing when you don&rsquo;t want to clutter the global namespace with a one-off module used only in a single test (works for <code>Class</code> too).<a href="#note-6"><sup>6</sup></a></p> <p>The method defined in the block in <code>AdderIncluder</code> is the same <code>+</code> method defined earlier in <code>AdderDefiner</code>, and in this sense, the two modules perform a very similar function. The key difference is that whereas <code>AdderDefiner</code> defines the method <em>on</em> the class, <code>AdderIncluder</code> <em>includes a module</em> with the method <em>into</em> the class.</p> <p>If you look up the chain of ancestors, you will see this module just after the <code>LineItem</code> class itself:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">LineItem</span>.ancestors <span style="color:#777">#=&gt; [LineItem,</span> <span style="color:#777">#&lt;Module:0x...&gt;,</span> ... </pre></div> </div> <p>Given this ancestor chain, it is clear why <code>super</code> works here: when you call <code>+</code> on <code>LineItem</code>, the method calls <code>super</code>, which continues up the class hierarchy to the anonymous module where our <code>+</code> method is defined. It then calculates and returns the result, which gets composed in the <code>LineItem</code> adder with the logging code.</p> <p>Although at first glance it might seem somewhat strange and unfamiliar, this technique of defining a bootstrap method to include a dynamically-generated anonymous module is in fact extremely useful, and widely used for this reason. This is particularly true for a flexible framework like Rails, where simply <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activerecord/lib/active_record/attribute_methods.rb#L13">defining a model</a> or <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/actionpack/lib/action_dispatch/routing/mapper.rb#L652">setting up routes</a> triggers calls to many bootstrap methods like <code>define_adder</code>. The technique can also be found in other gems that require a similar level of flexiblity (including <a href="https://github.com/shioyama/mobility">Mobility</a>).</p> <p>Bootstrapped module building can thus be considered a very important metaprogramming technique in Ruby. Indeed, it underpins the very dynamic nature that enables frameworks like Rails to work the way they do.</p> <p>That said, I hope to convince you here that the potential of module building goes well beyond the bootstrapping technique shown above, and as such that it is a pattern that deserves more attention. To do this, I&rsquo;m going to return to the Adder module one last time, and define it in a slightly different way.</p> <h2>The Module Builder</h2> <p>So far we&rsquo;ve looked at three &ldquo;adder&rdquo; modules:</p> <ol> <li><code>Addable</code>, which adds a <code>+</code> method on two variables <em>x</em> and <em>y</em>,</li> <li><code>AdderDefiner</code>, which adds a class method <code>define_adder</code> that defines a <code>+</code> method on an arbitrary set of variables, and</li> <li><code>AdderIncluder</code>, which also adds a class method <code>define_adder</code> that defines the <code>+</code> method, but in such a way that the method can be composed using <code>super</code>.</li> </ol> <p>Although not very flexible, note that in hindsight, the first <code>Addable</code> module had some good things going for it:</p> <ul> <li>It only included the <code>+</code> method, and left no other bootstrapping &ldquo;artifacts&rdquo; like <code>define_adder</code> (unlike both <code>AdderDefiner</code> and <code>AdderIncluder</code>)</li> <li>It allowed the parent class to access its methods using <code>super</code> (like <code>AdderIncluder</code> but not <code>AdderDefiner</code>)</li> <li>It had a <em>name</em>, and was therefore very easy to recognize in the chain of ancestors. While <code>AdderDefiner</code> and <code>AdderIncluder</code> <em>themselves</em> appear in the chain of ancestors, the fact that the bootstrap method was called, and <em>how</em> it was called, is hard or impossible to discern.</li> </ul> <p>In this section, I will show how we can solve the adder problem in a way that retains these benefits, while also offering the flexibility we sought in the last section.</p> <div class="img-center" style="margin: 70px 0 40px 0"> <img style="padding-bottom: 20px" alt="A Self-Replicating Machine" src="/img/selfreplicatingmachine-medium.png" /> <p class="caption">Module Builders</p> </div> <p>Let&rsquo;s start by defining an <code>AdderBuilder</code> class which inherits from <code>Module</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">AdderBuilder</span> &lt; <span style="color:#036;font-weight:bold">Module</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(*keys) define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other| <span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) })) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>This is a short but very dense little class. Before going into the details of what it&rsquo;s doing here, let&rsquo;s first check that it works. We&rsquo;ll include (not extend) an <em>instance</em> of this class (a module) into <code>LineItem</code> this time:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) include <span style="color:#036;font-weight:bold">AdderBuilder</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) <span style="color:#080;font-weight:bold">end</span> l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>) l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>) l1 + l2 <span style="color:#777">#=&gt; #&lt;LineItem:... @amount=25.98, @tax=3.9&gt;</span> </pre></div> </div> <p>So, as we can see, this little class does the same thing that we achieved with the <code>define_adder</code> bootstrap method in <code>AdderDefiner</code>, except <em>without ever defining a class method to do it</em>.</p> <p>How does it do this? First, it subclasses <code>Module</code>. Since we can call <code>new</code> on <code>Module</code>, it&rsquo;s not much of a stretch to imagine that we can also subclass it and define our own methods such as <code>initialize</code>. And indeed, as shown above, we <em>can</em> do this.</p> <p>Here&rsquo;s that initializer again:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(*keys) define_method <span style="color:#A60">:+</span> <span style="color:#080;font-weight:bold">do</span> |other| <span style="color:#069">self</span>.class.new(*(keys.map { |key| send(key) + other.send(key) })) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>There are two levels at play here, and it&rsquo;s important to recognize both of them. On one level, we are initializing a new instance of a class, <code>Module</code>. This instance is itself a module, so it&rsquo;s a collection of methods we will mix into other classes. We are then, as we initialize this module, defining one of these methods, a method called <code>+</code>.</p> <p>The method <code>+</code> itself is being defined <em>in terms of</em> the method names we have passed as arguments to <code>Module.new</code> (the &ldquo;keys&rdquo;, which were <code>:amount</code> and <code>:tax</code> in the example above).<a href="#note-7"><sup>7</sup></a> When we call <code>self.class.new</code> <em>in</em> the method, <em>this</em> <code>new</code> will be evaluated in the context of the class including this new module, and thus would evaluate to <code>LineItem</code> (or <code>Point</code>, or whatever class the module was included into).</p> <p>So <code>AdderBuilder.new(:amount, :tax)</code> evaluates to a module functionally equivalent to this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Module</span>.new <span style="color:#080;font-weight:bold">do</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) <span style="color:#069">self</span>.class.new(amount + other.amount, tax + other.tax) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>&hellip; which is what we had in <code>Addable</code>, with <code>x</code> and <code>y</code> swapped for <code>amount</code> and <code>tax</code>. But the power here comes from the fact that we can define our dynamic module in terms of <em>whatever variable names we want.</em></p> <p>And herein lies the essential ingenuity of this pattern: that <strong>it allows you to define modules not as fixed collections of methods and constants, but as configurable prototypes for building such collections</strong>. There is no need for class methods or other metaprogramming tricks to bootstrap the module-inclusion step; the builder builds modules directly, so they can be included in one step.</p> <p>Not only that, but the modules created in this process, unlike the anonymous modules generated by techniques described in the last section, actually have a name, so you can clearly see them in the chain of ancestors:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">LineItem</span>.ancestors [<span style="color:#036;font-weight:bold">LineItem</span>, <span style="color:#777">#&lt;AdderBuilder:0x...&gt;, Object, ... ]</span> </pre></div> </div> <p>This is a nice by-product of the encapsulation we have achieved by subclassing <code>Module</code>.<a href="#note-8"><sup>8</sup></a> With an anonymous module, this is much harder to debug:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">LineItem</span>.ancestors [<span style="color:#036;font-weight:bold">LineItem</span>, <span style="color:#777">#&lt;Module:0x...&gt;, Object, ... ]</span> </pre></div> </div> <p>Module Builders thus have the benefits of anonymous modules (their instances can be defined &ldquo;on the fly&rdquo; and do not clutter the global constant namespace), are easily traceable since their instances have a name (like &ldquo;normal&rdquo; modules), and yet do not require defining (and calling) new methods on the including class to bootstrap them.</p> <p>We can go a step further. We previously added some logging code to the including class. This logging code can also be rewritten as a module builder, which we will call <code>LoggerBuilder</code>:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LoggerBuilder</span> &lt; <span style="color:#036;font-weight:bold">Module</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(method_name, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">end_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#710">&quot;</span></span>) define_method method_name <span style="color:#080;font-weight:bold">do</span> |*args, &amp;block| print start_message <span style="color:#080;font-weight:bold">super</span>(*args, &amp;block).tap { print end_message } <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we can log <em>any included or inherited method</em> by creating an instance of <code>LoggerBuilder</code> with the name of the method to be logged:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) include <span style="color:#036;font-weight:bold">AdderBuilder</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) include <span style="color:#036;font-weight:bold">LoggerBuilder</span>.new(<span style="color:#A60">:+</span>, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Enter Adder...</span><span style="color:#b0b">\n</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">end_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Exit Adder...</span><span style="color:#b0b">\n</span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>And, presto, we&rsquo;ve got a logged addable line item:</p> <div class="CodeRay"> <div class="code"><pre> l1 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">9.99</span>, <span style="color:#60E">1.50</span>) l2 = <span style="color:#036;font-weight:bold">LineItem</span>.new(<span style="color:#60E">15.99</span>, <span style="color:#60E">2.40</span>) l1 + l2 <span style="color:#036;font-weight:bold">Enter</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#036;font-weight:bold">Exit</span> <span style="color:#036;font-weight:bold">Adder</span>... <span style="color:#777">#=&gt; #&lt;LineItem:... @amount=25.98, @tax=3.9&gt;</span> </pre></div> </div> <p>But of course, we can now also log any other included or inherited method we like:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">LineItem</span>.include(<span style="color:#036;font-weight:bold">LoggerBuilder</span>.new(<span style="color:#A60">:to_s</span>, <span style="color:#606">start_message</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Stringifying...</span><span style="color:#b0b">\n</span><span style="color:#710">&quot;</span></span>)) l1.to_s <span style="color:#036;font-weight:bold">Stringifying</span>... =&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">#&lt;struct LineItem amount=9.99, tax=1.5&gt;</span><span style="color:#710">&quot;</span></span> </pre></div> </div> <p>There are many other directions you can take this; I&rsquo;ve only scratched the surface here. For a relatively simple (yet powerful) example of this &ldquo;in the wild&rdquo;, take a look at <a href="https://github.com/dry-rb/dry-equalizer">Dry Equalizer</a>, which dynamically defines equality methods on a class using an instance of an <code>Equalizer</code> class that, like <code>AdderBuilder</code>, subclasses <code>Module</code>.</p> <h2>Module Builders and Composition</h2> <p>You may have noticed that I have used the word &ldquo;compose&rdquo; quite a lot in this post with respect to including modules into a class. Any time you include a module which overrides a method and calls <code>super</code>, you are in effect using function (method) composition, although it is not generally referred to as such.</p> <p>The same is true of our logging code in the last section. When we include an instance of both <code>AdderBuilder</code> <em>and</em> <code>LoggerBuilder</code>, we are building <code>+</code> as a composite function, which we could alternatively write as:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">LineItem</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:amount</span>, <span style="color:#A60">:tax</span>) <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">add</span>(other) <span style="color:#069">self</span>.class.new(amount + other.amount, tax + other.tax) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">log</span>(result) print <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Enter Adder...</span><span style="color:#710">&quot;</span></span> result.tap { print <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Exit Adder...</span><span style="color:#710">&quot;</span></span> } <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">+</span>(other) log(add(other)) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>So <code>super</code> is essentially taking the output of the last method call and incorporating it into the result of the current module&rsquo;s method.</p> <p>Composition happens to be a very interesting way to &ldquo;combine&rdquo; different instances of a module builder, like we combined logging and adding modules in the example above. With module builders, we can configure and combine many modules to build more complex methods and class behaviours.</p> <p>One way to do this is via a &ldquo;branching&rdquo; method flow. We define multiple modules that each override the same method, but only trigger some special processing if a condition on the method and its arguments matches the module&rsquo;s state; otherwise, the control flow continues up the class hierarchy to the next module via <code>super</code>.</p> <p>If you&rsquo;ve worked with Ruby for any length of time, this &ldquo;branching composition&rdquo; should make you think of one method: <a href="http://ruby-doc.org/core-2.4.1/BasicObject.html#method-i-method_missing">method_missing</a>. Rarely does anyone override <code>method_missing</code> without a call to <code>super</code> somewhere, since doing so would catch <em>any</em> method not defined on the class, which is generally not what you want.</p> <p>Instead, a typical <code>method_missing</code> override looks something like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">method_missing</span>(method_name, *arguments, &amp;block) <span style="color:#080;font-weight:bold">if</span> method_name =~ <span style="color:#036;font-weight:bold">METHOD_NAME_REGEX</span> <span style="color:#777"># ... do some special stuff ...</span> <span style="color:#080;font-weight:bold">else</span> <span style="color:#080;font-weight:bold">super</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>&hellip; where <code>METHOD_NAME_REGEX</code> is a regex used to determine if we should &ldquo;do some special stuff&rdquo;.</p> <p>I actually use exactly this branching flow in <a href="https://github.com/shioyama/mobility">Mobility</a> in a module builder called <a href="https://github.com/shioyama/mobility/blob/2c35c8aab08bbbbf43be41d05cbeab01a22ed3d6/lib/mobility/fallthrough_accessors.rb">FallthroughAccessors</a>. Each instance of <code>FallthroughAccessors</code> is initialized with a set of attributes (its &ldquo;state&rdquo;) and intercepts method calls where the method name is made up of one of these attributes plus a locale suffix (e.g. <code>title_fr</code> for the <code>title</code> attribute in French), which I refer to as an &ldquo;<a href="https://github.com/shioyama/i18n_accessors">I18n accessor</a>&rdquo;. If the method name matches, the attribute value in the suffix locale is returned, otherwise control continues up the class hierarchy.</p> <p>Composing <code>method_missing</code> in module builders this way results in an sequence of nested regex conditionals spread across a series of modules, which I call &ldquo;interceptors&rdquo;. The fact that these interceptors are entirely encapsulated, with no external dependencies on the class itself, enables Mobility to &ldquo;plug in&rdquo; i18n accessors for each translated attribute independently, at any time and in complete isolation from each other.</p> <p>Having found this pattern useful in Mobility, I extracted it into a gem called <a href="https://github.com/shioyama/method_found">MethodFound</a>. MethodFound is essentially one module builder, <a href="https://github.com/shioyama/method_found/blob/bcbd5a9d61455fee7cce3ce882077868c1115d82/lib/method_found/interceptor.rb">MethodFound::Interceptor</a>, which intercepts method calls that match a regex or proc (the &ldquo;state&rdquo; in this case) and passes the method name, any matches captured in the regex (or the return value of the proc), and any method arguments to a block (the &ldquo;do some special stuff&rdquo; in the conditional above). This block is evaluated in the context of the instance of the class including the module.</p> <p>Here&rsquo;s an example:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Greeter</span> &lt; <span style="color:#036;font-weight:bold">Struct</span>.new(<span style="color:#A60">:name</span>) include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">Interceptor</span>.new(<span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#D20">\A</span><span style="color:#808">say_([a-zA-Z_]+)</span><span style="color:#D20">\Z</span><span style="color:#404">/</span></span>) { |method_name, matches| <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>name<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20"> says: </span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>matches[<span style="color:#00D">1</span>].gsub(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20"> </span><span style="color:#710">'</span></span>).capitalize<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">.</span><span style="color:#710">&quot;</span></span> } include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">Interceptor</span>.new(<span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#D20">\A</span><span style="color:#808">scream_([a-zA-Z_]+)</span><span style="color:#D20">\Z</span><span style="color:#404">/</span></span>) { |method_name, matches| <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>name<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20"> screams: </span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>matches[<span style="color:#00D">1</span>].gsub(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20"> </span><span style="color:#710">'</span></span>).capitalize<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">!!!</span><span style="color:#710">&quot;</span></span> } include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">Interceptor</span>.new(<span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#D20">\A</span><span style="color:#808">ask_([a-zA-Z_]+)</span><span style="color:#D20">\Z</span><span style="color:#404">/</span></span>) { |method_name, matches| <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>name<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20"> asks: </span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>matches[<span style="color:#00D">1</span>].gsub(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20"> </span><span style="color:#710">'</span></span>).capitalize<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">?</span><span style="color:#710">&quot;</span></span> } <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Here we have composed three interceptors on <code>method_missing</code>. No other trace is left in <code>Greeter</code>, but you can peek at its ancestors to see directly the three module builders along with their regex matchers (interceptors override <code>Module#inspect</code> to show this):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Greeter</span>.ancestors =&gt; [<span style="color:#036;font-weight:bold">Greeter</span>, <span style="color:#777">#&lt;MethodFound::Builder:0x...&gt;,</span> <span style="color:#777">#&lt;MethodFound::Interceptor: /\Aask_([a-zA-Z_]+)\Z/&gt;,</span> <span style="color:#777">#&lt;MethodFound::Interceptor: /\Ascream_([a-zA-Z_]+)\Z/&gt;,</span> <span style="color:#777">#&lt;MethodFound::Interceptor: /\Asay_([a-zA-Z_]+)\Z/&gt;,</span> <span style="color:#036;font-weight:bold">Object</span>, ...] </pre></div> </div> <p>So when a method is called that the class does not know, it will bubble up first to the &ldquo;ask&rdquo; interceptor and see if it matches the regex. If it does, it will return a value; if not, it continues up to the &ldquo;scream&rdquo; interceptor and checks against the second regex, and so on.</p> <p>Here is the result:</p> <div class="CodeRay"> <div class="code"><pre> greeter = <span style="color:#036;font-weight:bold">Greeter</span>.new(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Bob</span><span style="color:#710">&quot;</span></span>) greeter.say_hello_world <span style="color:#777">#=&gt; &quot;Bob says: Hello world.&quot;</span> greeter.scream_good_morning <span style="color:#777">#=&gt; &quot;Bob screams: Good morning!!!&quot;</span> greeter.ask_how_do_you_do <span style="color:#777">#=&gt; &quot;Bob asks: How do you do?&quot;</span> </pre></div> </div> <p>Given this example using module builders and <code>method_missing</code>, here&rsquo;s a problem for you: how would you implement what we have shown above <em>without</em> module builders, but with the same level of flexibility?</p> <p>The first thing that happens when you take away the module builder is that you lose the home for your module state &ndash; &ldquo;normal&rdquo; modules have no place to put it. So those regexes and blocks in the interceptor will need to go somewhere else, and <em>the only other natural place to put them is the including class itself</em>.</p> <p>There are problems with this, which a real-world example should make clear.</p> <h2>Module Builders and Encapsulation</h2> <p>Let&rsquo;s now take everything we&rsquo;ve learned so far and apply it to some real living code at the core of Ruby&rsquo;s most popular application framework.</p> <p>Continuing from our discussion in the last section, I want to focus here on how modules are typically used to configure state, where that state is stored, and the problems with this approach to module configuration. I hope to convince you that module builders offer a much more natural way to encapsulate this state in the place where it best belongs: <em>in the module itself</em>.</p> <h3>ActiveModel::AttributeMethods</h3> <p>The module we will look at is ActiveModel&rsquo;s <a href="http://guides.rubyonrails.org/active_model_basics.html#attribute-methods">AttributeMethods</a>, which adds support for prefixed and suffixed attribute methods to Rails models<a href="#note-10"><sup>10</sup></a>. If you have worked with Rails for any length of time, you are probably familiar with these prefix/suffix patterns through their use in change-tracking methods like <code>name_changed?</code> in <a href="http://api.rubyonrails.org/classes/ActiveModel/Dirty.html">ActiveModel::Dirty</a>. The module implements both module bootstrapping (like <code>AdderIncluder</code>) and <a href="(https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L427">message interception using <code>method_missing</code></a> (like <code>MethodFound::Interceptor</code>).</p> <p>Let&rsquo;s start with a simple class that implements a method prefix and a method affix:<a href="#note-11"><sup>11</sup></a></p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Person</span> include <span style="color:#036;font-weight:bold">ActiveModel</span>::<span style="color:#036;font-weight:bold">AttributeMethods</span> attribute_method_affix <span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">reset_</span><span style="color:#710">'</span></span>, <span style="color:#606">suffix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_to_default!</span><span style="color:#710">'</span></span> attribute_method_prefix <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">clear_</span><span style="color:#710">'</span></span> define_attribute_methods <span style="color:#A60">:name</span> attr_accessor <span style="color:#A60">:name</span>, <span style="color:#A60">:title</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">attributes</span> { <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">name</span><span style="color:#710">'</span></span> =&gt; <span style="color:#33B">@name</span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">title</span><span style="color:#710">'</span></span> =&gt; <span style="color:#33B">@title</span> } <span style="color:#080;font-weight:bold">end</span> private <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">clear_attribute</span>(attr) send(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>attr<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">=</span><span style="color:#710">&quot;</span></span>, <span style="color:#069">nil</span>) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">reset_attribute_to_default!</span>(attr) send(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>attr<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">=</span><span style="color:#710">&quot;</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Default </span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>attr.capitalize<span style="font-weight:bold;color:#666">}</span></span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>This is how we would use these attribute methods:</p> <div class="CodeRay"> <div class="code"><pre> person = <span style="color:#036;font-weight:bold">Person</span>.new person.name = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">foo</span><span style="color:#710">&quot;</span></span> person.name <span style="color:#777">#=&gt; &quot;foo&quot;</span> person.clear_name person.name <span style="color:#777">#=&gt; nil</span> person.reset_name_to_default! person.name <span style="color:#777">#=&gt; &quot;Default Name&quot;</span> </pre></div> </div> <p>Just looking at the code above, it is clear that the module is storing state somewhere, since it needs to know what prefixes, suffixes and affixes have been defined for a given class. We can see where this state is stored in the code for <code>attribute_method_prefix</code>, which is called in <code>Person</code> to setup a method prefix:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">attribute_method_prefix</span>(*prefixes) <span style="color:#069">self</span>.attribute_method_matchers += prefixes.map! { |prefix| <span style="color:#036;font-weight:bold">AttributeMethodMatcher</span>.new <span style="color:#606">prefix</span>: prefix } undefine_attribute_methods <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>So the array of prefix strings are mapped to instances of a class, <code>AttributeMethodMatcher</code>, stored in an array <code>attribute_method_matchers</code> on the <code>Person</code> class. (The affix and suffix methods are similar.) These matchers are the module&rsquo;s core configuration, and they are <em>stored outside the module itself</em>.</p> <p>What&rsquo;s in one of these matchers? Let&rsquo;s have a look at <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L394-L399">the <code>initialize</code> method</a> of the class to get an idea:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">initialize</span>(options = {}) <span style="color:#33B">@prefix</span>, <span style="color:#33B">@suffix</span> = options.fetch(<span style="color:#A60">:prefix</span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#710">&quot;</span></span>), options.fetch(<span style="color:#A60">:suffix</span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#710">&quot;</span></span>) <span style="color:#33B">@regex</span> = <span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#808">^(?:</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span><span style="color:#036;font-weight:bold">Regexp</span>.escape(<span style="color:#33B">@prefix</span>)<span style="font-weight:bold;color:#666">}</span></span><span style="color:#808">)(.*)(?:</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span><span style="color:#036;font-weight:bold">Regexp</span>.escape(<span style="color:#33B">@suffix</span>)<span style="font-weight:bold;color:#666">}</span></span><span style="color:#808">)$</span><span style="color:#404">/</span></span> <span style="color:#33B">@method_missing_target</span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span><span style="color:#33B">@prefix</span><span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">attribute</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span><span style="color:#33B">@suffix</span><span style="font-weight:bold;color:#666">}</span></span><span style="color:#710">&quot;</span></span> <span style="color:#33B">@method_name</span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>prefix<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">%s</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>suffix<span style="font-weight:bold;color:#666">}</span></span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>So essentially, a matcher is two things: a prefix and suffix pair, and a regex generated from them (the rest is mostly for convenience). This internal state is used for two different but related purposes.</p> <p>The first of these two purposes is important but not well-documented at all; you can find it explained inline in a comment <a href="https://github.com/rails/rails/blob/46adb59b3474351e95ddc2357e67d169d6ffec51/activemodel/lib/active_model/attribute_methods.rb#L417-L422">here</a>. The purpose is to provide a way to convert a hash of keys and values returned from an <code>attributes</code> method into a set of first-class attribute methods, supporting all defined prefix, suffix and affix method patterns. In the example above, <code>attributes</code> returns a hash with keys <code>name</code> and <code>title</code>, so these become attributes that dispatch to methods like <code>clear_attribute</code> and <code>reset_attribute_to_default!</code>.</p> <p>Like <a href="https://github.com/shioyama/method_found">MethodFound</a>, this is implemented using <code>method_missing</code>: any method call which matches a matcher regex <em>and</em> whose target (captured from the regex) is a key in <code>attributes</code> is <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L427">intercepted and dispatched to an attribute handler</a>. So <code>reset_title_to_default!</code> is dispatched to <code>reset_attribute_to_default!</code>, with the name of the attribute (<code>"title"</code>) as its argument.</p> <p>Method interception is great for an open-ended, changing hash of attributes, but <code>method_missing</code> is slow as molasses. If you know what attributes you want, it&rsquo;s generally better to define the methods explicitly.</p> <p>This is the second purpose of the method matchers, triggered by calling <code>define_attribute_methods</code> with one or more attribute names <em>after</em> setting up prefixes, suffixes and affixes: to define methods for each combination of prefix and/or suffix on each attribute. <code>Person</code> defines methods for <code>name</code>, so <code>clear_name</code> and <code>reset_name_to_default!</code> dispatch to <code>clear_attribute</code> and <code>reset_attribute_to_default!</code>, respectively.</p> <p>These methods are bound to an anonymous module <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L332">defined and included</a> in another method added to the <code>Person</code> class, named <code>generated_attribute_methods</code>. As instance methods, they take precedence over method interception, so while both <code>clear_title</code> and <code>clear_name</code> will return the same result, the former falls through to <code>method_missing</code>, whereas the latter is handled (much more quickly) by the attribute method.</p> <div class="img-center" style="margin-top: 30px; margin-bottom: 30px"> <img style="margin-bottom: 20px" alt="ActiveModel AttributeMethods" src="/img/am-attributemethods.png"/> <p style="text-align:center"><small>ActiveModel's AttributeMethods</small></p> </div> <p>You can see the defined attribute methods if you grep the methods on an <code>Person</code> instance:</p> <div class="CodeRay"> <div class="code"><pre> person = <span style="color:#036;font-weight:bold">Person</span>.new person.methods.grep <span style="background-color:hsla(300,100%,50%,0.06)"><span style="color:#404">/</span><span style="color:#808">name</span><span style="color:#404">/</span></span> <span style="color:#777">#=&gt; [:name, :name=, :reset_name_to_default!, :clear_name, ...]</span> </pre></div> </div> <p>You can confirm these generated methods are defined on the module, and not somewhere else, by grabbing the module and checking its instance methods:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Person</span>.ancestors <span style="color:#777">#=&gt; [Person, #&lt;Module:Ox...&gt;, ActiveModel::AttributeMethods, ...]</span> <span style="color:#036;font-weight:bold">Person</span>.ancestors[<span style="color:#00D">1</span>].instance_methods <span style="color:#777">#=&gt; [:name, :name=, :reset_name_to_default!, :clear_name]</span> </pre></div> </div> <p>You can also find the <code>method_missing</code> override described earlier <a href="https://github.com/rails/rails/blob/00f1f95569a491c89bf78f08b78ebf0e6ff1d9d1/activemodel/lib/active_model/attribute_methods.rb#L427">here</a> on <code>ActiveModel::AttributeMethods</code>, just after the anonymous module in the class hierarchy.</p> <p>These two implementations of attribute methods, one using method interception with <code>method_missing</code> and the other using method definition, complement each other since they are suited to different access patterns (one for a large open-ended set of attributes, the other for a smaller fixed set of attributes). Mobility actually uses the same approach in defining i18n accessors.<a href="#note-12"><sup>12</sup></a> However, whereas AttributeMethods adds a dozen class methods and variables, Mobility adds none, instead hiding all state in its modules.</p> <p>Let&rsquo;s now look at an alternative implementation of AttributeMethods which, like Mobility, uses module builders to encapsulate state in the modules it builds.</p> <h3>MethodFound::AttributeMethods</h3> <p>This is where we bring all the ideas discussed so far together and apply them to a real application. Before jumping into more code, let&rsquo;s review the elements implementing attribute methods on a class:</p> <ol> <li>a <strong>class variable</strong> (<code>attribute_method_matchers</code>) holding an array of matchers for prefixes/suffixes/affixes</li> <li>a <strong>method_missing override</strong> to dispatch method calls matching the attribute matchers and the <code>attributes</code> hash to handlers accordingly</li> <li>an included <strong>anonymous module</strong> (<code>generated_attribute_methods</code>) holding defined attribute methods</li> </ol> <p>Now consider that <em>each of these core elements is located in a different place</em>:</p> <ul> <li>the method matchers are defined on the <strong>including class</strong></li> <li>the generated attribute methods are defined on an <strong>anonymous module</strong> included in this class</li> <li><code>method_missing</code> is defined on <strong>AttributeMethods itself</strong>, also included in the class</li> </ul> <p>So we have the three elements of our implementation, coupled to each other through the method matchers class variable, spread out across different parts of the implementing system. Not only that, but the method matchers and generated methods, which (as we will see below) are essentially functionally independent, are stored together in the <em>same place</em> (an array and an anonymous module, respectively).</p> <p>Let&rsquo;s consider an alternative implementation which distributes state in a different way. In this implementation, we will group the following elements together into a single module using a module builder:</p> <ul> <li>a single prefix/suffix pair, from which a matcher regex is generated</li> <li>a single <code>method_missing</code> override which uses the regex to intercept methods</li> <li>a method to define attribute methods for the given prefix/suffix pair</li> </ul> <p>The module builder is named <a href="https://github.com/shioyama/method_found/blob/9af678d8ff0400eb33f275cddc66ec3fd6d16ee8/lib/method_found/attribute_interceptor.rb">AttributeInterceptor</a>. It inherits from <code>MethodFound::Interceptor</code>, which we saw in the last section, customizing the initializer so that it accepts a prefix and/or suffix instead of the more open-ended format of the default interceptor builder. Although it is not very big, I won&rsquo;t include the actual code here so we can instead focus on what the class actually does.</p> <div class="img-center" style="margin-top: 30px; margin-bottom: 30px"> <img style="margin-bottom: 20px" alt="MethodFound AttributeMethods" src="/img/mf-attributemethods.png"/> <p style="text-align:center"><small>Implementing AttributeMethods with MethodFound interceptors</small></p> </div> <p>Let&rsquo;s first take a look at intercepting. We can reproduce this by replacing calls to <code>define_attribute_prefix</code> and <code>define_attribute_affix</code> in the <code>Person</code> class above with attribute interceptors, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Person</span> include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">AttributeInterceptor</span>.new(<span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">clear_</span><span style="color:#710">'</span></span>) include <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">AttributeInterceptor</span>.new(<span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">reset_</span><span style="color:#710">'</span></span>, <span style="color:#606">suffix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_to_default!</span><span style="color:#710">'</span></span>) attr_accessor <span style="color:#A60">:name</span>, <span style="color:#A60">:title</span> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">attributes</span> { <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">name</span><span style="color:#710">'</span></span> =&gt; <span style="color:#33B">@name</span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">title</span><span style="color:#710">'</span></span> =&gt; <span style="color:#33B">@title</span> } <span style="color:#080;font-weight:bold">end</span> <span style="color:#777"># ...</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>We are instantiating two attribute interceptors and including each of them into the class. There is no special case here for &ldquo;affix&rdquo;; we simply build an interceptor with both a prefix and suffix.</p> <p>With these modules included, <code>Person</code> behaves externally exactly the same as the earlier version using the ActiveModel module, but the internal implementation is quite different. This can be seen from its ancestors:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Person</span>.ancestors <span style="color:#777">#=&gt; [Person,</span> <span style="color:#777">#&lt;MethodFound::AttributeInterceptor: /\A(?:reset_)(.*)(?:_to_default!)\z/&gt;,</span> <span style="color:#777">#&lt;MethodFound::AttributeInterceptor: /\A(?:clear_)(.*)(?:)\z/&gt;,</span> ... ] </pre></div> </div> <p>Like the MethodFound interceptors described earlier, these two modules each override <code>method_missing</code> and branch the code path if the method name matches their regex (stored on the module).</p> <p>The resulting set of distributed nested conditionals is equivalent to <a href="https://github.com/rails/rails/blob/23aa0a2bb551717f153ac75f24c017c43ab853f2/activemodel/lib/active_model/attribute_methods.rb#L360">this line from ActiveModel</a>, which iterates through attribute method matchers checking for matches:</p> <div class="CodeRay"> <div class="code"><pre> matchers.map { |method| method.match(method_name) }.compact </pre></div> </div> <p>The key difference between these two implementations is that whereas this line is executed in <code>ActiveModel::AttributeMethods</code> (a module), using a class variable stored in <code>Person</code> (a class), the MethodFound version is executed <em>across</em> independent modules through method composition, using <code>super</code>.</p> <p>The implementation of generated attribute methods is similar. <code>AttributeInterceptor</code> has a method <code>define_attribute_methods</code> which takes one or more attribute names and defines attribute methods for each of them with the module&rsquo;s prefix and/or suffix, <em>on the module itself</em>. Again, because the module already contains its own prefix and suffix, it has all the information it needs to do this.</p> <p>So functionality is truly encapsulated: a single module contains its own prefix and suffix, the <code>method_missing</code> override to catch attribute method calls, and a method to generate its own attribute method.<a href="#note-13"><sup>13</sup></a></p> <p>Given this module builder, we can reproduce ActiveModel&rsquo;s implementation without any class methods or class variables, like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Person</span> [ <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">AttributeInterceptor</span>.new(<span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">clear_</span><span style="color:#710">'</span></span>), <span style="color:#036;font-weight:bold">MethodFound</span>::<span style="color:#036;font-weight:bold">AttributeInterceptor</span>.new(<span style="color:#606">prefix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">reset_</span><span style="color:#710">'</span></span>, <span style="color:#606">suffix</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">_to_default!</span><span style="color:#710">'</span></span>) ].each <span style="color:#080;font-weight:bold">do</span> |mod| mod.define_attribute_methods(<span style="color:#A60">:name</span>) include mod <span style="color:#080;font-weight:bold">end</span> <span style="color:#777">#...</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>MethodFound includes another module, <code>MethodFound::AttributeMethods</code>, which adds class methods do simplify the code above, so it can be used as a drop-in replacement for <code>ActiveModel::AttributeMethods</code>. The <a href="https://github.com/shioyama/method_found/blob/9532921d24d110e735d276c7e354c5dc0d0b40b1/lib/method_found/attribute_methods.rb#L65-L69">implementation</a> of <code>define_attribute_methods</code> in this module is interesting:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_attribute_methods</span>(*attr_names) ancestors.each <span style="color:#080;font-weight:bold">do</span> |ancestor| ancestor.define_attribute_methods(*attr_names) <span style="color:#080;font-weight:bold">if</span> ancestor.is_a?(<span style="color:#036;font-weight:bold">AttributeInterceptor</span>) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Since the class no longer stores the matchers in its state, the module cannot simply iterate through these matchers to define methods. Instead, it <em>iterates through its own ancestors</em>, which are module instances each with a prefix/suffix pair, and calls <code>define_attribute_methods</code> if the ancestor is an attribute interceptor. This generates attribute methods on each interceptor module, which can then be called by instances of the class.</p> <p>The resulting implementation encapsulates related elements together into separate modules without coupling via class variables or methods. This encapsulation means that we could now design totally different types of attribute interceptors and include them in the same way; as long as they have the same interface (for generating attribute methods), the internals can be entirely different and nothing else will need to change.</p> <p>This, it seems to me, is what a &ldquo;module&rdquo; should really be: an independent, interchangeable unit containing everything it needs to execute only one aspect of some desired functionality. Not only that, but the interface for building these modules falls directly out of Ruby&rsquo;s own object model, a model that has been around as long as the language itself. Ruby has had this trick up its sleeve for years, <em>most of us just never realized it</em>.</p> <h2>What&rsquo;s in a Name?</h2> <p>You may argue that the &ldquo;Module Builder Pattern&rdquo; I&rsquo;ve introduced is just a Ruby subclass where the class happens to be <code>Module</code>, and what&rsquo;s the big deal? And technically, you would be right. I&rsquo;m not the first to notice this idea, nor am I the first to write about it. Just <code>Foo &lt; Module</code>, and I could be done with it.</p> <p>But programming languages are written and read by and for humans, and to humans, names mean a lot. If you can&rsquo;t find the module with the methods on it buried in a list of ancestors, you won&rsquo;t notice it, and if you can&rsquo;t remember that subclassing thing some blogger wrote about, you won&rsquo;t use it. The fact that this pattern has been around for this long without almost anybody knowing about it is testament to this fact.</p> <p>So I gave it a name. A catchy one.</p> <p>Because as Rubyists, <strong>I think we need this pattern</strong>. Take a look at the code of a large Ruby project and how modules are used (and abused) in practice. Our modules trigger callbacks that bootstrap all kinds of extra state into our classes, coupling them to the modules they include. A module should be self-contained, but we use the excuse that we cannot configure modules at runtime to justify storing configuration all over the place.</p> <p>Ruby can do better than this, and we can do better than this. So take another look at that bloated module, the one with the tentacles that seem to have crept over every corner of your application, and give it a chance to be free. I think you will find that it &mdash; and you &mdash; will be all the better for it.</p> <hr /> <h4>Update (2017/10/20)</h4> <p><small>This post has been translated into Japanese: see <a href="https://techracho.bpsinc.jp/hachi8833/2017_10_12/46276">part 1</a>, <a href="https://techracho.bpsinc.jp/hachi8833/2017_10_19/46619">part 2</a> and <a href="https://techracho.bpsinc.jp/hachi8833/2017_10_26/46623">part 3</a>.</small></p> <h4>Update (2017/09/27)</h4> <p><small>Since writing this blog post, I have also <a href="https://www.youtube.com/watch?v=_E1yKPC-r1E">presented</a> the Module Builder Pattern at <a href="http://rubykaigi.org/2017/presentations/shioyama.html">RubyKaigi 2017</a> in Hiroshima. Slides from that talk can be found <a href="https://speakerdeck.com/shioyama/the-ruby-module-builder-pattern">on Speaker Deck</a>, and a video has also been posted to <a href="https://www.youtube.com/watch?v=_E1yKPC-r1E">YouTube</a>.</p> <h4>Notes</h4> <p><small><a name="note-1"><sup>1</sup></a> The first reference I can find to the idea of &ldquo;subclassing Module&rdquo; is in an <a href="http://solnic.codes/2012/08/13/subclassing-module-for-fun-and-profit">article in 2012</a> by Piotr Solnica. <a href="https://medium.com/@eric.programmer/arguments-for-included-modules-in-ruby-8056b9fa2743">Eric Anderson</a> and <a href="https://makandracards.com/makandra/43002-subclassing-module">Andreas Robecke</a> have written about it more recently, but hardly anything else even mentions it.</small><br /> <small><a name="note-2"><sup>2</sup></a> At the time of writing this post, it is used <a href="https://github.com/rails/rails/blob/0dd40fe72cfcf53b475a4084d61e9e931417c29e/activemodel/lib/active_model/validations/acceptance.rb#L25">here</a> and <a href="https://github.com/rails/rails/blob/0dd40fe72cfcf53b475a4084d61e9e931417c29e/activemodel/lib/active_model/type/helpers/accepts_multiparameter_time.rb#L4">here</a>. </small><br /> <small><a name="note-3"><sup>3</sup></a> By this I mean you can include modules on over another and use <code>super</code> to build complex composed method chains. I do a lot of this in Mobility.</small><br /> <small><a name="note-4"><sup>4</sup></a> And, to be precise, which also has an initializer which takes the values of these attributes as its arguments.</small><br /> <small><a name="note-5"><sup>5</sup></a> I&rsquo;ve used a trick here to add a module&rsquo;s methods as class methods to a class using <code>include</code>, by extending the class when the module is included. This is a common trick, if it looks a bit mysterious you might want to read up on the technique before continuing.</small><br /> <small><a name="note-6"><sup>6</sup></a> This &ldquo;no-clutter&rdquo; property of anonymous modules is also what makes them so handy in metaprogramming, since it means you don&rsquo;t need to worry about constant name collisions.</small><br /> <small><a name="note-7"><sup>7</sup></a> It is actually being defined from these keys using a <a href="https://en.wikipedia.org/wiki/Closure_(computer_programming">closure</a>.</small><br /> <small><a name="note-8"><sup>8</sup></a> Defining an <code>inspect</code> method in the module builder class which includes the variables (e.g. <code>amount</code> and <code>tax</code>) which define makes this ancestor chain even easier to read.</small><br /> <small><a name="note-9"><sup>9</sup></a> MethodFound actually does a bit more than this: it also &ldquo;caches&rdquo; the method name matched in the regex or proc and defines it on the interceptor module, so that future calls will be much faster. You can see this if you look at the methods on an interceptor after it has matched a method using <code>method_missing</code>.</small><br /> <small><a name="note-10"><sup>10</sup></a> ActiveRecord models in fact include <code>ActiveRecord::AttributeMethods</code>, which itself includes <code>ActiveModel::AttributeMethods</code> and overrides some of the methods discussed here, specifically <code>define_attribute_methods</code>. The relationship between these two modules, and between how persisted and non-persisted attribute methods are handled, is somewhat complex, but the ideas discussed here are relevant to both.</small><br /> <small><a name="note-11"><sup>11</sup></a> Note that I have slightly modified and simplified this example class from the one in the inline documentation to highlight some things that the standard docs do not mention.</small><br /> <small><a name="note-12"><sup>12</sup></a> Mobility handles these two cases with the module builders <code>FallthroughAccessors</code> and <code>LocaleAccessors</code>. I also extracted these from Mobility into a gem called <a href="https://github.com/shioyama/i18n_accessors">I18nAccessors</a>. </small><br /> <small><a name="note-13"><sup>13</sup></a> It actually also has a method <code>alias_attribute</code> to alias attributes, like <code>ActiveModel::AttributeMethods</code>.</small></br></p> <h4>Figures</h4> <p><small><sup>a</sup> &ldquo;Drawing Hands&rdquo; by M.C. Escher. [<a href="https://en.wikipedia.org/wiki/Drawing_Hands">reference</a>]</small><br /> <small><sup>b</sup> &ldquo;Automated space exploration and industrialization using self-replicating systems.&rdquo; From <em>Advanced Automation for Space Missions: 5. REPLICATING SYSTEMS CONCEPTS: SELF-REPLICATING LUNAR FACTORY AND DEMONSTRATION</em>. [<a href="http://www.islandone.org/MMSG/aasm/chapter5.htm">reference</a>]</small><br /> <small><sup>c</sup> &ldquo;Proposed demonstration of simple robot self-replication.&rdquo; [<a href="https://en.wikipedia.org/wiki/Self-replicating_machine">reference</a>]</small></p> Translating with Mobility https://dejimata.com/2017/3/3/translating-with-mobility 2017-03-03 2017-03-03 Chris Salzberg <p>In the first real post on this blog, I described my experience <a href="/2013/10/11/hacking-globalize">hacking globalize</a>, a <a href="https://rubygems.org/gems/globalize">gem</a> that calls itself the &ldquo;Rails I18n de-facto standard library for ActiveRecord model/data translation.&rdquo;</p> <p>Globalize was the first gem that I contributed to, and the experience was very important to me as a step toward becoming a better and more knowledgeable programmer.</p> <p>I am now one of the authors of that gem, having made <a href="https://github.com/globalize/globalize/commits?author=shioyama">many commits</a> to the repository. In the time since I first contributed to Globalize, I&rsquo;ve become keenly aware of the limitations of the gem, and of problems with existing solutions for managing translations in Rails projects.</p> <p>I&rsquo;d like to look back in this post to the problem of translation and to solutions devised so far, as motivation to introduce a gem I&rsquo;ve just released called <a href="https://rubygems.org/gems/mobility">Mobility</a>. Mobility attempts to solve some core problems with the multitude of Ruby-based translation gems out there right now.</p> <h2>The Problem</h2> <p>You have an application with user-generated content. You want to translate that content into multiple languages. And &ndash; importantly &ndash; you want to do it in such a way that you don&rsquo;t need to recode all your existing presentation logic (views, mailers, controllers, etc.) to handle the new languages.</p> <p>In other words, things should <em>just work</em> &ndash; more or less the same as they do with one language, except that switching the language should tell the application to switch context to read or write content stored in that language.</p> <p>So this:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first post.title </pre></div> </div> <p>&hellip; should return the title of the post, <em>in the language I am currently viewing</em>.</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:en</span> post.title <span style="color:#777">#=&gt; &quot;Mobility (noun): quality of being changeable, adaptable or versatile&quot;</span> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span> post.title <span style="color:#777">#=&gt; &quot;Mobility(名詞):動きやすさ、可動性&quot;</span> </pre></div> </div> <p>Likewise, I should be able to write translated values the same way:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:en</span> post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> post.save </pre></div> </div> <p>Now the English title has changed, but the Japanese one has not:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first post.title <span style="color:#777">#=&gt; &quot;Translating with Mobility&quot;</span> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span> post.title <span style="color:#777">#=&gt; &quot;Mobility(名詞):動きやすさ、可動性&quot;</span> </pre></div> </div> <p>For bonus points, this solution should also work for querying attributes, so something similar to this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span>) </pre></div> </div> <p>&hellip; should work even if <code>title</code> is a translated attribute.</p> <p>For this to happen, something has to hook in to attribute accessors, to detect changes in the locale and return language-specific content accordingly. This may seem easy to do at first glance, but it gets tricky.</p> <p>Notice that what we&rsquo;re asking for is similar but different from the problem solved by localization solutions such as the <a href="http://guides.rubyonrails.org/i18n.html">Rails I18n API</a>, which provide namespaced translation lookup (generally) for interface translation, typically with static data.</p> <p>With application content, we want translations at the class level, in the form of persisted attributes of a model. Ultimately we&rsquo;re talking about a storage strategy.</p> <p>As it turns out, there are a number of different strategies that different people have come up with to solve this problem. Let&rsquo;s take a look at them.</p> <h2>Solutions</h2> <h3><a name="strategy-1"></a>Strategy 1: Translatable Columns</h3> <p>The most obvious solution to this problem is to create columns on your model table in each locale you want to support, with some convention for identifying the locale of the column.</p> <p>Typically, this is done by appending a suffix to the column name made up of an underscore plus the locale, with any dashes in the locale converted to underscores.</p> <p>So <code>title</code> becomes <code>title_en</code>, <code>title_fr</code>, <code>title_ja</code>, and so on. Adding a new locale to an application then requires that you add columns for that locale to <em>each and every translated model table</em> (see figure below.)</p> <div class="img-center"> <img alt="Column Strategy" src="/img/column_strategy.png"></a> <p class="caption">Table structure of column strategy</p> </div> <p>For a small number of models, and a small number of locales, this is a reasonable solution. It has the advantage that the column structure is easy to understand, and querying by translated attribute is straightforward:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Mobility</span><span style="color:#710">&quot;</span></span>) <span style="color:#777">#=&gt; #&lt;Post:0x... id: 123 content_en: &quot;Mobility&quot; content_ja: &quot;モビリティ&quot;...&gt;</span> <span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title_ja</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">モビリティ</span><span style="color:#710">&quot;</span></span>) <span style="color:#777">#=&gt; #&lt;Post:0x... id: 123 content_en: &quot;Mobility&quot; content_ja: &quot;モビリティ&quot;...&gt;</span> </pre></div> </div> <p>Some key disadvantages of this approach:</p> <ul> <li>For many models and many locales, this approach requires many migrations, each time a locale or model is added.</li> <li>Space is not used very efficiently: every record has a column for every locale, even if the locale is not used.</li> </ul> <p>Despite these disadvantages, this strategy is appropriate for applications with a fixed number of locales. For a very nice implementation of this approach, see <a href="https://github.com/barsoom/traco">Traco</a>.</p> <p><em>Mobility supports this strategy with its <code>:column</code> backend.</em></p> <h3><a name="strategy-2"></a>Strategy 2: Model Translation Tables</h3> <p>Now consider that we have an application where we have many translated models, with a potentially open-ended list of locales we want to support. A service with user-generated content offered to users around the world would be a good example of this pattern. In this case, the first strategy becomes impractical as the number of locales grows; with, say, 50 locales, and only a small number of actually translations, the vast majority of translated columns would be null. On top of which, adding new locales to all the translated models would be a pain, not to mention error-prone.</p> <p>This brings us to the next strategy, shown in the database schema below. Here, each model table is assigned its own translations table. Each row in the translations table has:</p> <ul> <li>A column for the locale (<code>locale</code>)</li> <li>A foreign key pointing to the model table (<code>post_id</code>)</li> <li>One or more columns for translations (<code>title</code>, <code>content</code>)</li> </ul> <p>With this structure, we can now <em>join</em> the translation table (e.g. <code>post_translations</code>) to the model table (e.g. <code>posts</code>) on the locale we want to get the desired result: a model with translations in that locale.</p> <div class="img-center"> <img alt="Model Translation Strategy" src="/img/globalize_strategy.png"></a> <p class="caption">Table structure of model translation strategy</p> </div> <p>In ActiveRecord, we would have an association on the model <code>Post</code> to the model <code>PostTranslation</code>, such that we can fetch translations with <code>post.translations</code> and pick out the one we want:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first translation = post.translations.find <span style="color:#080;font-weight:bold">do</span> |translation| translation.locale == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">en</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#777">#=&gt; #&lt;PostTranslation:0x... id: 123 locale: &quot;en&quot; ...&gt;</span> translation.title <span style="color:#777">#=&gt; &quot;Translating with Mobility&quot;</span> translation.content <span style="color:#777">#=&gt; &quot;In the first real post on this blog...&quot;</span> </pre></div> </div> <p>With this approach, we&rsquo;ve nicely solved some of the problems with the first strategy:</p> <ul> <li>Since translations now have their own model and table, and we only create rows in that table when we save a translation in that locale, space usage is more efficient (no wasted columns/rows for locales that have no translation).</li> <li>We do not need any migrations to add locales: simply save the translation for a new locale and it becomes available. So scalability on the &ldquo;locale axis&rdquo; is very good.</li> </ul> <p>However, there are some clear disadvantages with this approach that should be kept in mind:</p> <ul> <li>Having translations on a different table implies a lot of joining, which (if not managed very carefully) can lead to performance degradation.</li> <li>Joining can also add significant complexity to the code dealing with translations. In particular, querying on translated attributes is much more complicated than it was with the first strategy, where we queried on the model column directly.</li> </ul> <p>Note also that although this strategy scales nicely along the &ldquo;locale axis&rdquo;, it nonetheless requires a table for every translated model, and thus a migration every time we add a new model or a new translated attribute to an existing model. So this strategy may not scale well if an application has many different forms of translated content, spread across a large and growing list of models.</p> <p>This strategy is implemented by <a href="https://github.com/globalize/globalize">Globalize</a>, the gem mentioned at the beginning of this post (it is also implemented by <a href="https://github.com/francesc/rails-translate-models">many</a> <a href="https://github.com/jo/puret">other</a> <a href="https://github.com/janne/model_translations">gems</a>). For the reasons mentioned above, Globalize has become the preferred option for model translation in Rails. Like the first strategy, it is a very useful approach for many (but not all) use cases.</p> <p><em>Mobility supports this strategy with its <code>:table</code> backend.</em></p> <h3>Strategy 3: Shared Translation Tables</h3> <p>My experience working with Globalize, and in particular with scalability issues mentioned above, prompted me to consider another strategy which would scale better with model creation. I wanted a strategy which would be usable <em>immediately</em> at the code level, without having to add a table or column every time a new model or translated attribute is added.</p> <p>This led me to the third strategy, which I&rsquo;ve called the &ldquo;key value&rdquo; strategy. The basic idea is simple: rather than using a dedicated table for each translated model, instead create shared tables to store translations across all models (as shown in the figure below). Two tables are required since we will be storing two types of values: strings and text. (If you wanted to store other column types, you could add more tables for each of those types, or cast strings to some other type).</p> <div class="img-center"> <img alt="KeyValue Strategy" src="/img/key_value_strategy.png"></a> <p class="caption">Table structure of key value strategy</p> </div> <p>The key thing here is that rather than using a normal association as we did in the last strategy, where <code>post_id</code> on the <code>post_translations</code> table pointed to the <code>posts</code> table, we will instead use a <a href="http://guides.rubyonrails.org/association_basics.html#polymorphic-associations">polymorphic association</a>. By doing this, we can share the translations tables with as many models as we like, migrating just once to create the shared tables.</p> <p>The new tables have the following columns:</p> <ul> <li>A column for the locale (<code>locale</code>)</li> <li>A foreign key pointing to the translated model id (<code>translatable_id</code>)</li> <li>A column holding the model class (<code>translatable_type</code>)</li> <li>A column holding the name of the attribute being translated (<code>key</code>)</li> <li>A column holding the value of the translated attribute (<code>value</code>)</li> </ul> <p>With these tables in place, we define a polymorphic relationship on the model, something like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span> has_many <span style="color:#A60">:text_translations</span>, <span style="color:#606">as</span>: <span style="color:#A60">:translatable</span>, <span style="color:#606">class_name</span>: <span style="color:#036;font-weight:bold">TextTranslation</span>, <span style="color:#606">inverse_of</span>: <span style="color:#A60">:translatable</span> <span style="color:#777">#...</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>and a translation class:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">TextTranslation</span> <span style="color:#069">self</span>.table_name = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">&quot;</span></span> belongs_to <span style="color:#A60">:translatable</span>, <span style="color:#606">polymorphic</span>: <span style="color:#069">true</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now that we have these models in place, we can easily fetch and set translations of a model class:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first post.text_translations.find <span style="color:#080;font-weight:bold">do</span> |translation| translation.key == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span> &amp;&amp; translation.value == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#777">#=&gt; #&lt;TextTranslation:0x...</span> <span style="color:#606">id</span>: <span style="color:#00D">123</span> <span style="color:#606">locale</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">en</span><span style="color:#710">&quot;</span></span> key=<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span> value=<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> ... &gt; </pre></div> </div> <p>To add translations to a new model, we simply add a polymorphic relationship like the one above on <code>Post</code>, and we can immediately add translations to the model.</p> <p>In practice, there are some other issues that need to be handled to make this work effectively, but done correctly this strategy has some notable benefits:</p> <ul> <li>No more migrations (after the first one), so an application can quickly add new translated models.</li> <li>As with Strategy 2, no extra effort to add new locales.</li> <li>Relatively efficient in terms of storage, in the sense that we only create records for translations in locales where they actually exist. However, since each attribute translation gets a row in the table, for multiple attributes on a single translation table, Strategy 2 may be more space efficient since it stores translated attributes together in a single row.</li> </ul> <p>Now for the disadvantages:</p> <ul> <li>The join complexity described in the last section is even worse now, because we have to join <em>every attribute separately</em> when searching for matches. This is quite tricky but can be done. I&rsquo;ll probably write about this in a future post.</li> <li>Since we have each attribute stored separately (as a key/value pair) it becomes trickier to maintain our database in a consistent state. For example, if we remove a model, its translations will remain in the translations table(s) unless we explicitly remove them. Likewise, if we decide to rename a translated attribute, we may end up with translations for the previous attribute name leftover in the table.</li> </ul> <p>Despite these disadvantages, I consider this strategy to be applicable to the widest variety of situations. It is the default strategy in Mobility, implemented as the <code>:key_value</code> backend.</p> <h3>Strategy 4: Serialized Data</h3> <p>Yet another set of strategies take the approach of jamming all the translations for an attribute into a single column. The first strategy following this approach is one which serializes translations as a hash and stores it on a text column.</p> <p>At the model layer, this is pretty trivial in Rails:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span> serialize <span style="color:#A60">:title_translations</span>, <span style="color:#036;font-weight:bold">Hash</span> <span style="color:#080;font-weight:bold">end</span> post = <span style="color:#036;font-weight:bold">Post</span>.new post.title_translations = { <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">en</span><span style="color:#710">&quot;</span></span> =&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> } post.save </pre></div> </div> <p>We can now fetch translations simply by fetching the value for the key matching a given locale:</p> <div class="CodeRay"> <div class="code"><pre> post.title_translations[<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">en</span><span style="color:#710">&quot;</span></span>] <span style="color:#777">#=&gt; &quot;Translating with Mobility&quot;</span> </pre></div> </div> <p>This strategy has the obvious advantage of simplicity: all our translations are stored together on the attribute column. Like the model translations and key-value strategies, locale scalability is not an issue since we can just add new hash keys as we need them.</p> <p>In addition, no additional migrations are necessary; simply adding a text column to every model for each translated attribute is enough to support any number of translations.</p> <p>Despite these advantages, the serialized approach is not generally recommended, for the key reason that <em>querying on serialized data is not possible</em>. Indeed, Rails itself has discouraged the use of serialized attributes in favour of database-specific storage solutions (described below).</p> <p>Nonetheless, there are valid use cases for this strategy. One gem that implements this approach is <a href="https://github.com/artworklv/multilang">Multilang</a>.</p> <p><em>Mobility supports this strategy with its <code>:serialized</code> backend.</em></p> <h3>Strategy 5/6: PostgreSQL Hstore/Jsonb</h3> <p>The last two strategies, <a href="https://www.postgresql.org/docs/current/static/hstore.html">Hstore</a> and <a href="https://www.postgresql.org/docs/current/static/datatype-json.html">jsonb</a>, require the use of PostgreSQL as your database. Hstore columns store depth-1 hashes, where keys and values of the hash must be of string type. Jsonb (&ldquo;binary json&rdquo;) columns store data in JSON format, which supports a much broader range of primitives (strings, numbers, booleans and null) as well as arrays, and unlimited depth. Jsonb is generally preferred over Hstore since it is more flexible.</p> <p>Further details on these data types can be found in the links above; for our purposes, the main point is that they both store complex data types in a single column, in a form that can be <em>easily and quickly queried</em>.</p> <p>An example of such a query (on a Jsonb column) looks like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">WHERE</span> (posts.title <span style="color:#F00;background-color:#FAA">@</span>&gt; (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">{&quot;en&quot;:&quot;Translating with Mobility&quot;}</span><span style="color:#710">'</span></span>)::jsonb) </pre></div> </div> <p>Querying on multiple locales at once is also simple given the flexibility of the data format.</p> <p>Gems that use Hstore as a storage format for translations include <a href="https://github.com/yabawock/trasto">Trasto</a>, <a href="https://github.com/bithavoc/multilang-hstore">Multilang Hstore</a> and <a href="https://github.com/Leadformance/hstore_translate">Hstore Translate</a>. I only know of one gem (as of yet) that uses Jsonb for storage, <a href="https://github.com/cfabianski/json_translate">Json Translate</a>.</p> <p><em>Mobility supports Hstore with its <code>:hstore</code> backend and Jsonb with its <code>:jsonb</code> backend.</em></p> <h2>Common Patterns</h2> <p>Having worked extensively on Globalize and having delved into the code of many of the other gems mentioned above, it became clear to me that each translation solution ended up implementing many of the same things, over and over again.</p> <p>The concept of a translation cache, for example, is implemented for <a href="https://github.com/barsoom/traco/blob/ec0ea6e33a606032ddf0064bc86ca89626b0b357/lib/traco/class_methods.rb#L39">translatable columns</a> in Traco, for <a href="https://github.com/globalize/globalize/blob/5b0ff2c92e7536f908dc82a4c2a15b04be79e821/lib/globalize/active_record/adapter.rb#L4">model table translations</a> in Globalize, and for <a href="https://github.com/bithavoc/multilang-hstore/blob/48ae91643a67941d8886325cdc67d357855ee546/lib/multilang-hstore/translation_keeper.rb#L3">hstore translations</a> in Multilang Hstore.</p> <p>The concept of fallbacks (originally described by Sven Fuchs in <a href="http://svenfuchs.com/2009/7/19/experimental-ruby-i18n-extensions-pluralization-fallbacks-gettext-cache-and-chained-backend">this blog post</a>) is likewise implemented <a href="https://github.com/barsoom/traco/blob/4c7039a0f786dd0528b2c070559d44ac901f6c5d/lib/traco/locale_fallbacks.rb#L3">again</a>, and <a href="https://github.com/globalize/globalize/blob/38443bcd07da78b7b8a9433e4c6c32dd51f964a3/lib/globalize.rb#L48">again</a>, and <a href="https://github.com/cfabianski/json_translate/blob/master/lib/json_translate/translates/instance_methods.rb#L16">again</a>.</p> <p>Locale accessors, methods which fetch an attribute in a particular locale with a dedicated method like <code>title_en</code>, is yet another repeated pattern found in many gems, for example <a href="https://github.com/globalize/globalize-accessors">Globalize Accessors</a> in the case of Globalize.</p> <p>Finally, tracking of changes (i.e. <a href="http://api.rubyonrails.org/classes/ActiveModel/Dirty.html">Active Model Dirty</a> in Rails) is another common pattern, found <a href="https://github.com/globalize/globalize/blob/5b0ff2c92e7536f908dc82a4c2a15b04be79e821/lib/globalize/active_record/adapter_dirty.rb#L3">here</a> in Globalize and <a href="https://github.com/cfabianski/json_translate/blob/17419b9043ae2d69d5fb9c82e13a17c844ef7ecf/lib/json_translate/translates/instance_methods.rb#L34">here</a> in Multilang Hstore, for example.</p> <p>What is important to note here is that while each of these is implemented in their own way, the repeated patterns (caching, fallbacks, locale accessors, dirty tracking, etc.) are not in general dependent on the storage implementation strategies described earlier.</p> <p>In fact, the core pattern of fetching and updating translations using getter and setter methods, which behind the scenes map to some storage format, is itself similar across translation gems, independent of the storage strategy.</p> <p>This code from <a href="https://github.com/yabawock/trasto/blob/develop/lib/trasto/translates.rb">Trasto</a> captures this core pattern succinctly:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_localized_attribute</span>(column) define_method(column) <span style="color:#080;font-weight:bold">do</span> read_localized_value(column) <span style="color:#080;font-weight:bold">end</span> define_method(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>column<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">=</span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">do</span> |value| write_localized_value(column, value) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>(Here, <code>read_localized_value</code> fetches the value of the translated attribute <code>column</code> in the current locale.)</p> <p>This commonality between translation solutions motivated me to consider whether it would be possible to separate the implementation of translation <em>as a high-level pattern</em> from the lower-level implementation of <em>storing translations</em> (an idea also discussed <a href="https://github.com/globalize/globalize/issues/500">here</a>). This led me to start developing Mobility.</p> <h2>Introducing Mobility</h2> <p>So what is Mobility? Well, I&rsquo;ve spent most of this post explaining a bunch of storage strategies and some things which are similar about them. Mobility takes this whole idea to the next level and turns the problem of translating attributes into the problem of mapping translated attribute accessors (readers and writers) to an abstract pluggable &ldquo;backend&rdquo;, which encapsulates (and isolates) all the storage logic.</p> <p>Rather than jump into the details of how these backends work (which would require a blog post of its own), let&rsquo;s look at how we can use Mobility to quickly implement any of the storage strategies above.</p> <h3><a name="getting-started"></a>Getting Started</h3> <p>First, add <code>mobility</code> to the Gemfile:</p> <p><pre type="ruby"> gem &lsquo;mobility&rsquo;, '</p> <p>In the first real post on this blog, I described my experience <a href="/2013/10/11/hacking-globalize">hacking globalize</a>, a <a href="https://rubygems.org/gems/globalize">gem</a> that calls itself the &ldquo;Rails I18n de-facto standard library for ActiveRecord model/data translation.&rdquo;</p> <p>Globalize was the first gem that I contributed to, and the experience was very important to me as a step toward becoming a better and more knowledgeable programmer.</p> <p>I am now one of the authors of that gem, having made <a href="https://github.com/globalize/globalize/commits?author=shioyama">many commits</a> to the repository. In the time since I first contributed to Globalize, I&rsquo;ve become keenly aware of the limitations of the gem, and of problems with existing solutions for managing translations in Rails projects.</p> <p>I&rsquo;d like to look back in this post to the problem of translation and to solutions devised so far, as motivation to introduce a gem I&rsquo;ve just released called <a href="https://rubygems.org/gems/mobility">Mobility</a>. Mobility attempts to solve some core problems with the multitude of Ruby-based translation gems out there right now.</p> <h2>The Problem</h2> <p>You have an application with user-generated content. You want to translate that content into multiple languages. And &ndash; importantly &ndash; you want to do it in such a way that you don&rsquo;t need to recode all your existing presentation logic (views, mailers, controllers, etc.) to handle the new languages.</p> <p>In other words, things should <em>just work</em> &ndash; more or less the same as they do with one language, except that switching the language should tell the application to switch context to read or write content stored in that language.</p> <p>So this:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first post.title </pre></div> </div> <p>&hellip; should return the title of the post, <em>in the language I am currently viewing</em>.</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:en</span> post.title <span style="color:#777">#=&gt; &quot;Mobility (noun): quality of being changeable, adaptable or versatile&quot;</span> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span> post.title <span style="color:#777">#=&gt; &quot;Mobility(名詞):動きやすさ、可動性&quot;</span> </pre></div> </div> <p>Likewise, I should be able to write translated values the same way:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:en</span> post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> post.save </pre></div> </div> <p>Now the English title has changed, but the Japanese one has not:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first post.title <span style="color:#777">#=&gt; &quot;Translating with Mobility&quot;</span> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span> post.title <span style="color:#777">#=&gt; &quot;Mobility(名詞):動きやすさ、可動性&quot;</span> </pre></div> </div> <p>For bonus points, this solution should also work for querying attributes, so something similar to this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span>) </pre></div> </div> <p>&hellip; should work even if <code>title</code> is a translated attribute.</p> <p>For this to happen, something has to hook in to attribute accessors, to detect changes in the locale and return language-specific content accordingly. This may seem easy to do at first glance, but it gets tricky.</p> <p>Notice that what we&rsquo;re asking for is similar but different from the problem solved by localization solutions such as the <a href="http://guides.rubyonrails.org/i18n.html">Rails I18n API</a>, which provide namespaced translation lookup (generally) for interface translation, typically with static data.</p> <p>With application content, we want translations at the class level, in the form of persisted attributes of a model. Ultimately we&rsquo;re talking about a storage strategy.</p> <p>As it turns out, there are a number of different strategies that different people have come up with to solve this problem. Let&rsquo;s take a look at them.</p> <h2>Solutions</h2> <h3><a name="strategy-1"></a>Strategy 1: Translatable Columns</h3> <p>The most obvious solution to this problem is to create columns on your model table in each locale you want to support, with some convention for identifying the locale of the column.</p> <p>Typically, this is done by appending a suffix to the column name made up of an underscore plus the locale, with any dashes in the locale converted to underscores.</p> <p>So <code>title</code> becomes <code>title_en</code>, <code>title_fr</code>, <code>title_ja</code>, and so on. Adding a new locale to an application then requires that you add columns for that locale to <em>each and every translated model table</em> (see figure below.)</p> <div class="img-center"> <img alt="Column Strategy" src="/img/column_strategy.png"></a> <p class="caption">Table structure of column strategy</p> </div> <p>For a small number of models, and a small number of locales, this is a reasonable solution. It has the advantage that the column structure is easy to understand, and querying by translated attribute is straightforward:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title_en</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Mobility</span><span style="color:#710">&quot;</span></span>) <span style="color:#777">#=&gt; #&lt;Post:0x... id: 123 content_en: &quot;Mobility&quot; content_ja: &quot;モビリティ&quot;...&gt;</span> <span style="color:#036;font-weight:bold">Post</span>.find_by(<span style="color:#606">title_ja</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">モビリティ</span><span style="color:#710">&quot;</span></span>) <span style="color:#777">#=&gt; #&lt;Post:0x... id: 123 content_en: &quot;Mobility&quot; content_ja: &quot;モビリティ&quot;...&gt;</span> </pre></div> </div> <p>Some key disadvantages of this approach:</p> <ul> <li>For many models and many locales, this approach requires many migrations, each time a locale or model is added.</li> <li>Space is not used very efficiently: every record has a column for every locale, even if the locale is not used.</li> </ul> <p>Despite these disadvantages, this strategy is appropriate for applications with a fixed number of locales. For a very nice implementation of this approach, see <a href="https://github.com/barsoom/traco">Traco</a>.</p> <p><em>Mobility supports this strategy with its <code>:column</code> backend.</em></p> <h3><a name="strategy-2"></a>Strategy 2: Model Translation Tables</h3> <p>Now consider that we have an application where we have many translated models, with a potentially open-ended list of locales we want to support. A service with user-generated content offered to users around the world would be a good example of this pattern. In this case, the first strategy becomes impractical as the number of locales grows; with, say, 50 locales, and only a small number of actually translations, the vast majority of translated columns would be null. On top of which, adding new locales to all the translated models would be a pain, not to mention error-prone.</p> <p>This brings us to the next strategy, shown in the database schema below. Here, each model table is assigned its own translations table. Each row in the translations table has:</p> <ul> <li>A column for the locale (<code>locale</code>)</li> <li>A foreign key pointing to the model table (<code>post_id</code>)</li> <li>One or more columns for translations (<code>title</code>, <code>content</code>)</li> </ul> <p>With this structure, we can now <em>join</em> the translation table (e.g. <code>post_translations</code>) to the model table (e.g. <code>posts</code>) on the locale we want to get the desired result: a model with translations in that locale.</p> <div class="img-center"> <img alt="Model Translation Strategy" src="/img/globalize_strategy.png"></a> <p class="caption">Table structure of model translation strategy</p> </div> <p>In ActiveRecord, we would have an association on the model <code>Post</code> to the model <code>PostTranslation</code>, such that we can fetch translations with <code>post.translations</code> and pick out the one we want:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first translation = post.translations.find <span style="color:#080;font-weight:bold">do</span> |translation| translation.locale == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">en</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#777">#=&gt; #&lt;PostTranslation:0x... id: 123 locale: &quot;en&quot; ...&gt;</span> translation.title <span style="color:#777">#=&gt; &quot;Translating with Mobility&quot;</span> translation.content <span style="color:#777">#=&gt; &quot;In the first real post on this blog...&quot;</span> </pre></div> </div> <p>With this approach, we&rsquo;ve nicely solved some of the problems with the first strategy:</p> <ul> <li>Since translations now have their own model and table, and we only create rows in that table when we save a translation in that locale, space usage is more efficient (no wasted columns/rows for locales that have no translation).</li> <li>We do not need any migrations to add locales: simply save the translation for a new locale and it becomes available. So scalability on the &ldquo;locale axis&rdquo; is very good.</li> </ul> <p>However, there are some clear disadvantages with this approach that should be kept in mind:</p> <ul> <li>Having translations on a different table implies a lot of joining, which (if not managed very carefully) can lead to performance degradation.</li> <li>Joining can also add significant complexity to the code dealing with translations. In particular, querying on translated attributes is much more complicated than it was with the first strategy, where we queried on the model column directly.</li> </ul> <p>Note also that although this strategy scales nicely along the &ldquo;locale axis&rdquo;, it nonetheless requires a table for every translated model, and thus a migration every time we add a new model or a new translated attribute to an existing model. So this strategy may not scale well if an application has many different forms of translated content, spread across a large and growing list of models.</p> <p>This strategy is implemented by <a href="https://github.com/globalize/globalize">Globalize</a>, the gem mentioned at the beginning of this post (it is also implemented by <a href="https://github.com/francesc/rails-translate-models">many</a> <a href="https://github.com/jo/puret">other</a> <a href="https://github.com/janne/model_translations">gems</a>). For the reasons mentioned above, Globalize has become the preferred option for model translation in Rails. Like the first strategy, it is a very useful approach for many (but not all) use cases.</p> <p><em>Mobility supports this strategy with its <code>:table</code> backend.</em></p> <h3>Strategy 3: Shared Translation Tables</h3> <p>My experience working with Globalize, and in particular with scalability issues mentioned above, prompted me to consider another strategy which would scale better with model creation. I wanted a strategy which would be usable <em>immediately</em> at the code level, without having to add a table or column every time a new model or translated attribute is added.</p> <p>This led me to the third strategy, which I&rsquo;ve called the &ldquo;key value&rdquo; strategy. The basic idea is simple: rather than using a dedicated table for each translated model, instead create shared tables to store translations across all models (as shown in the figure below). Two tables are required since we will be storing two types of values: strings and text. (If you wanted to store other column types, you could add more tables for each of those types, or cast strings to some other type).</p> <div class="img-center"> <img alt="KeyValue Strategy" src="/img/key_value_strategy.png"></a> <p class="caption">Table structure of key value strategy</p> </div> <p>The key thing here is that rather than using a normal association as we did in the last strategy, where <code>post_id</code> on the <code>post_translations</code> table pointed to the <code>posts</code> table, we will instead use a <a href="http://guides.rubyonrails.org/association_basics.html#polymorphic-associations">polymorphic association</a>. By doing this, we can share the translations tables with as many models as we like, migrating just once to create the shared tables.</p> <p>The new tables have the following columns:</p> <ul> <li>A column for the locale (<code>locale</code>)</li> <li>A foreign key pointing to the translated model id (<code>translatable_id</code>)</li> <li>A column holding the model class (<code>translatable_type</code>)</li> <li>A column holding the name of the attribute being translated (<code>key</code>)</li> <li>A column holding the value of the translated attribute (<code>value</code>)</li> </ul> <p>With these tables in place, we define a polymorphic relationship on the model, something like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span> has_many <span style="color:#A60">:text_translations</span>, <span style="color:#606">as</span>: <span style="color:#A60">:translatable</span>, <span style="color:#606">class_name</span>: <span style="color:#036;font-weight:bold">TextTranslation</span>, <span style="color:#606">inverse_of</span>: <span style="color:#A60">:translatable</span> <span style="color:#777">#...</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>and a translation class:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">TextTranslation</span> <span style="color:#069">self</span>.table_name = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">&quot;</span></span> belongs_to <span style="color:#A60">:translatable</span>, <span style="color:#606">polymorphic</span>: <span style="color:#069">true</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now that we have these models in place, we can easily fetch and set translations of a model class:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first post.text_translations.find <span style="color:#080;font-weight:bold">do</span> |translation| translation.key == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span> &amp;&amp; translation.value == <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#777">#=&gt; #&lt;TextTranslation:0x...</span> <span style="color:#606">id</span>: <span style="color:#00D">123</span> <span style="color:#606">locale</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">en</span><span style="color:#710">&quot;</span></span> key=<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title</span><span style="color:#710">&quot;</span></span> value=<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> ... &gt; </pre></div> </div> <p>To add translations to a new model, we simply add a polymorphic relationship like the one above on <code>Post</code>, and we can immediately add translations to the model.</p> <p>In practice, there are some other issues that need to be handled to make this work effectively, but done correctly this strategy has some notable benefits:</p> <ul> <li>No more migrations (after the first one), so an application can quickly add new translated models.</li> <li>As with Strategy 2, no extra effort to add new locales.</li> <li>Relatively efficient in terms of storage, in the sense that we only create records for translations in locales where they actually exist. However, since each attribute translation gets a row in the table, for multiple attributes on a single translation table, Strategy 2 may be more space efficient since it stores translated attributes together in a single row.</li> </ul> <p>Now for the disadvantages:</p> <ul> <li>The join complexity described in the last section is even worse now, because we have to join <em>every attribute separately</em> when searching for matches. This is quite tricky but can be done. I&rsquo;ll probably write about this in a future post.</li> <li>Since we have each attribute stored separately (as a key/value pair) it becomes trickier to maintain our database in a consistent state. For example, if we remove a model, its translations will remain in the translations table(s) unless we explicitly remove them. Likewise, if we decide to rename a translated attribute, we may end up with translations for the previous attribute name leftover in the table.</li> </ul> <p>Despite these disadvantages, I consider this strategy to be applicable to the widest variety of situations. It is the default strategy in Mobility, implemented as the <code>:key_value</code> backend.</p> <h3>Strategy 4: Serialized Data</h3> <p>Yet another set of strategies take the approach of jamming all the translations for an attribute into a single column. The first strategy following this approach is one which serializes translations as a hash and stores it on a text column.</p> <p>At the model layer, this is pretty trivial in Rails:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span> serialize <span style="color:#A60">:title_translations</span>, <span style="color:#036;font-weight:bold">Hash</span> <span style="color:#080;font-weight:bold">end</span> post = <span style="color:#036;font-weight:bold">Post</span>.new post.title_translations = { <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">en</span><span style="color:#710">&quot;</span></span> =&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> } post.save </pre></div> </div> <p>We can now fetch translations simply by fetching the value for the key matching a given locale:</p> <div class="CodeRay"> <div class="code"><pre> post.title_translations[<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">en</span><span style="color:#710">&quot;</span></span>] <span style="color:#777">#=&gt; &quot;Translating with Mobility&quot;</span> </pre></div> </div> <p>This strategy has the obvious advantage of simplicity: all our translations are stored together on the attribute column. Like the model translations and key-value strategies, locale scalability is not an issue since we can just add new hash keys as we need them.</p> <p>In addition, no additional migrations are necessary; simply adding a text column to every model for each translated attribute is enough to support any number of translations.</p> <p>Despite these advantages, the serialized approach is not generally recommended, for the key reason that <em>querying on serialized data is not possible</em>. Indeed, Rails itself has discouraged the use of serialized attributes in favour of database-specific storage solutions (described below).</p> <p>Nonetheless, there are valid use cases for this strategy. One gem that implements this approach is <a href="https://github.com/artworklv/multilang">Multilang</a>.</p> <p><em>Mobility supports this strategy with its <code>:serialized</code> backend.</em></p> <h3>Strategy 5/6: PostgreSQL Hstore/Jsonb</h3> <p>The last two strategies, <a href="https://www.postgresql.org/docs/current/static/hstore.html">Hstore</a> and <a href="https://www.postgresql.org/docs/current/static/datatype-json.html">jsonb</a>, require the use of PostgreSQL as your database. Hstore columns store depth-1 hashes, where keys and values of the hash must be of string type. Jsonb (&ldquo;binary json&rdquo;) columns store data in JSON format, which supports a much broader range of primitives (strings, numbers, booleans and null) as well as arrays, and unlimited depth. Jsonb is generally preferred over Hstore since it is more flexible.</p> <p>Further details on these data types can be found in the links above; for our purposes, the main point is that they both store complex data types in a single column, in a form that can be <em>easily and quickly queried</em>.</p> <p>An example of such a query (on a Jsonb column) looks like this:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">WHERE</span> (posts.title <span style="color:#F00;background-color:#FAA">@</span>&gt; (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">{&quot;en&quot;:&quot;Translating with Mobility&quot;}</span><span style="color:#710">'</span></span>)::jsonb) </pre></div> </div> <p>Querying on multiple locales at once is also simple given the flexibility of the data format.</p> <p>Gems that use Hstore as a storage format for translations include <a href="https://github.com/yabawock/trasto">Trasto</a>, <a href="https://github.com/bithavoc/multilang-hstore">Multilang Hstore</a> and <a href="https://github.com/Leadformance/hstore_translate">Hstore Translate</a>. I only know of one gem (as of yet) that uses Jsonb for storage, <a href="https://github.com/cfabianski/json_translate">Json Translate</a>.</p> <p><em>Mobility supports Hstore with its <code>:hstore</code> backend and Jsonb with its <code>:jsonb</code> backend.</em></p> <h2>Common Patterns</h2> <p>Having worked extensively on Globalize and having delved into the code of many of the other gems mentioned above, it became clear to me that each translation solution ended up implementing many of the same things, over and over again.</p> <p>The concept of a translation cache, for example, is implemented for <a href="https://github.com/barsoom/traco/blob/ec0ea6e33a606032ddf0064bc86ca89626b0b357/lib/traco/class_methods.rb#L39">translatable columns</a> in Traco, for <a href="https://github.com/globalize/globalize/blob/5b0ff2c92e7536f908dc82a4c2a15b04be79e821/lib/globalize/active_record/adapter.rb#L4">model table translations</a> in Globalize, and for <a href="https://github.com/bithavoc/multilang-hstore/blob/48ae91643a67941d8886325cdc67d357855ee546/lib/multilang-hstore/translation_keeper.rb#L3">hstore translations</a> in Multilang Hstore.</p> <p>The concept of fallbacks (originally described by Sven Fuchs in <a href="http://svenfuchs.com/2009/7/19/experimental-ruby-i18n-extensions-pluralization-fallbacks-gettext-cache-and-chained-backend">this blog post</a>) is likewise implemented <a href="https://github.com/barsoom/traco/blob/4c7039a0f786dd0528b2c070559d44ac901f6c5d/lib/traco/locale_fallbacks.rb#L3">again</a>, and <a href="https://github.com/globalize/globalize/blob/38443bcd07da78b7b8a9433e4c6c32dd51f964a3/lib/globalize.rb#L48">again</a>, and <a href="https://github.com/cfabianski/json_translate/blob/master/lib/json_translate/translates/instance_methods.rb#L16">again</a>.</p> <p>Locale accessors, methods which fetch an attribute in a particular locale with a dedicated method like <code>title_en</code>, is yet another repeated pattern found in many gems, for example <a href="https://github.com/globalize/globalize-accessors">Globalize Accessors</a> in the case of Globalize.</p> <p>Finally, tracking of changes (i.e. <a href="http://api.rubyonrails.org/classes/ActiveModel/Dirty.html">Active Model Dirty</a> in Rails) is another common pattern, found <a href="https://github.com/globalize/globalize/blob/5b0ff2c92e7536f908dc82a4c2a15b04be79e821/lib/globalize/active_record/adapter_dirty.rb#L3">here</a> in Globalize and <a href="https://github.com/cfabianski/json_translate/blob/17419b9043ae2d69d5fb9c82e13a17c844ef7ecf/lib/json_translate/translates/instance_methods.rb#L34">here</a> in Multilang Hstore, for example.</p> <p>What is important to note here is that while each of these is implemented in their own way, the repeated patterns (caching, fallbacks, locale accessors, dirty tracking, etc.) are not in general dependent on the storage implementation strategies described earlier.</p> <p>In fact, the core pattern of fetching and updating translations using getter and setter methods, which behind the scenes map to some storage format, is itself similar across translation gems, independent of the storage strategy.</p> <p>This code from <a href="https://github.com/yabawock/trasto/blob/develop/lib/trasto/translates.rb">Trasto</a> captures this core pattern succinctly:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">define_localized_attribute</span>(column) define_method(column) <span style="color:#080;font-weight:bold">do</span> read_localized_value(column) <span style="color:#080;font-weight:bold">end</span> define_method(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="background-color:hsla(0,0%,0%,0.07);color:black"><span style="font-weight:bold;color:#666">#{</span>column<span style="font-weight:bold;color:#666">}</span></span><span style="color:#D20">=</span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">do</span> |value| write_localized_value(column, value) <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>(Here, <code>read_localized_value</code> fetches the value of the translated attribute <code>column</code> in the current locale.)</p> <p>This commonality between translation solutions motivated me to consider whether it would be possible to separate the implementation of translation <em>as a high-level pattern</em> from the lower-level implementation of <em>storing translations</em> (an idea also discussed <a href="https://github.com/globalize/globalize/issues/500">here</a>). This led me to start developing Mobility.</p> <h2>Introducing Mobility</h2> <p>So what is Mobility? Well, I&rsquo;ve spent most of this post explaining a bunch of storage strategies and some things which are similar about them. Mobility takes this whole idea to the next level and turns the problem of translating attributes into the problem of mapping translated attribute accessors (readers and writers) to an abstract pluggable &ldquo;backend&rdquo;, which encapsulates (and isolates) all the storage logic.</p> <p>Rather than jump into the details of how these backends work (which would require a blog post of its own), let&rsquo;s look at how we can use Mobility to quickly implement any of the storage strategies above.</p> <h3><a name="getting-started"></a>Getting Started</h3> <p>First, add <code>mobility</code> to the Gemfile:</p> <div class="CodeRay"> <div class="code"><pre> gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">mobility</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">&gt; 0.1.3</span><span style="color:#710">'</span></span> </pre></div> </div> <p>In Rails, you can use a generator to create an initializer file and migrations for the shared translation tables for the key value strategy:</p> <pre> rails generate mobility:install </pre> <p>The initializer file has the line:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Mobility</span>.config.default_backend = <span style="color:#A60">:key_value</span> </pre></div> </div> <p>With this configuration, the key value backend will be our default backend. Run the two migrations to generate string and text translation tables with <code>rake db:migrate</code>, and we&rsquo;re ready to go.</p> <p>Creating a translated attribute in a model is as easy as including (or extending) the <code>Mobility</code> module and declaring translated attributes (similar to Globalize and other gems):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span> include <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">type</span>: <span style="color:#A60">:string</span> translates <span style="color:#A60">:content</span>, <span style="color:#606">type</span>: <span style="color:#A60">:text</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we can use the attributes:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.new <span style="color:#777">#=&gt; #&lt;Post id: nil&gt;</span> post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Mobility</span><span style="color:#710">&quot;</span></span> post.title <span style="color:#777">#=&gt; &quot;Mobility&quot;</span> <span style="color:#036;font-weight:bold">I18n</span>.locale = <span style="color:#A60">:ja</span> post.title <span style="color:#777">#=&gt; nil</span> post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">モビリティ</span><span style="color:#710">&quot;</span></span> post.title <span style="color:#777">#=&gt; &quot;モビリティ&quot;</span> post.save </pre></div> </div> <p>When you call <code>post.title</code>, you will see the SQL:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">translatable_id</span><span style="color:#710">&quot;</span></span> = <span style="color:#00D">1</span> <span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">translatable_type</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Post</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_string_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">key</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">title</span><span style="color:#710">'</span></span> </pre></div> </div> <p><code>post.content</code> looks similar, but instead fetches from the <code>mobility_text_translations</code> table:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">translatable_id</span><span style="color:#710">&quot;</span></span> = <span style="color:#00D">1</span> <span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">translatable_type</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Post</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">mobility_text_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">key</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">content</span><span style="color:#710">'</span></span> </pre></div> </div> <p>It&rsquo;s also possible to query for a post with finder methods using the <code>i18n</code> scope:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.i18n.find_by(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Mobility</span><span style="color:#710">&quot;</span></span>) <span style="color:#777">#=&gt; #&lt;Post id: 1&gt;</span> </pre></div> </div> <p>The SQL for this query is a somewhat tricky table join, which is necessary since translations for this backend are stored on the shared translations table. (I&rsquo;ll leave the details of that for another blog post.)</p> <p>What is important to note here is that although the format for declaring translated attributes (with <code>translates</code>) is similar in form to other translation gems, the implementation is totally different. In particular, Mobility strives to keep the accessor and setup code for a given set of attributes <em>totally isolated</em> from attributes declared separately. This is what allows us to call <code>translates</code> twice in the model code above, one for each type of attribute (string and text).</p> <h3>Translation Options</h3> <p>Behind the scenes, when <code>translates</code> is called, Mobility creates an anonymous backend class which optionally includes modules for caching, fallbacks, and dirty accessors. Which of these modules is included can be customized by passing options:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span> include <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">type</span>: <span style="color:#A60">:string</span>, <span style="color:#606">fallbacks</span>: { <span style="color:#606">en</span>: <span style="color:#A60">:ja</span> } translates <span style="color:#A60">:content</span>, <span style="color:#606">type</span>: <span style="color:#A60">:text</span>, <span style="color:#606">fallbacks</span>: { <span style="color:#606">en</span>: <span style="color:#A60">:fr</span> } <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>I wouldn&rsquo;t want to set up fallbacks like this in a real application, but regardless, it actually works: if we call <code>post.title</code> and the title is not set in English, it will fall back to Japanese. However, <code>post.content</code> will fall back to French, since this is how we have configured the attributes.</p> <p>The <code>fallbacks</code> option is an example of a shared option which can be included for any backend. Others of this kind are <code>cache</code> (<code>true</code> by default), <code>dirty</code> (limited to classes which include <code>ActiveModel::Dirty</code>) and <code>locale_accessors</code>.</p> <p>Here is an example of adding locale accessors to attributes:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span> include <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">type</span>: <span style="color:#A60">:string</span>, <span style="color:#606">locale_accessors</span>: [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>] translates <span style="color:#A60">:content</span>, <span style="color:#606">type</span>: <span style="color:#A60">:text</span>, <span style="color:#606">locale_accessors</span>: [<span style="color:#A60">:en</span>, <span style="color:#A60">:ja</span>] <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now we can use the accessors to access attribute values in English or Japanese:</p> <div class="CodeRay"> <div class="code"><pre> post = <span style="color:#036;font-weight:bold">Post</span>.first post.title_en <span style="color:#777">#=&gt; &quot;Mobility&quot;</span> post.title_ja <span style="color:#777">#=&gt; &quot;モビリティ&quot;</span> </pre></div> </div> <p>For more details on how these options are set, see the API documentation on the <a href="http://www.rubydoc.info/gems/mobility/0.1.3/Mobility/Attributes">Mobility::Attributes</a> class.</p> <h3>Translation Backends</h3> <p>As mentioned in <a href="#getting-started">Getting Started</a>, the default backend is set in the initializer file to <code>:key_value</code>. But we can override it to use any backend, for any attribute:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span> include <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">backend</span>: <span style="color:#A60">:column</span> translates <span style="color:#A60">:content</span>, <span style="color:#606">backend</span>: <span style="color:#A60">:table</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>In order for this to work, we will need to create the necessary columns for the column backend (as described in <a href="#strategy-1">Strategy 1</a>, here for three locales):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">AddColumnsToPosts</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Migration</span>[<span style="color:#60E">5.0</span>] <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">change</span> add_column <span style="color:#A60">:posts</span>, <span style="color:#A60">:title_en</span>, <span style="color:#A60">:string</span> add_column <span style="color:#A60">:posts</span>, <span style="color:#A60">:title_ja</span>, <span style="color:#A60">:string</span> add_column <span style="color:#A60">:posts</span>, <span style="color:#A60">:title_fr</span>, <span style="color:#A60">:string</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>&hellip;and translations table for the table backend (as described in <a href="#strategy-2">Strategy 2</a>):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">CreatePostTranslations</span> &lt; <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Migration</span>[<span style="color:#60E">5.0</span>] <span style="color:#080;font-weight:bold">def</span> <span style="color:#06B;font-weight:bold">change</span> create_table <span style="color:#A60">:post_translations</span> <span style="color:#080;font-weight:bold">do</span> |t| t.string <span style="color:#A60">:locale</span> t.integer <span style="color:#A60">:post_id</span> t.text <span style="color:#A60">:content</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>With these columns and table, we can use the attributes:</p> <div class="CodeRay"> <div class="code"><pre> post.title = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> =&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span> post.content = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">&quot;</span></span> =&gt; <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">&quot;</span></span> </pre></div> </div> <p>If we now save, this is the generated SQL:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">INSERT</span> <span style="color:#B06;font-weight:bold">INTO</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title_en</span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">VALUES</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">'</span></span>) <span style="color:#B06;font-weight:bold">INSERT</span> <span style="color:#B06;font-weight:bold">INTO</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">locale</span><span style="color:#710">&quot;</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_id</span><span style="color:#710">&quot;</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">content</span><span style="color:#710">&quot;</span></span>) <span style="color:#080;font-weight:bold">VALUES</span> (<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span>,<span style="color:#00D">5</span>,<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">'</span></span>) </pre></div> </div> <p>So each attribute is treated entirely in isolation, and we can have multiple backends operating on the same model without any conflicts.</p> <p>This works as well for queries:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">Post</span>.i18n.find_by(<span style="color:#606">title</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">&quot;</span></span>, <span style="color:#606">content</span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">&quot;</span></span>) <span style="color:#777">#=&gt; #&lt;Post id: 5, title_en: &quot;Translating with Mobility&quot;, title_ja: nil, title_fr: nil&gt;</span> </pre></div> </div> <p>Note the SQL:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#B06;font-weight:bold">SELECT</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.* <span style="color:#080;font-weight:bold">FROM</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">INNER</span> <span style="color:#080;font-weight:bold">JOIN</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">ON</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_id</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">id</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">locale</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">en</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">WHERE</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">posts</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">title_en</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Translating with Mobility</span><span style="color:#710">'</span></span> <span style="color:#080;font-weight:bold">AND</span> <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">post_translations</span><span style="color:#710">&quot;</span></span>.<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">content</span><span style="color:#710">&quot;</span></span> = <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">In the first real post on this blog...</span><span style="color:#710">'</span></span> </pre></div> </div> <p>So components for each backend are combined to generate the final query. This applies to any combination of backends.</p> <h3>Mobility = Freedom</h3> <p>Why go to the trouble to implement things this way? There are a few reasons, but the driving motivation is to free the developer from having to constantly think about the details of the underlying translation implementation when working with translations. As its name implies, Mobility also strives to make it easy to change these details (e.g. change strategy) without having to rewrite an entire code base.</p> <p>This in fact goes further than the examples described here. If you look at Mobility&rsquo;s <a href="https://github.com/shioyama/mobility/blob/master/mobility.gemspec">gemspec</a>, you will see that it does not depend on activerecord or on activesupport. These are optional dependencies, and indeed the basic framework for setting up translated attributes with a backend (including caching, fallbacks and locale accessors) is totally independent of Rails. (Dirty tracking can also be used on a non-persisted class as long as it inherits from <code>ActiveModel::Dirty</code>.)</p> <p>Thanks to this isolation of dependencies, Mobility can also support other ORM, and currently does: all strategies described here are also implemented for <a href="http://sequel.jeremyevans.net/">Sequel</a> models. So this will work (provided translation tables have been created):</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#080;font-weight:bold">class</span> <span style="color:#B06;font-weight:bold">Post</span> &lt; ::<span style="color:#036;font-weight:bold">Sequel</span>::<span style="color:#036;font-weight:bold">Model</span> include <span style="color:#036;font-weight:bold">Mobility</span> translates <span style="color:#A60">:title</span>, <span style="color:#606">type</span>: <span style="color:#A60">:string</span> translates <span style="color:#A60">:content</span>, <span style="color:#606">type</span>: <span style="color:#A60">:text</span> <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Mobility detects the parent class and generates a backend instance using a class namespaced accordingly.</p> <p>Aside from keeping with the &ldquo;DRY&rdquo; philosophy of not repeating yourself, having a shared framework which works with many different configurations (ActiveRecord models, Sequel models, PORO, etc.) creates an opportunity to build something more interesting on <em>top</em> of the translation framework. I think there are some really interesting possibilities in this space, but current solutions being spread across more than a dozen different gems makes it impossible to explore those possibilities.</p> <h3>Try it!</h3> <p>So, try it and let me know what you think! I&rsquo;d be really eager to hear from anyone interested in building interesting projects using Mobility for translation. While this blog has been nearly dead for a long time, I plan to follow up this post with more details on how Mobility works, other use cases, and on further developments of the gem. Stay tuned!</p> Debugging Spree https://dejimata.com/2014/4/17/debugging-spree 2014-04-17 2014-04-17 Chris Salzberg <p><a href="https://github.com/spree/spree">Spree</a> is the most popular e-commerce platform built on Ruby on Rails. It&rsquo;s made up of a set of different gems implementing different functions of the platform: a <a href="https://github.com/spree/spree/tree/master/frontend">front</a> and <a href="https://github.com/spree/spree/tree/master/backend">back</a> end, an <a href="https://github.com/spree/spree/tree/master/api">API</a>, <a href="https://github.com/spree/spree/tree/master/cmd">command-line tools</a> and a <a href="https://github.com/spree/spree/tree/master/core">core</a> powering a cart and payment system&hellip;</p> <p><a href="https://github.com/spree/spree">Spree</a> is the most popular e-commerce platform built on Ruby on Rails. It&rsquo;s made up of a set of different gems implementing different functions of the platform: a <a href="https://github.com/spree/spree/tree/master/frontend">front</a> and <a href="https://github.com/spree/spree/tree/master/backend">back</a> end, an <a href="https://github.com/spree/spree/tree/master/api">API</a>, <a href="https://github.com/spree/spree/tree/master/cmd">command-line tools</a> and a <a href="https://github.com/spree/spree/tree/master/core">core</a> powering a cart and payment system.</p> <div class="img-right"> <a href="http://spreecommerce.com"> <img src="/img/spree.png"></a> <p class="caption">Spree: e-commerce solution for Ruby on Rails</p> </div> <p>Until recently although I had heard of it, I&rsquo;d never had the chance to deploy an application with Spree, let alone dig into its internals.</p> <p>That all changed with my new job at <a href="http://www.degica.com/">Degica</a>, where Spree is a core element of almost everything we do. Not only do we use Spree to power our carts, we&rsquo;ve also extended it, built on it, transformed it and twisted it to make it do things it was probably never intended to do.</p> <p>Putting aside the question of whether this kind of &ldquo;adaptation&rdquo; is a good thing (and you could fairly argue that in some cases it is not), as a programmer it presents an interesting challenge: to ensure that the core Spree engine and everything we&rsquo;ve built on top of it are working smoothly together. The more complex our adaptations become, the more difficult this can be.</p> <p>One bug we hit on recently brought this challenge home to me, and got me thinking about the whole process of programming and how it is affected by our environment.</p> <h2>A transaction in limbo</h2> <p>The bug was brought to our attention by our customer support team, who reported that some customers would make an order, get a confirmation email with payment instructions, pay for the product, but then never receive the expected license key(s) from us.</p> <p>When we checked our system, we saw that the payment attached to the order was in a &ldquo;failed&rdquo; state. Spree&rsquo;s cart makes extensive use of the <a href="https://github.com/pluginaweek/state_machine"><code>state_machine</code></a> gem to manage <a href="http://guides.spreecommerce.com/developer/payments.html">transitions</a> between the sequence of events in the cart checkout and payment process. The &ldquo;failed&rdquo; state in this machine is a final state with no transitions &ndash; once payment reached this state, the order was effectively killed on our side.</p> <div class="img-left" style="max-width: 440px"> <a href="http://guides.spreecommerce.com/developer/payments.html"> <img src="/img/payment_flow.png"></a> <p class="caption">Transitions between payment states in the Spree payment flow</p> </div> <p>The mystery deepened when we checked with the gateway and learned that, while the order had failed on our side, the payment had gone through on theirs without any issues. Customer support made a note of the bug on the order and manually triggered delivery of the keys as we began the process of combing our logs to figure out what had happened.</p> <p>Eventually we tracked the problem down to &ldquo;double submission&rdquo; by the client: for whatever reason (perhaps an accidental double-click or javascript bug), the customer sent two requests to our server with the same order number. Since our servers run multiple workers (<a href="http://unicorn.bogomips.org/">unicorns</a>), each request triggered a parallel request to the payment gateway with the same information.</p> <p>The gateway accepted the first request, returning a success response and triggering our system to send out the first confirmation email and mark payment as &ldquo;pending&rdquo;. But when the second request hit the gateway, it returned an error complaining that the payment had already been processed. This in turn caused the second worker to <em>overwrite</em> the previous pending state with a &ldquo;failed&rdquo; state.</p> <p>Through this sequence of events, the order was left in limbo: its payment had been accepted by the gateway but marked as failed on our system, so it was unable to transition to the &ldquo;complete&rdquo; state and deliver the requested keys.</p> <h2>The ghost in the (state) machine</h2> <p>At this point, based on log data and a bit of guesswork, we had narrowed down the <em>what</em> of the bug, but not the <em>why</em>. I found that I could actually reproduce the bug running multiple unicorns on my laptop, but the deep integration of the <code>state_machine</code> gem into Spree &ndash; not to mention the commits we had added in <a href="https://github.com/freerunningtech/spree">our fork</a> &ndash; made it hard to pinpoint where the bug was actually coming from.</p> <p>The <a href="http://guides.spreecommerce.com/developer/payments.html">Spree documentation</a> described a &ldquo;payment processing&rdquo; state, &ldquo;intended to prevent double submission&rdquo; (see diagram above). The state is triggered by the <code>started_processing</code> method, which is called in two key places: in the <a href="https://github.com/spree/spree/blob/93c2081064bd386cc48d0e758f7afd2c1fe1525b/core/app/models/spree/payment/processing.rb#L25"><code>authorize!</code></a> and <a href="https://github.com/spree/spree/blob/93c2081064bd386cc48d0e758f7afd2c1fe1525b/core/app/models/spree/payment/processing.rb#L30"><code>purchase!</code></a> methods of the <code>Spree::Payment::Processing</code> module, included in <code>Spree::Payment</code>. Payment processing is wrapped in the <code>process!</code> method in a <a href="https://github.com/spree/spree/blob/93c2081064bd386cc48d0e758f7afd2c1fe1525b/core/app/models/spree/payment/processing.rb#L4">conditional</a> which checks that it is not already in that state &ndash; this is the mechanism to prevent double submission mentioned in the documentation.</p> <p>The conditional should block double submission, but when I dug deep enough, I found that this was not happening. By placing a <code>binding.pry</code> call in the <code>authorize!</code> method, I could get two parallel processes simultaneously into the <code>processing</code> state, which by all accounts should not be possible.</p> <p>Running slower through the code with parallel pry sessions, I found that although the first process would see the payment state as <code>processing</code>, the second process would see it as <code>pending</code> until it too entered the processing state. With both processes processing, I then opened a mysql session and found that &ndash; contrary to what both pry sessions were telling me &ndash; the state in the database was still <code>pending</code>.</p> <p>The processes, in other words, were running in bubbles. Or, in database-speak, &ldquo;transactions&rdquo;.</p> <h2>Transactions in transactions</h2> <p>A <a href="https://en.wikipedia.org/wiki/Database_transaction">transaction</a> is a unit of work performed in a database which can be rolled-back at any point prior to completion. Two processes running in transactions are isolated until the point when the transaction ends and data is written to the database.</p> <p>A quick check from either pry session confirmed that the payment processing code was indeed wrapped in a transaction:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>.connection.open_transactions <span style="color:#777">#=&gt; 1</span> </pre></div> </div> <p>Before moving on, I wrote a <a href="https://github.com/shioyama/spree/commit/54f6df0b4aad720eb50a07ca9423532d29264287">quick failing test</a> isolate the problem:</p> <div class="CodeRay"> <div class="code"><pre> it <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">does not process payment within transaction</span><span style="color:#710">&quot;</span></span> <span style="color:#080;font-weight:bold">do</span> <span style="color:#777"># Make sure we are not already in a transaction</span> <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>.connection.open_transactions.should == <span style="color:#00D">0</span> <span style="color:#036;font-weight:bold">Spree</span>::<span style="color:#036;font-weight:bold">Payment</span>.any_instance.should_receive(<span style="color:#A60">:authorize!</span>) <span style="color:#080;font-weight:bold">do</span> <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>.connection.open_transactions.should == <span style="color:#00D">0</span> <span style="color:#080;font-weight:bold">end</span> order.payments.create!({ <span style="color:#A60">:amount</span> =&gt; order.outstanding_balance, <span style="color:#A60">:payment_method</span> =&gt; payment_method, <span style="color:#A60">:source</span> =&gt; creditcard}) order.next! <span style="color:#080;font-weight:bold">end</span> </pre></div> </div> <p>Now all I had to do was get this test to pass. To do this, I had to dig into <code>state_machine</code>, particularly the <a href="https://github.com/pluginaweek/state_machine/commit/fb0259ea5aa17b2924a7007d6d1e2a12f743a301"><code>1.1.2</code></a> release which Spree <code>1-3-stable</code> uses. As it turns out, there are some <a href="https://github.com/pluginaweek/state_machine/blob/fb0259ea5aa17b2924a7007d6d1e2a12f743a301/lib/state_machine/integrations/active_record.rb#L131">inline comments</a> that were a big help:</p> <div class="CodeRay"> <div class="code"><pre> <span style="color:#777"># To turn off transactions:</span> <span style="color:#777">#</span> <span style="color:#777"># class Vehicle &lt; ActiveRecord::Base</span> <span style="color:#777"># state_machine :initial =&gt; :parked, :use_transactions =&gt; false do</span> <span style="color:#777"># ...</span> <span style="color:#777"># end</span> <span style="color:#777"># end</span> <span style="color:#777">#</span> <span style="color:#777"># If using the +save+ action for the machine, this option will be ignored as</span> <span style="color:#777"># the transaction will be created by ActiveRecord within +save+. To avoid</span> <span style="color:#777"># this, use a different action like so:</span> <span style="color:#777">#</span> <span style="color:#777"># class Vehicle &lt; ActiveRecord::Base</span> <span style="color:#777"># state_machine :initial =&gt; :parked, :use_transactions =&gt; false, :action =&gt; :save_state do</span> <span style="color:#777"># ...</span> <span style="color:#777"># end</span> <span style="color:#777">#</span> <span style="color:#777"># alias_method :save_state, :save</span> <span style="color:#777"># end</span> </pre></div> </div> <p>So here was my answer: add the <code>:use_transactions =&gt; false</code> and <code>:action =&gt; :save_state</code> options and alias <code>save</code> to <code>save_state</code>. (The details of why we have to alias <code>save</code> to <code>save_state</code> are pretty involved, but the basic point is that <code>state_machine</code> runs its transition callbacks as ActiveRecord callbacks if the action is <code>save</code>, and these callbacks are called <em>within</em> the transaction. State transition callbacks on any other action are run <em>around</em> the action, so are not within the transaction.)</p> <p>With these small changes, my test passed and the processing state actually began to do its job. I&rsquo;m happy to say that those changes have now been incorporated into the master (<a href="https://github.com/spree/spree/pull/4542">PR</a>) and <code>1-3-stable</code> (<a href="https://github.com/spree/spree/pull/4499">PR</a>) branches of Spree, and I can count myself as one of the contributors to the project. (Thanks in particular to <a href="https://github.com/huoxito">@huoxito</a> who read and responded to my <a href="https://github.com/spree/spree/pull/4499#issuecomment-39213981">rambling comments</a> on the pull request.)</p> <h2>Epilogue</h2> <p>This little &ldquo;bug hunting&rdquo; expedition was quite an eye-opening experience for me. The bug itself of course taught me a lot about the internals of Spree. More than that, though, it highlighted to me how essential it is that programmers have the space and freedom to dive deep into code &ndash; as deep as necessary &ndash; to solve difficult, often unexpected problems.</p> <p>Here is a case where a silently failing feature in the core of a widely-used library emerges as an annoying double-click bug, one that could easily be ignored or patched at the surface level. To actually <em>fix</em> the bug required the patience to yank it out by its roots, delving through application code, to a dependency on a forked core library (spree), to the interaction of that library with its own dependency (state machine).</p> <p>At any point, it would be very easy for something (an impatient manager, impending deadline or unexpected event) to interrupt the bug hunt and leave the core library broken. No disaster would have resulted had that happened in this case, and I could have spared myself days of frustrating debugging.</p> <p>But in the long run, this would leave us with a broken payment system at the core of all our e-commerce applications. If anyone underestimated the importance of having reliable open-source libraries, <a href="http://heartbleed.com/">recent events</a> have provided ample evidence to convince them otherwise.</p> <p>Disregarding the bug would also have left me, the programmer, with a gaping hole in my understanding. It would have robbed me of a great chance to understand how the system works. And it would have left me with the disempowering sense that the bug was beyond my understanding, when in fact it was not.</p> <p>I&rsquo;m glad that didn&rsquo;t happen to me in this case. But I have to wonder how often it does, and how that type of failure &ndash; coming back empty-handed after hours or even days of fruitless bug hunting &ndash; must affect programmers and the code they write. The experience has given me a greater appreciation of how much a work environment can contribute to code quality. And it has taught me to <em>slow down</em>: to look at problems carefully before rushing in to solve them, to think carefully before rushing to write code.</p> <p>Here&rsquo;s hoping those lessons will lead to more successful bug hunts, and less bugs.</p> Software, the Moving Target https://dejimata.com/2013/11/27/software-the-moving-target 2013-11-27 2013-11-27 Chris Salzberg <p>Yukihiro &ldquo;Matz&rdquo; Matsumoto, creator of the Ruby programming language, gave a fascinating <a href="https://www.evernote.com/shard/s54/sh/9cd05cf6-6119-4baa-a1ae-31bd2f23ae34/16638f82bd08ac21c5ad4206d81630de">keynote talk</a> at this year&rsquo;s <a href="http://www.rubyworld-conf.org/en/">RubyWorld conference</a> the other day that left me thinking. It was a talk about Ruby, about the recent history of software, but also about much more than that&hellip;</p> <p>Yukihiro &ldquo;Matz&rdquo; Matsumoto, creator of the Ruby programming language, gave a fascinating <a href="https://www.evernote.com/shard/s54/sh/9cd05cf6-6119-4baa-a1ae-31bd2f23ae34/16638f82bd08ac21c5ad4206d81630de">keynote talk</a> at this year&rsquo;s <a href="http://www.rubyworld-conf.org/en/">RubyWorld conference</a> the other day that left me thinking. It was a talk about Ruby, about the recent history of software, but also about much more than that.</p> <div class="img-right"> <a href="http://instagram.com/p/g9YcEsCqFa/"> <img alt="Yukihiro Matsumoto at RubyWorld 2013" src="/img/rubyworld-aiming-moving-target.png"></a> <p class="caption">Yukihiro "Matz" Matsumoto at RubyWorld 2013<br /> <small>(Photo by Osamu Fukui)</small></p> </div> <p>In his trademark style, Matz recounted this history through the story of his own life. That story begins just before he first created Ruby, 20 years ago, in an era where hardware was expensive, software took years to create, and documentation was measured not in number of pages but in centimeters of thickness.</p> <h2>&#8220;20 years ago, we should have admitted our ignorance.&#8221;</h2> <p>Software was like nothing mankind had ever created before, a digital embodiment of human thought. But too much money was invested in hardware and infrastructure for companies to admit to clients that they knew nothing about how to build it. Failure was just too costly.</p> <p>Now, looking back from an era where hardware is cheap, software is commonly developed in weeks or months, not years, and documentation is measured in commits, not pages, some things become clear.</p> <p>The first is that our basic assumptions were wrong, foremost among them the assumption that &ldquo;we know what we should make&rdquo;. Were this true, software development would simply be a matter of specifying requirements, setting out a plan based on those requirements, and executing the plan.</p> <p>Except, that doesn&rsquo;t work. Or at least, it doesn&rsquo;t work very well.</p> <div class="img-left"> <a href="http://www.hsbt.org/diary/20131121.html"><img alt="Moody skies at RubyWorld 2013" src="/img/rubyworld.png"></a> <p class="caption">Moody skies at RubyWorld 2013<br /><small>(Photo by Hiroshi Shibata)</small></p> </div> <p>Before creating Ruby, Matz himself worked at a company producing enterprise software this way, with large projects executed over many years and code written to specification, <a href="https://en.wikipedia.org/wiki/Waterfall_model">waterfall</a>-style. Clients said what they wanted, specifications were drawn up, and programmers wrote code to satisfy those specs.</p> <p>And in end, something was produced. It may not have been very satisfactory, or even satisfactory at all, but there was <em>something</em> to show for the many years of time and resources invested, and that something was produced on schedule. That was the minimum requirement.</p> <p>But twenty years ago, Matz already felt that something was wrong. The big projects at his company treated software just like automobiles, or cabinets, or toasters, something to be designed, manufactured and delivered. But code is not physical, it&rsquo;s digital, and it follows a different set of laws.</p> <p>The challenge of grappling with those laws was vastly underestimated then just as it is now. The word &ldquo;software&rdquo; itself, coinded to describe the stuff being coded, evoked the wrong image: software was not &ldquo;soft&rdquo; at all, but hard. Very hard.</p> <div class="img-right"> <a href="https://commons.wikimedia.org/wiki/File:NASAComputerRoom7090.NARA.jpg"><img alt="IBM 7090 computer" src="/img/ibm-7090.png"></a> <p class="caption">Once upon a time, computing was expensive</p> </div> <h2>&#8220;Software may be the most complex thing ever created by human beings.&#8221;</h2> <p>It is fascinating to me to think that software is at once one of the most complex things we have ever created, and also something that defies our basic instincts about what it is to &ldquo;create&rdquo;. Thousands of years of blood and sweat have been spent learning to build things in the physical world, so it&rsquo;s not exactly surprising that we want to apply these lessons to the digital world.</p> <p>But they fail us, Matz said, and not in a good way &ndash; at high cost, without teaching us anything. They fail us because we are not developing software in a predictable, cartesian world like the one we live in, but in an abstract world that is unfamiliar and constantly changing.</p> <p>In this world, where our real-world instincts are often wrong, we need to try and fail often, in a repeatable way, and cheaply. Large projects of the kind Matz had been working on 20 years ago do exactly the opposite.</p> <p>The message was not exactly new, but Matz expressed it with such simplicity that it seemed self-evident. He did not profess to have the solution to the problem, or even suggest that there was one solution to the problem, but only offered some hints drawn from experience.</p> <p>&ldquo;Keep moving,&rdquo; he said, advising against the conservative &ldquo;ostrich&rdquo; strategy of putting your head in the sand in the hope that things will return to normal. (They won&rsquo;t.)</p> <p>&ldquo;Work hard to code less,&rdquo; he said. Collaborate, leverage others' work, use existing tools. In other words, stand on the shoulder of giants. This made a lot of sense to me.</p> <div class="img-left"> <a href="https://www.facebook.com/photo.php?fbid=470262379759385&set=pb.214817381970554.-2207520000.1385528208.&type=3&theater"><img alt="Yukihiro Matsumoto at RubyWorld 2013: Write great code, change the world" src="/img/rubyworld-change-the-world.png"></a> <p class="caption">Write great code, change the world</p> </div> <h2>&#8220;Write great code. Change the world.&#8221;</h2> <p>Ultimately though, this was the most powerful message. Ruby was designed to make this possible, by <a href="http://www.artima.com/intv/rubyP.html">maximizing programmer happiness</a>. But Matz, to his credit, hardly mentioned Ruby. He was thinking more broadly.</p> <p>And when thinking more broadly, something struck me as curious about the whole gathering at RubyWorld.</p> <p>Here we were, brought together by a programming language named Ruby. Ruby was designed by one programmer in his free time at a large corporation producing enterprise software to specification. Twenty years later, this programming language is used around the world, forms the foundation for one of the world&rsquo;s most popular web frameworks, and has arguably elevated Japan&rsquo;s status and visibility in the global technology sphere.</p> <p>The fact that this programmer had the free time to develop Ruby was pure luck. Certainly nobody at his company ever allocated time to develop a quirky programming language to build better software by increasing programmer happiness.</p> <p>The situation today is not so different: industry and government invest massive sums of money in <a href="http://www.fujitsu.com/global/about/tech/k/">large, expensive projects</a> destined to the dustbin of history, while largely ignoring a homegrown success story right at their doorstep.</p> <p>Now here was the creator of Ruby telling us that the existing approach to building software is wrong. Were Matz not such a gifted storyteller, this message might come off as arrogant or paternalistic. But it didn&rsquo;t.</p> <p>And yet.</p> <p>I watch everybody nodding their heads in agreement. But are they really listening?</p> <p>For a core group of hackers and entrepreneurs mostly in Tokyo, early adopters of every new technology, what he&rsquo;s saying is old news. They&rsquo;ve heard the message before &ndash; by now, many have internalized it. They&rsquo;re looking into their laptops as he speaks, acting on his advice without even hearing it.</p> <p>Matz isn&rsquo;t really speaking to them. He&rsquo;s speaking to everyone else. And he&rsquo;s trying, in his very personable, self-effacing style, to get a message across to them.</p> <div class="img-right"> <a href="https://www.facebook.com/photo.php?fbid=470262379759385&set=pb.214817381970554.-2207520000.1385528208.&type=3&theate://commons.wikimedia.org/wiki/File:Alice_%26_Red_Queen.jpg"><img alt="Red Queen and Alice in Alice in Wonderland" src="/img/red-queen.png"></a> <p class="caption">The Red Queen from Alice in Wonderland</p> </div> <h2>&#8220;If you want to get somewhere else, you must run at least twice as fast as that! &#8221;</h2> <p>Software is not just something which affects software developers. Not anymore. It&rsquo;s not just in the computers, it&rsquo;s in the automobiles and the toasters, too. It won&rsquo;t be long before it&rsquo;s in almost everything.</p> <p>And just as it is hard for software developers to build software, so it is hard for everyone else to understand software, and the potential of software. People look to lessons from the past. When those lessons do not provide useful clues, they stick their head in the sand, hoping things eventually return to normal.</p> <p>But they won&rsquo;t.</p> <p>Software is fundamentally different. It&rsquo;s a moving target. Simply to keep up takes enormous effort &ndash; to actually push the boundaries, you need to run twice as fast, think twice as hard, stretch your imagination twice as far &ndash; so most people just stick to doing the same thing, and demanding more of the same thing.</p> <p>We need to change more than just how we create software. We need to change how people understand software.</p> <p>And in that sense, watching his talk, it seemed to me that Matz the storyteller was at least as influential in today&rsquo;s world as Matz the programmer was 20 years ago. Ruby is a language that changed how programmers write code. The lesson of Ruby is a story that could change how people understand code.</p> <p>That&rsquo;s a story that I want to read.</p> Bundler, Meet Bower https://dejimata.com/2013/11/4/bundler-meet-bower 2013-11-04 2013-11-04 Chris Salzberg <p><a href="http://bundler.io/">Bundler</a> is a package management tool for ruby gems&hellip;</p> <p><a href="http://bundler.io/">Bundler</a> is a package management tool for ruby gems. It takes in a ruby file with a list of gems and version numbers, like this:</p> <div class="CodeRay"> <div class="code"><pre> source <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">https://rubygems.org</span><span style="color:#710">'</span></span> gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">rails</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">4.0.0</span><span style="color:#710">'</span></span> gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">sqlite3</span><span style="color:#710">'</span></span> gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">sass-rails</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">~&gt; 4.0.0</span><span style="color:#710">'</span></span> gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">uglifier</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">&gt;= 1.3.0</span><span style="color:#710">'</span></span> gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">coffee-rails</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">~&gt; 4.0.0</span><span style="color:#710">'</span></span> gem <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">jquery-rails</span><span style="color:#710">'</span></span> </pre></div> </div> <p>and a single command:</p> <div class="CodeRay"> <div class="code"><pre> bundle install </pre></div> </div> <p>and magically downloads all the dependencies for your project, so you don&rsquo;t have to hunt them down yourself or store them in your git repository.</p> <p>That means that when you update a dependency, rather than see all the changes to that dependency in your git history, you see this:</p> <div class="CodeRay"> <div class="code"><pre> -gem 'rails', '4.0.0' +gem 'rails', '4.0.1' </pre></div> </div> <p>Beautiful! Elegant and succinct &ndash; everything that I want to know expressed in one line.</p> <div style="text-align: center"> <img src="/img/bundler.png"> <p><small>Bundler: The best way to package your application's dependencies.</small></p> </div> <p>But now wait, what about all those pesky assets? What do we do when we want to update our version of <a href="http://backbonejs.org/">Backbone.js</a>? Well, without a package manager, we&rsquo;ll end up with an ugly commit like <a href="https://github.com/netalab/cojiro/commit/19e9afc6e5865a0b5a7fb5ec87c4425a33eb23c8#diff-01501b24a7a876fc3ee38323019212d8">this</a>.</p> <p>What? Did I make all those changes? No, but it sure looks like I did.</p> <p>And how do I know that Backbone.js depends on jQuery and Underscore.js? I just need to know. And if the dependencies change, I&rsquo;ll have to go hunt down the documentation to see what the new ones are.</p> <p>Yuck. Not elegant at all.</p> <p>It&rsquo;s amazing to me that most of the Rails community seems satisfied with this sorry state of affairs. I supect that this is one of many factors contributing to the slow uptake of Javascript frameworks like Backbone.js by Rails developers.</p> <p>If assets are first-class citizens, then they deserve to be managed as such.</p> <p>Enter <a href="http://bower.io/">Bower</a>, a package management tool for javascript libraries. It takes in a JSON file with a list of javascript dependencies and version numbers, like this:</p> <div class="CodeRay"> <div class="code"><pre> { <span style="color:#606"><span style="color:#404">&quot;</span><span>name</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">bower dependencies</span><span style="color:#710">&quot;</span></span>, <span style="color:#606"><span style="color:#404">&quot;</span><span>dependencies</span><span style="color:#404">&quot;</span></span>: { <span style="color:#606"><span style="color:#404">&quot;</span><span>backbone</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">1.0.0</span><span style="color:#710">&quot;</span></span>, <span style="color:#606"><span style="color:#404">&quot;</span><span>backbone-relational</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">0.8.5</span><span style="color:#710">&quot;</span></span>, <span style="color:#606"><span style="color:#404">&quot;</span><span>jquery</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">1.10.1</span><span style="color:#710">&quot;</span></span>, <span style="color:#606"><span style="color:#404">&quot;</span><span>jquery-timeago</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">https://raw.github.com/rmm5t/jquery-timeago/b311508ee019650200f552015a48f69de18c5857/jquery.timeago.js</span><span style="color:#710">&quot;</span></span>, <span style="color:#606"><span style="color:#404">&quot;</span><span>underscore</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">1.4.4</span><span style="color:#710">&quot;</span></span>, <span style="color:#606"><span style="color:#404">&quot;</span><span>jquery-masonry</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">http://www.fishspotr.com/jquery.masonry.js</span><span style="color:#710">&quot;</span></span>, <span style="color:#606"><span style="color:#404">&quot;</span><span>jquery-autosize</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">latest</span><span style="color:#710">&quot;</span></span> }, <span style="color:#606"><span style="color:#404">&quot;</span><span>devDependencies</span><span style="color:#404">&quot;</span></span>: { <span style="color:#606"><span style="color:#404">&quot;</span><span>sinon</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">http://sinonjs.org/releases/sinon-1.4.2.js</span><span style="color:#710">&quot;</span></span>, <span style="color:#606"><span style="color:#404">&quot;</span><span>jasmine-sinon</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">0.1.0</span><span style="color:#710">&quot;</span></span> } } </pre></div> </div> <p>and a single command:</p> <div class="CodeRay"> <div class="code"><pre> bower install </pre></div> </div> <p>and magically downloads all the dependencies for your project, so you don&rsquo;t have to hunt them down yourself or store them in your git repository.</p> <p>That means that when you update a dependency, rather than see all the changes to that dependency in your git history, you see this:</p> <div class="CodeRay"> <div class="code"><pre> - &quot;backbone&quot;: &quot;1.0.0&quot;, + &quot;backbone&quot;: &quot;1.1.0&quot;, </pre></div> </div> <p>Beautiful! Elegant and succinct &ndash; everything that I want to know expressed in one line.</p> <div style="text-align: center"> <img src="/img/bower.png"> <p><small>Bower: A package manager for the web.</small></p> </div> <p>The amazing thing is that these two package managers work beautifully in tandem: bundler for backend packages (gems) and bower for front-end assets.</p> <p>I recently added bower to our pipeline and <a href="https://github.com/netalab/cojiro/commit/12bb5a00fa4eb95fde3023e8e5cefec63edc8371">zapped 8,935 lines in one commit</a>. Those were all lines of code from assets that should never have been in our repository in the first place. All of that code is now expressed in a single <code>bower.json</code> file, used at build-time to download all packages and dependencies, just like you would with bundler.</p> <p>Now that we&rsquo;ve switched to bower, I can&rsquo;t believe that anyone would manage assets without it.</p> <hr /> <p>For those interested in trying out bower with a Rails stack, here are a few steps to get you going. First, assuming you have <a href="http://nodejs.org">node and npm</a> installed (they&rsquo;re packaged together nowadays, so installing once will get you both), you&rsquo;ll can install bower with:</p> <div class="CodeRay"> <div class="code"><pre> npm install -g bower </pre></div> </div> <p>Next, create a directory in <code>vendor/assets</code> called <code>bower_components</code>. Then add a <code>.bowerrc</code> file to your root directory to tell bower to install assets in that directory:</p> <div class="CodeRay"> <div class="code"><pre> { <span style="color:#606"><span style="color:#404">&quot;</span><span>directory</span><span style="color:#404">&quot;</span></span>: <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">vendor/assets/bower_components</span><span style="color:#710">&quot;</span></span> } </pre></div> </div> <p>You&rsquo;ll probably want to ignore that directory in your <code>.gitignore</code> file since you the whole point is not to commit these dependencies to the git repository.</p> <p>Now you&rsquo;ll need to describe your dependencies in a JSON file named <code>bower.json</code>. The key element of that file is the <code>dependencies</code> hash, which contains all your apps dependencies and their version numbers (see the example above).</p> <p>Once you have that file, just run: <code>bower install</code> to pull them all in with whatever their dependencies.</p> <p>There are a couple other caveats. You have to add <code>bower_components</code> to the <code>assets.paths</code> config to get the asset pipeline to pick it up. In <code>config/application.rb</code>, add the line:</p> <div class="CodeRay"> <div class="code"><pre> config.assets.paths &lt;&lt; <span style="color:#036;font-weight:bold">Rails</span>.root.join(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">vendor</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">assets</span><span style="color:#710">'</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">bower_components</span><span style="color:#710">'</span></span>) </pre></div> </div> <p>Also, since bower packages are each given their own directory, you will need to require them with <code>jquery/jquery</code> rather than just <code>jquery</code> as you normally would, e.g.:</p> <div class="CodeRay"> <div class="code"><pre> #= require jquery/jquery </pre></div> </div> <p>Incidentally, if you dig into <a href="https://github.com/sstephenson/sprockets">Sprockets</a>, the Rack-based asset packaging system used by the Rails asset pipeline, you&rsquo;ll see that it has in fact recently <a href="https://github.com/sstephenson/sprockets/commit/5dad425ed66eeb94223c266cbd2b1a1086d8252a">added support for <code>bower.json</code> configuration files in assets</a> and if it finds one, will let you load the package with its name rather than the full path. However, many bower packages continue to name their configuration file <code>package.json</code> instead of <code>bower.json</code>, which sprockets will not find, so this setup will not work universally. (I submitted a <a href="https://github.com/sstephenson/sprockets/pull/484">pull request</a> to change this but it was rejected. Bah.)</p> <p>Also, somewhat annoyingly, the bower assets will be given a low priority in the asset search paths, so if you have gems that include assets, those assets will take precedence. Better just to use the full path (i.e. <code>jquery/jquery</code>) and avoid any potentially confusion bugs down the line.</p> <p>That being said, using bower with a Rails stack is easy peasy and brings the full ecosystem of asset packages to your fingertips. Take the plunge, you won&rsquo;t regret it!</p> <p>Further reading: <a href="https://shellycloud.com/blog/2013/09/how-to-manage-front-end-packages-in-rails-with-bower">How to manage front-end packages in Rails with Bower</a></p>