Mastering ClojureScript Routing with Secretary and goog.History

June 24, 2015

Summary: The Google Closure Library provides a nice interface to the HTML5 History API. Coupling it with Secretary is very easy. But not all browsers support HTML5 History. In this post I'll talk about one way to make sure you have client-side routing in all browsers.


About a year ago I was working for a company of three people. Two coders and one business person. I was developing a consumer product and the other programmer was building a related B2B product. We were as agile as could be: no planning meetings, no prioritized list of features, just a shared vision. I was working in Clojure and ClojureScript and getting paid to do it.

That job eventually disappeared. But the amount of code I produced and the dark corners of features I explored still surprises me. I discovered (uncovered?) a lot of gems of ClojureScript in that time. This post is about one of them.

Update: Andre Rauh pointed out that I was using a require when I should use an import for goog.history.EventType. I fixed it in the code. Thanks!

Browser History

In a project I did about a year ago, we wanted the speed of a single page application but we wanted the back button to work and we wanted the URL to reflect where the reader was in the app. We turned to the HTML5 History API.

The HTML5 History API is an API for manipulating the browser's history without making a request to the server and loading a new page. The idea is that your Javascript application can keep all of its state in memory, but still change the URLs and keep the back button working. You have to code it up yourself, but it gives you fine-grained control over what exactly the back button does.

Luckily (and not surprisingly), the Google Closure Library has a nice way to access the History API. It's in a class called goog.history.Html5History. That gives you events about when the URL changes. We used that along with Secretary to parse, interpret, and dispatch on the URL.

The code

First, we set up our ns declaration.

(ns history.core
   [secretary.core :as secretary :refer-macros [defroute]]
   [goog.history Html5History EventType]))

We need a function that will get the current path fragment to switch on. We'll just use the path and the query string.

(defn get-token []
  (str js/window.location.pathname js/

Now we define how to instatiate the history object.

(defn make-history []
  (doto (Html5History.)
    (.setPathPrefix (str js/window.location.protocol
    (.setUseFragment false)))

Let's make a couple of simple routes. I won't go into how to make routes with Secretary in this post.

(defroute home-page "/" []
  (js/console.log "Homepage!"))

(defroute default-route "*" []
  (js/console.log (str "unknown route: " (get-token))))

Now a handler for what to do when the URL changes.

(defn handle-url-change [e]
  ;; log the event object to console for inspection
  (js/console.log e)
  ;; and let's see the token
  (js/console.log (str "Navigating: " (get-token)))
  ;; we are checking if this event is due to user action,
  ;; such as click a link, a back button, etc.
  ;; as opposed to programmatically setting the URL with the API
  (when-not (.-isNavigation e)
    ;; in this case, we're setting it
    (js/console.log "Token set programmatically")
    ;; let's scroll to the top to simulate a navigation
    (js/window.scrollTo 0 0))
  ;; dispatch on the token
  (secretary/dispatch! (get-token)))

Now we set up our global history object. We use defonce so we can hot reload the code.

(defonce history (doto (make-history)
                   ( EventType.NAVIGATE
                                       ;; wrap in a fn to allow live reloading
                                       #(handle-url-change %))
                   (.setEnabled true)))

And we will want a function to programmatically change the URL (and add to the history).

(defn nav! [token]
  (.setToken history token))

Incidentally, my links look like this in Om:

  #js {:href "/some/page"
       :onClick #(do
                   (.preventDefault %)
                   (nav! "/some/page"))}
  "some page")

That is, I try to follow the principle of graceful fallback. If Javascript fails for some reason, the href is still valid. It will make a request to the server and fetch the page. But if Javascript is working, we override it.

On the server side, I make sure that the same routes exist and that they return valid pages that include this script. When the page loads, the EventType.NAVIGATE event will fire, and so Secretary will route it. This usually means a repaint, but it's very quick and acceptable.

Add the requires:

   [om.core :as om]
   [om.dom :as dom]

And the Om code to render and get it started:

(defonce state (atom {}))

(defn cmp-link [cursor owner]
    (render [_]
       #js {:href "/some/link"
            :onClick #(do
                        (.preventDefault %)
                        (nav! "/some/link"))}
       "some link"))))

(om/root cmp-link state
         {:target (. js/document (getElementById "app"))})

When you click the link, you should see a message in the console saying it's navigating to /some/link.

A hitch

I was using this for a while when I got a message about it not working for someone. After a little investigation, it turned out they were using an older version of IE. :( IE <= 9 does not support HTML5 History. In fact, according to, only 88.2% of users have a browser with HTML5 support. That means that 12 out of every 100 visitors can't use what we just wrote.

What a lot of people would do at this point is just to use the hash-based history wrangling that 93% of the internet supports. But I wanted to do better without punishing people who upgrade their browsers.

Here's what I did: the server still serves content at URLs as normal. The routes on the client stay the same. But I used feature detection to determine if the browser supports HTML5 History. If it does support it, it runs the code above. If it doesn't, it uses the hash API. Lucky for me, Google Closure has a class called goog.History that is interface-compatible with goog.history.Html5History. So 90% of the work was done.

First, we need to add this import:

  [goog History]

goog.history.Html5History required a tiny little patch to work.

;; Replace this method:
(aset js/goog.history.Html5History.prototype "getUrl_"
      (fn [token]
        (this-as this
          (if (.-useFragment_ this)
            (str "#" token)
            (str (.-pathPrefix_ this) token)))))

I was very reluctant to do that, but it was the only solution I found to making it work consistently with the query string. Unfortunately, it was done a year ago and I don't remember the exact reason.

Now we need to modify get-token so it works in both cases. In the case HTML5 History is not supported, the token is everything after the # if we're on /.

(defn get-token []
  (if (Html5History.isSupported)
    (str js/window.location.pathname js/
    (if (= js/window.location.pathname "/")
      (.substring js/window.location.hash 1)
      (str js/window.location.pathname js/

make-history is different, too. If we don't support HTML5 History, we check if we're on /. If not, we redirect to / with the token. If we are, we construct an instance of goog.History.

(defn make-history []
  (if (Html5History.isSupported)
    (doto (Html5History.)
      (.setPathPrefix (str js/window.location.protocol
      (.setUseFragment false))
    (if (not= "/" js/window.location.pathname)
      (aset js/window "location" (str "/#" (get-token)))

Everything else is the same! You can even test out what happens without the HTML5 History API by replacing the (Html5History.isSupported) with false in both places in the code above. You'll see it start to use the # fragment when you click the link!


I figured out all of this stuff incrementally by experimentation. I wanted to share this with you because I think it's valuable. The biggest lesson to take away is that the Google Closure Library is very complete and well-built. We should lean on it as much as we can from ClojureScript.

If you're interested in learning some ClojureScript, Om, and how to make Single Page Applications, I have to recommend my LispCast Single Page Applications with ClojureScript and Om course. It's interactive with lots of animations, exercises, screencasts, and code. It's designed to get you up and running with a smooth dev process all the way through deploying code to production. It won't teach you everything about ClojureScript and Om, but it will get you over lots of the major hurdles we all encounter.

You might also like

How I made my Clojure database tests 5x faster

June 17, 2015

Summary: Setting up and tearing down a test database can be slow. Use a rolled back transaction to quickly reset the database to a known state. You can do that in an :each fixture to run each test in isolation.

On one of my projects, I wrote a bunch of tests that had to hit the database. There was a :once fixture to create all of the tables anew and an :each fixture to delete everything in the tables before each test. That ensured that I was always working with a known empty database. Overall, the tests took about 10 seconds. Woah! That's a long time. But I lived with it.

(defn clear
  "Delete all rows before and after, just for good measure.
  (cleardb db) ;; delete all rows from all tables
      (cleardb db))))

(defn setupdb [tests]
  (initdb db) ;; create the tables
      (teardown db)))) ;; drop the tables

(use-fixtures :each clear)
(use-fixtures :once setupdb)

Then I remembered a technique someone once mentioned where you use a transaction that you roll back instead of starting with a fresh db each time. It's supposed to be a lot faster.

After a little experimentation, I came up with this:

(defn clear [test]
  (sql/with-db-transaction [db db]
    (sql/db-set-rollback-only! db)
    (binding [db db] ;; rebind dynamic var db, used in tests

We open a transaction, immediately set it to rollback (which it will do when the transaction closes). Then we have to rebind our dynamic db var, which holds the current connection. And inside of that we run the test. Inside of the transaction, anything you write to the database will be available to read. When the test ends, the transaction closes and it rolls back all of the changes, leaving the database empty again.

The result? Running the tests went from 10 seconds to 2 seconds. They still start and end with a clean database, but it's done faster with a transaction.

The one gotcha that I ran into was that the PostgreSQL function now() was always returning the same time within the transaction. I had made an assumption (that was true before) that different calls would happen at different times. That assumption was no longer true inside the transaction. I had to fix the code to not rely on time.

The other part of this technique, which I did not really have to use, was that you can set up your database with test data in the :once fixture. It's costly to set up the test data, but because you're rolling back transactions, once it's set up it's quick to reset it.

If you'd like to learn more about testing in Clojure, you might be interested in my LispCast Intro to clojure.test. In it, we cover test namespaces, assertions, running your tests, and of course fixtures. It's an interactive course with exercises, screencasts, animations, and code. You should also check out the free cheatsheet below.

You might also like

TDD Workflow in Clojure using Emacs with CIDER

June 08, 2015

Summary: TDD is about fast feedback. CIDER tightens the feedback loop with quick commands for running tests and a powerful test reporting system.


I've always been into flow. One of the key aspects of flow is a short feedback loop. Test Driven Development (TDD) is partially based on flow, too. You write a new test, then you write code to satisfy the test, then you refactor. You cycle quickly with very small steps. Great for flow!

Now, I'm not going to be a pedant in this post about what is and what is not TDD. Sometimes I like to adhere to a strict discipline of TDD. And sometimes I like to code fast and loose. But as a working definition, for the purposes of this article, I'll define TDD as writing code and tests incrementally, and running the tests fairly often. The last thing you want is to have to wait for those tests to run.

Luckily, CIDER1 has been optimized to make the whole process smooth, fast, and feedbacky. You can learn about installing CIDER here. Or if you run into trouble.

What you need to know

First, you'll need CIDER connected to the REPL (usually just C-c M-j).

Besides the basic commands for switching buffers (C-c b), I use just one command a ton while I'm TDD'ing:

That will give you just the feedback you need: a green status report in the status bar if everything passes. And a new buffer with a failure and error report if it's not passing. You can do some cool stuff in that buffer, like jumping to the test definition, rerunning individual tests, and seeing diffs of actual vs. expected output. Look here for a quick reference to the available key bindings.

But mostly I'm editing code, compiling it (with C-c C-k), and running the tests (C-c ,) to make sure they pass now. Unlike with running lein test at the command line, this command only runs the tests for that specific namespace. This is usually what you want while you're editing code in the namespace. After you're done, you'll want to rerun all of the tests in a fresh JVM at the command line.


Getting a productive workflow set up is really important. It's hard on our nerves to be waiting through long feedback cycles. CIDER tightens those loops down to human scale so we can focus on the work of making the world better.

If you'd like to learn more about testing in Clojure, including how to write tests so that they work seamlessly with Leiningen and CIDER, I have to recommend my LispCast Intro to clojure.test course. It covers the most important and fundamental concepts and skills for testing in Clojure. You should not miss the free clojure.test cheat sheet below.

You might also like

  1. CIDER is an Emacs package for rocking Clojure code.

Lambda Abstraction

May 17, 2015

Summary: Lambda abstractions are always leaky, but some are leakier than others. Clojure programmers recommend keeping most of your functions pure and containing the leaks as much as possible.

Lambda abstraction refers to something we do all of the time. Let's say I have some code:

(+ 1 2)

I'm adding the number 2 to a number, in this case, 1. I could abstract that into a lambda:

(defn add2 [x] (+ x 2))

Now it's a function, which I can apply to 1. (add2 1). I can apply it to any number I want. The actual thing I am adding 2 to is abstracted away and replaced by the variable x. Lambda abstractions are just functions.

Functional programming is at its best when lambda abstractions are referentially transparent. That means that given the same arguments, a function will always return the same value. Being referentially transparent makes a software function more like a mathematical function. And that lets you reason about your code.

But there's a very real difference between software functions and mathematical functions: mathematical functions take no time or energy to "compute". They are defined abstractly, with no notion of computation. In contrast, software functions always take some time to compute. Sometimes the clearest way to write a function takes enough time that the illusion of mathematical functions is shattered. The abstraction is leaky.

So software functions are already a leaky abstraction, even if they are referentially transparent. Clojure (like most programming languages) opens the leak even further: you can put stuff that's not referentially transparent right in your function. For instance, you can write a "function" that reads from the disk or makes a web request. Making the same request twice can obviously return different values.

What most people programming Clojure recommend is to program mostly with pure functions (that means referentially transparent). You still have to deal with time, but that's way easier than dealing with the chaos of the world outside. That leaves a small bit of your code to deal with mutation, input/output, and the disk. It's still a lambda abstraction (function) but it's just leakier. Clojure simply leaves the decision up to you where to draw the line. Clojure tries to make pure functions easy, even when not everything fits into pure functions.

The takeaway of functional programming is the same recommendation: write most of your code as referentially transparent functions. The degree to which a language helps you do that is how "functional" the language is.

If you'd like to learn more about Clojure and pure functions, check out LispCast Introduction to Clojure. It's 2.5 hours of high quality video. You probably haven't seen anything like this! There's animations, exercises, characters, and screencasts. It takes you from no knowledge to a deep experience, all while having fun!

Learn Functional Programming using Clojure with screencasts, visual aids, and interactive exercises
Learn more

Infinite Application

May 17, 2015

Summary: Function application is a key concept in lambda calculus. While it is commonly expressed using parentheses in Clojure, it is also reified into a function which itself can be applied to another function.

Function application in Clojure is easily expressed with a couple of parentheses:

(foo 1 2 3)

That's the function foo applied to the arguments 1, 2, and 3. But let's say we have those numbers in a vector somewhere, and we want to call foo on them.

(def some-numbers [1 2 3])

We could manually pull out the arguments from the vector like this:

(foo (some-numbers 0) (some-numbers 1) (some-numbers 2))

Great! That should work. But more commonly you see this:

(apply foo some-numbers)

apply means take the function (the first argument) and apply it to the arguments which are in the list (the last argument). apply pulls out the values from the list internally so you don't have to.

apply is a function you'll see in many Lisps. It plays a key role in the meta-circular evaluator as defined in The Structure and Interpretation of Computer Programs (SICP). In the meta-circular evaluator, eval and apply are defined in terms of each other.

The way eval is defined classically, (foo 1 2 3) gets turned into (apply foo [1 2 3]) internally. This means that you can replace (foo 1 2 3) with (apply foo [1 2 3]) in the program without changing the meaning.

But! Since apply is a function, (apply foo [1 2 3]) is equivalent to (apply apply [foo [1 2 3]]), which is equivalent to (apply apply [apply [foo [1 2 3]]]). And you can expand that out forever. (Please don't!).

apply is something I really love about Lisp. It takes one of the main parts of lambda calculus (function application) and reifies it. Function application is available as a function, which can be passed around, composed, etc, just like any other value. I love it!

If you're in love with this idea, too, you might want to check out LispCast Introduction to Clojure. It's my video course about, you guessed it, Clojure. It takes you from absolute parenthesis-phobe to I-never-knew-it-could-be-this-way lisper by using animations, exercises, and screencasts.

Learn Functional Programming using Clojure with screencasts, visual aids, and interactive exercises
Learn more

You might also like

But the World is Mutable

May 11, 2015

Summary: The world may be mutable, but people have been using the notion of immutability to build reliable systems for a long time.

Immutability is a hard topic to breach. As a programmer used to modeling the world, you might object to immutable data structures. How do you model a changing world? Why would you choose to use immutable data structures when everything in the world is changeable?

Let's do a little thought experiment. Let's look at a nice mutable system: paper and pencil. You can write, erase, and write again. It's very convenient. It lets you correct mistakes. And when you don't need something anymore, you can easily erase it.

Now answer this: would you trust a bank that used pencils to record transactions? It would be easy: whenever you would withdraw money, they would erase the old balance and write the new balance. And if you transferred money from one account to another, they'd erase two balances and write the new ones in. It may sound great, but there's a reason banks don't use pencils: they want to be sure nothing has changed. That sounds like immutability.

Bank ledger (photo credit)

Bank ledger (photo credit)

This is a bank ledger. Each transaction gets its own line. Always done in pen. It's an example of an append-only data structure. You can answer questions about the past like "How much money was in the account at the close of last Tuesday?" by going up lines until you find the last entry for Tuesday. And you can do that because you never modify existing entries. You only add new entries on blank lines.

Medical record system (photo credit)

Medical record system (photo credit)

This is another example of an append-only data structure in the real world: medical records. Each patient gets a file that everything is added to. You never modify old records. That way, everything is recorded, even the wrong diagnoses (mistakes) of the doctor.

It turns out that traditional systems that need a high degree reliability create immutable records out of mutable paper. Even though you could in theory scratch out some data and write it again, or white it out, or find some other way to mutate the document, a mark of professionalism in the job is to discipline yourself to adhere to strict append-only behaviors.

Wouldn't it be nice if the machine took care of the discipline for us? Even though RAM and disk are mutable like paper and pen, we can impose a discipline inside of our program. We could rely on the programmer to never accidentally overwrite existing data. But that's just shifting the burden. Instead, we can build in immutability into our data structures and make a paper that cannot be overwritten.

That's how immutable data structures work. All new pieces of information are written to new locations in memory. Only when it is proven that a location is never going to be used again is it reused.

Reliable paper-based systems use immutable data. There was a time when computer memory was expensive when we had to reuse storage, so we couldn't make immutable systems. But RAM is cheap now! We should be using immutable data, just as banks have done for hundreds of years. Ready to join the 13th century?1

If you're interested in a language with a very cool set of powerful immutable data structures, probably the most cutting edge immutable data structures in any language, you're in luck! You can get LispCast Introduction to Clojure. It's a video course with animations, exercises, and screencasts that teaches you Clojure so you'll learn it and remember it.

Learn Functional Programming using Clojure with screencasts, visual aids, and interactive exercises
Learn more

You might also like

  1. The Double-entry method of accounting can trace its history back to 13th century Florence.

clojure.test cheatsheet

May 03, 2015

Summary: I made a clojure.test cheatsheet that you can get for free.

Cheatsheets are a great way to bootstrap new skills. Because at first there is so much to learn, we need all the support we can get. Instead of asking a beginner to scour through docs, hand them a good cheatsheet and they can get something accomplished. They'll remember the details eventually, with practice.

And as part of my campaign to rock the Clojure world with cool stuff, I made a cheatsheet for clojure.test, the built-in and oh-so-popular testing library for Clojure. Just sign up for the email list and it will be yours for free, along with the core.async reference sheet and the one-page Ring Spec.

You might also like

Pre-West Interview: Ron Toland

April 21, 2015


Ron Toland was gracious enough to agree to an interview. He is giving a talk at Clojure/West about building large systems in Clojure. The background to his talk is available, if you like.

Interview with Ron Toland

LispCast: How did you get into Clojure?

Ron Toland: I got into Clojure through Scheme.

There was a senior engineer at my first programming gig that kept raving about Scheme and the book Structure and Interpretation of Computer Programs (SICP). After several months of listening to him, I finally bought a copy and read through it. I was blown away by how very simple building blocks could be used to do very complicated things.

In particular, it struck me that the authors covered the map-reduce paradigm in only chapter 2 of their book; given that at the time I was reading (2009-2010) hadoop was all the rage, and the book was written in the 1980s, I started wondering "what else does lisp have that we're only now re-discovering"?

After doing some digging into currently used Lisps, I came across Clojure. Since I was already familiar with Java, it seemed like a perfectly pragmatic Lisp to me: stealing all the thunder of the JVM, but stacking on top of that the beauty and power of Lisp. It's been my favorite programming language ever since.

LC: You've been using Clojure at Sonian for 6 years. When you started, Clojure was very new. What made you choose Clojure at such an early point?

RT: I wasn't at Sonian when the decision was made to use Clojure, but I do know the background.

Basically the first version of everything was written in Ruby, which worked fine until they needed to scale things out and up to handle a larger volume of email.

They knew they needed to rebuild it in a different language, so they looked at several contenders: JRuby, Java, Clojure, Scala, even Erlang. They held a big meeting of all the devs to decide which one to go with, and it came down to Clojure or Erlang. Clojure, because several of the devs were familiar with Common Lisp or Scheme and liked it, and Erlang, because the architecture they were designing around is actually pretty close to the way you'd build it in Erlang.

Clojure won because of the JVM. It turns out that email is terrible, with multiple standards used by different programs at different times over the decades, and so processing it into something searchable is a nightmare. Java has a lot of libraries for processing email that have been built up over the years to handle most of email's craziness. Erlang doesn't (or at least didn't at the time).

Since with Clojure they could use those Java libraries directly, while if they used Erlang they'd have to write their own, they went with Clojure.

LC: What is the SAFE codebase?

RT: SAFE is the primary application we use at Sonian for email ingestion. archiving, and indexing into Elasticsearch.

LC: What is the most important thing to get right early when writing a large Clojure system?

RT: I wasn't part of the team that originally wrote SAFE, so I can't speak much about what to get right at the start of building a large system in Clojure.

I can say that a lot of large systems don't start out that way; they begin as smaller applications that grow over time, as needs arise. The trick is to keep it from becoming a mess by constantly revisiting how it's put together and looking for better -- usually more abstract, but not always -- ways to express what it needs to do.

LC: What do you wish could be better in Clojure for large systems?

RT: I'm not sure there's anything in the language itself I wish was better about Clojure for building large systems. Getting logging configured correctly always seems to be a pain; it'd be nice to have a single logging library that could hide the ugliness of the underlying java libs from us.

LC: Is there anything else you'd think would be useful for people to know before they watch your talk?

RT: I think that pretty much covers it. I'd encourage people to attend the Trapper Keeper talk today, so they can compare the two approaches.

LC: Where can people follow you online?

RT: I blog at, and my twitter handle is @mindbat.

LC: If Clojure were a food, what food would it be?

RT: If Clojure were a food, it'd be coffee. Once you use it long enough you can take it for granted, but when you don't have it you can't seem to get anything done.

This post is one of a series called Pre-West Prep.

You may like the Clojure Gazette

For more inspiration, history, interviews, and trends of interest to Clojure programmers, get the free Clojure Gazette.

Learn More

Clojure pulls in ideas from many different languages and paradigms, and also from the broader world, including music and philosophy. The Clojure Gazette shares that vision and weaves a rich tapestry of ideas from the daily flow of library releases to the deep historical roots of computer science.

Clojure/West is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and helpers) curating and organizing public information about the conference.

You might also like

Pre-West Interview: Nathaniel Smith and Ruth Linehan

April 21, 2015


Nathaniel Smith and Ruth Linehan generously agreed to do an interview about their talk at Clojure/West. They will be talking about Trapperkeeper. The background to their talk is available, if you like.


LispCast: How did you get into Clojure?

Ruth Linehan: I started using Clojure about a year and a half ago, when my team at Puppet Labs started working on a new project in it. Previously I had mostly worked in Ruby and Javascript, and while switching to Clojure required a bit of a mind shift, I really enjoy it!

Nathaniel Smith: I had never used it, but was offered a job at Puppet Labs two years ago to write it full time. I thoroughly enjoyed learning it on the 5 hour flight to my first interview there and have loved writing it ever since (in other words, I got the job :) )

LC: What does Trapperkeeper offer the Clojure developer?

RL: Great question! Trapperkeeper is great for complicated Clojure applications. It allows you to easily break up your code into modular pieces (what we call "services"), and it serves as a binder for these pieces of code. Any state a service needs can be stored in a map in the service context, rather than kept globally. It provides a consistent way for expressing the lifecycle of these services - what happens at startup and shutdown - and for managing dependencies between services. It has built in config file parsing, and it initializes a logging system using logback for you, so that you don't have to worry about this. It also has some useful test helpers that allow you to start up a Trapperkeeper application in code and test the whole system.

NS: A better way to compose the various services (Database, job queue, web server, logging, configuration...) that one has in a large, long running Clojure application. Trapperkeeper leads to more maintanable and reusable code.

LC: How does breaking everything into small services help build a large application?

NS: You have well-defined, protocol-enforced APIs for you various services and get to leverage TK's dependency management to ensure clean statups/shutdowns of all of your JVM infrastructure bits.

RL: Frequently, some of the code that's necessary to build a large application is code you (or someone else) had to write to build a different application. Breaking it down into smaller, more modular services means that you can more easily reuse that code across multiple applications. Furthermore, each service manages the state that it needs, so you don't end up with a huge mess of global state. It also makes testing much easier - you can test a service on its own, rather than having to set up the entire system to test it.

LC: Are there any similar systems in Clojure or elsewhere?

NS: A few, but none of them met our needs at Puppet Labs. We think that Immutant and Components and other similar projects are great, but they weren't a good fit for our needs.

RL: Trapperkeeper was heavily influenced by Stuart Sierra's "Clojure, Reloaded" workflow, and it has a number of similarities with other frameworks motivated by this, such as Jig and Component. Our way of turning on and off services and managing service dependencies borrows a lot from OSGi's service registry.

LC: Where can people follow you online?

RL: Nathaniel and I are both on twitter: I'm @ruthlinehan, he's @nate_smith. If you want to follow Trapperkeeper, we ("we" meaning many of the folks at Puppet Labs who work on or with Trapperkeeper) hangout on IRC on freednode in #trapperkeeper. We'd love to see folks there!

NS: I publish poetry and other esoterica at

LC: If Clojure were a food, what food would it be?

RL: Hm... maybe a melon? From the outside it seems hard and impenetrable, but once you get inside it's awesome and delicious! Also, a slice of melon looks like a parenthesis. :)

NS: Delicious whole wheat bread (ooh, or oat bread) used to make function sandwiches.

This post is one of a series called Pre-West Prep.

You may like the Clojure Gazette

For more inspiration, history, interviews, and trends of interest to Clojure programmers, get the free Clojure Gazette.

Learn More

Clojure pulls in ideas from many different languages and paradigms, and also from the broader world, including music and philosophy. The Clojure Gazette shares that vision and weaves a rich tapestry of ideas from the daily flow of library releases to the deep historical roots of computer science.

Clojure/West is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and helpers) curating and organizing public information about the conference.

You might also like

Pre-West Interview: Leon Barrett

April 19, 2015


Leon Barrett was gracious enough to agree to an interview. He is giving a talk at Clojure/West about parallel programming in Clojure. The background to his talk is available, if you like.

Interview with Leon Barrett

LispCast: How did you get into Clojure?

Leon Barrett: Actually, I wasn't even aware of Clojure until I started at The Climate Corporation 2.5 years ago. I had used Lisp a number of times in school, so it wasn't too foreign, but I'd never imagined that I'd write Lisp for a living.

LC: Clojure is great for shared-memory parallelism. Is it good for distributed programming?

LB: It can be; the same set of abstractions that are helpful locally can be helpful when distributed. For instance, there are some nice Hadoop MapReduce tools (though I happen to dislike Cascalog--it doesn't mesh with my mental model of mapreduce). Of course, you have to do some extra work to support distributed computing, but all in all it's relatively pleasant.

LC: What are your preferred distributed Clojure abstractions?

LB: You know, I don't feel qualified to opine on this too much. I mostly end up working on tasks that are fairly embarrassingly parallel, and I spend more of my time worrying about single-machine parallelism (hence my need for Claypoole). The core idea in the better Clojure distribution work I've done was, for both Storm and Hadoop, to just write simple, functional data transformations and then let the distribution framework worry about everything else.

LC: Where should someone get started with distributed programming in Clojure?

LB: I think both Parkour and Storm are very nice, though I had some issues using Parkour on Amazon's EMR. Just working with those, it's pretty easy to write distributed tools without worrying about the hard distributed parts (data movement, reliability, etc) yourself.

LC: What makes Clojure great for parallel programming?

LB: Clojure is great for parallel programming because of three things: Immutability, good core libraries, and macros.

First, the bane of parallel programming is mutable state; state makes parallel programs much harder to reason about. While it's possible to avoid shared state by writing functional programs with immutable data even in other languages, it's much easier in Clojure because all the standard libraries support it, so Clojure is easier to do parallelism with from the very beginning. Also, Clojure's parallelism features, such as future and deref, are well-designed and easy to use, making it very easy to get started with parallelism. Finally, macros make it possible to write parallel things without so much fuss; for instance, Clojure's futures (built with macros) are easier to use than Java's futures because they don't require any boilerplate. Similarly, the library core.async uses macros to add amazing parallel and asynchronous features as a library, whereas in other languages such features would need to be designed into the core language.

LC: Can you briefly explain Claypoole? What does it do? Why did you write it?

LB: I wrote Claypoole because I was dissatisfied with Clojure's built-in pmap for several reasons, including 2 in particular: First, I wanted to control the degree of parallelism, but core pmap's parallelism is determined only by the number of CPUs. Second, I wanted my pmaps to get things done as fast as possible, but core pmap is lazy and may be inefficient when the individual tasks have a high variance in duration.

Claypoole's core feature is a pmap that meets my demands--it's eager, and I can control the degree of parallelism, even across multiple simultaenous pmaps. As a bonus, it turns out that once one has good control over sharing threadpools in pmaps, it's easy to add other such features, so Claypoole also does a number of related, handy things, such as parallel for (pfor), unordered pmaps that return results in the order they're completed (upmap), and so on. I see Claypoole as a tool to provide an advanced degree of control over parallelism.

LC: Those sound really handy! Do you use reducers in your work? Where have you found them most helpful?

LB: I don't actually use reducers a lot. I used them while working with Parkour, and they were very cute there. However, in my own work I've tended to prefer Claypoole. Reducers is good because it has much less overhead than chained maps, which is great for functionally combining small tasks. However, I find that I tend map bigger tasks, so I benefit more from fine-grained control of parallelism in Claypoole than I would from the efficiency of reducers.

LC: Do you have any good background materials for people who want to do a little pre-reading/watching?

LB: Nope. My talk will start with some basics to make sure that everyone's on the same page, and my initial test audiences seemed to indicate that my intro works well for both beginners and more advanced programmers.

LC: How can people help with Claypoole?

LB: I love pull requests, and I appreciate bug reports (obviously, repeatable test cases make my life easier). But mostly, my goal with open-sourcing Claypoole was to have the community do this stuff right once, rather than having lots of people making partial reimplementations that work on just their case. So, just use it! I want people to use Claypoole rather than reimplementing it.

LC: Where can people follow you online?

LB: I'm not terribly vocal online. I guess I post to the Climate Corp engineering blog every so often.

LC: If Clojure were a food, what food would it be?

LB: One could claim that it'd be Cajun blackened catfish, because it's lean and full of flavor.

This post is one of a series called Pre-West Prep.

You may like the Clojure Gazette

For more inspiration, history, interviews, and trends of interest to Clojure programmers, get the free Clojure Gazette.

Learn More

Clojure pulls in ideas from many different languages and paradigms, and also from the broader world, including music and philosophy. The Clojure Gazette shares that vision and weaves a rich tapestry of ideas from the daily flow of library releases to the deep historical roots of computer science.

Clojure/West is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and helpers) curating and organizing public information about the conference.

You might also like