How to Write Software

Summary: Writing software alone and in a long term sustainable manner requires a lot of discipline. Upfront planning and avoidance of feature creep can turn a stressful project into a pleasure.

UPDATE: I originally wrote this piece back in 2010. I was recently reminded of it and thought it would be good to revive this article from my old blog.

A guide for the solo programmer

I have created a planning and organization process for developing software. The process works for me. I need to organize my work ahead of time or I get stuck working on unimportant details.

I have built minimalism into the process. I tend to work alone. Solo programming imposes very real limitations on the scope and completeness of software. One man can only do so much.

My process includes nothing that cannot be found in Worse is Better, The Cathedral and the Bazaar, Paul Graham’s writing, and Getting Real. These are well-written and well-known works. I thought I understood them when I read them, but I was wrong. It wasn’t until recently, after projects stalled and failed due to feature creep, that I understood the essence of the release early, release often idea. I have attempted to formalize the idea to be more actionable instead of philosophical.

Here is my hard-earned process for writing software. Note that this is not a strict algorithm. Sometimes I will apply filters from Step 2 when I am doing experiments in Step 3.

  1. Choose a purpose.

    The software must serve a single purpose. Clearly defining it will illuminate the entire software process.

  2. For each release:

    1. List features.

      List out all of the features you want in the software. Not all of these will make it into the release, but if it is not on this list, it will not make it into the release.

    2. Apply the following filters and transformations, in any order, repeatedly, until there is nothing further to filter out from the list of features.

      • Break features into subfeatures.

        If a feature can be broken into two features, do it. All features must be atomic. For instance, an easy one would be if I want to send messages by email and through a web API, that’s two features. Rule of thumb: if it can break, it’s a separate feature.

      • Eliminate unnecessary features.

        Features that are not essential to the purpose are unnecessary. Features that can be put off until the next version are unnecessary. In the above example, maybe email can wait until the next release. If it can wait, it will wait. You are trying to make a B line to release.

      • Fill in “hidden features”.

        Remember, if it can break, it’s a feature. Sometimes there are features that need to be listed because of the existence of other features. If you support two different communication channels, you need a way to choose between them. Whatever mechanism presents that choice is a feature, too. List it.

      • Simplify features.

        If a feature can be done more simply or more easily than what you originally intended, replace it. The primary way to release quickly is to simplify the release. Do you really need it to be customizable?

    3. Design and carry out experiments.

      Within your minimal set of features, there are often questions as to the best means of implementation. The experiments determine the answers to those questions.

    4. Order the features in order of dependency.

      The first feature is the one that does not depend on any others. Dependency is a complex and sneaky issue. You might say that a function needs a button to activate it. But the button is useless without the function. Which depends on which? Personally, dependency means dependent on another feature for usefulness. If I can activate a function in a very simple way (like immediately when the program starts), then that function does not depend on the button. The button depends on the function for its usefulness.

    5. Sketch out the first remaining feature on paper.

      Sketch out the code. The experiments should have instructed you in how to implement the feature. Rewrite the code repeatedly, eliminating fluff and abstraction until the code is minimal and perfect.

    6. Type the code in.

      Build it and test it.

    7. Make the code rock solid.

      Clean it up. As a solo programmer, you are never going to come back to this code and clean it up. You won’t have time. You will never have the time to hunt for bugs in this code. Make sure it works in every possible case. The code must catch every exception. It must handle the null case. The feature is not done until it is unbreakable. If you are doubting whether it’s worth your time to work this much on one feature, maybe you should eliminate the feature.

    8. Loop to Step 5 until you run out of features.

    9. Release.

The essence of this process is to follow the release early, release often philosophy by reducing the amount of development between releases. I mistakenly thought that release often meant “write code really fast” in the past. Now I understand that it means write less but more important code between releases but write it correctly.

Data > Functions > Macros. But why?

Summary: "Prefer data over functions" is a common adage in Clojure circles. It is poorly debated because it is a terse statement in generalities. A valuable perspective is that data is transparent at runtime, while functions are not. This perspective gives a firm ground for discussion and design.

There's a design rule of thumb in Clojure that says that we should prefer functions to macros and to prefer data to functions. People talk about why, people react saying that functions are data, etc. It's all true, but it's all missing the point. It doesn't get at the fundamental, structural difference. And I think the discussion breaks down because people speak in generalities and not much is made measurable and concrete. But the pregression from macros to functions to data is, in my opinion, increasing in one important aspect, and that's the availability at runtime. Discussions should hinge on whether availability at runtime is desirable, which of course needs to be determined on a case-by-case basis.


Macros in Clojure are simply not available at runtime. By definition, they are expanded at compile time. It would be hard to make sense to be passing around macros as runtime values. Try to pass a macro as an argument to a function. You can't.

Further, once a macro has been expanded, it is no longer clear that it's expanded code even came from that macro. Macros are opaque at runtime.


Functions are first-class values. They can be passed around to functions, stored in maps, etc. They are totally available at runtime to be called.

But, calling them is about all you can do with them (besides building new functions with them, like with composition). You can't even easily inspect the code that's inside them, nor get at the environment they have closed over. And that's kind of the point. The function is a useful unit of computation. It's not a unit of semantics.

A function, too, is opaque in its way. A function, at runtime, is a black box. What does it do? You can't tell. You can't even tell how the function got there. Was it a fn defined in code? Or was it the result of function composition using comp? That information is not available at runtime.


In Clojure, by data, people are really talking about the immutable data structures available in Clojure. Just to be concrete, let's narrow the definition of Clojure data down to edn.

Edn data is available at runtime. It's first-class in the way that functions are and macros are not. But it is also transparent. The structure of the data is completely available at runtime, unlike the structure of the function. This is why data is preferable.

Because data is available at runtime, you can do many things with it. Does the data describe a computation? Well, write an interpreter. Interpreters are much easier to write than compilers (macros) because the interpreter runs at runtime in the dynamic environment of the program. Compilers (and macros) separate out the two phases of compile-time and runtime. You have to keep track of the difference in your code. And Lisp is totally well-suited for writing interpreters. The first Lisp was defined in itself, for crying out loud.

And since it's just edn, it can also be manipulated using all of the tools available in the language. Maps can be assoced to. Sequences can be iterated. Transfering it over the wire is easy. Storing it to disk in a way that can be read in is easy. Try pretty-printing a function. But pretty-printing a data structure? Easy.

And, finally, the other thing about data is that it can be interpreted in different ways by different interpreters. I'm not trying to say that you might implement two algorithms for doing the same thing. What I'm getting at is that you can compute from it, or analyze it, or algebraically transform it, etc. It has become a semantic system in its own right.


Why not work with code, which is data in Clojure?

You can argue that since code is data, why should I use data structures likes maps and vectors instead of actual code, represented as lists? This is actually a very valid point, and I think this is one of the better arguments. Lisp was defined in terms of an interpreter for data structures which together provide a powerful programming model. It would be foolish to discard this power and define our own for no reason.

The best reason I can think of is that our data is usually a very restricted form of code, many times not Turing complete. Turing complete code is proven to be impossible to analyze. But our restricted data model is powerful in exactly the way we need it (for our specific problem), but not generally powerful (as in Turing complete). So we can design it to be analyzable.

If we can restrict the power to be less-than-Turing-complete, we can analyze it at runtime. If analyzing it at runtime is desirable, then it is desirable to represent it as data.

The semantics of data is vague, while code is well-defined. Why should we use data instead of code?

Ok, this is also a good point. Clojure code, in theory, is well defined. At the very least, it is defined as whatever the compiler does. Most of the time, it is well-documented and well-understood. But your ad-hoc data structure, which represents some computation, has all sorts of assumptions baked in, like what keys are valid when, that are undocumented, have poor error messages, and maybe corner cases.

Wow, such a good point. When you're designing a DSL, this is always a challenge. But, just like restricting your power to below Turing complete can make your analysis way easier, keeping your semantic model simple and well-defined is the key to making it worthwhile. If the semantic model is simple, it could be beneficial to have it available at runtime. For instance, you could create a custom editor for it. If it's just functions, that's out.

Hasn't this discussion happened many times before? I mean, Ant started off ok, but then it was its own programming language and it sucked.

This is very true. People have talked before about the difference between internal and external DSLs, and how external DSLs eventually lose because they need all sorts of conditionals and loops, which the internal DSLs had by default. In my experience, this is true.

My personal guideline is that I only prefer data after I have bound the problem to a well-understood domain. That means that I have to write the program first in code, using functions, before I realize that, yes, this could be described very succinctly and declaratively as data. This took a long time and lots of mistakes to understand. I essentially will only refactor to a data-driven approach after I already have it written and working.


I've been coding in Lisp for a long time, so I've internalized this idea of data-driven programming. It's the main idea of Paradigms of Artificial Intelligence Programming, one of the best Lisp books out there, and a big influence on me. What the guideline of "Prefer data over functions" means to me is that when it's beneficial, one should choose data, even if functions are easy to write. Data is more flexible and more available at runtime. It's one of those all-things-being-equal situations. But all things are rarely equal. Data is often more verbose and error-prone than straight code.

But there is a sweet spot where data is vastly superior. In those cases, it makes your code more readable. It is more amenable to analysis. It can be passed over the wire. When I find one of those cases, you bet I'll prefer data over functions.

I think that doing data-driven programming is one of the things Clojure excels at, even more than other Lisps, because of its literal data structure syntax. Data-driven programming is one of the deep experiences that I wish everyone could have. And it's the primary goal of LispCast Introduction to Clojure, my 1.5 hour video course filled with visuals, animations, exercises, and screencasts. Check out the preview.

Learn Functional Programming using Clojure with screencasts, visual aids, and interactive exercises
Learn more

You might also like

The Ultimate Guide to Learning Clojure for Free

Summary: There are many great resources out there for learning Clojure. Some of the resources are paid, but fortunately many of the best resources are absolutely free.

Are you interested in learning Clojure? Would you like to get started without risking your money on a book? Is there something that can help you even know whether you want to learn Clojure?

Well, yes.

I learned Clojure for free. In fact, I'm a bit of a cheapskate. For many years I would avoid spending money if there was a free alternative. I also keep an eye on resources that pop up around the internet.

I have assembled what I believe to be the best resources to start with if you want to get into Clojure but want to spend $0.


Start here if you have never programmed in Clojure before.


The ClojureBridge curriculum is a community effort that is informed by real-world workshops. It is excellent for starting.

A fun and adventurous guide through the Clojure journey.

A direct approach to bringing more people (especially underrepresented groups) to the Clojure community. The style of this book is very direct and approachable.


For beginners, these two exercise sets will give you plenty of practice with the details of the language.

Exercises you can do right in your browser.

Exercises you do with git and an editor on your local machine.

Editors and Installation

The best install guide is from ClojureBridge. If you follow this completely you'll have Light Table installed, which is a great (and open source) editor that makes Clojure development very easy.


An organized list of built-in Clojure functions and macros.

A searchable version of the cheatsheet with documentation, source code, and examples.

Workshops and Events

ClojureBridge hosts free workshops around the world to encourage diversity in the Clojure community. Go to one if you can.

There are many groups on that are interested in Clojure and functional programming. Find one in your area and get in touch.

Online Community

Always hundreds of people on and someone to help you if you get stuck.

Clojure discussion group. This contains everything from announcements of new library releases to beginners asking questions.

Ask questions on stackoverflow. They are usually answered very quickly.


A great high-level introduction to the philosophy behind Clojure.


Once you've got the basics and need to go deeper.


An open source, community-built book (also available for purchase). It includes many real-world, practical "recipes".


Once you're writing Clojure code, do these exercises and get comments on your style.

These exercises are not Clojure-specific. They are small problems but are usually tricky enough to require deep thinking. I cut my Clojure teeth on Project Euler before other systems existed.


The official documentation.

A searchable community documentation project.

Cross-references between Clojure libraries. Very interesting stuff.


At this point, you may want to think about dedicating some time to learning a different editor. Emacs is the most popular, and many people use vim.

An Emacs plugin called CIDER is the most popular way to use Emacs with Clojure. It gives you lots of functionality.

Many people use vim, so if you like it, here's a setup guide.


There is a ton of value in these free resources. But if you'd like to venture into something that is tailored for bringing up up to speed in Clojure from nothing to writing code and you don't mind paying, check out LispCast Introduction to Clojure. It's 95 minutes of screencasts, exercises, and visuals and there's nothing else like it.

Learn Functional Programming using Clojure with screencasts, visual aids, and interactive exercises
Learn more

You might also like

Pre-conj: Scheme Workshop


Well, just at the last minute, this interview was finished. Jason Hemann is the organizer of this year's Scheme and Functional Programming Workshop that is happening just before the Conj. Tickets are very affordable and still available.

The schemers are inviting everyone to attend Programming Enthusiasts Unite for Great Justice, an after-hours event at 8:30 on Wednesday, November 19 at the Mezzanine Lounge, Capitol City Brewing Company, 1100 New York Ave. I'm going to be there, so say hi!

On with the interview!


LispCast: Tell me a bit about the Scheme Workshop.

Jason Hemann: The Scheme Workshop (or Scheme and Functional Programming Workshop, to use the full title) is an annual meeting for the community of folk interested in Scheme and functional programming generally. The purpose is to report recent developments in the language, to foster collaboration among the members of the community, and for researchers, implementers, and users to present recent work.

The first workshop was held in 2000, making this the Scheme Workshop's 15th anniversary. In previous years we've hosted it with PLI, ICFP, and SPLASH, among other venues. We're excited this year to again be co-located with Clojure/conj.

LC: What is something that Clojure (as a language or as a community) can learn from Scheme?

JH: I think the Schemer's view of the world offers an interesting comparison for Clojurers. To me, Scheme is a tiny extensible core, one basic data structure (the cons pair), tail recursion, and the ability to redefine almost everything in the language. It is amazing how few primitives you can get away with using in practice. To quote from a recent standard document,

"Scheme demonstrates that a very small number of rules for forming expressions, with no restrictions on how they are composed, suffice to form a practical and efficient programming language...."

From this vantage point, Clojure feels like an extremely large and complex language (this is not to say it's the correct or only point of view, of course).

Historically, the design of Scheme has had relatively few compromises---some might say because there wasn't much language to begin with. This feels very different than Clojure, which readily compromises in order to interact with Java seamlessly. For example, Scheme implementations must handle tail calls properly. This isn't to say that Clojure makes the wrong decision--Clojure has different design goals, and unoptimized tail calls makes sense when interacting with Java. But by comparison Scheme feels very uncompromised, and it's fun to play around in such a minimal language with relatively few corner cases.

That design fits well with the Scheme workflow for writing small programs. Many Schemes exist as single standalone executables; you simply run it, and the REPL starts instantly. And then you're programming. There aren't any toolchains to build, projects to setup, or package dependencies to fight with. Compared with other languages' toolchains and workflows, there is very little to get in the way of thinking about a problem, and Scheme feels lightweight and nimble by comparison.

There are obviously tradeoffs to this design, and there can be a strain when trying to move to medium to large scale projects. Scheme has no large standard library, or standard build and versioning tools, or canonical directory layout, or anything like that. When creating large- or even medium-sized applications, the support you get in the Clojure world can be very useful. But when trying to quickly experiment with a new idea, Scheme doesn't have the encumberance of these (admittedly sometimes useful) features.

Scheme has been a useful tool for exploring and explaining new ideas, some of which have found their way into Scheme itself. Its relatively small size has made it a proving ground for language features. The Scheme approach has been to lean toward adding fewer, more highly expressive primitives. An example of this is the Scheme operator call/cc, which subsumes what would otherwise be a variety of distinct primitives.

Another idea that Schemers have explored and then added to the language is hygienic macros. While traditional unhygienic and pseudo-hygienic macros do great things for other lisps, Scheme's hygienic macros and the more recent procedural hygienic macro systems designed and developed in Scheme have really pushed the state of the art forward. The macro systems available in Scheme seem more usable than those of other lisps, and lead to more readable and more reliable code as well.

LC: So, now the opposite: what can Scheme learn from Clojure?

JH: Well, one thing I haven't mentioned yet is that Scheme is in the process of growing. The Scheme steering committee hopes both to keep Scheme as this small, lightweight language, but also make it a language able to meet "the practical needs of modern software development". The groups working on the latest language standard are producing both a spec for a small language and a spec for a large language, which is to be, by some metrics, larger than Common Lisp. The small language report is finished and was announced at last year's Scheme Workshop. This year, we will hear an update from John Cowan, the head of the large language working group. I'm excited to hear what they've been up to.

I mention this all because Clojure is a recent, comparatively large lisp designed with the practical considerations of real software development in mind. Clojure provides an example of one way to build a large language and ecosystem, and from which we can learn. The first thing that comes to mind is persistent data structures. Clojure comes out-of-the-box with a suite of persistent data structures that'd make programmers in a lot of other languages envious. The R7 large group has an immutable data structures proposal on the table, and I think that would be a nice addition. Clojure has a number of libraries that could be good additions to the large language.

Too, for a (comparatively) young language you all have---and have had---a broad, vibrant, and growing community. The conj itself is a big event. The recipe for that kind of enthusiasm and growth is something I hope the Scheme community can bring back with us.

LC: What are some resources for Clojurists who want to learn about Scheme to prep for the workshop?

JH: Clojurists are already about as well prepared as anybody to learn about Scheme. That's the wonderful thing about being close linguistic cousins. A classic introductory text that uses Scheme to talk about some really neat concepts is Abelson & Sussman's "Structure and Interpretation of Computer Programs". Another good resource is "The Little Schemer" by Friedman & Felleisen. It's cute, and as much puzzle book as it is textbook (full disclosure, my advisor is one of the authors). As far as online references,,, and have enough information to keep one busy for quite a while. Also, we've just put up on the Scheme '14 website the papers that will be presented. Those still hungry for more might be interested in the proceedings of previous years' Scheme Workshops, available at

LC: Thanks for the great interview!

This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.

That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.

Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.

You might also like

Elm FRP in Clojure core.async

Summary: Elm is an exciting FRP language. I implemented the FRP part in Clojure using core.async.

I like to read research papers. I have ever since I was in high school. I've always wanted it to be pretty easy to just translate the pseudocode for an algorithm for a paper and then have it working without any trouble.

Of course, this rarely happens. Either the pseudo-code leaves out important details, or it's expressed in terms that are not available abstractions in any language I know. So then I am left to puzzle it all out, and who has the time?

A few months ago I read the fantastic thesis by Evan Czaplicki1 where he introduces Elm. It includes a novel way to express asynchronous Functional Reactive Programming (which he calls Concurrent FRP) and a way to express GUI layouts functionally.

I highly recommend the thesis. It is very clear and readable, and points you to valuable resources.

The reason I was reading it was because I think FRP has a bright future. It's definitely got potential for simplifying the way we write interactive applications. I wanted to learn how it works so that I could build bigger castles in the sky.

Elm FRP is pretty cool. It is, first of all, very few parts. You build bigger abstractions from those small parts. It is built so that events trickle through the graph in order. In other words, they're synchronized.

But the other interesting thing is that sometimes you don't want things to be synchronized. What if one of the calculations takes a long time? You don't want the GUI to stop responding to mouse events while that slow thing is happening. Elm lets you segment the graph into different subgraphs that can propagate concurrently.

Seriously, just read the thesis.

While I was reading it, I got to this page where he lays out all of the FRP functions in Concurrent ML. It's not uncommon for entire Computer Science theses to be written, passed, and published without a line of runnable code. But here it was, in real code (no pseudocode). I don't know ML, but I do know Haskell, which is related. And I started puzzling through it, trying to understand it.

Elm FRP in Concurrent ML

Elm FRP in Concurrent ML

By flipping between the text and the code, I got it. And then I realized: everything I needed to write this was in Clojure with core.async. So I got to work.

It took a while of playing with different representations, but I got something I can use and that is pretty much a direct translation of the Concurrent ML code from the thesis. I toyed with some deeper factorizations, but I think they only muddy what's going on. And it runs in both Clojure and ClojureScript. It's available on Github.

And, of course, I needed to test my creation, so I ported the Mario example from the Elm website. You have to click on it to start it. Use the arrow keys to walk left/right and up to jump. It captures your keystrokes. Click outside of the box to get your scrolling keys back.

The code for Mario.

Now, I didn't write all of the cool GUI stuff from the second part of the thesis. I was learning Om so I decided to use that. That's why that part is so long. Writing Om components is basically as long as writing HTML + CSS.

And Elm is a language built around these FRP abstractions, so a lot of the signals are built in. I had to write event handlers to capture the mouse clicks and keyboard events. But right in the middle of mario.cljs, you'll see a very straightforward translation of the Mario example from Elm.

There are a few differences between the Elm version and the Clojure version. Clojure is untyped, so I was able to eliminate a channel used only to carry an input node id. That instead is passed along with the message on a single channel.

Also, I added a node type called "pulse" which I use for emitting time deltas for calculating physics. I'm not sure if Elm internally does something similar for its timers but it seems like the correct solution. The trick is you want to emit a zero time delta while other events are firing, and an actual time delta when the timer ticks.

Finally, instead of the graph being global as in the thesis, it's all tied to an "event stream", which is where you send the events which get channeled to the correct input signal. You can have multiple event streams and hence multiple graphs.

The implementation is very straightforward. You have to know the basics of Clojure core.async, plus mult and tap. core.async brings Clojure a new set of abstractions that let us tap into more research and learn from more languages. And that makes me happy :)

If you'd like to learn core.async, I have to recommend my own video course LispCast Clojure core.async. It teaches the most important concepts from core.async using story, animation, and exercises. It is no exaggeration to say that it was the most anticipated Clojure video when it came out. Go check out a preview.

You might also like

  1. Go say Hi to him!

Conveyor Belts: Nature's core.async Channels

A true history

Summary: Conveyor belts are strikingly similar to Clojure core.async channels. While it could be a coincidence, there is speculation that conveyor belts were influenced by a deep understanding of core.async.

Who invented the conveyor belt? No one knows for sure. Many historians believe that it was Henry Ford, seeking to mechanize the transportation of car parts from one part of the factory to another. This conservative view is plausible. Henry Ford designed much of the assembly line in his factories.

Despite the simplicity of this explanation, a small minority of researchers believe that there was a man behind Henry Ford who was the true inventor of the conveyor belt. Not much is known about him, but historians have pieced together a different story. This article is about that story.

Henry Ford in front of his invention.

Henry Ford in front of his invention.

This is a classic picture of Henry Ford standing in front of his creation. Recent advances in digital imaging enhancement have allowed us to learn a little more about the people that influenced Ford who, until recently, have lived within his shadow.

Strange man who is suspected true inventor of conveyor belt.

Strange man who is suspected true inventor of conveyor belt.

While the identity of this man is not known, historians have reassembled a story from bits and pieces of Henry Ford's journals and quotes. It is believed (by those few, brave historians) that this man is the true inventor of the conveyor belt. He looks eerily similar to the famed inventor of Clojure, Rich Hickey. However, the chronologies do not match up, so this hypothesis is easily ruled out.

Conveyor belt similarities to core.async channels

It is well understood that the conveyor belt is a poor, physical compromise on the much more reasonable abstraction of the Clojure core.async channel. As lossy as the copy is, it is still quite useful. core.async channels and conveyor belts have many properties in common.

Things can be put onto and taken off of a conveyor belt

The similarity here is uncanny. It is almost as if the inventor of the conveyor belt had some knowledge of the core.async operations.

The conveyor belt serves as a buffer

Again, it is striking that the conveyor belt can serve nearly the same purpose as a core.async channel. On long conveyor belts, the people taking things off and putting things on might not even know each other. And for things that take an unknown amount of time, you can just point to the conveyor belt and say "Wait here for the thing you need." They don't have to know when or from where it is coming.

Things can be taken off in the same order they are put on

This is something that perhaps the conveyor belt has lost. While core.async channels always maintain their own semantics for choosing which thing is taken next, conveyor belts are basically dumb and expose all of their contents. However, it is clear that you could enforce the common "first-in-first-out" queue behavior by always taking the last item from a conveyor belt.

When the conveyor belt is full, the "putter" has to wait

What happens when the conveyor belt fills up? Well, you wait for some room. Everyone has had this experience where you're at the supermarket and the cashier is taking too long to ring up your food so the conveyor belt fills up. You just have to wait with a dumb look while they look up the code for celery.

When the conveyor belt is empty, the "taker" waits

Conversely to the full belt situation, when the belt is empty the taker must wait. This is what happens when the cashier is faster than the shopper at the supermarket. There's nothing to do but wait for some groceries to reach the end of the belt.

Differences between conveyor belts and core.async channels

Though the evidence that core.async influenced the development of the conveyor belt is overwhelming, there are some differences due to the physical limitations of time and space.

Non-instantaneous transfer

Though not truly instantaneous, core.async channels do operate at the speed of light. When a value is put onto a channel, it is immediately available to be taken. Contrast this with conveyor belts, which slowly move objects along a physical belt.

Limited in length

Conveyor belts have finite length. But core.async channels can be passed around just like any other object. They, in fact, are not limited by space at all. Also, the length of a conveyor belt is equivalent to its size as a buffer. But core.async channels have properly decomplected buffer size from "distance" of travel.

No dropping/sliding semantics

It is quite curious that this important feature of core.async channels has not been copied to conveyor belts. core.async channels can be set up to drop either the newest value added or the oldest value added. Conveyor belts that do that are considered defective. The best explanation is that physical goods are so expensive compared to computing ephemeral values that it is uneconomical to trow away any items after they are made.

The jury is still out as to whether core.async influenced the development of conveyor belts. Was Tony Hoare, who invented Communicating Sequential Processes (on which core.async is based) given the secret to CSP by this same man? This hypothesis becomes more convincing when we see the immense similarities between conveyor belts and core.async channels.

Here is a picture of Rich Hickey being picked up by a friend after Strange Loop 2014.

Rich Hickey being picked up by his friend after Strange Loop

Rich Hickey being picked up by his friend after Strange Loop

If you would like to explore the mysteries of the strange and tangled similarities between conveyor belts and core.async, you might want to try LispCast Clojure core.async. It's a video series that introduces the concepts using animations, code, and exercises.

You might also like

Token Buckets with core.async

Summary: Token Bucket is a simple algorithm for rate limiting a resource. It's easy to understand because you can reason about it in terms of real-world objects. core.async makes this algorithm very clear and easy.

Token + BucketImage credits: token and bucket.

Token + Bucket1

You know you've got a good abstraction when you find lots of algorithms that are easy and clear to write using them. One algorithm that I really like because it's practical and simple is called the Token Bucket. It's used for rate limiting a resource. And it's very easy to write in core.async.

Let me digress a little. Here in the US, we have a complicated voting system. Not only do your presidential votes count only indirectly, but the counting is done with machines that vary from state to state. Some of those systems are so complicated. It's a little scary because it's hard to know how those machines work.

I mean, relatively hard. It's obviously possible to know, if you study enough. But compare it to papers in a box. I know there's all sorts of room for corruption in the papers in a box system. But it's understandable by children. Well, as long as they can count.

The token bucket is similarly simple. There's a bucket. You put tokens in it at a certain rate (for instance, once per hour). If you have a token, you can take an action (ride the subway, send a packet, read a file), then you lose the token (it's good for one ride). If you don't have a token, you can wait. Everything slows down to the rate of tokens falling into the bucket.

It's so simple. It so easily corresponds to a real-world situation you can imagine. I love that type of algorithm.

An implementation

Let's build Token Bucket in core.async.

First, we need a bucket. For that, we'll use a core.async channel with a buffer. Let's just start with size 10.

(def bucket (chan 10))

Bucket is done. Now, we need something to add tokens to the bucket at a given rate.

  (while true
    (>! bucket :token)
    (<! (timeout 1000))))

That will add one token to the bucket every second.

We can rate limit an existing channel by forcing it to take a token before values get through.

(defn limit-rate [c]
  (let [out (chan)]
      (loop []
        (let [v (<! c)]
          (if (nil? v) ;; c is closed
            (close! out)
              (<! bucket) ;; wait for a token
              (>! out v)

Corner cases

Ok, it's not that simple. There are two corner cases.

  1. What happens when nobody takes a token out of the bucket? Do you keep putting coins in?

The answer is yes. In the next hour, there are two tokens, so two can come through. But then . . .

  1. What do you do when the bucket gets really full and a whole bunch of people take tokens out at the same time?

Well, you let them all through. One token, one ride. There's no other coordination. And that means it's really important to choose the size of your bucket.

The number of tokens your bucket can hold is called the burstiness. It's because when the bucket is full, you could get a rampage of people trying to get on the subway. How many people should be allowed through at that point? The burstiness is the maximum that should come through at a time.

We have our two parameters: the rate and the burstiness. Let's incorporate all of that.

(defn limit-rate [c r b]
  (let [bucket (chan b) ;; burstiness
        out (chan)]
      (while true
        (>! bucket :token)
        (<! (timeout (int (/ 1000 r)))))) ;; rate
      (loop []
        (let [v (<! c)]
          (if (nil? v) ;; channel is closed
            (close! out)
              (<! bucket) ;; wait for a token
              (>! out v)

The burstiness is taken care of in the size of the buffer. The buffer will fill up if no one takes a token. Since we're using blocking buffers, putting tokens into a full bucket will block until something takes a token--exactly as the algorithm describes.

Well, that's it. It's easy. And now we have a way to limit the rate of a channel. We can use it to limit the rates of other things, too, like a function call or access to a database. I use it to rate limit an API.

core.async makes this algorithm nice and easy to use. There's a library that does this for you in a very convenient package. It's called Throttler. Bruno Vecchi has done the work of making this work well as a library. If you'd like to learn core.async, I recommend my LispCast Clojure core.async videos. It's a gentle and fun introduction to a great topic. You will learn everything you need to write Token Bucket and more!

You might also like

  1. Image credits: token and bucket.

Pre-Conj Interview: Steven Yi


Steven Yi is next up on the interviews. He will talk at Clojure/conj about music systems in Clojure. Read the background to his talk.


LispCast: How did you get into Clojure?

Steven Yi: I think I came across it online about two years ago when it started to get a bit of a buzz. I had long wanted to learn a Lisp, but Common Lisp and Scheme never really clicked when I tried. Clojure was different though, and since I do a fair amount of Java programming, the possibility to integrate Clojure with my JVM music software Blue helped motivate me to get into it.

LC: Can you briefly describe Blue, Pink, and Score?

SY: Blue is a music composition environment, written in Java using the Netbeans RCP. It is a graphical environment with a score timeline, graphical instruments, mixer and effects, and more. It uses Csound as its audio engine and signal processing system. (Disclosure: I'm also one of the core developers of Csound.)

Pink is a Clojure library for building music systems. It features a low-level audio engine, higher-order events, and includes signal generation and processing functions.

Score is a Clojure library for higher-level music work, particularly for generating notes and scores using various strategies. It is designed to be generic and usable with multiple backends. Using Score and Pink together can be particularly effective and expressive, having one language for the full musical stack.

LC: Overtone has been getting a lot of attention lately. How do Pink and Score compare to Overtone?

SY:Ah yes, certainly the elephant in the room. :) I'll certainly address comparisons to Overtone in more detail during the talk. In a nutshell, Overtone builds upon SuperCollider. Of those two, Pink would compare more to SuperCollider than to Overtone, as it is concerned with low-level audio engine design. Score deals with higher-level abstractions in music, many of which are covered in Overtone. However, because of the full-stack Clojure design between Pink and Score, Score can interact with Pink in ways Overtone/SuperCollider can not. Pink/Score are currently not as mature nor as performant as Overtone/SuperCollider. There's a lot of overlaps between the various systems, and use cases and requirements for a user's project will guide them to one system or the other.

Hopefully that's enough to get going! I think the breakdown of music systems design I'm planning for the talk should hopefully make things clear.

LC: Is there any resource that could help someone new to Clojure, or new to music systems, get the most out of your talk?

SY: think it's a tough question about music systems as I don't know of any resources out there that really introduce the subject. Having some experience with Overtone would certainly be useful. I can say I'll probably mention SuperCollider, Csound, Common Lisp Music, Nyquist, and Common Music during the talk:

and perhaps Incudine and OpusModus:

But that's a long list. :P

Actually, the Wikipedia link to Music-N would probably be a good place to get going.

LC: Where can people follow you on your adventures?


LC: Thanks, Steven, for a great interview! It was a pleasure.

SY: And thank you too!

This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.

That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.

Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.

You might also like

Pre-Conj Interview: Bozhidar Batsov


Bozhidar Batsov agreed to an interview about his talk at Clojure/conj about CIDER, the Clojure Emacs IDE. Read the background to his talk.


LispCast: How did you get into Clojure?

Bozhidar Batsov: I’m very fond of Lisps and I love exploring new programming languages. In 2009 Common Lisp was my favourite Lisp dialect and I was often solving small programming exercises in it. I wanted to do some “real work” in CL, but the ANSI standard had left out basic things like networking, GUI, etc. and relying on distribution-specific extensions didn’t feel right to me. I was working as a Java developer at the time, so when I stumbled upon a news about the release of Clojure 1.0 I was super excited. A practical Lisp seemed like a dream come true to me! I immediately started playing with Clojure and enjoyed it immensely. I was impressed by the immutable data structures, the seamless Java interop and the powerful concurrency primitives. Along the way I developed a deep appreciation for the merits of functional programming and explored a few other functional programming languages.

LC: What is CIDER and how did you wind up as its maintainer?

BB: CIDER is an interactive Clojure development environment for Emacs. It’s pretty similar to SLIME (which I consider CIDER’s spiritual ancestor) for Common Lisp and Geiser for Scheme/Racket. People who are not familiar with Emacs should think of CIDER as a Clojure IDE plugin, that provides a ton of useful (Emacs-flavoured) features like interactive code evaluation, code completion, documentation lookup, code navigation and many others.

CIDER is built on top of the popular nREPL server and replaces SLIME + swank-clojure as the standard way to write Clojure programs in Emacs. A lot of the functionality in implemented in terms of reusable nREPL middleware, which makes the project pretty accessible to Clojurists who are afraid of Emacs Lisp. Portions of CIDER’s middlewares are used in other dev tools like vim’s fireplace and Gorilla REPL.

I was an early adopter of CIDER (which was named nrepl.el back then) as I was pretty excited to use a development environment tailored specifically for Clojure (unlike SLIME). The project felt pretty barebone (compared to SLIME) and my Emacs skill were pretty decent, so I started contributing features and improvements to it (I think the first major thing I contributed were the compilation error highlights). About one year after the project started, the original maintainer Tim King had to step back and handed over the maintenance to me. I’ve been hacking on CIDER pretty actively ever since and I have epic plans for its future! :-)

LC: What is the best resource for someone who wants to get started with CIDER?

BB: I guess that for people familiar with Emacs the best resource would be the project's README itself.

Users who are new to Emacs should probably start with some gentler tutorial like or perhaps

While not strictly necessary, I’d encourage people to also have a look at cider-nrepl’s README.

A more extensive user guide (perhaps in texinfo format) and a few screencasts are on the roadmap, but I cannot promise when/if they’ll become a reality. Lots of bugs to squash and features to add prevent me from spending more time on improving the documentation.

LC: What is the best resource for someone (like me) who has migrated from SLIME and might not know all the cool stuff that's in there now?

BB: The README lists all prominent config options and commands (or at least I think it does). The changelog is also a great place to discover new features.

There’s a wiki entry on differences with SLIME, but it hasn’t been updated in a while, so people won’t learn much from it. Guess I should find some time to update it!

I’m always planning to write blog posts and do short screencasts for important new features, but unfortunately I never find the time to do so. Who knows, maybe next year...

LC: Where can people follow your adventures online?

BB: There are plenty of opportunities for people to stalk me online. :-) Here are some of the best options:

LC: One last question: If Clojure were a zoo animal, what would it be and why?

BB: A bear! Bears are (arguably) cute, smart and powerful. They are good at pretty much everything - they can run very fast, climb trees and swim. Does this remind you of a certain programming language?

LC: This was a great interview and I enjoyed it!

This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.

That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.

Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.

You might also like

Pre-Conj Interview: Paul deGrandis


Paul deGrandis gave a nice interview about his talk at Clojure/conj about data-driven systems. Read the background to his talk.


LispCast: How did you get into Clojure?

Paul deGrandis: I was a very early Clojure adopter. I came from Python (and before that a mix of C/C++/Java/Python), but I had always used Common Lisp to prototype ideas. Once I saw Clojure publicly announced, I gave it a shot and instantly identified with the Clojure Trinity: Simplicity, Power, and Focus. I began using it in all of my proof-of-concept work, and as I became confident in the language, integration with the platform, and my own skill set, I also started putting it into production.

LC: Can you define Data-Driven systems briefly?

PdG: Data-driven systems capture their interactions and operations as values - they are one possible conclusion of programming with values. We see this done in Datomic - queries are expressed as data. This concept is easily mapped to other systems - services, client-side applications, and more. Note that is more than just, "data all the things," this is, "data all the thing's things." Another example: Pedestal routes are represented as data ("data all the things"), but imagine if we captured the expression of an entire service as data. Imagine the properties that naturally fall out from that design decision. My talk walks through a real project that pushed this design decision as far as possible, and the steps that other developers can follow to do the same for their own systems.

LC: At Cognitect, they talk about Data > Functions > Macros. Is that what your talk is about?

PdG: My talk, at a very high level is about "Data > Functions > Macros," but it's mostly about modeling entire systems (not just interfaces) as data, why you'd want to, and why it's hard. It definitely challenges the previous notions (established by existing tooling) of what the conclusions of "programming with values" actually are (or could/should be). Tangentially, I'll talk about DSLs, but only in regards to constraining your domain as a means to increase expressiveness.

LC: I'm a big fan of working with data. But I've also seen its dark side. I've seen it pushed too far, or in the wrong area, to where it would have been better to use code. Have you noticed this? How do you avoid that?

PdG: I have indeed noticed it and it's a topic within my talk. Two crucial pieces of any great design (in software or otherwise) are the external constraints imposed by the problem, and the internal constraints imposed by the designer. Both are important, but it seems common that people overlook self-imposing constraints - and one needs them to achieve a really great design. I don't necessarily agree that capturing a system in data can be pushed too far, but I think it can be done without constraint or design intent - which produces a useless system. The crux is knowing that limitation and working within it. Outside of those limitations and constraints, one should fall back to foundation - regular Clojure code, etc.

LC: What is one thing you want someone to be able to do after they watch your talk?

PdG: I want someone to come away with a set of techniques to envision, constrain, and design data-driven systems. I want to create a spark that encourages Clojure developers to look beyond the initial merits of programming with values, and see further-reaching impact within their own systems - how to turn our ecosystem into a set of super powers.

I hope to provide a compelling example of how this approach plays out in production systems, and the general metrics and project impact it has.

LC: What resources would you recommend to a beginner so they could maximize their understanding of your talk?

PdG: My talk touches upon a lot of themes covered in some inspiring and informative talks: Tim Ewald's keynote from last year's Conj (Programming with Hand Tools), Rich Hickey's GOTO conference Keynote (The value of values), and previous Conj talks. I'm still searching for literature and books that I like on the topic, but I haven't hit any that I'd strongly recommend.

LC: Where can people follow your adventures online?


LC: If Clojure rode an animal, what animal would it ride?

PdG: A very serious pony.

LC: Thanks for a great interview. It was informative.

PdG: Thanks a ton! I really appreciate all of this!

This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.

That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.

Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.

You might also like