Nil Punning (Or Null Pointers Considered Not So Bad)

Null pointers are considered by their inventor to be a huge mistake. Clojure inherits its null pointer, called nil in Clojure, from the JVM. In contrast to Java1, Clojure seems to embrace the null pointer. In this post, I'd like to explore how Clojure uses the null pointer in what is often called nil-punning.

Nil-punning has its roots in the very first Lisps, where nil was both false (the boolean value) and the end of a list (the empty list). It was also often used to represent "no answer", as in what is the first element of an empty list. It is called punning because you can use it to mean different things in different contexts.2

In Clojure, nil, as a value, is nearly void of meaning. And it is all pervasive, because it can be returned from any Clojure function or Java method.

Let's go through that last part bit by bit.

  1. It is a value. Java made the mistake of making null a lack of object even though it was pointed to by an object pointer. You can't call methods on it. It is not an object. It has a weird nameless type. Clojure did not make this mistake. It is a first-class value and type3, meaning it can be compared to other values, it can implement protocols, and be used as the key or value of a map, etc. Using nil where it doesn't make sense in Clojure code is usually a type error, not a NullPointerException, just as using a number as a function is a type error.
  1. It is nearly void of meaning. It means "no answer", but not much more. Because of this, it can take whatever form fits the context. With proper wisdom in choosing what form it takes, nil can become an asset instead of a liability. Clojure takes nil-punning to an extreme.

nil can be many things. To name but a few, nil plays false as a boolean. It plays the empty seq as a seq4. It plays the empty map as a map. Because nil has a role to play in most of the major abstractions of core Clojure, it rarely leads you into an error situation. An unexpected nil can surprise a good programmer, just as much as an unexpected Nothing from a Haskell function can bewilder even the most experienced Haskeller. 5 Finding out where a nil came from is the hardest problem that nils present.

  1. It is all-pervasive. nils are normal parts of Clojure programs. They are not anomalous as in Java, where you often have to check it everywhere. This means it is always on the experienced programmer's mind. nils flow like water through s-expressions.

first has nothing to return if the seq is empty, and so it returns nil. Because nil is a seq, first works on nil, and returns nil. 6. rest works on nil as well, because nil is a seq.

These examples show the best of nil-punning. When nil-punning works right, nils are expected and they give the expected results.

nil is everywhere, but it can be used mostly everywhere as well--without error and often with exactly the desired result. There are many abstractions that nil does not participate in (for instance IFn, which is Clojure's interface for things that can be called like functions). In these places, nil can present a problem--a problem of type, the same as if you tried to call a number as a function.

The best thing to do, in my experience, is simply to wrap the expression in a (when ) to catch the nil cases, if appropriate, while also preserving it. Otherwise, perhaps letting the Exception bubble up is the best answer. If you got a nil where you couldn't use one, the stack trace is probably your best clue to where it came from.

After a bit of experience with Clojure, I rarely have difficult problems with out-of-place nils in pure Clojure code. However, there is often some Java interop--namely, calling Java methods directly--that will cause a NullPointerException if the object of the method call is nil. In these cases, wrapping a Java method call in a (when ) is often appropriate. But sometimes not, and the NullPointerException is welcome.

There are some decisions in Clojure that I think make poor use of nil-punning. These places actually make working with Clojure more difficult than they need to be. For instance, (str nil) is the empty string. Printing this out prints nothing--a form of silence, which is rarely what you want so you have to check for nils in these cases. But nil is not the empty string, like it is the empty seq. (clojure.string/trim nil) throws a NullPointerException. This is inconsistent behavior. When nil acts inconsistently, nil-punning does not work right. nils need to be checked. In the worst cases, nils fail silently. While I have learned to deal with these situations, they are a wart on the language. The fact that nils are so common does help surface the bugs sooner. A small consolation.

Let me make it clear: null pointers are still a costly problem in Clojure. But I can make a claim similar to what Haskellers claim about the type system: nil-punning eliminates a certain class of errors. A fortuitous set of decisions in Clojure has reduced the magnitude of the problem. And some decisions have made the problem worse by hiding it. In general, I find that by embracing nil-punning, my code gets better.

  1. I don't mean to pick on Java alone. I just wanted to be specific.

  2. Note that this is very different from weak typing as you find in Javascript or Python. Nil-punning is more like polymorphism.

  3. The type of nil is nil

  4. Clojure's core is built on several small, powerful abstractions. The most prominent abstraction is seq, which stands for sequence. seq basically has two operations, first and rest. The most obvious use for them is to iterate through items of a collection. There are built-in implementations for lists, vectors, sets, hashmaps, and even strings. But anything that has a notion of sequential values can implement seq, including Java Iterators. I would also like to posit that the most important and often overlooked implementation of seq is for nil itself.

  5. Even the best Haskellers complain about not knowing where a Nothing came from.

  6. You might look at this as nil-preserving behavior--much like the Nothing-preserving behavior of the Maybe Monad

Tap Into Your Social Brain

Let's look at this logic problem:

A logic problem involving cards

Try to determine the answer.

Hint: It's presented visually to try to trick you (as we saw last time).

The answer is two cards, the A and the 3.

Try this one:

You are a police officer enforcing this law:

It is only legal for minors to drink non-alcoholic beverages.

You are busy, so you need to quickly assess each bar with the minimum amount of checks.

You walk into a bar, and you see this scene:

There are four people in the bar. One is a teenage boy, but you can't see his drink. One is an old man, and you can't see his drink. The third person has a coke1, but he/she is behind a column so you can't see their age. The last person has a big pint of beer, but he/she is also blocked by a column.

Did you get the answer? You need to check two people. You don't need to check the old man, he's obviously not a minor. You don't need to check the person with a coke, even though he might be underage. That leaves the other two. The teenage boy on the left might have an alcoholic drink (which you can't see), and the person you can't see has a beer, so he/she might be a minor.

Was that easier? Would it surprise you if I told you that the card problem and the bar problem are equivalent?2 Why is it easier to solve the bar problem with almost no effort?

There are two reasons: one, it calls on years of built-up, real-world experience. Second, the problem is social--that is, it is about people. We humans are built to handle complex relationships between people. Our reasoning power is somehow magnified and clarified when phrased in terms of people in a familiar situation.

Moral: When possible present a logic problem as a problem people have no difficulty solving.

What's interesting is that you instantly know the shape of the solution when presented with the bar version of the problem. You think "I'm looking for minors and alcoholic beverages." Whereas my first reaction to the card version was "I'm looking for vowels and even numbers" (which is wrong). Only after careful, slow, deliberate thinking was I able to see the I should be looking for vowels and odds.

If you present material in the right way, it will help you teach the material better. You've likely had this experience before. Did your teacher ever do a math problem in terms of buying something and making change? Somehow, kids who score poorly in math class can still do the same problem when it's presented as a human to human exchange!

Social problems are not the only ones that we are hyper-capable of solving. There are also other situations that we are also hardwired to understand better than symbolic puzzles. Spatial orientation (for instance, that the arm sticking out from behind the column is attached to a person) and movement are also easy to solve, and luckily some of the most interesting math problems are equivalent to orientation and motion.

By converting a symbolic problem to one that is a familiar, real-world situation, you are tapping into many different parts of the brain. The key to a good "conversion" is whether the problem solver can properly simulate the situation themselves. The bar problem is good because it's something we can all imagine.

This is one thing I try to take advantage of in my videos. Yes, you are learning Clojure to solve a very complex problem. However, the problem is familiar to most, as it involves many metaphors and simulated situations. You are teaching someone to bake. A function is like a recipe. Pure functions are like doing a calculation in your head. Side effects move you around or use up ingredients, etc. I worked hard to make it seamless to learn.

Now that we know that it's easier to learn something if we can already simulate it, the next question is how to convert math/logic problems from their symbolic form into something more suitable for whole-brain simulation. This post is long enough already, so I'll address that next time.

  1. Yes, I come from that part of the country.

  2. Here's the conversion: minors are vowels, legal-aged are consonants. Alcoholic drinks are odd numbers, non-alcoholic drinks are even numbers. And legal is true and illegal is false. (That is the big one!)

A Personal Lisp Crisis

There are other languages with healthier communities, more momentum, cleaner cores, and features on par with CL. So I have to ask myself, why bother with CL?

I had a similar experience with Common Lisp. Great language, snobby community, little progress, needs more modern amenities.

Why technical explanation alone is not enough

Here's a problem you might see on a standardized test:

Choose the answer that makes the following statement FALSE: If the Triangle is Red, then the Square is Blue.

Go ahead and try to solve it. Choose an answer.

When people answer this question on a test, the most common answer is C. Yellow triangle and red square. They try to make the statement false by negating all of the colors.

This is wrong. Actually, the correct answer is A. It is the only one that makes the statement false. Why?

This is a propositional logic problem. The colors and shapes are a red herring. The rule for implication (if-then statement) is:

An implication is false if the antecedant is true and the consequent is false.

Kind of boring and therefore hard to understand if you don't already know the rule. It is unlikely that any amount of explanation will help, though you should try to understand if you don't already. Here, try this. If the triangle isn't red, then the statement doesn't apply, so it is true by irrelevance. That eliminates C and D. B obviously makes the statement true, so it must be A.1

Researchers found that explanations alone do not help. I read an article2 about a study where similar questions were asked to a group of students. The students were hooked up to EEG (Electroencephalography) machines to measure what parts of their brains were active during each question. They also asked them to say which ones they thought they got right.

EEG setup also labeling reasoning/visual areas.
Photo credit: Petter Kallioinen

What they found was that most of the people who answered A were using their frontal and temporal lobes: the parts of the brain responsible for logic, reasoning, and language. Most of the people who answered C were using their occipital lobes, which is responsible for vision.

Moral #1: Recognize what type of problem you need to solve. If possible, use the right part of your brain for that problem.

The people who used their vision centers were doing visual pattern matching, looking for shapes and colors. This is typical of a fight-or-flight response, where you are running from a tiger and need to quickly spot it in the jungle.

Moral #2: Don't get nervous during a test. Anxiety makes some parts of your brain more active--usually the wrong ones.

In addition, the people who used their vision center to answer the question thought they got the answer right just as much as the people who actually got it right.

Moral #3: The different parts of your brain are good at different tasks. If you ask the occipital lobe to do pattern matching, it will, and it will think it did a good job--even if it is not a pattern matching task.

The second part of the study tried to teach those who got it wrong to do it better next time. They broke them into a control group and an experimental group. The control group got an explanation like the boring one above on how to make an implication false. The experimental group got the same explanation plus they were trained with a little bio-feedback to use the right part of the brain.3

The result was that the control group didn't make any gains. And they still thought they got it right. The experimental group did significantly better. They scored higher and were more likely to say they got the wrong answer when they did get the wrong answer.

Moral #4: Explanation alone is basically worthless. Even practice is not enough. The key is to recognize what type of problem it is, use the right part of your brain, and don't listen to the rest of your brain. Only then is explanation useful.

Now, this is easier said than done. I don't have an EEG machine. I've never done bio-feedback. But the idea has helped me tremendously when teaching math. It is not hard to recognize when someone is using the wrong part of their brain. First, they confidently get the wrong answer. Second, they are getting the basic mechanics of the problem wrong. They're looking for something. They find the closest thing to what they want. They're done.

Logic problems aren't about looking. They're more abstract. Common helpers for doing logic problems are mouthing words, making gestures, and looking away from the problem.

Recognizing whether someone is using the right part of their brain is easy. But now you know explaining won't help. The hard part is to get them to use the correct part of the brain. That's what I'll talk about next time.

  1. If you still don't get it, I'll share one more trick. Convert it into a purely symbolic puzzle to switch off your occipital lobe.

    Choose the answer that makes the following statement FALSE:

    X -> Y
    A.  X & ~Y  
    B.  X &  Y  
    C. ~X & ~Y  
    D. ~X &  Y  
  2. I read it back in 2002 and have lost the reference. I tried searching for it and could not find it. The reason I bring it up, even without a citation, is that it has been very important to my thinking about learning. If you know this or a similar study, please let me know.

  3. Yes, it was double-blind. In fact, if I remember correctly, they were all in the same classroom listening to the same lecture. The control group was moved to a room to practice. The experimental group was moved to a different room to practice with EEG bio-feedback machines. Then they took a similar test again.

Functors, Applicatives, And Monads In Pictures

This is a nice, short post. It does the job well, though it might need a few readings if you have never seen these concepts before.

Learn these ideas and get on with your life. The sooner we get over this Functor/Applicative/Monad obsession, the better.

3 Things Java Programmers Can Steal from Clojure

The other day I wrote about some principles that programming in Clojure makes very clear. Those principle could be applied just as well in Java, and often are. However, there are some things that make Clojure distinct.

Three of those distinctions are the way it deals with state change (using an STM), the Persistent Data Structures, and the literal syntax for data with a reader (now called edn). Diving into the source code for Clojure, I realized that these three bits were written in Java. And that means that they can be used from Java. It is certainly not as easy as using them in Clojure, but they are all three powerful enough to warrant using them if you are using Java. You simply need to add one more JAR to your project (or add a maven dependency). I have constructed a few minimal examples of their use.

1. Persistent Data Structures

Clojure comes with several powerful and fast collection classes. The interesting thing about them is that they are immutable. If you want to add an object to a list, you actually create a new list containing the old elements and the new element. Instead of using copy-on-write, it reuses most of the internal structure of the original list, so only a small number of objects need to be allocated. It turns out that this can be done very quickly, comparable to using an ArrayList.

The following example illustrates three of the more useful data structures: Vector, HashMap, and HashSet.

package persistent;

import clojure.lang.IPersistentMap;
import clojure.lang.IPersistentSet;
import clojure.lang.IPersistentVector;
import clojure.lang.PersistentHashMap;
import clojure.lang.PersistentHashSet;
import clojure.lang.PersistentVector;

public class PersistentTest {
  public static void main(String[] args) {
    IPersistentMap m = PersistentHashMap.create("abc", "xyz");
    m = m.assoc(1, 4); // add a new key/value pair
    m = m.assoc("key", "value");
    m = m.without("abc"); // remove key "abc" 
    IPersistentVector v = PersistentVector.create(1, 2, 3);
    v = v.assocN(0, "a string"); // change index 0
    v = v.cons("should be last"); // add a string at the end
    IPersistentSet s = PersistentHashSet.create("a", "b", "c");
    s = (IPersistentSet) s.cons("d"); // add d to the set
    s = (IPersistentSet) ((IPersistentMap) s).without("a"); // remove an element
    s.contains("g"); // should return false

Now, it ain't pretty. But it's actually no worse than quite a few native Java libraries I've seen. There may be a better way to do this, but this one works.

2. Software Transactional Memory

Clojure uses Multiversion concurrency control to provide a safe way to manage concurrent access to state shared between threads. In Clojure, they are called refs. I won't go very deep into how it works. Suffice it to say that Clojure refs gives you non-blocking reads and transactional updates without having to do locking yourself. There are two caveats: 1 is that the value you give to the ref has to be immutable. 2 is that you should not perform IO (or perform any mutation) inside of the transaction.

package stm;

import java.util.concurrent.Callable;

import clojure.lang.LockingTransaction;
import clojure.lang.Ref;

public class STMTest {
  public static void main(String[] args) {
    // final needed to be used in anonymous class
    final Ref r = new Ref(1);
    final Ref s = new Ref(5);

    try {
      // run this in a transaction
      // don't do IO inside
        new Callable<Object>() {
          public Object call(){
            s.set((Integer)r.deref() + 10);
            return null;
    } catch (Exception e) {

3. Extensible Data Notation

With Clojure 1.5, edn has become a standard part of the language. Edn is like an extensible JSON where the keys of objects can be any value (not just strings). It is based on the Clojure literal syntax, much in the same way that JSON is based on Javascript literal syntax. It is a nice way to serialize data. And since you already have the JAR in your project, it's a no brainer to use it.

package edn;


import clojure.lang.EdnReader;
import clojure.lang.PersistentHashMap;

public class EDNTest {
  public static void main(String[] args) {
    // reading from a string
      EdnReader.readString("{\"x\" 1 \"y\" 2}", PersistentHashMap.EMPTY));

    // reading from a Reader
    // really, you can use any Reader wrapped in a PushbackReader
    System.out.println( PushbackReader(new StringReader("#{10 2 3}")),

4 Things Java Programmers Can Learn from Clojure (without learning Clojure)

I was trained in Java at University. The OOP matrix was firmly implanted in my thinking. I wanted to share some things that I have learned from Clojure that were certainly possible in Java but never became fundamental to my programming practice.

Clojure certainly has learned a lot from Java. It might be cool if the learning went both ways. These are universal principles. In fact, these principles are actually well known in the OOP world. You probably already know them, so learning Clojure is not required (but it is recommended!).

1. Use immutable values

One of Clojure's claim to fame is its immutable data structures. But immutable values were appreciated even in the very early days of Java. String is immutable and was a bit controversial when Java came out. Back then, C and C++ strings were simply arrays which could be changed. Immutable Strings were seen as inefficient.

However, looking back, immutable Strings seem to have been the right choice. Many of the mutable Java classes are now seen as mistakes. Take, for example, java.util.Date. What does it mean to change the month of a date?

Let's go a little further. Let's imagine that I'm an object. You ask me when my birthday is. I hand you a piece of paper with July 18, 1981. You take that home, store it somewhere, and even let other people access that piece of paper.

One of those people says "cool, a date!" And changes it to his birthday, April 2, 1976 using setTime. Now the next person who asks for my birthday actually gets that guy's birthday. What a disaster! Why did I give away that magic paper that changes my birthday?

By making values mutable, this magical-changing-at-a-distance is always a possibility. One way to look at the reason it is actually wrong to use mutable values is that it breaks the information hiding principle. My birthdate is part of the state of my object. By giving direct access to the month, day, and year, I'm actually letting any class have direct access to my internal state.

The answer, of course, is to not have any setters on an object. After construction, the object can't change. That way, my internal state remains encapsulated.

This applies to collections as well. Have you ever read the docs for Iterator. Can you tell what happens when the underlying list changes? Neither can I. An immutable list would not have such a complicated interface.

Solution: Don't write setter methods. For collections, you have a couple of options. One easy thing to do is use the Google Guava Immutable... classes. If using Guava is not an option, whenever you are returning a collection, make a copy, wrap it in a java.util.Collections.unmodifiable...(), and throw away the reference to the copy.

public static Map immutableMap(Map m) {
  return Collections.unmodifiableMap(new HashMap(m));

To learn more about immutable values, I suggest watching The Value of Values, a talk by Rich Hickey, the creator of Clojure.

2. Do no work in the constructor

Imagine this situation. Your Person class has a constructor that takes a bunch of information (first name, last name, address, etc.) and stores it in the object's state. Someone on your team needs to store that data to a file, so stores it as JSON. For convenience in creating a Person, you add a constructor that takes an InputStream and parses it as JSON, then sets up the state. Just because, you also add one that takes a File, reads in the file, then parses it. And then one that reads in a web request given a URL. Great! You've got a very convenient class.

But wait! What is the responsibility of the Person class? Originally, it was "represent personal information about a person". Now it is also responsible for:

  • Parsing JSON
  • Making web requests
  • Reading files
  • Handling errors

What's more is that the class is now harder to test. How can you test the File constructor? First, you write a temporary file to the file system. Not too bad. How do you test the web request? Set up a web server, configure it to serve the file, then call the constructor.

The problem is that Person violates the single responsibility principle. Person is about keeping bits of information together, not permanent storage or serialization. It should be a data object, no more.

UPDATE: Solution: Keep your constructor free of logic. Separate out convenience constructors (like the one that parses JSON) into static factory methods.

To learn more about this idea (and more!), I suggest you watch OO Design for Testability by Miško Hevery, the creator of AngularJS.

3. Program to small interfaces

One thing that Clojure has done very well is to define a set of very powerful, small interfaces which abstract a pattern of access. The interface allows many different types to participate in an "ecosystem". Any function which applies to the interface can act on any type that implements that interface. Any new type can take advantage of all of the existing functionality already built in.

Take for instance the Iterable interface. It generalizes anything that can be accessed sequentially (such as a list or set). If all a method needs to do is operate on something sequentially, it only needs to know that it implements this interface. That means it can operate on types that were not known to the programmer when the method was written.

This aspect follows from the dependency inversion principle which states that high-level logic should be written in terms of abstractions instead of the details of the lower-level logic. Interfaces capture this principle well. High-level logic should operate on interfaces which are implemented by the lower-level logic.

Solution: Think hard about the access patterns for classes and see if you can't abstract out small interfaces which pinpoint those access patterns. Then program to those interfaces. Remember, it takes two to use an interface: the implementor and the client. Make sure you use them from both sides as much as possible.

Nothing increases maintainability and the future cost of code more than good interfaces. To learn more about this, I suggest watching How To Design A Good API and Why It Matters. It's an older (but good) talk by Joshua Bloch.

4. Represent computation, not the world

When I was in college, the teacher taught us that you should use classes to model objects in the world. The quintessential modeling problem was students registering for courses.

A course can have many students and a student can be registered in many courses. A many-to-many relationship.

The obvious choice is to make a Student class and a Course class. Each has a list of the other. Inclusion in that list represents registration. Methods like register and listCourses let a Student register or list the courses he's registered in.

Professors would present this problem in order to discuss the tradeoffs of different design choices. None of the configurations of Student and Course were ideal. An astute data modeler would see the pattern of the many-to-many relationship and abstract that out. You can create a class called ManyToMany<X,Y> that manages the relationship. You can create a ManyToMany<CourseID, StudentID> and it solves your problem exactly.

The issue is that this directly contradicts the teacher's lesson. A relationship is not an object in the real world. At best it is an abstract concept.

What's more is that it solves the more general abstract problem as well. The ManyToMany class can be reused anywhere it is suited. Even better would be to make ManyToMany an interface with many possible implementations.

I think my professor was wrong. The Java standard library contains many classes that are purely computational. Why can't application programmers write them as well? Further, look at the GOF Design Patterns Book. Most (if not all) of the patterns are about abstracting computation, not "objects in the real world". Take, for instance, Chain-of-responsibility, which Wikipedia describes as "Avoid coupling the sender of a request to its receiver by giving more than one object a chance to handle the request."

Solution: look for repetitive patterns of code and build classes that take care of those patterns. Use those classes instead of repeating the pattern over and over in code.

One reason I like Clojure is that it has a lot to teach. If you'd like to learn Clojure, you might like my Introduction to Clojure video course.

Existing Clojure Introductory Videos

As you probably know, I am running a Kickstarter project to create Introduction to Clojure videos. The project is still going (it runs until March 14, 2013). Please support it if you want the videos.

I know that there are many great videos out there that teach the basics of Clojure. Allow me to present my selection.

Jim Slaterry is creating video walkthroughs of the Clojure Koans.

Full Disclojure was a long series of screencasts by Sean Devlin that dives deep into Clojure. There are many lessons about Clojure and functional programming, screencasts explaining how to install editors, and a few explorations of katas. Most of the material should still be current.

Brian Will created a series of videos introducing Clojure back in 2009 which explains the basics of the language. The basics have changed somewhat, but this is still a good way to get into the language.

PeepCode has a video for $12 for Clojure beginners.

Rich Hickey gave some talks a long time ago teaching Clojure. Again, a little old, but I learned a lot from them when they were new.