Ring 1.3

Summary: Ring is great because it closely models the HTTP message format using native Clojure data structures. It strictly defines a message format that any software can use and rely on. With Ring 1.3, the specification has gotten even closer to the HTTP spec.

A couple of months ago, Ring 1.3 was released without much fanfare. It included a few improvements and updates, but in general, not much had changed.

One change, though, is very significant: the specification is shorter. It's simpler. Three keys were deprecated in the Ring request map (:content-type, :content-length, and :character-encoding). These keys were unnecessary because their values were in the headers, which are also in the Ring request. Equivalent utility functions have been added for pulling the data out of the headers.

Why is this important? While many libraries get more complex and overburdened, it is refreshing to see a library going in the correct direction of shedding complexity. It does not significantly impact application development. Nor does it reduce the already low barrier to entry. Still, I welcome this kind of change.

Ring is the central specification that ties most of the Clojure web ecosystem together. The spec should be minimal. And a mark of good software is that it models the problem very closely without unnecessary abstraction. Ring merely defines a common format (using Clojure data structures) that mirrors the text-based HTTP message format. That's why Ring has worked so well thus far and why it is appreciated in Clojure.

Because I was so happy about the change, I decided to update my Ring Spec to Hang on the Wall PDF. The newly deprecated keys are gone. It used to be two pages long. The Ring Request took up an entire page, and the Response took up about half of one. But now, with three keys removed and a little tweaking of the font sizes, everything fits on one page.

One page in big, readable fonts, with just the information you need for quick reference. I like it. I'm printing one out right now to tack on the wall. You can get a free copy for yourself by getting on the PurelyFunctional.tv mailing list here.

Master the fundamental tools and concepts of web development in Clojure
Learn more

Church vs Curry Types

Summary: Static vs dynamic typing debates often flounder because the debators see from two different perspectives without knowing it. Learning to identify the two perspectives can calm the discussion. The tension between the two perspectives has led to Gradual Typing and other technologies.

Many discussions about type systems around the internet fail to be interesting because one or both parties are not versed in type theory. There's a less common (yet related) reason, which I have begun to notice more and more: people who are not familiar with the difference between Church Types and Curry Types. These are also known, respectively, as Intrinsic Types and Extrinsic types. Because the participants are not aware of the two perspectives, they blame the other one for ignorance, when in fact they just have a different perspective.

Church Types are what Haskell has. Church types are named for Alonzo Church, the inventor of the lambda calculus. In a Church-style system, types are an intrinsic part of the semantics of the language. The language would be different without the types--it may even be meaningless. With an intrinsic type system, the meaning of the program is different from the runtime behavior of the program. One way to think of this is that so much of the meaning of the program occurs at compile time that you can begin to think of the program having properties you can reason about even if you never run the program.

The other kind of types are Curry types (aka extrinsic types). They are named for Haskell Curry, the man the Haskell language is named after. Curry-style types is when a system of types is applied that is not part of the semantics of the language. This is what Clojure has in core.typed. The meaning of a Clojure program is not dependent on it passing the type checker--it can be run without it. The type checker simply alerts you to type errors in your code. Note that you could consider your type checker to be your own head, which, as flawed as it may be, is what most Clojure programmers use. The types could be anywhere outside of the language.

Each perspective is valuable and bears its own fruit. Intrinsic types are great because you are guaranteed to have a mathematically-sound1 safety net at all times. You always have something you can reason about. Such is not guaranteed for extrinsic type checkers. They may not be able to reason about your code at all.

Extrinsic types are useful because you can apply multiple type systems to your code2--or even write something that you don't know how to prove is sound. There are more benefits on both sides, but you get the idea.

We now have a new perspective which is slightly "higher" than either of them. We can now see that both perspectives exist and talk about them as such. What can we see/say now that we couldn't before?

A famous article by Robert Harper exemplifies the Church perspective very well. It argues that untyped programs are a subset of typed programs. They are programs that have a single type and all values are of that one type. So instead of being liberating, dynamic languages restrict you to one type. Notice the assumption that languages have a type system by default which is typical of the Church-style perspective. We can now say "This reasoning is correct given that perspective."

On the other side, you'll often see a dynamic typist say the exact opposite: that well-typed programs are a subset of dynamically typed programs. In other words, well-typed programs are just dynamic programs with fewer errors. Curry-style to the core: static type errors are something that is added onto the semantics of the language. We can now see that they are right, from their perspective.

Here's a diagram:

Church vs Curry Types Language Subset Diagrams

Church vs Curry Types Language Subset Diagrams

Notice how they're isomorphic? That means something, I just don't know what :)

My ambitious hope is that this perspective will quiet a lot of the fighting as people recognize that they are just perpetuating a rift in the field of mathematics that happened a long time ago. The perspectives are irreconcilable now, but that could change. A paper called Church and Curry: Combining Intrinsic and Extrinsic Typing builds a language with both kinds of types. And Gradual Typing and Blame Calculus are investigating the intersection of static and dynamic typing. Let's stop fighting, make some cool tools and use them well.

If you like this article, you might want to subscribe for free to the Clojure Gazette.

Learn More

  1. As sound as the type system in the language.

  2. See Liquid Haskell for an example of applying an extrinsic type system on Haskell.

JSON Serialization for APIs in Clojure

Summary: Clojure is well-suited for processing JSON, but there are some decisions you have to make to suit your application. The major decisions are actually easy, though they took me a while to figure out.

I tend to use JSON instead of edn for an API serialization format, if only because JSON is more readily readable from other languages. I could do both, but it's good to eat your own dogfood and test those JSON code paths.1

edn is better, but JSON is generally pretty good. However, JSON's expressibility is decidedly a subset of the Clojure data structures, so there is considerable loss of information when going from Clojure to JSON. That information is not recovered in a round-trip, at least not automatically. There are lots of decisions that have to go into how to, at least partially, recover this.

One bit of information that is lost is the type of keys to a map. JSON only allows strings as keys. Clojure allows anything. But most of the time, I find myself using keywords for keys. I say most, but really, it's the vast majority. Maps are bundles of named values pretty much all the time.2 So the optimal decision, after trying lots of combinations, is to convert keywords to strings (the default in JSON libraries I've seen) when emitting JSON; and to convert map keys (always strings in JSON) to keywords (also known as keywordize-keys) when parsing JSON. That covers nearly all cases, and pinpointed special cases can cover the rest.

But that's not the end of the keyword/string story. What about namespaces? Surprisingly, the two major JSON libraries, clojure.data.json and cheshire handle things differently. How do you parse a JSON key that has a slash in it, indicating a namespace? If we're keywordizing (as I suggest above), they both give namespaced keywords (keyword will parse around the /). But when emitting JSON, they act differently. clojure.data.json will emit the name of the keyword (and ignore the namespace) while cheshire emits a string with "namespace/name".

I like to keep the namespace, or, put another way, I like to drop as little information as possible. So I prefer the namespace approach. I'm not sure how to get clojure.data.json to do that. I just use cheshire.3 The other gotcha for namespaces is that ClojureScript's clj->js and js->clj are similarly asymetrical.4

Keywords in other places besides the keys of maps will just get turned into strings, but they don't get converted back to keywords. That sucks, but it's minor. You'll just have to convert them back some other way. At work, we use Prismatic Schema's coercions. They do the trick nicely, in a declarative way.

So, back to other JSON issues. The other issue is other data types. Dates, URI's, and UUID's are in our data as well. Dates, well, it's up to you how to encode them. I've always been a fan of the Unix timestamp. It's not exactly human readable, but it's universally parseable. There's also the ISO datetime format, which is probably a better bet--it's human readable and agreed upon among API designers. You could emit that as a string and coerce back to a Date object later.

URI's and UUID's are by definition strings, so that's easy. How do you set up cheshire to handle the encoders? It's pretty simple, really.

(cheshire.generate/add-encoder java.net.URI cheshire.generate/encode-str)

That means add the encoder for the java.net.URI type to be encoded as a JSON string. str will be called on the value. You can figure out the other types you need. There are some JSON emission settings built-in, including Date (the ISO string format) and UUID. Weirdly URI is not in there, so you have to add it.

What's next? Oh, pretty-printing. Yeah, I pretty-print my JSON to go over the wire. It's nice for debugging. I mean, who wants to curl one long, 1000-character line of JSON? Put some whitespace, please! How to do that?

(cheshire.core/generate-string mp {:pretty true})

That's right, it's basically built in, but you have to specify it. But, oh man, that's long. I don't want to type that, especially because my lazy fingers are going to not do it one time, then I'm going to look at the JSON in my browser and see a one-line JSON mess. So, what do I do? I put all my JSON stuff for each project in json.clj. It's got all my add-encoder stuff, and it's got two extra functions, just for laziness5:

(defn parse [s]
  (cheshire.core/parse-string s true))

(defn gen [o]
  (cheshire.core/generate-string o {:pretty true}))

Or of course whatever options you want to pass those functions. This one is my choice--you make your choice. But these two functions are all I use most of the time. Parsing strings and generating strings. Woohoo! Much shorter and less to keep in my little head.

Well, that just about wraps up my JSON API story. There's the slight detail of outputting JSON from liberator, which is its own blog post. And there's a bit of generative testing I do to really challenge my assumptions about how I set up the round-tripping. But that, too, is another blog post for another day. Oh, and what about all that JSON middleware? Again, another post.

If you like peanut butter and you like jelly, you will probably like peanut butter and jelly sandwiches. If you like web and you like Clojure, you will most definitely like Web Development in Clojure, which is a gentle, soothing, visually rich video course ushering in the fundamentals of Clojure web development through your eyes and ears and down through your fingertips and into your very own Heroku-hosted web server. At least watch the preview!

Master the fundamental tools and concepts of web development in Clojure
Learn more

  1. It may be hubristic to think anyone else will use my API.

  2. There are exceptions, but these typically are not communicated to the outside. Those that are need special-casing. C'est la vie!

  3. :key-fn in clojure.data.json only works for keys, not all keywords. Emitting a keyword in any other place emits the str of it, which includes the : in the string. Ick.

  4. clj->js uses name for keywords and str for symbols, so keywords lose their namespace when emitting JSON, but retain their namespace when parsing key strings as keywords.

  5. also known as ease, peace of mind, DRY, and cleanliness

What is Functional Programming?

Summary: I prefer to define Functional Programming as making a distinction between pure and impure code. With this definition, you can program functionally in any language. What differentiates the functional languages is how much help they give you to make the distinction.

There are a lot of conflicting definitions of Functional Programming out there. I'd like to share mine, which serves me well. It explains why Haskell is more functional than Scheme, and also how you can program functionally in a non-functional language like Java.

Functional programming means programming with a distinction between pure code and impure code. Pure code has no side effects. It's referentially transparent. It means the same thing every time you run it. Impure code contains side effects, so running it twice is different from running it once.

The distinction between pure code and impure code uniquely identifies functional programming and distinguishes it from other paradigms such as procedural and Object Oriented. Procedural is about modeling your solution as sequential steps. Object Oriented is about modeling your solution as communicating objects. Functional programming is about modeling your solution as pure functions.

Now, this definition is very practical. Notice that it's not about choice of language. You can write functional code in any language, just as you can code up an object system in C and say you're doing OO. The question is how much the language helps you write functional code or OO code.

On one extreme, you've got Haskell. There is no doubt that Haskell is a functional language. How does it help you write functional code? It has no mutable values and side-effects are confined to a single type: IO. The language forces you to make the distinction between pure and impure.

On the other extreme, you've got machine code or assembly. At the lowest level, the language pushes you to avoid the distinction. All operations are about changing at least one location in memory. It could be a register or the top of the stack or something. But you are forced to change something. However, with a lot of super-human discipline, you could keep the distinction in your head. You might create a little heap and keep the discipline "a procedure can only write to memory it allocates directly". And this way, you make a bit of room for some functional programming. But that language is not giving you any help.

So why functional programming? Well, it turns out that knowing that running code twice will produce the same result makes it very easy to reason about it. And reasoning about code is basically our job as software engineers. What's more, the kind of reasoning you can do with functional programs can reach all the way up to the highest forms of reasoning, like math. That's where Haskell really shines. All of the category theory stuff (monads, functors, applicatives, etc) is an expression of that--mathematical concepts that are applicable in Haskell code.

That's it. That's my definition. The definition is inclusive yet gets at the essence. Functional Programming is a perspective that makes code easier to understand and maintain as it's being used in a system complex beyond your possible comprehension. And at its most sublime and abstract levels, Functional Programming approaches mathematical reasoning.

If you'd like to get started with Functional Programming in Clojure, you can do worse than using the LispCast Introduction to Clojure video course.

Learn Clojure with screencasts, visual aids, and interactive exercises
Learn more

core.async Code Style

Summary: If your functions return core.async channels instead of taking callbacks, you encourage them to be called within go blocks. Unchecked, this encouragement could proliferate your use of go blocks unnecessarily. There are some coding conventions that can minimize this problem.

I've been using (and enjoying!) core.async for about a year now (mostly in ClojureScript). It has been a huge help for easily building concurrency patterns that would be incredibly difficult to engineer (and maintain and change) without it.

Over that year, I've developed some practices for writing code with core.async. I'm putting them here as an invitation for discussion.

Use callback style, if possible

A style develops when using core.async where you convert what would in regular ClojureScript be a callback style with return-a-channel style. The channel will contain the result of the call when it is ready.

Using this style to keep you out of "callback hell" is overkill. "Callback hell" is not caused by a single callback. It is caused by the eternal damnation of coordinating multiple callbacks when they could be called in any order at any time. Callbacks invert control.

core.async quenches the hellfire because coordinating channels within a go block is easy. The go block decides which values to read in which order. Control is restored to the code in a procedural style.

But return-a-channel style is not exactly free of sin. If you return a channel too much, the code that calls those functions will likely end up in a go block.

go blocks will proliferate. go blocks incur extra cost, especially in ClojureScript where they happen asynchronously, meaning at the next iteration of the event loop, which is indeterminately far away.

Furthermore, go blocks might begin nesting (a function whose body is a go block is called by another function whose body is a go block, etc), which is correct semantically but probably won't give you the performance you're looking for. It's best to avoid it.

"How?" you say? The most important rule is to only use core.async in a particular function when necessary. If you can get by with just a callback, don't use core.async. Just use a callback. For instance, let's say you have an ajax function that takes a callback and you're trying to make a small API wrapper for convenience. You could make it return a channel like this:

(defn search-google [query]
  (let [c (chan)]
    (ajax (str "http://google.com/?q=" query) #(put! c %))
    c))

The interesting thing to note is that core.async is not being used very well above. Yes, you get rid of a callback, but there isn't much coordination happening, so it's not needed. It's best to keep it straightforward, like this:

(defn search-google [query cb]
   (ajax (str "http://gooogle.com/?q=" query) cb))

You're just doing one bit of work here (basically constructing a URL), which is a good sign. But how do you "lift" this into core.async?

<<<

There's a common pattern in Javascript (not ubiquitous, but very common) to put the callback at the end of the parameter list. Since the callback is last, you can easily write something to add it automatically.

(defn <<< [f & args]
  (let [c (chan)]
    (apply f (concat args [(fn [x]
                             (if (or (nil? x)
                                     (undefined? x))
                                   (close! c)
                                   (put! c x)))]))))

This little function is very handy. It automatically adds a callback to a parameter list. You call it like this:

(go
  (js/console.log (<! (<<< search-google "unicorn droppings"))))

This function lifts search-google, a regular asynchronous function written with callback style, into core.async return-a-channel style. With this function, if I always put the callback at the end, I can use my functions from within regular ClojureScript code and also from core.async code. I can also use any function (and there are many) that happen to have the callback last. This convention has two parts: always put the callback last and use <<< when you need it. With this function, I can reserve core.async for coordination (what it's good at), not merely simple asynchrony.

<convention

There are times when writing a function using go blocks and returning channels is the best way. In those cases, I've adopted a naming convention. I put a < prefix in front of functions that return channels. I tried it at the end of the name, but I like how it looks at the beginning.

(go
  (js/console.log (<! (<do-something 1 2 3))))

The left-arrow of <do-something fits right into the <!. It also visually matches (<<< do-something 1 2 3), so it makes correct code look correct and wrong code look wrong. The naming convention extends to named values as well:

(def <values (chan))

(go
  (while true
    (js/console.log (inc (<! <values)))))

Conclusion

These conventions are a great compromise between ease of using core.async (<<<) and universality (callbacks being universal in JS). The naming convention (< prefix) visually marks code that should be used with core.async. These practices have taken me a long way. I'd love to discuss them with you here.

If you know Clojure and you are interested in learning core.async in a fun, interactive style, check out the LispCast Clojure core.async videos.

Learn real-world core.async through interactive video lessons
Learn more

Regexes in Clojure

Summary: With a few functions from the standard library, Clojure lets you do most of what you want with regular expressions with no muss.

Clojure is designed to be hosted. Instead of defining a standard Regular Expression semantics that works on all platforms, Clojure defers to the host's semantics. On the JVM, you're using Java regexes. In ClojureScript, it's Javascript regexes. That's the first thing to know.

Other than the semantics of the regexes themselves, the API is standardized across all platforms in the core library. And the syntax is convenient because you don't need to double escape your special characters.

Literal representation

Regexes can be constructed in Clojure using a literal syntax. Strings with a hash in front are interpreted as regexes.

#"regex"

On the JVM, the above line will create an instance of java.util.regex.Pattern. In ClojureScript, it will create a RegExp. Remember, the two regular expression languages are similar but different.

Matching (with groups)

There is a nice function that matches the whole string. It is called re-matches. The return is a little complex. If the whole string does not match, it returns nil, which is nice because nil is falsey.

=> (re-matches #"abc" "zzzabcxxx")
   nil

If the string does match, and there are no groups (parens) in the regex, then it returns the matched string.

=> (re-matches #"abc" "abc")
   "abc"

If it matches but there are groups, then it returns a vector. The first element in the vector is the entire match. The remaining elements are the group matches.

=> (re-matches #"abc(.*)" "abcxyz")
   ["abcxyz" "xyz"]

The three different return types can get tricky, but in general I do have groups, so it's either a vector or nil, which is easy to handle. You can even destructure it before you test it.

(let [[_ fn ln] (re-matches #"(\w+)\s(\w+)" full-name)]
  (if fn ;; successful match
    (println fn ln)
    (println "Unparsable name")))

Matching substrings

re-matches matches the whole string. But often, we want to find a match within a string. re-find returns the first match within the string. The return values are similar to re-matches.

No match returns nil

=> (re-find #"sss" "Loch Ness")
nil

Match without groups returns matched string

=> (re-find #"s+" "dress")
"ss"

Match with groups returns a vector

=> (re-find #"s+(.*)(s+)" "success")
   ["success" "ucces" "s"]

Finding all substrings that match

The last function from clojure.core I use a lot is re-seq, which returns a lazy seq of all of the matches, not just the first. The elements of the seq are whatever type re-find would have returned.

=> (re-seq #"s+" "mississippi")
   ("ss" "ss")

Replacing regex matches within a string

Well, matching strings is cool, but often you'd like to replace a substring that matches with some other string. clojure.string/replace will replace all substring matches with a new string. Let's take a look:

=> (clojure.string/replace "mississippi" #"i.." "obb")
   "mobbobbobbi"

This function is actually quite versatile. You can refer directly to the groups in the replacement string:

=> (clojure.string/replace "mississippi" #"(i)" "$1$1")
   "miissiissiippii"

You can also replace with the value of a function applied to the match:

=> (clojure.string/replace "mississippi" #"(.)i(.)"
     (fn [[_ b a]]
       (str (clojure.string/upper-case b)
            "--"
            (clojure.string/upper-case a))))
   "M--SS--SS--Ppi"

You can replace just the first occurence with clojure.string/replace-first.

Splitting a string on a regex

Let's say you want to split a string on some character pattern, like one or more whitespace. You can use clojure.string/split:

=> (clojure.string/split "This is a string    that I am splitting." #"\s+")
   ["This" "is" "a" "string" "that" "I" "am" "splitting."]

Nice!

Other functions

Those are all of the functions I use routinely. There are some more, which are useful when you need them.

re-pattern

Construct a regex from a String.

re-matcher

This one is not available in ClojureScript. On the JVM, it creates a java.util.regex.Matcher, which is used for iterating over subsequent matches. This is not so useful since re-seq exists.

If you find yourself with a Matcher, you can call re-find on it to get the next match (instead of the first). You can also call re-groups from the most recent match. Unless you need a Matcher for some Java API, just stick to re-seq.

Conclusion

Well, that's regexes as I use them. They're super useful and easy to use in Clojure once you get the hang of them.

If you're interested in learning the fundamentals of Clojure, may I suggest my own LispCast Introduction to Clojure video series. It guides you through a deep experience of the language. You'll learn REPL skills, how to set up a project, and how to develop a DSL, all in a fun, interactive way.

Learn Clojure with screencasts, visual aids, and interactive exercises
Learn more

Warty Lists in Clojure

Summary: Lists are kind of warty in Clojure. Care should be taken, especially by those coming from other Lisps.

One of the things that still trips me up in Clojure is the actual types of lists. I used to program in Common Lisp, where things are a bit easier to understand: something is either a list or an atom. Lists are built out of conses and end with nil. Everything else is an atom.

Coming from Common Lisp, one might expect this to work:

=> (listp (cons 1 nil))

In CL, it will be true. In Clojure, it is also true. (Try it out yourself!) Launch a REPL. Go to Try Clojure.

=> (list? (cons 1 nil))

What about this:

=> (list? (cons 1 (cons 1 nil)))

We're consing onto a list, should be a list, no?

In Common Lisp, yes. In Clojure?

NO!

That's not a list. Go ahead, try it at your Clojure REPL.

What gives? How is that possible? Are we living in a Kafkaesque ECMAScript World?

Well . . . probably.

What's going on? Put on your Indiana Jones fedora. We're going on an adventure deep into the heart of the Clojure JVM implementation.

First, how is list? defined?

In clojure.core:

(defn list?
  "Returns true if x implements IPersistentList"
  {:added "1.0"}
  [x] (instance? clojure.lang.IPersistentList x))

Great, list? is just an instance check. What classes implement clojure.lang.IPersistentList? According to the javadoc on the Clojure sources, there are two: PersistentQueue and PersistentList. There's also a static inner class in PersistentList called EmptyList.

Let's see those at the REPL:

=> (type ())
  clojure.lang.PersistentList$EmptyList
=> (list? ())
  true

=> (type (list 1 2 3))
  clojure.lang.PersistentList
=> (list? (list 1 2 3))
  true

=> (type clojure.lang.PersistentQueue/EMPTY)
  clojure.lang.PersistentQueue
=> (list? clojure.lang.PersistentQueue/EMPTY)
  true

These all return true when given to list?. What about (cons 1 nil)?

=> (type (cons 1 nil))
  clojure.lang.PersistentList
=> (list? (cons 1 nil))
  true

Great. Consing onto nil gives you a list. Let's cons onto a list:

=> (type (cons 0 (list 1 2 3)))
  clojure.lang.Cons
=> (list? (cons 0 (list 1 2 3)))
  false

Oh, no! Why does consing onto nil create a list, but consing onto a list create a cons? Poisonous dart averted! Back to the source code!

In clojure.core:

(def
 ^{:arglists '([x seq])
    :doc "Returns a new seq where x is the first element and seq is
    the rest."
   :added "1.0"}

 cons (fn* cons [x seq] (. clojure.lang.RT (cons x seq))))

So it calls clojure.lang.RT/cons. We can look that up:

static public ISeq cons(Object x, Object coll){
    //ISeq y = seq(coll);
    if(coll == null)
        return new PersistentList(x);
    else if(coll instanceof ISeq)
        return new Cons(x, (ISeq) coll);
    else
        return new Cons(x, seq(coll));
}

Wow! The code is clear: if the second argument is null (nil), it makes a PersistentList. Otherwise, it constructs a clojure.lang.Cons, which is not a list! We're at the root of our wart, but there's nothing we can really do about it except keep exploring.

If I have a list (according to list?) and I want to add an element to the front to make a new list, how do I do that?

Well, the answer is a little disappointing:

=> (def ls (list 1 2 3))
=> (def ls2 (conj ls 0))
=> (list? ls2)
  true

conj will maintain the type, cons will not. What happens if I conj onto a Cons?

=> (def c (cons 1 (cons 2 nil)))
=> (conj c 0)
  (0 1 2)
=> (type conj c 0)
  clojure.lang.Cons

So, there you have it. It's a bit of a wart having all of these slightly different types and predicates like list? that slice them up in odd ways. Sometimes it's like running away from a giant paper-mache ball.

In Clojure's defense, I will say that I rarely use list?, if at all. It's not a very useful function in clojure.core. I'm usually working at a much higher level than that, thinking in terms of sequences (an abstraction), not their concrete implementations. And in the end, you never get out of danger.

There you have it. If you'd like to learn more Clojure, I have a nice video series:

Learn Clojure with screencasts, visual aids, and interactive exercises
Learn more

CSS and the Lambda Calculus

Summary: Using LESS, we can almost achieve the expressive power of the Lambda Calculus as applied to styling. The expressive power is enough to create reusable styles applied to reusable HTML components.

Let's continue our exploration and analysis of CSS and LESS. This series is about Functional CSS. Our aim is to determine a good way to use HTML and CSS so that we can reuse both. We can't talk about "Functional" without talking about the Lambda Calculus.

The Lambda Calculus includes three things: variables, abstractions, and applications. Let's look at some Javascript.

(function(x) { return x + x; })(10);

In the above code, x is a variable, the function definition is a lambda abstraction, and calling the function (with the parens at the end) is called application. Luckily, Javascript gives us an additional kind of abstraction where we can name an expression to be reused:

var f = function(x) { return x + x; };
f(10);

We can name the function f, then apply it by referring to the name. What's more, we can compose them pretty well.

var g = function(x) { return x * x; };
var h = function(x, y) { return g(x) + g(y); };
h(10, 20);

We're composing function g by applying it inside of the definition of h. You're probably saying "duh!"--and rightly so. It's so common to do.

Which is why it's hard to understand why CSS does not include all of these parts. Let's at least try to decompose CSS into some parts.

.some-class { width: 10px; height: 20px; }

Here, we're applying the rule which contains two properties (width and height) to all of the elements that have the class .some-class. .some-class is the argument to the rule's application. In fake Javascript, it would look something like this1:

function(element) {
  element.width  = 10;
  element.height = 20;
}(document.getElementsByClassName('some-class'));

That is, we are immediately applying the abstraction to the argument. CSS has no way, as Javascript does, of naming the abstraction for use later. We definitely want that. What's more, there's no way to compose two rules together. In Javascript, we could refer to g within h. But in CSS, there's no way to do that.

LESS does have this ability. Since LESS is a superset of CSS, we can start with the above CSS code. Let's repeat it:

.some-class { width: 10px; height: 20px; }

Now we abstract it by naming it and then apply the name:

.small-box() { width: 10px; height: 20px; }
.some-class { .small-box(); }

That's somewhat better. We've got a reusable component (.small-box) and we've also added a bit more meaning to our code because we have a meaningful name. What's more, LESS lets you compose:

.small-box() { width: 10px; height: 20px; }
.my-button() { .small-box(); border: 1px solid black; }

.some-class { .my-button(); }

Here we've made an abstraction called .my-button by using .small-box inside. That's composition. And that's on par with our Javascript examples above.

There's one more thing that we might want that even Javascript doesn't have. Javascript has first-class functions. That's really nice. But there are many things in Javascript that are not first-class. For instance, the arithmetic operators (+, *, -, /) cannot be passed to a function or assigned to variables. They are treated differently by the language. They are a bit like properties in CSS: all you can do with them is refer to them directly in an expression.

But what if Javascript did have first-class arithmetic functions? We can certainly fake them out.

var plus  = function(a, b) { return a + b; };
var times = function(a, b) { return a * b; };
var minus = function(a, b) { return a - b; };
var over  = function(a, b) { return a / b; };

Now we have functions that act just like the operators--and they're first-class. It's also way more consistent: every operator is a function.

We can do something similar in LESS. Imagine we took every property and did what I do to these:

.width(@w)  { width:  @w; }
.height(@h) { height: @h; }
. . .

Now we have the consistency that everything is at least referrable as a mixin. This is pretty good.

We can now define our styles--and name them--in this subset of LESS. Styles may reference other styles. And rules with selectors may reference styles. You could import pure styles from an external library, then refer to them in your rules, where you apply them to selectors. The HTML does not have to change, and neither do the styles.

Rules--where styles and selectors are tied together--will change the most. As common sets of styles are used together often, you might think about factoring them out into a new style, which would be added to your organization's standard style library. Also, as HTML components solidify, you could begin to firm up the class names and their structure, reusing them in a more permanent way. Yet, even though these two assets (the standard styles and the standard class names / HTML structures) are permanent, you can still change how they are styled by changing the rules. Both assets retain their value over time.

A fly in the ointment

Even though we have tremendous power over that given by CSS, we are not at the level of first-class styles. The following will NOT work in LESS, though we would expect it to.

/* refer to variable like a mixin */
.apply(@style, @arg) { @style(@arg); }

.width(@w) { width: @w; }

/* pass in .width mixin */
#xyz { .apply(.width, 10px); }

The problem is that you cannot use a mixin named with a variable. The same limitation exists in SCSS (SASS). The following is invalid.

@mixin apply($s, $a) { @include $s($a); }

Why can't a mixin be assigned to a variable? Mixins appear to be in a different namespace from variables, and only mixin syntax can appear in the mixin position. Though I don't think apply itself would be very useful, it would indicate a recursive abstraction power that could be used well.

Not being able to write apply hints that the developers of these two languages are thinking at the syntax level. They have added some great features, but they have not truly modeled the problem semantically. Instead, it feels like a mix of special-cased syntax rules and string interpolation. You can assign a ruleset to a variable, but not a mixin?

Conclusion

LESS (and SASS, etc) have some very powerful features that are miles above CSS in terms of abstraction and composition. I hate using plain CSS after using LESS--LESS is just so much more expressive. You can do some impressive and useful things with them, and I continue to use them. I want to write about how I use a subset of LESS and a strict discipline in HTML to make styling easier and more maintainable. In addition, I'd like to explore what a better-designed style language might look like.


  1. Please ignore the fact that CSS does operate on sets of elements, not simple elements.

Separation of Presentation and Content

Summary: One reason to separate style from content is to reuse HTML or CSS. Ultimately, we would like a solution where we can reuse both.

Reusable Content

There is an economic reason to separate presentation from content. Publishers have thousands of pages of HTML on their site, yet they want to enhance the style of their pages over time. It would cost a lot of money to change every single page to match their new style. So they invest a little more time writing each page so that the HTML markup does not refer to styles but to the semantics of the content (referred to as semantic HTML). Then they hire a designer to write CSS to make their existing content look new. The HTML is permanent and reusable, and the CSS is temporary and not-reusable. The separation is only one way: the HTML doesn't know the CSS, but the CSS does know the HTML.

Examples: CSS Zen Garden, newspaper websites, blogs

Characteristics: Semantic markup, CSS tailored to classes/structure of HTML

Reusable Styles

Yet another economic reason is a relatively newer phenomenon. It has become very easy to create a new web site/application. Writing (or generating) lots of HTML is cheap, and it changes often during iterative development. What is relatively expensive is to design each of those pages each time the pages change. CSS is not good at adapting to page structure changes. So people have built CSS frameworks where the CSS is (relatively) permanent and the HTML is temporary. In these cases, the HTML knows the CSS, but the CSS doesn't know the HTML. The separation is again one way--this time the other way.

Examples: Open Source CSS, Bootstrap, Foundation, Pure

Characteristics: HTML tailored to classes/structure of CSS, Reusable CSS

Reusable Content and Styles

What if a newspaper site, with millions of existing HTML pages, could cheaply take advantage of the reusable styles of frameworks like Bootstrap? That is the Holy Grail of separation of concerns. What would be required to do that?

What we really want is a two-way separation. We want HTML written in total isolation and CSS written in total isolation. We want permanent HTML and permanent CSS. How can the style and content, each developed separately, finally be brought together? The answer is simple: a third document to relate the two.

We have already seen that CSS is not good at abstraction. CSS cannot name a style to use it later. However, LESS does have powerful forms of abstraction. LESS has the ability to define reusable styles and apply them to HTML that did not have those styles in mind. If you put the definition of reusable styles in one document and the application of those styles in another document, you achieve true separation. And it is already happening a little bit. You can do it in your own code.

It is a bit like a software library. We put the reusable bits in the library, and their specific use in the app.

Examples: Compass, Semantic Grid System

Characteristics: Semantic markup, Reuseable Styles, Tie-in document to relate Style to Content

Conclusion

CSS preprocessors, which began as convenience tools, is actually powerful enough to solve fundamental problems with HTML and CSS. While it is still early, LESS and other CSS preprocessors, if harnessed correctly, could dramatically transform how we build and design web sites. Typography, grids and layout, and other design concerns can be used as plugable libraries. And other languages that are specifically designed to do that may emerge. What would a systematic, analytical approach to such an approach look like?

Clojure Web Security

Summary: Use the OWASP Top Ten Project to minimize security vulnerabilities in your Clojure web application.

Aaron Bedra gave a very damning talk about the security of Clojure web applications. He went so far as to say that Clojure web apps are some of the worst he has seen. You should watch the talk. He has some good recommendations.

One of the jobs of web frameworks is to handle security concerns inherent in the web itself. Because most Clojure programmers build their own web stack, they often fail to look at the security implications of their application. They do not protect their site from even the easiest and most common forms of vulnerabilities. These vulnerabilities are problems with the way the web works, not with the particular server technology, yet it has become the server's responsibility to mitigate the vulnerabilities. Luckily, the vulnerabilities are well-studied and there are known fixes.

The Open Web Application Security Project (OWASP) does a very good job of documenting common web vulnerabilities and providing good fixes for them. They have a project called the Top Ten Project which every web developer should refer to regularly and use to improve the security of their app. You should also run through the Application Security Verification Standard checklists to audit your code. But the Top Ten should get you to understand the basics.

Warning: I am not a security expert. You should do your own research. The code I present here is my own interpretation of the OWASP recommendations. It has not been audited by experts. Do your own research!

Also, security is an ongoing concern. If you have any comments, suggestions, or questions, please bring them up!

Here is the Top Ten 2013 with a small breakdown and a Clojure solution, if applicable.

A1. Injection

If a server accepts input from the outside and then parses and interprets that input as a scripting or query language, it is open to attack. The most common form is SQL Injection, where an input form is posted to the server, the value of that form is concatenated into a string to make a SQL statement, and then the SQL statement is sent to the database to be executed. What happens if a malicious user types in "'; DELETE FROM USERS;"?

My preferred solution to SQL Injection in Clojure is to always use parameterized SQL statements. clojure.java.jdbc, supports these directly. The parameters will be escaped, making injection impossible.

Another problem is if you want to read in some Clojure data from the client, and you call clojure.core/read-string on it. read-string will execute arbitrary Java constructors. For instance:

#java.io.FileWriter["myfile.txt"]

This will create the file myfile.txt or overwrite if it already exists. Also, there is a form (called read-eval form) to execute code at read-time:

#=(println "Hello, vulnerability!")

Read in that string, and it will print. Any code could be in there.

The solution is to never use clojure.core/read-string. Use clojure.edn/read-string, which is a well-documented format. It does not run arbitrary constructors. It has no read-eval forms.

Summary: Always use parameterized SQL and use clojure.edn/read-string instead of clojure.core/read-string on edn input.

A2 Broken Authentication and Session Management

Authentication

This is a big topic and I can't address it all here. Clojure has the Friend library, which is the closest thing we have to a de facto standard. My suggestion is simply to read the entire Friend README and evaluate whether you should use it. This is serious stuff. Read it.

Session Management

Ring provides a session system which is fairly good. It meets many of the OWASP Application Security Verification Standard V3 requirements. But it does not handle all of them automatically. You still need code audits. For instance, if you are logging requests, OWASP recommends against logging the session key. You must ensure that the session key is added after the request is logged.

The ASVS also recommends expiring your sessions after inactivity and also after a fixed period, regardless of activity. Ring sessions do not do this automatically (the builtin mechanism has no notion of expiration) and the default implementations of session stores will store and accept sessions indefinitely. A simple middleware will do the trick of expiring them in both cases:

(defn wrap-expire-sessions [hdlr & [{:keys [inactive-timeout
                                            hard-timeout]
                                     :or {:inactive-timeout (* 1000 60 15)
                                          :hard-timeout (* 1000 60 60 2)}}]]
  (fn [req]
    (let [now (System/currentTimeMillis)
          session (:session req)
          session-key (:session/key req)]
      (if session-key ;; there is a session
        (let [{:keys [last-activity session-created]} session]
          (if (and last-activity
                   (< (- now last-activity) inactive-timeout)
                   session-created
                   (< (- now session-created) hard-timeout))
            (let [resp (hdlr req)]
              (if (:session resp)
                (-> resp
                    (assoc-in [:session :last-activity] now)
                    (assoc-in [:session :session-created] session-created))
                resp))
            ;; expired session
            ;; block request and delete session
            {:body "Your session has expired."
             :status 401
             :headers {}
             :session nil}))
        ;; no session, just call the handler
        ;; assume friend or other system will handle it
        (hdlr req)))))

Set the HttpOnly attribute on the session cookie. Very important for preventing stealing of session ids from XSS attacks.

Do not set the Domain attribute, and do set the Path if you want something more restrictive than / (the Ring session default).

Do not set the Expire and Max-Age attributes. Setting them makes the browser store the session id on disk, which simply expands the number of ways an attacker can get ahold of it.

Change the session cookie name to something utterly generic, like "id". You don't want to leak more information than necessary about how your sessions work.

Use HTTPS if you can and set the Secure attribute of the cookie.

Do not use in-cookie sessions. In-memory are good but they can't scale past one machine. carmine has a redis-based session implementation.

Summary: Here's how I use Ring sessions (with carmine) based on these OWASP recommendations.

(session/wrap-session
   (wrap-expire-sessions
    handler
    {:inactive-timeout 500
     :hard-timeout 3000})
   {:cookie-name "id"
    :store (taoensso.carmine.ring/carmine-store redis-db
             {:expiration-secs (* 60 60 15)
             :key-prefix ""}) ;; leak nothing!
    :cookie-attrs {:secure true :httponly true}})

A3 Cross-Site Scripting (XSS)

Whenever text from one user is shown to another user, there is the potential for injecting code (HTML, JS, or CSS) that is run in the victim's browser. Imagine if Facebook allowed any HTML in the post submission form. A malicious user could add a <script> tag with some keystroke logging code. Anybody who viewed that post in their feed would also get the key logger installed. That would be bad.

XSS is common because of how easy it is to make an app that stores user input (from a form post) in a database, then constructs the page out of stuff from the database. If you're not extremely careful, you could create a place where people can exploit each other.

The solution is to only use scrubbed or escaped values to build HTML pages. Because HTML pages can include different languages (HTML, CSS, JS), text needs to be scrubbed differently in each context. OWASP has a set of rules to follow which will guarantee XSS prevention.

hiccup.util/escape-html (also aliased as hiccup.core/h) will escape all dangerous HTML characters into HTML entities. JS and CSS still need to be handled, and rules for HTML attributes need to be followed.

If you want to allow some HTML elements, you will need to do a complex scrub. Luckily, Google has a nice Java library that sanitizes HTML. Use it.

Summary: Validate and scrub input from the user and scrub/escape text on output.

A4 Insecure Direct Object References

This one is a biggie: each handler has to do authentication. Does the particular logged in user have access to the resources requested? There's no way to automate this with a middleware. But having some system is better than doing it ad hoc each time. Remember: an attacker can construct any URL, including URLs with a database key in it. Don't assume that just because a request contains a key, the user must have the rights to it.

Summary: Always check the authority of the requesting session before performing an action.

A5 Security Misconfiguration

This is about keeping your software up to date and making sure the settings of all software makes sense.

A6 Sensitive Data Exposure

Having data is risky. Don't let it leak out.

A7 Missing Function Level Access Control

Use an authorization system (Friend) and audit the roles used for access control.

A8 Cross-Site Request Forgery (CSRF)

Let's imagine you have a bank account at Bank of Merica. You just checked your balance and didn't log out. Then you go to some public forum, where someone has posted a cool file. There's a big download button. You click it, and the next thing you know, you're on your bank page and all of your money has been transfered out of your account.

What happened?

The download button said "Download" but it was really a form submit button. The form had hidden fields "to-account", and "amount". The action of the form was "http://www.bankofmerica.com/transfer-money". By clicking that button, the form was posted to the bank, and because you were just logged in, oops, it transfered all your money away.

The solution is that you only want to accept form posts that come directly from your site, which you control. You don't want some random person to convince people to click on other sites to be able to transfer people's money like that.

There are several possible solutions. One approach is to add a secret to the session and also insert that secret into every form. That is the approach taken by the ring-anti-forgery library.

The solution that I like is to do a double-submit. This means you submit a secret token in the cookie (sent with each web request) and in a hidden field in the form. The server confirms that the cookie and the hidden field match. But the hidden field in the form is added by a small Javascript script which reads it from the cookie. Browsers don't allow Javascript to read cookies from other sites, so you guarantee that they form was posted from your site.

There are three parts to the solution.

  1. Install a secret token as a cookie.
  2. Install a script to add the hidden field to all forms.
  3. Check that the field matches the cookie on POSTs.

Here is some code to do 1 and 3.

(defn is-form-post? [req]
  (and (= :post (:request-method req))
       (let [ct (get-in req [:headers "content-type"])]
         (or (= "application/x-www-form-urlencoded" ct)
             (= "multipart/form-data" ct)))))

(defn csrf-tokens-match? [req]
  (let [cookie-token (get-in req [:cookies "csrf"])
        post-token   (get-in req [:form-params "csrf"])]
    (= cookie-token post-token)))

(defn wrap-csrf-cookie [hdlr]
  (fn [req]
    (let [cookie (get-in req [:cookies "csrf"]
                         (str (java.util.UUID/randomUUID)))]
      (assoc-in (hdlr req) [:cookies "csrf"] cookie))))

(defn wrap-check-csrf [hdlr]
  (fn [req]
    (if (is-form-post? req)
      (if (csrf-tokens-match? req)
        ;; we're safe
        (hdlr req)
        ;; possible attack
        {:body "CSRF tokens don't match."
         :status 400
         :headers {}})
      ;; we don't check other requests
      (hdlr req))))

The Javascript should be something like this:

(def csrf-script "(function() {
  var cookies = document.cookie;
  var matches = cookies.match(/csrf=([^;]*);/);
  var token   = matches[1];
  $('form').each(function(i, form) {
    if(form.attr('method').toLowerCase() === 'post') {
      var hidden = $('<input />');
      hidden.attr('type', 'hidden');
      hidden.attr('name', 'csrf');
      hidden.attr('value', token);
      form.append(hidden);
    }
  })
}());")

You should add it to all HTML pages. Note that this example script requires jQuery. Put it right before the </body>.

[:script csrf-script]

The nice thing about this solution is that it is strict by default. If you don't include the script, form posts won't work (assuming wrap-check-csrf is in your middleware stack).

Summary: CSRF attacks take advantage of properties of the browser (instead of properties of your server), so their defense can largely be automated.

A9 Using Components with Known Vulnerabilities

Software with known vulnerabilities is easily attacked using scripts. You should ensure that all of your software is up-to-date.

A10 Unvalidated Redirects and Forwards

One common pattern for login workflow is to have a query parameter that contains the url to redirect to. Since it's a user parameter, it's open to the world and could be a doorway for attackers.

For example, let's say someone sends an email to someone asking them to log in to their bank account. In it, there's this link:

http://www.bankofmerica.com/login?redirect=http://attackersite.com

What happens when they click? They see the legitimate site of their bank, which they trust. But it redirects them to the attacker's site, which has been designed to look like the bank site. The user might miss this change of domains and unwittingly reveal private information.

What can you do?

OWASP recommends never performing redirects, which is impractical. The next best thing is to never base the redirect on a user parameter. This would work, but puts a lot of trust in the developers and security auditors to check that the policy is enforced. My preferred solution allows redirects that conform to a whitelist of patterns.

(def redirect-whitelist
  [#"https://www.bankofmerica.com/" ;; homepage
   #"https://www.bankofmerica.com/account" ;; account page
   ...
  ])

(defn wrap-authorized-redirects [hdlr]
  (fn [req]
    (let [resp (hdlr req)
          loc (get-in resp [:headers "Location"])]
      (if loc
        (if (some #(re-matches % loc) redirect-whitelist)
          ;; redirect on our whitelist, it's ok!
          resp
          ;; possible attack
          (do
            ;; log it
            (warning "Possible redirect attack: " loc)
            ;; change redirect back to home page
            (assoc-in resp [:headers "Location"] "https://www.bankofmerica.com/")))
        resp))))

Summary: Redirect attacks can largely be avoided by checking the redirect URL against a whitelist.

Conclusion

Web security is hard. It takes education and vigilance to keep our servers secure. Luckily, the main security flaws of the web are well-understood and well-documented. However, this is only half of the work. These need to be translated into Clojure either as libraries and simply as "best practices". Further, these libraries and practices need to be discussed and kept top-of-mind.

If programming the web in Clojure interests you, you might be interested in my Web Development in Clojure video series. It covers all of the basics of web development, building a foundation to understand the entire Clojure web stack.

Master the fundamental tools and concepts of web development in Clojure
Learn more