Eric Normand
PodcastClojureCoursesBook

JSON Serialization for APIs in Clojure

Written by Eric Normand. Published: July 30, 2019.

Summary: Clojure is well-suited for processing JSON, but there are some decisions you have to make to suit your application. The major decisions are actually easy, though they took me a while to figure out.

I tend to use JSON instead of edn for an API serialization format, if only because JSON is more readily readable from other languages. I could do both, but it's good to eat your own dogfood and test those JSON code paths.^1

edn is better, but JSON is generally pretty good. However, JSON's expressibility is decidedly a subset of the Clojure data structures, so there is considerable loss of information when going from Clojure to JSON. That information is not recovered in a round-trip, at least not automatically. There are lots of decisions that have to go into how to, at least partially, recover this.

One bit of information that is lost is the type of keys to a map. JSON only allows strings as keys. Clojure allows anything. But most of the time, I find myself using keywords for keys. I say most, but really, it's the vast majority. Maps are bundles of named values pretty much all the time.^2 So the optimal decision, after trying lots of combinations, is to convert keywords to strings (the default in JSON libraries I've seen) when emitting JSON; and to convert map keys (always strings in JSON) to keywords (also known as keywordize-keys) when parsing JSON. That covers nearly all cases, and pinpointed special cases can cover the rest.

But that's not the end of the keyword/string story. What about namespaces? Surprisingly, the two major JSON libraries, clojure.data.json and cheshire handle things differently. How do you parse a JSON key that has a slash in it, indicating a namespace? If we're keywordizing (as I suggest above), they both give namespaced keywords (keyword will parse around the /). But when emitting JSON, they act differently. clojure.data.json will emit the name of the keyword (and ignore the namespace) while cheshire emits a string with "namespace/name".

I like to keep the namespace, or, put another way, I like to drop as little information as possible. So I prefer the namespace approach. I'm not sure how to get clojure.data.json to do that. I just use cheshire.^3 The other gotcha for namespaces is that ClojureScript's clj->js and js->clj are similarly asymetrical.^4

Keywords in other places besides the keys of maps will just get turned into strings, but they don't get converted back to keywords. That sucks, but it's minor. You'll just have to convert them back some other way. At work, we use Prismatic Schema's coercions. They do the trick nicely, in a declarative way.

So, back to other JSON issues. The other issue is other data types. Dates, URI's, and UUID's are in our data as well. Dates, well, it's up to you how to encode them. I've always been a fan of the Unix timestamp. It's not exactly human readable, but it's universally parseable. There's also the ISO datetime format, which is probably a better bet---it's human readable and agreed upon among API designers. You could emit that as a string and coerce back to a Date object later.

URI's and UUID's are by definition strings, so that's easy. How do you set up cheshire to handle the encoders? It's pretty simple, really.

(cheshire.generate/add-encoder java.net.URI cheshire.generate/encode-str)

That means add the encoder for the java.net.URI type to be encoded as a JSON string. str will be called on the value. You can figure out the other types you need. There are some JSON emission settings built-in, including Date (the ISO string format) and UUID. Weirdly URI is not in there, so you have to add it.

What's next? Oh, pretty-printing. Yeah, I pretty-print my JSON to go over the wire. It's nice for debugging. I mean, who wants to curl one long, 1000-character line of JSON? Put some whitespace, please! How to do that?

(cheshire.core/generate-string mp {:pretty true})

That's right, it's basically built in, but you have to specify it. But, oh man, that's long. I don't want to type that, especially because my lazy fingers are going to not do it one time, then I'm going to look at the JSON in my browser and see a one-line JSON mess. So, what do I do? I put all my JSON stuff for each project in json.clj. It's got all my add-encoder stuff, and it's got two extra functions, just for laziness^5:

(defn parse [s]
  (cheshire.core/parse-string s true))

(defn gen [o]
  (cheshire.core/generate-string o {:pretty true}))

Or of course whatever options you want to pass those functions. This one is my choice---you make your choice. But these two functions are all I use most of the time. Parsing strings and generating strings. Woohoo! Much shorter and less to keep in my little head.

Well, that just about wraps up my JSON API story. There's the slight detail of outputting JSON from liberator, which is its own blog post. And there's a bit of generative testing I do to really challenge my assumptions about how I set up the round-tripping. But that, too, is another blog post for another day. Oh, and what about all that JSON middleware? Again, another post.

If you like peanut butter and you like jelly, you will probably like peanut butter and jelly sandwiches. If you like web and you like Clojure, you will most definitely like Web Development in Clojure, which is a gentle, soothing, visually rich video course ushering in the fundamentals of Clojure web development through your eyes and ears and down through your fingertips and into your very own Heroku-hosted web server. At least watch the preview!

  1. It may be hubristic to think anyone else will use my API.
  2. There are exceptions, but these typically are not communicated to the outside. Those that are need special-casing. C'est la vie!
  3. :key-fn in clojure.data.json only works for keys, not all keywords. Emitting a keyword in any other place emits the str of it, which includes the : in the string. Ick.
  4. clj->js uses name for keywords and str for symbols, so keywords lose their namespace when emitting JSON, but retain their namespace when parsing key strings as keywords.
  5. also known as ease, peace of mind, DRY, and cleanliness