intro⌗

JSON serves as the back bone of most RPC data formats. It is embedded in modern day JavaScript and thousands of other languages by default in the standard library. Common Lisp is different because it was standardized 30+ years ago and JSON wasn’t around back then.

However, old age is never a problem for a language like Lisp as it evolves with the times. First, we’ll start off with encoding JSON by using classes. Second, we’ll decode JSON data into normal primitive lists. Lastly, we’ll expand this functionality to unmarshal JSON data directly into an instance of a class.

pick a package⌗

You have a list of choices here, but I recommend cl-json:

cl-json stands out to me because it supports dumb and smart decoding. Dumb decoding meaning decoding an object into an ordinary list of values. Smart decoding meaning decoding an object into a Common Lisp Object System class(CLOS).

encoding json⌗

Encoding json is extremely simple. Call cl-json:encode-json-to-string like so:

(cl-json:encode-json-to-string `(("key" . 123) ("key2" . 124) (1 2 3 4)))
;; produces this:
;; {"key":123,"key2":124,"1":[2,3,4]}

How about classes? Well, all class slots are encoded into their respective types without any need to do extra configuration. Class slot names are formatted using kebab-case.

decoding json⌗

Like mentioned before, cl-json, by default, decodes an object into an ordinary list of items.

Objects like {"key": 123} take the form of (("key" . 123)), while lists [123, "key"] are unmarshalled to normal lists (123 "key").

going deeper : unmarshalling json⌗

Classes come in handy while working in Common Lisp, luckily the package cl-json facilitates a way to decode json into an object.

While other languages & libraries unmarshal json into an already-existing object, cl-json creates an empty object and then assigns values to it. This tiny detail will come handy later.

However, this functionality isn’t on by default and requires a bit of tinkering with the inner workings of cl-json.

The way cl-json parses objects onto Common Lisp classes is through the prototype field. The prototype field must contain two string fields which are: lispClass and lispPackage.

lispPackage denotes the lisp package, which usually is common-lisp-user. lispClass denotes the name of the class, i.e user.

Let’s test out this idea. So, first, we must define a class that we can unmarshal to. Second, we have to input the prototype field with the appropriate class and package information. Lastly, we need to wrap the decoding function cl-json:decode-json-from-string with a special method that decodes onto classes.

defining a class⌗

I won’t go too much in depth of how to define a class in Common Lisp. I’ll just define the schema in JSON and contrast that with the Common Lisp class definition.

We’ll define a user class containing two string fields: username and password.

{
    "username": "John Doe",
    "password": "password"
}

(defclass user ()
    ((username :initarg :username :initform "" :type string :accessor user-username)
     (password :initarg :password :initform "" :type string :accessor user-password)))
; #<STANDARD-CLASS COMMON-LISP-USER::USER>

Now, you can create an instance of the class user through the make-instance function like so:

(make-instance 'user)
; #<USER {1003D71D73}>

creating our json data⌗

At this point, we need to create our own JSON data containing the class and package information. This part is very easy as all you need to do is literally copy and paste the previous json data and modify it slightly.

This is our starting point:

{
    "username": "John Doe",
    "password": "password"
}

Now, let’s add the class name and package name. We already know the class name because we defined it before; it’s the symbol after the function defclass.

As for the package, you can know the current package you’re using by typing (package-name *package*).

(package-name *package*)
; "COMMON-LISP-USER"

We have our fields’ values and we need to wrap them in a prototype object at the root of our json data, like so:

{
    "prototype": {
        "lispClass": "user",
        "lispPackage": "COMMON-LISP-USER"
    },
    "username": "John Doe",
    "password": "password"
}

quote blah blah blah

wrapping the decode method⌗

We mentioned before that when you want to decode something in cl-json, you use cl-json:decode-json-from-string. Well, to decode it into a CLOS object, you wrap that same method with cl-json:with-decoder-simple-clos-semantics, like so:

(cl-json:with-decoder-simple-clos-semantics
  (cl-json:decode-json-from-string "{\"prototype\": {\"lispClass\": \"user\", \"lispPackage\": \"common-lisp-user\"},
\"username\": \"John Doe\", 
\"password\": \"password\"}")) 
; #<USER {1004AB6FA3}>

That’s basically it. Congratulations, you can now unmarshal json, nifty, ain’t it?

But, this is all useless compared to what other languages offer. I mean, in golang, you wouldn’t have to include the silly prototype field, you’d just pass a pointer to the data structure and voila, you have your object. Surely, there must be a cleaner and more elegant way of unmarshalling json.

And there is, enter the cl-json:prototype class. The prototype class binds to the prototype JSON field that we used to get our CLOS object.

*end-of-object-handler*⌗

cl-json:end-of-object-handler is a designator for a function without arguments that’s called at encountering a closing brace for an object. By using cl-json:end-of-object-handler, we can inject the prototype field without any string manipulation to get the result we want.

But first we need to create our prototype class. To do that we have to use make-instance like so:

(defvar *user-prototype* (make-instance 'json:prototype :lisp-class 'user))

Then, we wrap end-of-object-handler with our version that sets the prototype for the object, and call the original end-of-object-handler.

;; memorize the old end-of-object-handler
(let ((fn json:*end-of-object-handler*))
  ;; lambda is a definition for a function
  (lambda ()
    ;; set the global variable prototype
    ;; to our prototype
    (let ((json:*prototype* *user-prototype*))
      (funcall fn))))

We could wrap this operation into a function:

(defun wrap-for-class (class)
  (let ((prototype (make-instance 'json::prototype :lisp-class class))
        (fn json:*end-of-object-handler*))
    (lambda ()
      ;; dynamically rebind *prototype* right around calling fn
      (let ((json::*prototype* prototype))
        (funcall fn)))))

*json-symbols-package*⌗

For cl-json to get access to the class accessors (writers and readers), it must be in its package. So, building on our last function, we need to define our final function called unmarshal-json.

What unmarshal-json does is

  1. Tell cl-json what package the class belongs to
  2. Wrap the decoding operation in CLOS semantics
  3. Inject our prototype
  4. Decode the actual data
(defun unmarshal-json (class data &optional (pkg-name :cl-user))
  ;; what package to use
  (let ((json:*json-symbols-package* (find-package pkg-name)))
    ;; wrap the decoding operation
    (json:with-decoder-simple-clos-semantics
      ;; inject our prototype
      ;; via *end-of-object-handler*
      (let ((json::*end-of-object-handler* (wrap-for-class class)))
        ;; decode the data
        (json:decode-json-from-string data)))))

glory⌗

Now, if you call the function unmarshal-json, you’ll get fancy unmarshalling like so:

(defvar *my-user* 
   (unmarshal-json 'user "{\"username\": \"John Doe\", \"password\": \"password\"}"))
; #<USER {1003819C73}>
(user-username *my-user*)
; "John Doe"
(user-password *my-user*)
; "password"

Getting JSON to work properly with Common Lisp was a hassle for me.

I started out by making my own unmarshalling macro for weeks on-end without any proper results, it’s only when I stumpled onto this question on stackoverflow that I finally understood how powerful cl-json truly is.