intro⌗

Common Lisp is one of those languages that seem bad at first, but offers a lot of upsides when you use it. At first glance, this is such a bad language: It doesn’t have a lot of libraries, the libraries it has are outdated and possibly insecure, and at-last there aren’t a lot of developers using this esoteric language.

However, all these downsides can be countered with one upside and that is Power. Common Lisp provides the developer with so much power over the language itself that you don’t necessarily need a ton of developers, robust libraries, or anything of that sort.

The language itself can be modified by the developers using the language. You want strong pattern matching? You got it through the library trivia. What about strong typing? Sure, just use the library Coaltion.

When it comes to the web, the number one platform for software & app disturbition, Common Lisp of-course fails to provide any good libraries.

Whilst programming languages like Python, Java, or even Go offer robust, fast and secure libraries for http server development. Common Lisp has the old Hunchentoot and Woo libraries which only support HTTP version 1.1.

Luckily, there is an open standard called CGI(Common Gateway Interface) that allows applications to skip over the nitty gritty of the HTTP protocol and instead map functions to strings.

In today’s article, I’ll go over the advantages of using FastCGI in any language, better ways to compose web apps, and lastly applying Common Lisp to these principles to create a real web app with REST-like functionality.

CGI: Why?⌗

Well, it is far better to depend on an open standard that is implemented by popluar servers than depend on your standard libraries.

Sure, if you use a standard library, it is most likely more concise and cleaner than the CGI alternative. But, in-terms of robustness, servers like nginx have implemented CGI and FastCGI for a long duration(15+ years, older than a lot of porgramming languages).

Simply put, the standard has stood the test of time. In terms of robustness, it is better. In terms of performance, it is performant enough.

There is another point about mixing languages together. CGI and FastCGI allow easier mixing of languages because:

  1. You don’t depend one each language’s HTTP implementation; which could be a liability
  2. Unix Socket > Net Socket: don’t waste ports on unnecessary things.

Another point is that FastCGI supports having a unified code base. It is 2022, why should we program the server in language X, while the client is programmed in language Y. It is far easier to share languages because it allows for shared variables and data structures(less boilerplate).

FastCGI allows the developers to not worry too much about their language and therefore give more potential to historically more client-based languages like Dart.

Do note: FastCGI wasn’t created for maintaining a unified code base. You gain that because FastCGI makes it easy to implement server functionality.

Lastly, it is horizontally scalable. Servers essentially function as workers. Mount the workers’ sockets as files(because the workers should have file-servers) and voila, you have a horizontally scalable web app.

CGI vs FastCGI⌗

FastCGI is obviously faster than CGI. How? Well, FastCGI has one long-living process that takes in requests through either a network-socket(tcp) or a unix socket(file-based socket). In contrast, CGI launches the process per each request, which makes it cheaper for the operating system.

Thus, FastCGI is not only faster but more efficient in-terms of resources. In-case of performance issues, you only have to worry about your application’s algorithms rather than the Operating System and environment.

Better ways to compose web apps⌗

I have a rather striking idea that I want to detail on, which is: We build web apps the wrong way. Yes, all of us. We don’t know how to create web apps.

My theory is that web apps are built wrong because the tools are wrong. We ravel in how big our web app is, how complex and how much functionality it offers, when in-reality that’s not the purpose.

In a sense, I think web apps should be composed in container or a layer-like structure. So, the authentication layer must be seperated from the functionality layer and the functionality layer must be seperated from the protocol layer.

Web Apps are an interface to your program, not the program itself

With that being said, create a good program and expose it to the web. Do not create a web app with the idea of transforming it into a program.

Why? Because it is better. Web Apps are relatively new compared to the history of programming itself. There are tons of well developed books about program structure, not a lot of well developed books about web apps.

Testing is easier with a program. Performance tuning is easier with a program. A web app, not so much.

Web Apps: Conform or Die⌗

There is a trend I’ve began to notice amongst web developers and that is conformity. We all use the same tools, frameworks and languages, regardless of their merit.

That is a dangerous idea in of itself because it doesn’t reward creativity. Spending time learning about the newest frameworks, tools, and languages isn’t worthwhile because most likely it doesn’t offer anything ground-breaking.

Frameworks and libraries are in a constant state of conforming to each other rather than create something new. Take, for example, the popular library React.

For those who do not know, React makes building user interfaces easy because it allows developers to map data to a component, allowing for data updates to change the actual component. It is splendid at that.

React doesn’t have anything bad in itself, at-least not to my knowledge, but it is overly promoted. So much so that libraries like snabbdom (a 200 LOC react alternative) do not get any recognition. Not to claim anything about snabbdom being better than React, I haven’t tried snabbdom myself but new approaches like this aren’t covered that much due to companies emphasizing on using standard libraries even when normal libraries could possibly serve better.

To be completely fair: This behaivor isn’t exclusive to web apps, programming or anything else, but is instead a side effect on people collaborating. We naturally get lazy. This is something to consider when composing web apps: The standard will not necessarily serve you better than creating your own thing.

Why Common Lisp?⌗

Common Lisp can be used as a server-side language for your web app. Why Common Lisp? For easier debuggability, better performance over other languages and cleaner code.

Easier Debuggability⌗

Easier debuggability comes in the form of remote server introspection. Let’s demonstrate this better with a comparison with another language. Go, for example, can be debugged through running the server locally and trying to re-create the results that the user experienced again.

But, most Common Lisp implementations can be introspected right from your editor. Yes, with a little messing around, you could practicallity introspect, create new code and test it out without stepping in your local machine.

All you have to do is setup Swank on your remote machine, connecting through slime(swank's client) or something similar and voila, you could debug the server’s actual state, no need to re-create the server’s environment.

Add to that the fact that you could tell the compiler to help you with debugging through using the declaim function. This allows the compiler to give you a full stack trace of functions and their results.

This doesn’t awesome lest you actually try it, you never know how good debuggability in Common Lisp is until you actually try to use it.

Better Performance⌗

Common Lisp is easily faster than Python, Go, or similar languages. If not, it can be optimized for it through the declaim function or through using a different Common Lisp Implementation.

Even if Common Lisp was dramatically worse than the language you were using, you have better ways to tune it. Through the function disassemble, you could actually view the assembly code that a function generates.

Not only that but with implementations like SBCL, you could actually execute assembly code from the compiler.

Cleaner Code⌗

Through macros, one could easily make the codebase a lot more readable. A good example of this is using pattern matching to help parse time, telephone or even emails.

With other languages, pattern matching is either built in or not. There is no opt-out or opt-in. With Common Lisp, you could implement pattern matching if you want to use it.

(ql:quickload :trivia)
;; pattern-matching through a library
(defun mul (x)
 (trivia:match x
  (0 1)
  (_ (* x (mul (- x 1))))))
;; normal if statement
(defun mul (x)
 (if (= x 0)
  1
  (* x (mul (- x 1)))))

(defmacro ? (&body v)
    `(if ,@body))

;; or through question mark
(defun mul (x)
 (? (= x 0) 1
  (* x (mul (- x 1)))))

Another example where Common Lisp clearly shines is list processing. Imagine you have a list of numbers that you’d like to have modified so that every number in that list is multiplied by itself.

Essentially, you want this: list of numbers -> list of squared numbers. In something like C, C++, or Go, you’d loop over the elements, multiply the elements and lastly set the new values.

In Common Lisp, it is done the same way but most of the list processing is abstracted away. Take a look at the following code snippets:

// go
func modifyList(arr []int) {
 for i := 0; i < len(arr); i++ {
  arr[i] = arr[i] * arr[i]
 }
}

(defun square (x) (* x x))

(defun modifyList (arr)
 (mapcar #'square arr))
;; or a one-liner
(defun modifyList (arr)
 (mapcar #'(lambda (x) (* x x)) arr))

Let’s not forget about the Common Lisp Object System because it drives much of the core of Common Lisp. Essentially, the pros of the CLOS are:

  1. Classes can be redefined even when there are existing instances of the old class definition
  2. Classes can inherit multiple other classes
  3. Setters and Getters can be easily modified
  4. Wrappers around class definitions can be easily written to provide more functionality.
  5. Methods can have methods run before or after them

I failed to mention other benefits of the CLOS, I suggest you research the CLOS so that you fully comprehend how powerful the object system really is.

To illustrate how powerful the CLOS can be, I’ll use the package mito. Mito is a package that translates objects to SQL table definitions. Take a look at the following code:

(mito:connect-toplevel :mysql :database-name "myapp" :username "fukamachi" :password "c0mon-1isp")
;=> #<DBD.MYSQL:<DBD-MYSQL-CONNECTION> {100691BFF3}>

(mito:deftable user ()
  ((name :col-type (:varchar 64))
   (email :col-type (or (:varchar 128) :null))))
;=> #<MITO.DAO.TABLE:DAO-TABLE-CLASS COMMON-LISP-USER::USER>

(mito:table-definition 'user)
;=> (#<SXQL-STATEMENT: CREATE TABLE user (id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY, name VARCHAR(64) NOT NULL, email VARCHAR(128))>)

(mito:deftable tweet ()
  ((status :col-type :text)
   (user :col-type user)))
;=> #<MITO.DAO.TABLE:DAO-TABLE-CLASS COMMON-LISP-USER::TWEET>

(mito:table-definition 'tweet)
;=> (#<SXQL-STATEMENT: CREATE TABLE tweet (id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY, status TEXT NOT NULL, user_id BIGINT UNSIGNED NOT NULL, created_at TIMESTAMP, updated_at TIMESTAMP)>)

Which doesn’t look much different than the regular defclass function:

(defclass user ()
 ((name :type string :initarg :name :accessor user-name)
  (email :type string :initarg :email :accessor user-email)))

(defclass tweet ()
 ((status :type string :initarg :status :accessor tweet-status)
  (user :type user :initarg :user :accessor tweet-user)))

This section could be its own article. Common Lisp is beyond any other language in-terms of being concise and cohesive.

Using SB-FASTCGI⌗

SB-FASTCGI is a library that uses the C library libfcgi to create fastcgi based applications. Using it is quite simple, just import the library, link it to the actual library, create a socket with a function and voila you’re done.

Do note that this only works for SBCL(Steel Bank Common Lisp) hence the name sb-fastcgi. If this doesn’t suite you, you could use cl-fastcgi which is similar but offers worse performance.

;; import the library
(ql:quickload 'sb-fastcgi)
;; load the C library
(sb-fastcgi:load-libfcgi "/usr/lib/libfcgi.so.0.0.0")
;; define the main function
(defun main (req)
 (sb-fastcgi:fcgx-puts req "Content-Type: text/plain

hello from lisp"))

(sb-fastcgi:socket-server-threaded #'main :inet-addr "127.0.0.1" :port 9090)
;; or using a unix socket
(sb-fastcgi:socket-server-threaded #'main :sock-path "web.socket")

Afterwards, you could use a program like ashd to create a one-line fastcgi server like so:

htparser plain:port=8080 -- callfcgi -t 127.0.0.1:9090

Though, it doesn’t seem clear how to write files to the request, I think through writing directly to the t stream, one could write the file. But first, one shouldn’t forget to write the content-type of the file in the response header.

conclusion⌗

Common Lisp alongside FastCGI can provide for extremely pleasant web development. Common Lisp’s development environment is unmatched, and with FastCGI is becomes relatively easy to create backends for web applications.