This is a description of the Common Lisp ecosystem, as of August 2015, from the perspective of a user and contributor.
The purpose of this article is both to give an overview of the ecosystem, and to help drive consolidation in each domain.
Each application domain has recommendations for consolidating that part of the ecosystem, and pointers for interesting future work.
Application Domains
Web Development
Backend
Clack, the equivalent of WSGI/Rack has existed since 2009, and is throughly tested and battle-tested. Three web frameworks – Caveman2, Ningle, and Lucerne – are built on top of it.
Clack is an HTTP server abstraction, that allows the user to write web applications (or, more reasonably, web application frameworks) without depending on a particular server.
The importance of using Clack cannot be understated: If you build an application directly on, say, Hunchentoot, you’re tied to Hunchentoot, and if a new, faster server – like Woo – comes out, you have to rewrite the entire application to use it. If you write a plugin for Clack – like clack-errors – it is automatically usable by all applications, regardless of framework, that are built on Clack, reducing useless duplication of code.
With Clack, switching from Hunchentoot to Woo, and enjoying the incredible speedup, is a simple matter of installing libev and changing a keyword argument.
Consolidation:
Stop using Hunchentoot directly. Use Clack, or even better, one of the frameworks built on it.
Future Work:
The foundation is finished, now it’s time to write higher-level layers. An extensible administration framework for Clack applications, like Django’s Admin, would be a good example.
Frontend
This is bound by Common Lisp’s ability to compile to JavaScript. Common Lisp being rather a sizeable language, the options are rather limited:
-
Parenscript: A DSL that compiles a subset of Common Lisp to idiomatic JavaScript.
-
JSCL: A CL-to-JS compiler designed to be self-hosting from day one. Lacks CLOS,
formatandloop.
Consolidation:
The best way to help consolidation is to drive one of the existing CL-to-JS implementations forward.
Future Work:
Something like CFFI, but for CL-to-JavaScript implementations, so bindings to JavaScript libraries don’t have to be rewritten if a new implementation comes out.
A tool for compiling Common Lisp to JavaScript files, independent of the actual implementation, to make it easier to go from Parenscript to JSCL or to some other implementation.
Command Line
Over the years some tools have cropped up in this area, the latest, and the one that sees to have gained most momentum, is Roswell, an implementation manager/installer and script runner. One neat feature is support for very easily compiling tiny scripts into executables, e.g., for building documentation.
It has recently also been used to install implementations in Travis, thus circumventing some of the problems of cl-travis.
Consolidation:
Kill cl-launch, use Roswell.
Future Work:
More Roswell scripts.
GUI
For quite a while, a major concern was the lack of a complete GUI solution. Well, now we have it: CommonQt plus Qtools. The former has years of use in real-life applications, and the latter is a layer to make everything simpler.
The biggest problem when using CommonQt is it requires Smoke to run, and getting the libraries can be difficult, especially on systems other than Linux. This is solved by Qtools, which depends on the qt-libs library. It downloads the Smoke libraries for whatever platform you’re on, and this makes setting it up and deploying applications easier.
Consolidation:
Focus on CommonQt, and help improve cl-cffi-gtk, but other libraries should be considered deprecated.
CLIM is interesting and was the last attempt to do any kind of research on how user interfaces should be built, but is not a viable option in 2015.
Future Work:
More tutorials and examples of using CommonQt and Qtools.
Machine Learning
CLML is a fairly extensive solution. It was developed by Mathematical Systems Inc., a Japanese company. Mike Maul then took it to GitHub and cleaned it up a little. A tutorial on time series is available.
Another candidate in this area is mgl, used by its author to win the Higgs Boson Machine Learning Challenge.
In the area of numerical code, a library I’ve always though was interesting in this domain is Antik, but sadly it depends on the GNU Scientific Library, making it GPL. There’s also mgl-mat and LLA.
Databases
cl-dbi provides a uniform interface to the various database server-specific libraries (cl-postgres, cl-mysql, etc.). SxQL provides a DSL for building safe, automatically parameterized SQL queries.
There are two fairly complete ORMs: Crane, by yours truly, and Integral, by the author of cl-dbi.
Consolidation:
Discourage using anything other than cl-dbi.
Future Work:
Bindings for other database systems, e.g. Oracle, exist. Writing drivers for cl-dbi would be the best course of action and help consolidation.
Graphics
I’ve never done any graphics programming, so my knowledge of this area is lacking. There’s CEPL, and its sister project, Varjo, which have a nice collection of video tutorials. Of course, there are lower level libraries, like cl-opengl and cl-sdl2.
Consolidation:
Promote the use of CEPL, because it’s fairly complete.
Future Work:
A high-level OpenGL library, like pg, would be great.
More libraries in this area, especially for manipulating different mesh or other 3D formats.
Concurrency
cl-async is probably the most complete solution to anything concurrency related. And it’s built on libuv, the library that powers Node.js.
Other libraries of interest in this area are,
-
STMX: Provides support for software transactional memory, which is pretty impressive.
-
lparallel: A very complete framework for parallel programming.
Libraries like legion simplify concurrency for specific use cases.
Consolidation:
Future Work:
There’s plenty of room for new ideas in this area.
File Formats
There exist Common Lisp libraries for all the major file formats:
A new player in the field of JSON libraries is Jonathan, a very fast JSON encoder and decoder.
Consolidation:
There are too many XML and JSON libraries, this leads to choice paralysis.
Future Work:
A YAML parser so that cl-yaml doesn’t depend on the libyaml library
would make distribution far simpler.
System
UIOP, ASDF’s portable compatibility layer, contains a large set of tools for portably doing everything from querying the hostname to running external programs to manipulating environment variables.
Future Work:
An interesting project to develop in this area would be a DSL for building LLVM IR, either as a string or using the LLVM API bindings. This would allow you to use Common Lisp as an assembly language with a very powerful macro system. A use case: JIT-compiling highly-optimized, SIMD-vectorized numerical code.
Other
- Benchmarking
- Building a framework like Haskell’s Criterion would be a good step forward.
- Parsing
- My go-to library in this area is esrap, which generates parsers from Lispy grammar definitions. More diversity in this area would be welcome. Recently, proc-parse, a fast vector parsing library, was released.
- Logging
- There is not much to say about this, except that Common Lisp has good tools in this area, like log4cl and vom.
Development
Type System
There’s not much to say here, except that Common Lisp has a pretty great type system that is not exploited nearly enough.
Consolidation:
Not applicable. CLOS has no competition, e.g. from revived versions of Flavors, and is basically the only OOP system there is in Common Lisp.
Future Work:
I often reach for trivial-types, a library that contains a bunch of simple type specifiers. A larger library, e.g. including this CDR, would be good.
Testing
There are many existing test frameworks, the main ones being FiveAM and the much newer Prove.
Of the existing libraries, Prove is a good candidate for the one to settle on. Extensible test reporters are a great idea, and the general approach to extensibility makes it a good solution to the problem of test frameworks.
Other, older libraries depend on test frameworks like rt or lisp-unit, but
these aren’t relevant anymore.
Consolidation:
Discourage using testing frameworks other than FiveAM and Prove.
Future Work:
Writing any new test frameworks will be counter-productive at this point. Work should focus on consolidating the existing ones.
Testing on the Cloud
Common Lisp has good support for using services like Travis and Coveralls, for testing code and tracking code coverage in the cloud, respectively. The two main libraries for this are cl-travis and cl-coveralls, and they are both very easy to use. Testing and tracking the coverage of a Common Lisp project is certainly easier than doing the same with a Python project, in my experience.
Consolidation:
cl-travis has some problems, namely, it requires sudo to install
implementations, which means it can’t use Travis’ new Docker-based
infrastructure.
Roswell solves this problem, so using Roswell should be encouraged over cl-travis. A tutorial on using Roswell with Travis is here.
Future Work:
Supporting more services, like Circle CI, is always useful. Tutorials are always a good addition.
Documentation
For online, automatically updated documentation, in the style of Read the Docs, we have Quickdocs, which extracts project API references using docparser.
There aren’t many documentation generators, surprisingly. I use Codex to generate my documentation. It’s written on CommonDoc, a library that provides a format-agnostic internal representation for documents. Codex is designed along the lines of Sphinx: documentation is written in prose, and you insert automatically-extracted API documentation (functions, macros, classes, etc.) into the text using macros.
Consolidation:
Promote the use of Quickdocs by linking to it in library READMEs, to new users, etc.
Future Work:
I’ll continue working on Codex and CommonDoc, and any improvements to CommonDoc (mainly, more output formats and more macros) will be reflected in Codex.
Unicode
This is essentially a solved problem in most implementations. Sometimes you can get tripped up if an implementation’s encoding format is set to something like Latin-1 instead of UTF-8, but those are easily fixed, e.g. in SBCL:
(setf sb-impl::*default-external-format* :utf-8)
Other than that, I’ve never run into any Unicode problems, even when interfacing with C libraries.
Consolidation:
Some implementations lag behind others in Unicode support, e.g. ABCL.
Future Work:
There is some work to be done, in the form of missing features of cl-unicode.
Package Management
Quicklisp is the de-facto package manager for Common Lisp. And that’s pretty much it. Don’t disrupt what works.
Build System
ASDF is the de-facto build system of Common Lisp. Everyone uses it.
Every project has an .asd file, called a system definition file, which defines
project metadata (author, maintainer, homepage, etc.) and the components.
This, to me, is one of the major selling points of Common Lisp. With languages like Python, every file imports whatever it needs, and your project becomes a massive graph of interdependent files. In ASDF, you basically list the files in your project in the order in which they are defined. Or, you can specify the dependencies between the files, and let ASDF figure out a linear ordering. The point is that dependencies are explicit, and clearly spelled out.
But, enough proselytizing. The point is that there is no competition to ASDF, not because nobody has bothered to create a competitor, but because ASDF out-competed all the other alternatives years ago. And that’s a good thing.
Consolidation:
Not applicable.
Future Work:
More ASDF components, e.g. for building C/C++ files. A platform-independent package manager for downloading the external C libraries required by a Lisp library would be amazingly useful.
IDEs
SLIME is a Common Lisp IDE built on Emacs, and the most widely-used CL IDE. The fact that it’s built on Emacs is a problem, as telling a new user they have to learn Emacs before using SLIME (which is not strictly true) is a significant artificial barrier to entry.
Consolidation:
Reduce the barrier to entry to using SLIME.
Future Work:
Non-Emacs IDEs can’t hurt. An Atom plugin that interfaces with a Swank server, for instance, would be pretty great.
Conclusion
Specific Problems
Communication
The number of people building web applications in Common Lisp who don’t know about Clack is too high. Obviously, there’s a problem in reaching potential users with good library choices.
Choice Paralysis
When someone asks what library to use to write code in a given domain, only the best library in that domain should be recommended. Saying “you also have X, Y and Z” only worsens the paradox of choice.
If someone asks what GUI toolkit to use, answer “CommonQt with Qtools”, don’t add “but you can also use cl-cffi-gtk or LTK…”
If someone asks what to use to build web applications, link them to one of the frameworks built on Clack. Don’t talk about Hunchentoot or Wookie or Woo or some other server.
If someone asks what IDE to use, point them to SLIME. Don’t tell them about some obscure abandoned project from 2005 that only runs CLISP.
And, most importantly: when someone asks which implementation to use, just say SBCL or CCL. Don’t tell them, “well, you can use ECL to embed Lisp or ABCL to run on the JVM”. Those are for more niche users, not new users asking how to get started.
Growth
Quicklisp Downloads
Below is the total number of downloads, of the top 100 most popular projects on Quicklisp, between January and July:
Repos
I am subscribed to this feed of new Common Lisp repos on GitHub. I wrote some code to query my Newsbeuter database:
(ql:quickload (list :dbi :yason :local-time :group-by))
(defvar *connection*
(dbi:connect :sqlite3
:database-name (merge-pathnames #p".newsbeuter/cache.db"
(user-homedir-pathname))))
(let* ((output (merge-pathnames #p"lisp-github.json"
(user-homedir-pathname)))
(query (dbi:prepare *connection*
"SELECT pubDate FROM rss_item WHERE feedurl = ?"))
(results (mapcar #'(lambda (result)
(local-time:unix-to-timestamp
(getf result :|pubDate|)))
(dbi:fetch-all
(dbi:execute query "http://planet.lisp.org/github.atom")))))
(with-open-file (stream output
:direction :output
:if-exists :supersede
:if-does-not-exist :create)
(yason:encode
(mapcar #'(lambda (group)
(list
(local-time:format-timestring nil
(first group)
:format (list :year #\- :month #\- :day))
(1- (length group))))
(group-by:group-by-repeated results
:keys (list #'identity)
:tests (list #'(lambda (a b)
(= (local-time:day-of a)
(local-time:day-of b))))))
stream)))
Acknowledgements
Thanks to Javier Olaechea for his comments on Unicode support, Eitaro Fukamachi and François-René Rideau, and Gabriel Gonzalez for the original State of the Haskell Ecosystem article.