This is a description of the Common Lisp ecosystem, as of August 2015, from the perspective of a user and contributor.
The purpose of this article is both to give an overview of the ecosystem, and to help drive consolidation in each domain.
Each application domain has recommendations for consolidating that part of the ecosystem, and pointers for interesting future work.
Clack, the equivalent of WSGI/Rack has existed since 2009, and is throughly tested and battle-tested. Three web frameworks – Caveman2, Ningle, and Lucerne – are built on top of it.
Clack is an HTTP server abstraction, that allows the user to write web applications (or, more reasonably, web application frameworks) without depending on a particular server.
The importance of using Clack cannot be understated: If you build an application directly on, say, Hunchentoot, you’re tied to Hunchentoot, and if a new, faster server – like Woo – comes out, you have to rewrite the entire application to use it. If you write a plugin for Clack – like clack-errors – it is automatically usable by all applications, regardless of framework, that are built on Clack, reducing useless duplication of code.
With Clack, switching from Hunchentoot to Woo, and enjoying the incredible speedup, is a simple matter of installing libev and changing a keyword argument.
Stop using Hunchentoot directly. Use Clack, or even better, one of the frameworks built on it.
The foundation is finished, now it’s time to write higher-level layers. An extensible administration framework for Clack applications, like Django’s Admin, would be a good example.
JSCL: A CL-to-JS compiler designed to be self-hosting from day one. Lacks CLOS,
The best way to help consolidation is to drive one of the existing CL-to-JS implementations forward.
Over the years some tools have cropped up in this area, the latest, and the one that sees to have gained most momentum, is Roswell, an implementation manager/installer and script runner. One neat feature is support for very easily compiling tiny scripts into executables, e.g., for building documentation.
It has recently also been used to install implementations in Travis, thus circumventing some of the problems of cl-travis.
Kill cl-launch, use Roswell.
More Roswell scripts.
For quite a while, a major concern was the lack of a complete GUI solution. Well, now we have it: CommonQt plus Qtools. The former has years of use in real-life applications, and the latter is a layer to make everything simpler.
The biggest problem when using CommonQt is it requires Smoke to run, and getting the libraries can be difficult, especially on systems other than Linux. This is solved by Qtools, which depends on the qt-libs library. It downloads the Smoke libraries for whatever platform you’re on, and this makes setting it up and deploying applications easier.
Focus on CommonQt, and help improve cl-cffi-gtk, but other libraries should be considered deprecated.
CLIM is interesting and was the last attempt to do any kind of research on how user interfaces should be built, but is not a viable option in 2015.
More tutorials and examples of using CommonQt and Qtools.
CLML is a fairly extensive solution. It was developed by Mathematical Systems Inc., a Japanese company. Mike Maul then took it to GitHub and cleaned it up a little. A tutorial on time series is available.
Another candidate in this area is mgl, used by its author to win the Higgs Boson Machine Learning Challenge.
In the area of numerical code, a library I’ve always though was interesting in this domain is Antik, but sadly it depends on the GNU Scientific Library, making it GPL. There’s also mgl-mat and LLA.
cl-dbi provides a uniform interface to the various database server-specific libraries (cl-postgres, cl-mysql, etc.). SxQL provides a DSL for building safe, automatically parameterized SQL queries.
There are two fairly complete ORMs: Crane, by yours truly, and Integral, by the author of cl-dbi.
Discourage using anything other than cl-dbi.
Bindings for other database systems, e.g. Oracle, exist. Writing drivers for cl-dbi would be the best course of action and help consolidation.
I’ve never done any graphics programming, so my knowledge of this area is lacking. There’s CEPL, and its sister project, Varjo, which have a nice collection of video tutorials. Of course, there are lower level libraries, like cl-opengl and cl-sdl2.
Promote the use of CEPL, because it’s fairly complete.
A high-level OpenGL library, like pg, would be great.
More libraries in this area, especially for manipulating different mesh or other 3D formats.
cl-async is probably the most complete solution to anything concurrency related. And it’s built on libuv, the library that powers Node.js.
Other libraries of interest in this area are,
STMX: Provides support for software transactional memory, which is pretty impressive.
lparallel: A very complete framework for parallel programming.
Libraries like legion simplify concurrency for specific use cases.
There’s plenty of room for new ideas in this area.
There exist Common Lisp libraries for all the major file formats:
A new player in the field of JSON libraries is Jonathan, a very fast JSON encoder and decoder.
There are too many XML and JSON libraries, this leads to choice paralysis.
A YAML parser so that cl-yaml doesn’t depend on the
would make distribution far simpler.
UIOP, ASDF’s portable compatibility layer, contains a large set of tools for portably doing everything from querying the hostname to running external programs to manipulating environment variables.
An interesting project to develop in this area would be a DSL for building LLVM IR, either as a string or using the LLVM API bindings. This would allow you to use Common Lisp as an assembly language with a very powerful macro system. A use case: JIT-compiling highly-optimized, SIMD-vectorized numerical code.
- Building a framework like Haskell’s Criterion would be a good step forward.
- My go-to library in this area is esrap, which generates parsers from Lispy grammar definitions. More diversity in this area would be welcome. Recently, proc-parse, a fast vector parsing library, was released.
- There is not much to say about this, except that Common Lisp has good tools in this area, like log4cl and vom.
There’s not much to say here, except that Common Lisp has a pretty great type system that is not exploited nearly enough.
Not applicable. CLOS has no competition, e.g. from revived versions of Flavors, and is basically the only OOP system there is in Common Lisp.
There are many existing test frameworks, the main ones being FiveAM and the much newer Prove.
Of the existing libraries, Prove is a good candidate for the one to settle on. Extensible test reporters are a great idea, and the general approach to extensibility makes it a good solution to the problem of test frameworks.
Other, older libraries depend on test frameworks like
these aren’t relevant anymore.
Discourage using testing frameworks other than FiveAM and Prove.
Writing any new test frameworks will be counter-productive at this point. Work should focus on consolidating the existing ones.
Testing on the Cloud
Common Lisp has good support for using services like Travis and Coveralls, for testing code and tracking code coverage in the cloud, respectively. The two main libraries for this are cl-travis and cl-coveralls, and they are both very easy to use. Testing and tracking the coverage of a Common Lisp project is certainly easier than doing the same with a Python project, in my experience.
cl-travis has some problems, namely, it requires
sudo to install
implementations, which means it can’t use Travis’ new Docker-based
Supporting more services, like Circle CI, is always useful. Tutorials are always a good addition.
For online, automatically updated documentation, in the style of Read the Docs, we have Quickdocs, which extracts project API references using docparser.
There aren’t many documentation generators, surprisingly. I use Codex to generate my documentation. It’s written on CommonDoc, a library that provides a format-agnostic internal representation for documents. Codex is designed along the lines of Sphinx: documentation is written in prose, and you insert automatically-extracted API documentation (functions, macros, classes, etc.) into the text using macros.
Promote the use of Quickdocs by linking to it in library READMEs, to new users, etc.
I’ll continue working on Codex and CommonDoc, and any improvements to CommonDoc (mainly, more output formats and more macros) will be reflected in Codex.
This is essentially a solved problem in most implementations. Sometimes you can get tripped up if an implementation’s encoding format is set to something like Latin-1 instead of UTF-8, but those are easily fixed, e.g. in SBCL:
(setf sb-impl::*default-external-format* :utf-8)
Other than that, I’ve never run into any Unicode problems, even when interfacing with C libraries.
Some implementations lag behind others in Unicode support, e.g. ABCL.
There is some work to be done, in the form of missing features of cl-unicode.
Quicklisp is the de-facto package manager for Common Lisp. And that’s pretty much it. Don’t disrupt what works.
ASDF is the de-facto build system of Common Lisp. Everyone uses it.
Every project has an
.asd file, called a system definition file, which defines
project metadata (author, maintainer, homepage, etc.) and the components.
This, to me, is one of the major selling points of Common Lisp. With languages like Python, every file imports whatever it needs, and your project becomes a massive graph of interdependent files. In ASDF, you basically list the files in your project in the order in which they are defined. Or, you can specify the dependencies between the files, and let ASDF figure out a linear ordering. The point is that dependencies are explicit, and clearly spelled out.
But, enough proselytizing. The point is that there is no competition to ASDF, not because nobody has bothered to create a competitor, but because ASDF out-competed all the other alternatives years ago. And that’s a good thing.
More ASDF components, e.g. for building C/C++ files. A platform-independent package manager for downloading the external C libraries required by a Lisp library would be amazingly useful.
SLIME is a Common Lisp IDE built on Emacs, and the most widely-used CL IDE. The fact that it’s built on Emacs is a problem, as telling a new user they have to learn Emacs before using SLIME (which is not strictly true) is a significant artificial barrier to entry.
Reduce the barrier to entry to using SLIME.
Non-Emacs IDEs can’t hurt. An Atom plugin that interfaces with a Swank server, for instance, would be pretty great.
The number of people building web applications in Common Lisp who don’t know about Clack is too high. Obviously, there’s a problem in reaching potential users with good library choices.
When someone asks what library to use to write code in a given domain, only the best library in that domain should be recommended. Saying “you also have X, Y and Z” only worsens the paradox of choice.
If someone asks what GUI toolkit to use, answer “CommonQt with Qtools”, don’t add “but you can also use cl-cffi-gtk or LTK…”
If someone asks what to use to build web applications, link them to one of the frameworks built on Clack. Don’t talk about Hunchentoot or Wookie or Woo or some other server.
If someone asks what IDE to use, point them to SLIME. Don’t tell them about some obscure abandoned project from 2005 that only runs CLISP.
And, most importantly: when someone asks which implementation to use, just say SBCL or CCL. Don’t tell them, “well, you can use ECL to embed Lisp or ABCL to run on the JVM”. Those are for more niche users, not new users asking how to get started.
Below is the total number of downloads, of the top 100 most popular projects on Quicklisp, between January and July:
I am subscribed to this feed of new Common Lisp repos on GitHub. I wrote some code to query my Newsbeuter database:
(ql:quickload (list :dbi :yason :local-time :group-by)) (defvar *connection* (dbi:connect :sqlite3 :database-name (merge-pathnames #p".newsbeuter/cache.db" (user-homedir-pathname)))) (let* ((output (merge-pathnames #p"lisp-github.json" (user-homedir-pathname))) (query (dbi:prepare *connection* "SELECT pubDate FROM rss_item WHERE feedurl = ?")) (results (mapcar #'(lambda (result) (local-time:unix-to-timestamp (getf result :|pubDate|))) (dbi:fetch-all (dbi:execute query "http://planet.lisp.org/github.atom"))))) (with-open-file (stream output :direction :output :if-exists :supersede :if-does-not-exist :create) (yason:encode (mapcar #'(lambda (group) (list (local-time:format-timestring nil (first group) :format (list :year #\- :month #\- :day)) (1- (length group)))) (group-by:group-by-repeated results :keys (list #'identity) :tests (list #'(lambda (a b) (= (local-time:day-of a) (local-time:day-of b)))))) stream)))
Thanks to Javier Olaechea for his comments on Unicode support, Eitaro Fukamachi and François-René Rideau, and Gabriel Gonzalez for the original State of the Haskell Ecosystem article.