Topi Kettunen

Sila Dev Log: New Beginning in LLVM

Some time ago, I was quite actively working on my very own compiler project whenever I had some spare time. Unfortunately, due to personal time constraints, this project has been on the back-burner for a while. I have recently started writing and recording a new solo music project, which has taken a good chunk of my time.

On top of recording music, I’ve been diving quite deeply into Opusmodus for composing and analyzing music, a beautiful example of what Common Lisp can enable in creative domains.

Another reason for the pause was that I had coded myself a little bit into a corner, mostly due to early architectural decisions that didn’t scale well. At one point, I even attempted a refactor. Thankfully, the codebase was still relatively small, but even then, it proved to be a frustrating experience.

A big part of that frustration came from the dynamic nature of Common Lisp, which, at least in my personal experience, made larger-scale refactoring feel “unnecessarily” difficult. Of course, I fully admit this is largely a skill issue, not an inherent flaw of the language. I still love Lisp. Its flexibility, its macro system, and its directness are all things I deeply value.

Also, I’ve just run into a situation, where I happen to need quite a bit of C++ and also LLVM. So this naturally got me thinking that maybe I should look into just rewriting my compiler in those. Another reason was the fact that I just recently got a new Apple Silicon machine and most of the compiler was written for x86-64. So now I would need to write a new backend for aarch64, which is why LLVM started to look like a feasible choice.

Of course going back to C++ from Common Lisp is in many ways a step back. I love utilizing Common Lisp’s dynamic nature pretty thoroughly and especially when working as a solo developer, it offers great perks. Due to this, I played around with the idea of just generating LLVM IR from Common Lisp without using the LLVM API directly, but then decided to go with the C++ API route, mainly since I just wanted to refresh some memories of C++ and also at least by doing that, I get exposed to the LLVM API completely. Which I find beneficial, especially when trying diving into LLVM more.

There was a moment when I considered switching the project to another language, something statically typed, like Haskell. But for me, despite appreciating Haskell and especially its contributions to PLT, it doesn’t really spark similar joy as programming Common Lisp does.

Now I do think Haskell offers something great in the form of its type system, which then made me into start looking at Coalton , which is essentially programming language that lives in the userspace of Common Lisp supercharging it by providing static typing to your regular Common Lisp programming.

Decisions to start over are never easy, but often necessary for progress. By doing this, I tend to address some of the shortcomings of the initial design. But also since I’m going with LLVM, I can also focus on some of the more interesting problems in the compiler construction.

The old Common Lisp is now archived in https://github.com/topikettunen/cl-sila and the future C++ and LLVM based development will be happening in https://github.com/topikettunen/sila