Lisp, Programming

Meditations on Lisp

Lisp is a fantastic programming language, one which I’ve toyed with for many years, but have been taking far more seriously in recent months.  One of the more interesting secrets about Lisp is that singular moment of clarity that hits every aspiring Lisper, where they finally begin to ‘grok’ Lisp, the universe unfolds about them and even the toughest programming challenges suddenly become a mere macro away.  Then they start thinking about all programming languages in terms of Lisp.  Then they start dreaming in Lisp.  Then they see their own mind unfold as a Lisp program in endless recursion.  The world and the mind become more beautiful and intricate as they dance among the cons cells to the rhythm of cadaddr and lambda calculus…

Well, unless you’re already a Lisper, I’ve probably lost you.  Even if you’re a professional developer like me, it’s unlikely that you’ve communed with your code so deeply that you’ve found yourself reduced to self-referential software.  This is normally an exclusive gift of functional programming languages, such as the Lisp family (including Common Lisp, Scheme, Clojure, Arc, etc.), Haskell, and OCaml.

A Little (Too Much) History

I’ve been programming since I was 5 years old.  It should be no surprise then that I achieved my first ‘moment of clarity’ with PHP and XML right around my 20th birthday.  At the time I was using PHP for everything—of course, PHP is a tangled, bastardized mess of CGI, Perl, C, C++, TCL, and ASP, and the hypnagogic vision of myself reduced to XML-parsing PHP4 code was nothing short of a waking nightmare.  It was almost enough to turn me off development altogether.  Obviously—since I work as a professional contract developer, primarily porting proprietary software and web apps to open source technologies and linux platforms—I dusted myself off, went back to C and C++, and added Objective-C and Python to my repertoire; but I still find myself occasionally tangling with PHP from time to time.  It’s a useful language to know, and once you see your entire mind output in it, you can’t help but be damn good at it.  Anyway, this isn’t a post about C-family languages, Python, or PHP—it’s a post about Lisp.

Even though I’m a good enough programmer to write, parse, and/or compile Python, PHP, and C in my head, ultimately seeing the source code as a computer would, and even though Python used to be my favourite programming language, I was beginning to find something negatively nihilistic about all of my previous top choices.  Coming from a GW-Basic, TurboPascal, Visual Basic, and Visual C++ programming background as a child and teenager, the C family and Python are obvious and robust choices for me.  But something still felt missing.  The code felt empty, and pointless.  It was as if programming was just something I happened to be good at, but it had nothing to do with my true self.

As for Lisp, I think I first stumbled upon it when I was 7 or 8, as I trolled various branches of the Winnipeg Public Library looking for exciting programming books.  It seemed interesting, but I kept being swayed away and tempted by the flash and bang of Pascal, the proclaimed power of C++, and the ease and speed of Basic.  My raven’s eye continued leading me astray well into the 90s—Visual Basic 3, Visual C++ 4, Visual Basic 5—and then the internet revolution struck.  I just had to learn Graphic Design, HTML, CSS, JavaScript, VBScript, ASP, ColdFusion, Java, and CGI/Perl.  And I certainly did.  At 14 I started my first business, a little bedroom web design company, which I’m still proud of to this day.  It didn’t end well, since within a year my over-the-age-of-majority business partner decided that, as a minor, I had no right to own the business I founded and tried to turn me into an employee.  Needless to say, I quit and let the business implode—but at the same time, I wasn’t ignoring my software development, hardware, and physics interests.  It was also around this time that I designed my own programming language (I called it SIMPL, occasionally adding and subtracting a final E for style only, the Structured and Integrated Markup and Programming Language) and drew up the specs for a hybrid optical and quantum computer (The VoRTeX), that called for a Linux-based OS written in SIMPL.  I still have one of my journals from that time where I noted down the basic ideas and sketches I had in mind, and a very early code sample from the first draft specification of SIMPL.

As I’ve said elsewhere on this blog, I used to be a member of the Nano Computer Dream Team online community; and once again Lisp poked out it’s head at me, took a couple sniffs, and then went back into hiding.  I noticed but didn’t pursue, since the fall-out from my former web design company left me looking for work; and since I was always building my own computers, I ended up doing that on a contract basis for friends, and friends of friends, until it took off.  I had already been playing around with game AI for a few years, and became more than a little inspired by my online associates in the NCDT; Lisp is the mother tongue of AI research and development, so obviously it came up in conversation on a regular basis.  I might have even started looking seriously into Lisp then, if it wasn’t for my sudden need to extract myself forcefully from my unfortunate and heavily constraining Mormon upbringing, which was becoming increasingly antagonistic to all my real interests (especially in theoretical physics, AI, and robotics).  The only relevant part of that major segue in my life to this story is that it took me away from computers for a couple years, until I got reasonably settled enough in Toronto to start looking for programming and web design jobs.  Enter PHP and XML.  They were all the rage in the early 2000s, and every other coder and hacker I knew was building their own CMS in PHP.  It had to be done, as if it was a rite of passage.  Of course, none of those CMSs were even half as good as a Django app you can throw together in half-an-hour now; it was a simpler time then, you had to do everything from scratch, but PHP gave you leverage over the dusty CGI/Perl and ASP powered sites.  It made once impossible web development tasks a mere challenge, one which we were all hungry to take on.  I admit, I probably took PHP too far, didn’t sleep enough, and drank way too much coffee while hacking away at my own CMS—but as I said, I eventually had that terrifying vision of my entire mind reduced to PHP and XML, scrolling along the ceiling in bright, blood-red monospace as sleep eluded me.

It is this very reason, the hypnagogic PHP+XML vision, that when I got into UofT I chose not to study Computer Science, even though it was the most obvious choice for me.  The CS department certainly called to me, all my best friends in school were in the CS department, I continued building my own computers, setting up totally customized linux boxes, designing and developing all my own web sites—but PHP had left me with a bad taste in my mouth.  I scorned it and Python (since it was being marketed at the time primarily to PHP developers, as a better alternative to PHP or Perl), and turned to the Mac world, while also taking my music, art, and writing more seriously.  I was especially pleased with the earliest versions of OS X—the sleek look of Mac OS with all the power of BSD; Cocoa, a powerful built-in open-source style application framework that made COM and the (at the time) new .NET framework look like primitive artifacts; and Objective-C, a sexy, true object-oriented programming language enough like my old friend C++ that I felt right at home, but without all the ridiculous nonsense that made C++ a headache in its own right.

It wasn’t long after this that another internet revolution struck.  AJAX and Ruby.  Then out of nowhere the mobile app boom hit.  I had to know everything, do everything.  For some mysterious reason, I felt that I had to stay on top of all of these technologies, and still find time for school, a full-time job, and all my art projects.  I forgot entirely about Lisp, AI, and robotics.  My raven’s eye had taken me as far away from my true self as I have ever been.  And then the recession hit—which, granted, most of the world is still dealing with—I had to buckle down and take whatever job I could get.  There was no time for programming anymore, but I did finally manage to squeeze out my first novel, Placeholder.

Lisp poked it’s head out at me again while I was writing Placeholder.  By the way, just to set the record, the “QCL” language I refer to throughout the narrative is Quantum Common Lisp.  While writing and researching it, I remembered the love of Lisp prevalent in the Nano Computer Dream Team, read an excellent article on Lambda Calculus being the most natural expression for Quantum algorithms, was somewhat saddened about how Lisp had completely fallen out of favour, and was reminded of its raw power of expression.  It seemed like the perfect fit for the quantum computer programming language that Konrad Schreiber would obsess over, both for its expressive power as well as the difficulty other programmers would have with it—especially in a future history where it had nearly disappeared entirely.  And even though I liberally used elements of C as well, the Placeholder theory itself is a lot more Lispy than it is C-like.

Yet still I did not start programming in Lisp.  In fact, it wasn’t until a year and a half ago (just a few months after the initial release of Placeholder in Hardcover, when I finally got a development job again), that I suddenly remembered that Lisp was almost exactly like the programming language I designed as a teenager, and I began to study it with the rigour it deserved.  My day job prevented me from making as much progress as I would have liked, as I had returned to contract PHP and Python development, with the occasional jQuery-powered interactive application prototype to keep things interesting; but I kept up my studies, Lisping away my nights, until I finally had a few bug-free macro powered terminal apps to show for it.  But ultimately, I was still spending most of my time with imperative languages, and I could feel inside that I wasn’t thinking in Lisp.  But it was this knowledge, this instinctual awareness of the incorrect ordering of my thoughts, that led me the right way.

This year alone, I’ve had at least three major ‘moments of clarity’ that preceded the most productive programming periods of my life, all triggered by meditating seriously over Lisp.  Even just thinking about Lisp is enough to reenergize and enliven the soul.  But all these moments, the major and the minor moments of clarity, are so much different than that hypnagogic nightmare in PHP, or even my most inspired Python or C code.  With Lisp, there is no emptiness inside, no horror lurking behind the blood-red glowing code.  There is only endless indigo, warm, inviting, and ever inspirational.

Lisp As Psychology

This trend has led me to an interesting line of reasoning.  According to Julian Jaynes, the brilliant psychologist behind The Origin of Consciousness in the Breakdown of the Bicameral Mind, consciousness developed in part as a result of language.  The idea was not so popular in his time, but now the principle has become ubiquitous and self-evident.  Language both limits and refines the manner in which an individual can describe the world and themselves; the development and increasing sophistication of language assists in the smooth functioning and technological advancement of society; as society evolves, so does consciousness—globalization, for example, has taken all the tiny tribes of the insecure, xenophobic, and horrendously inbred human race, and given us a wide view of culture through exchange.  We cannot help but be bettered as individuals as our culture and perspective expands to accept diversity in all its wondrous forms.  This multicultural expansion also gives us new words and languages, new ways to think and create, thousands of new worlds to explore every day.  And all of this is only possible through language, and the power it gives us to see ourselves and rewrite our own minds by only thinking the right words.

As most of my readers will already be aware, I talk on and on about the Unified Consciousness—the new psychology that will finally eradicate the old bicameral constraints of a divided mind.  Even though we all as humans in the 21st century are, to some degree or other, sentient, self-aware beings, the old bicameral paradigm still lingers everywhere.  We, like Tukulti of ancient Assyria, still point to the empty thrones of our false gods—but many, many people point while thinking that their god must still be there, if they could only strain their eyes and pray hard enough.

I think the problem lies in the nature of our current psychology, which is itself a limitation of our language.  The mind, as it exists now, is divided into a Conscious portion, variously called the Ego, Self, or Analog “I”, and an extremely complex and mysterious Sub/Un-conscious which we only ever get to experience in our dreams, periods of heightened stress, and sleeplessness.  But given the right language, we could access the full power of our minds, see ourselves more clearly than ever before possible in human history, and of course, experience the world with pure, unfiltered senses.

I spent quite a lot of time developing a Con-Lang that I named Phoeronitic (for obvious reasons), for this express purpose—to define the unified syntax and semantics necessary for developing a unified mind.  But all this time, the language I was trying to invent has been around since the beginning of Computer Science as we know it.  It is—what else?—Lisp.

Lisp—the Language of the Universe

There is a reason that Lisp gives programmers such clarity, and that every Lisper eventually experiences true gnosis with the language, while only the most obsessed and caffeine-addled non-lisp programmers ever experience with their language of choice what I did with PHP.  First of all, Lisp is organic, fluid, chaotic, while remaining mathematically pure; it doesn’t lock its programmers into the fascist rigidity of functional programming as manifested in Haskell—it is multi-paradigm, so you can program how you want.  Every implementation of ANSI Common Lisp comes with the most powerful macro system ever devised by humankind, enabled by the ever-pervasive duality of syntax, and more importantly, the unification of syntax and semantics, the very same goal I had in mind with my Con-Lang, and the goal I accidentally and intuitively achieved with SIMPL, the programming language I designed as a teenager.

Lisp is true to nature, and nature is best described through the language of physics.  Particularly, the language of Lambda Calculus gives us an insight into the functional and recursive nature of the universe as presented in an M-Theory driven fractal cosmology.  And in the programming world, only Lisp retains the spirit and soul of mathematics and physics without attempting to impose an artificial order upon us.

As I’ve quoted before, Aleister Crowley said that the one who lives their true will has the power of the universe behind them.  It strikes me deeply that one cannot then know their true will unless they speak the language of the universe.  They are all connected after all—Lisp is the language of the Universe, the voice and will of the true self.

Meditations on Lisp

The clarity of mind that comes with meditating on Lisp is undeniable.  The moment is so vivid and obvious that it’s the equivalent of a dose of NZT-48 as depicted in the movie ‘Limitless’: you know exactly what to do, you instantly solve a number of completely unrelated problems that have been tormenting you, you write better software in every programming language you know, but most importantly, you know yourself.  It therefore stands to reason that you can only really know yourself if you’re a Lisper.  Granted, that may be a purposeful overstatement, and I expect serious blowback from any Haskell devotees that stumble across this—but that aside, I think my point stands.

By way of self-initiation, I have taken up the daily study of Lisp and Lambda Calculus, particularly before bed so that I dream in Lisp every night, instead of just whenever my unconscious mind feels like it.  This is to keep both my conscious and unconscious minds focused on the pursuit of clarity through Lisp.  I also take the time every day to review the source code of various open-source Lisp projects and compilers online, to familiarize myself with a variety of implementation styles, functions, and macros.  While I can not, and do not desire to escape from my day job working primarily with Python, I have mostly stopped using Python on my personal projects, and intend to rewrite everything in Lisp.  I have already dabbled with reimplementing SIMPL, under a different name—that’s an important lesson for every Lisper, to write your own interpreter and compiler for an arbitrary language (whether your own or one you know), as well as writing domain-specific language macros to output source code in any given language translated from pure Lisp source.  I also intend to bridge Lisp to D-Wave’s Python Pack and Quantum Cloud API, so that I can start programming in Quantum Lisp, just like my fictional character in Placeholder—if this endeavour interests anyone else, please feel free to tell me so in the comments.

But I digress.  Lisp isn’t just about what you can do, it’s really about who you are.  This is why the meditations on Lisp are most important, more important than traditional methods of learning.  For each new aspect of Lisp I learn, it’s important that I take the time to mull over it for hours, grokking it in isolation, as well as in combination with everything else I’ve already grokked about Lisp.

Lisp programming is all about refinement, after all.  You don’t just write your functions and classes, use them until they’re useless and throw them away like in imperative languages and frameworks—no, Lisp is a friend that you play with for hours and never want to leave, no matter how late at night, and think about whenever you’re forced to be apart.  Lisp is a friend that you grow old with, and stay true to all the way to your grave.  Lisp is a legacy that never ages, shining all the brighter for every programmer that knows its name and the mysteries it contains.

Lisp gives to you as much as you give to it.  Conrad Barski was right, in both cases, when he said Lisp was alien technology, and pure magic(k)—if you give Lisp the thoughtful attention it deserves, it will give you your true self, and all the power of the universe that comes with it.

For one final treat, here is my favourite xkcd comic ever: