Tag Archives: cl

top: The poor man’s performance analyzer

hunchentoot logoFor the sake of the future of all mankind, I wrote a tiny web server for proving that Hunchentoot works with certain system specs. “doeshunchentootwork” is only 77 lines long, serves a singe page and its favicon, and the application is compiled, not interpreted. So why does top show it using so many CPU cycles?

Daemonization may have been a bad idea, at least this early in development. The process spends a lot of time… doing what exactly?

Since the app is only 77 lines long, debugging wasn’t that hard. Long story short, CL loads the program and quits, so I was using (loop) to keep the program running. I know, I know, terrible. (read) is a better choice, since it doesn’t waste CPU by looping but merely blocks for command line input that will never arrive to a daemonized web server.

With the new code, top shows almost no activity for doeshunchentootwork unless someone is currently requesting the webpage. Whew! Now the resources can be wasted on other servers.

I just want Hunchentoot!

hunchentoot logoThere’s something to be said for tutorials like Lisp for the Web. Adam Peterson’s guide to the Hunchentoot web server is easy and fun. Trouble is, gathering equipment is difficult.

Setup should be relatively simple. Equipment needed: Common Lisp, Quicklisp, and Hunchentoot.

  1. Pick your Common Lisp: SBCL, CLISP, ECL, ABCL, CCL, LispWorks, AllegroCL, XCL, WCL, CMUCL, GCL, etc.
  2. Install Quicklisp.
  3. Run (ql:quickload "hunchentoot").
  4. Run (hunchentoot:start (make-instance 'hunchentoot:acceptor :port 4242)).

Depending on which operating system you have and which Common Lisp you used, you’ll get different errors when you follow those instructions.

Trouble is, getting a Common Lisp implementation with CFFI and thread support is difficult. I regularly program on Mac OS X, Linux, and Windows; I need languages which work well on all three operating systems. Sadly, there are few Common Lisps that have working ports.

SBCL comes closest, but it crashes on Xen (so no Hunchentoot on my Gandi.net webserver).

CLISP has CFFI on Mac OS X, but only if you use "fink install ffcall && fink install clisp +dynffi to get it. MacPorts CLISP is no go. Even with Fink, CLISP lacks thread support. Homebrew CLISP isn’t any different, and I doubt DarwinPorts is either.

ECL lacks thread support on Mac OS X and Ubuntu.

ABCL takes time to setup: Install sun-java6-jdk. Download ABCL. Unzip ABCL. Move JAR to home. Add JAR to CLASSPATH. Create alias for JAR command. Install rlwrap. Generate completions. Modify alias to use rlwrap. Jump off bridge. Reload shell configuration. And after all that, ABCL lacks thread support according to Bordeaux Threads/Quicklisp.

CCL’s installation procedure is involved.

LispWorks and AllegroCL are proprietary. Obtaining valid license files is bothersome, and I don’t trust the projects to stay current.

OpenMCL and CMUCL are out of date and unavailable for modern Ubuntu versions.

XCL and WCL are maintained by very few people, and they must be manually compiled from source.

GCL is out of date, it fails to build with MacPorts, and Fink doesn’t even know it exists.

newlisp logoFark it, newLISP has a built-in web server, and it runs on Mac OS X, Windows, and Linux.

 

ccl logoUpdate: Lispbox is just what the doctor ordered. It has working CCL, Emacs, SLIME, Quicklisp, CFFI, and threads.

Writing Scripts with Common Lisp

lisp alienScripting languages such as Python and Ruby naturally lend themselves to the task of writing command line utilities. Lisp is much older than these languages, so it’s not surprising that some Lisp implementations are more difficult than others for scripting. Some lisps have a library too small to be useful. Some lisps cannot access files. But most of all, many lisps have trouble with shebangs and scripted main.

Shebangs

Add the following to ~/.cmucl-init.lisp (or your initialization file):

;;; Play nice with shebangs
(set-dispatch-macro-character #\# #\!
 (lambda (stream character n)
  (declare (ignore character n))
  (read-line stream nil nil t)
  nil))

This instructs CL to ignore all lines beginning with #!, even in the REPL.

Scripted Main

Java, Python, Perl, Ruby, Lua, Haskell, and newLISP all have scripted main.

One of the most useful tools in a scripting language is a scripted main. I invented this term out of necessity: it’s hard to Google something without a name. Scripted main refers to a special function, typically called main, which is executed when a script is run from the command line (e.g. python myscript.py or ./myscript.py). If the script is imported by another script, main() is not called. Scripted main is extremely useful for writing programs that do this:

$ ./welcome.py
Usage: ./welcome.py <name>
$ ./welcome.py Brandon
Welcome Brandon!

If another script greeter.py imports code from welcome.py, we don’t want greeter to call welcome‘s main function. The code in welcome that prints out Usage: ./welcome.py <name> probably calls sys.exit(0) soon after, which would kill greeter immediately after importing welcome.

How to do it

The next few examples are specific to CLISP. If you use SBCL, Clojure, or another implementation, you’ll need to modify the shebang lines accordingly.

Save as hello.cl:

#!/bin/bash
#|
exec clisp -q -q $0 $0 ${1+"$@"}
exit
|#
(defun hello-main ()
  (format t "Hello Main!~%")
  (quit))

(hello-main)

Run:

$ chmod +x hello.cl
$ ./hello.cl
Hello Main!

The chmod command marks hello.cl as a self-running executable so that ./hello.cl is allowed.

What if another script greeter.cl loads the code from hello? Let’s write a greeter that lets us choose which greeting to print.

Save as greeter.cl:

#!/bin/bash
#|
exec clisp -q -q $0 $0 ${1+"$@"}
exit
|#

(load "hello")

(defun greeter-main ()
  (format t "Ready to greet.~%")
  (quit))

(greeter-main)

Run:

$ chmod +x greeter.cl
$ ./greeter.cl
Hello Main!
Ready to greet.

The problem with this logic is that hello.cl‘s main function will be called as soon as the file is loaded. We don’t want this to happen, so we will use some special CMUCL code:

Resave as hello.cl:

#!/bin/bash
#|
exec clisp -q -q $0 $0 ${1+"$@"}
exit
|#

(defun hello-main (args)
  (format t "Hello from main!~%"))

;;; With help from Francois-Rene Rideau
;;; http://tinyurl.com/cli-args
(let ((args
       #+clisp ext:*args*
       #+sbcl sb-ext:*posix-argv*
       #+clozure (ccl::command-line-arguments)
       #+gcl si:*command-args*
       #+ecl (loop for i from 0 below (si:argc) collect (si:argv i))
       #+cmu extensions:*command-line-strings*
       #+allegro (sys:command-line-arguments)
       #+lispworks sys:*line-arguments-list*
     ))

  (if (member (pathname-name *load-truename*)
              args
              :test #'(lambda (x y) (search x y :test #'equalp)))
    (hello-main args)))

Resave as greeter.cl:

#!/bin/bash
#|
exec clisp -q -q $0 $0 ${1+"$@"}
exit
|#

(load "hello")

(defun greeter-main (args)
  (format t "Ready to greet.~%")
  (quit))

;;; With help from Francois-Rene Rideau
;;; http://tinyurl.com/cli-args
(let ((args
       #+clisp ext:*args*
       #+sbcl sb-ext:*posix-argv*
       #+clozure (ccl::command-line-arguments)
       #+gcl si:*command-args*
       #+ecl (loop for i from 0 below (si:argc) collect (si:argv i))
       #+cmu extensions:*command-line-strings*
       #+allegro (sys:command-line-arguments)
       #+lispworks sys:*line-arguments-list*
     ))

  (if (member (pathname-name *load-truename*)
              args
              :test #'(lambda (x y) (search x y :test #'equalp)))
    (greeter-main args)))

Now the mains function are only called when hello.cl orgreeter.cl are run directly, not when it is loaded.

Run:

$ ./hello.cl
Hello Main!
$ ./greeter.cl
Ready to greet.

This is the desired behavior. For more examples, check out problem.cl and sigil-clean.cl.

Syntactic barriers to learning Common Lisp

lisp logoFunctional programming is blessed to have several hardy languages. I have only tried two: Haskell and Common Lisp. From the few tutorials I’ve completed, it seems that Haskell is easy to learn, and Lisp hard to learn.

This difference is, I believe, due to Lisp’s various and sundry dialects, implementations, and macros. In Haskell, there are two major ways to specify local variables: let and where. Common Lisp has at least four: let, let*, flet, and labels. CL has a retarded number of loop constructs. I’m currently reading a sample chapter from Land of Lisp. Here is a code snippet:

(defun find-islands (nodes edge-list)
  (let ((islands nil))
    (labels ((find-island (nodes)
	       (let* ((connected (get-connected (car nodes) edge-list))
		      (unconnected (set-difference nodes connected)))
		 (push connected islands)
		 (when unconnected
		   (find-island unconnected)))))
      (find-island nodes))
    islands))

The defun syntax is intuitive; it’s like any other language’s function creator. And let makes sense: create some local variables and use them in sub-expressions. But labels baffles me. Would let have sufficed? Why or why not? After googling for a while (there is NO useable, complete Common Lisp reference document, and that includes HyperSpec and the Simplified Common Lisp Reference), I stumbled on a half-readable description of labels.

LABELS is special form for local function binding.

I could have told you that.

Bindings can be recursive and can refer to each other.

So labels is just like defun, defvar, and friends?

Each binding contains function name, arguments, and function body.

It seems that labels is just a special way to do defun locally. If Lisp is so powerful, why doesn’t Barkski do this:

(defun find-islands (nodes edge-list)
  (let ((islands nil))
    (defun find-island (nodes)
      (let* ((connected (get-connected (car nodes) edge-list))
	     (unconnected (set-difference nodes connected)))
	(push connected islands)
	(when unconnected
	  (find-island unconnected))))
    (find-island nodes)
    islands))

Nested functions don’t require special syntax in Haskell, Python, or Lua. Why should they in the premier functional language? I think the answer is that CL allows nested defuns, but lispers aren’t happy with that–they want a closure they can use in if, cond, and other non-code-block areas. A combination of let and lambda would do just that, but macro-addiction dictates that a new syntax be created. Now we have labels.

So that’s defun, let, and labels. Then there’s let*. Apparently let* is similiar to flet, which is similiar to labels , which is similar to let, except that I’ve forgotten where the fuck I was going with that. These macros are simply variations of let, some more powerful than others.

Land of Lisp does explain these macros, but if they weren’t used in the code, they wouldn’t have to be explained. Or, Barski could have used a separate example for each macro, but that would have taken more pages. There must be a balance between code simplicity and time to get the game running. I understand his dilemma. But if he errs to often on the side of code obscurity, then readers will simply close the book. I don’t know if I can write an informative and entertaining programming tutorial as well as Barski does, but I know what I like in one, and it’s not a never-ending stream of new syntax. I’m still going to buy the book, and I’m still going to love it. It’s just going to take longer for me to parse than I had expected.

Debriefing

sigilIf, like me, you have trouble editing ePub books with Sigil, then you’ll enjoy sigil-clean. Your ebook’s code blocks will return to their original, tidy state. sigil-clean is a Common Lisp script made available by yours truly, with help from Zach Beane.

Before:

As it turns out, it’s trivially easy to get the REPL to print âhello, world.â

 

CL-USER> "hello, world"
"hello, world"

After:

As it turns out, it’s trivially easy to get the REPL to print “hello, world.”

CL-USER> "hello, world"
"hello, world"

Common Lisp Travels

Intro

Tooling around with my Nook, I discovered a bug in Sigil, an ePub editor. Whenever a user toggles between the HTML code section and the book view section, a newline is inserted before every PRE tag. This means every code block in an ebook gets shoved way down the page. I thought that’s not so bad, filed a bug report, and began manually undoing those newlines.

Open ePub. Double-click chapter. Find a code block. Delete newlines. Repeat. After editing several chapters of Practical Common Lisp this way, I got an idea. Why not write a program to do this for me? The quest had begun.

Allegro CL seemed a likely choice. Sure, I have reservations about a commercial programming language, but Allegro was good enough for both Practical Common Lisp and Casting SPELs in Lisp. Common Lisp is difficult to install, so Lispbox is suggested.

Lispbox

My Internet connection was slow, but steady. I had Lispbox-0.7-with-acl81_express.dmg in no time. Double-click, double-click, drag & drop… done.

Once again, Allegro CL is commercial: free, but not open source. In particular, Allegro CL requires a license file. The Lispbox version of Allegro contains a program called newlicense. When executed, newlicense informs the user that Allegro is too old to warrant a license. Great.

This was too bad, as Lispbox comes with everything a Common Lisper could want: CL, Emacs, SLIME, and ASDF. Lispbox has, apparently, moved to http://common-lisp.net/project/lispbox/. I downloaded Lispbox-0.7-with-ccl-1.5-darwinx86.dmg, which contains a massive 400MB installation, which fails to accept any license file. Oh well, I’ll just have to install Common Lisp, etc. etc. manually.

Installing Common Lisp, etc. etc. Manually

The Allegro CL homepage offers a–surprise!–working edition of Allegro CL v8.2. The edition contains a working license as well as ASDF. Two down, six to go. I also downloaded Aquamacs (did I mention I was on Mac OS X?), which installed and ran equally well.

I don’t know the several hundred shortcut keys for Emacs, so I was eager to install SLIME, a plugin with such helpful features as code highlighting, indentation, compilation, execution, and debugging. Most CL implementations have only a basic interpreter; SLIME adds code completion as well.

Installing SLIME manually is moderately difficult. One must download the files, place them in an appropriate directory, and edit a ~/.emacs configuration file to inform Emacs that SLIME is in fact installed. After much retooling, here is my .emacs:

;; Initialize slime.
(setq inferior-lisp-program "/Applications/AllegroCL/alisp")
(add-to-list 'load-path "/Library/Application Support/Aquamacs Emacs/SLIME")
(require 'slime-autoloads)
(eval-after-load "slime"
  '(progn (slime-setup '(slime-fancy))))

;; Automatically use slime mode on .cl and .lisp files.
(add-to-list 'auto-mode-alist '("\\.cl$" . common-lisp-mode))
(add-to-list 'auto-mode-alist '("\\.lisp$" . common-lisp-mode))

;; Start slime.
(slime)

This code is scavenged from several different websites and a trip to Freenode. Lines 1-3 inform Emacs of the locations of a CL installation and a SLIME installation. Line 4 loads the SLIME framework. Lines 5-6 prepare SLIME such that it loads all of its features, including new keyboard shortcuts. Lines 8-10 specify that .CL and .LISP files should automatically open in SLIME mode so that the aforementioned SLIME features are applied to them. Finally, line 12 starts SLIME automatically just like Lispbox’s Emacs does.

That was a chore.

At this point, I had a working and cooperating installation of Allegro CL, Aquamacs, and SLIME. As a result, I was able to read and work through several examples in Practical Common Lisp. I even managed to pre-empt some of Seibel’s code, writing smaller/cleaner code before I saw his. Alas, it was at this point that I must have run (hubris), because everything I did after that was doomed to failure.

Doomed to Failure

Yeah, my code was sleek, but I had a ways to go before I could edit ebooks (remember the intro). ePub files are essentially zipped XHTML documents, ergo I need a Lisp library capable of reading and writing to zip files. As luck would have it, a Common Lisp ZIP library exists. It sports a tidy API and a tidy setup tutorial.

There are two ways to install the ZIP library: automatic and manual. The automatic way requires ASDF-Install. The manual way requires ASDF. Either way, I was destined to become familiar with ASDF and its subsidiaries.

ASDFFFFFFFUUUUUUUU

rageguy

Had to get that out of the way.

ASDF is nearly impossible to install and use. Fortunately, it comes with Lispbox. Unfortunately, Lispbox is broken. Fortunately, ASDF comes with Allegro CL. Unfortunately, that version of ASDF is crippled–it can’t do any of the commands I see in ASDF tutorials across the web. Fortunately, Allegro successfully loads ASDF. Unfortunately, Allegro fails to load ASDF-Install no matter where I place the files or how I configure ~/.clinit.cl. Fortunately, ASDF comes with SBCL. In fact, the Mad Scientist expressed similar misgivings about ASDF problems and reported that SBCL quickly resolved them.

SBCL

Lost in Technopolis has an excellent tutorial for installing SBCL on Mac OS X. The basic idea is:

  1. Install MacPorts.
  2. Use MacPorts to install Emacs, SBCL, and SLIME in one fell swoop.

This all ran so smoothly, I wondered why I had spent so much time and effort on Allegro/Aquamacs. In a few minutes, SBCL finished compiling and was ready for duty. The interpreter was a bit finnicky: the debugger constantly gets in your way, and the command to exit SBCL is hidden deep in some underwater cave. Eventually, (quit) worked.

SBCL does indeed come with ASDF, and indeed SBCL easily installs ASDF-Install, indeed. Very soon indeed, ASDF-Install began installing ZIP, the last remaining sidequest to editing ebooks with Lisp. Very soon indeed, ASDF-Install, or ZIP, or ZIP’s several dependencies failed, and the ZIP install borked with errors such as

Cannot open: file exists

and

gray.lisp.lisp No such file or directory.

No dice. Throughout the day I have posted bug reports on Stack Overflow and two mailing lists. One mailing list was very active, and I soon got an email reply suggesting I drop ASDF for Quicklisp.

Quicklisp, or the End of my Tether

An oasis in the desert of broken software, Quicklisp is. The homepage is simple and elegant. The installation process is as easy as downloading a .LISP file and loading it.

Upon running the tutorial’s commands, I receive the error

==================================================
163,840 bytes in 0.09 seconds (1720.43KB/sec)
; Fetching #
; 2.78KB
==================================================
2,846 bytes in 0.001 seconds (2779.30KB/sec)
debugger invoked on a SB-INT:SIMPLE-FILE-ERROR:
  can't create directory /Users/andrew/.cache/common-lisp/sbcl-1.0.44-
darwin-x86-64/Users/andrew/quicklisp/

Meanwhile my roommate has everything working on his Kubuntu setup with apt-get install emacs clisp slime. It’s just not fair.

Candide is Becoming More Relevant by the Second

MacPorts knows SBCL but not CMUCL. It knows CLISP but not GCL. While CLISP installs, I begin manual installation of GCL.

  1. Download GCL source from five years ago (really).
  2. Run ./configure.
  3. ./configure error’s.
  4. Follow ./configure‘s helpful advice.
  5. Run ./configure --enable-machine=FreeBSD, the closest thing to an Intel Mac version.
  6. ./configure error’s.

As a side note, Dr. Edmund Weitz offers yet another Common Lisp bundle, called Starter-Pack. It uses LispWorks, which automatically and forcibly exits after a couple hours. Also, Starter-Pack is Windows-only.

Let’s check up on CLISP.

It Works

Wait, what?

Your Life Is Meaningful Again

Oh, joy! Let’s test this.

$ cd ~/Downloads
clisp
[1]> (load "quicklisp.lisp")

...

[2]> (quicklisp-quickstart:install)

...

==================================================
2,846 bytes in 0.001 seconds (2779.30KB/sec)
Upgrading ASDF package from version 2.004 to version 2.009
; Fetching #
; 0.40KB
==================================================
408 bytes in 0.003 seconds (132.81KB/sec)

  ==== quicklisp installed ====
[3]> (ql:quickload "zip")
To load "zip":
  Load 1 ASDF system:
    zip
; Loading "zip"

("zip")
[4]>

Yes, yes, a thousand times yes! Now back to helping Seibel organize his pop music.