Category Archives: Computers

Relating to computers and technology

Lying system calls

Only Debian hackers are likely to grok this.

I just spent a week, maybe more, trying to look into a variety of very bizarre build failures for some packages, after building and installing newer versions of coreutils, dpkg and debhelper. Things like “install -d” dying because its destination wasn’t a directory (I’d check afterwards, it was), tar complaining about a file changing out from under it (always about symlink files, where the block of code in tar that could generate that error could never be evaluated for symlink files), and dpkg-deb –build complaining that a pre-installation script wasn’t executable, when it clearly was.

Of course, I was mostly treating these as separate, bizarre problems that happened to start up within a similar timeframe. Tonight it suddenly dawned on me that maybe all these system calls appear to be lying to me, because they are. They are, and building a package in fact typically involves running a program whose job is specifically to make system calls lie: “fakeroot”.

Fakeroot is of course only intended to lie just enough so that programs think they’re running as root, even though they really aren’t. But fakeroot’s documentation clearly states that it will not work correctly if the programs running underneath it are linked against a different version of the system libraries. The seeds of this problem were sown some years ago when we built and installed a newer version of libc6, but the problem didn’t manifest itself until now, because the most commonly-used utilities for building packages are dpkg, debhelper, and coreutils, all or most of which were not updated until I updated them just a week or so ago. Gah!

Announcement: Niwt (Nifty Integrated Web Tools)

Alright, I’ve been working my ass off lately on this project, which I haven’t wanted to say a lot about until i had something reasonable to show for it. I now feel that I’ve reached this point (barely, and depending on your point of view).

Niwt is a project that aims to (eventually) reproduce most of the functionality of GNU Wget (and some additional), but with a radically different design philosophy—namely, that it is built entirely around Unix pipelines, and facilities to easily swap out or extend every existing piece of functionality with an alternative (or additional) program that offers equivalent (or improved) functionality. It is expected that this will result in a big trade-off between, the relative efficiency, lower resource consumption, and portability to other systems that Wget enjoys (which Niwt will certainly not), versus extreme and relatively easy customization. If a highly customizable tool is what you need, Niwt may (when it’s finished) fit your needs; if efficiency and general leanness are what is called for, it most likely will not.

In terms of functionality, Niwt has virtually nothing to offer at the moment. It can download files. It doesn’t have Wget’s automatic connection recovery (yet), nor does it have timestamping or recursion (yet). The point of this pre-pre-pre-prerelease is not to demonstrate what Niwt does, but what it could eventually do, and how it will allow you to do it. Every bit of Niwt’s operation is open and transparent to the user, and modifiable in every way.

To find out more about this project, please visit http://niwt.addictivecode.org/Niwt, and especially http://niwt.addictivecode.org/TryingOutNiwt to get an idea of how it works (though that page is best enjoyed with your copy of niwt already installed, which you can get from http://niwt.addictivecode.org/InstallingNiwt).

I’ve set up an IRC channel, #niwt @ irc.freenode.net, where I’ll try to be available when I can, and a users’ discussion mailing list at http://addictivecode.org/mailman/listinfo/niwt-users/ .

Try it out and let me know what you think!

Niwt’s source code is free and open source software, and is available under the MIT (simple BSD-style) license.

Japanese Mail Archive Character-Set Transcoder

I have no idea if this will be helpful to anyone else, but I’ll just throw this out there for the search engines to pick up, just in case. I wrote a Perl script to take a Sendmail mbox archive of email messages, and transcode the text of all their bodies, and the Subject and From headers, to UTF-8. This script may be had here. It reads in the archive on standard input, and spits the transcoded archive on standard output.

I’m subscribed to a few Japanese-language mailing lists (well, more accurately, one Japanese-language mailing list, a daily mail of the Slashdot Japan headlines, and Google Alerts for wget and tmux). The idea for this was for me to get Japanese practice by reading regular Japanese content on subjects I’m interested in.

The problem is, I just don’t read it when it comes in. The best time for me to practice Japanese is on the train, during my work commute. Which is why I have a Kindle 3, so I can browse Japanese websites on the train. So, I need these mails web-accessible. No problem, I can use a mail web archive tool, like hypermail.

But hypermail doesn’t like dealing with mbox files that consist of messages that are in various incompatible encodings; some of my mail arrives in UTF-8 (unicode), and others in ISO-2022-JP (a popular encoding for Japanese-language text). Hypermail doesn’t deal with encoded characters in the mail headers, and also I didn’t know what character encoding to configure Apache to tell the browser, because it differed from one mail to the next.

So I wrote this transcoder tool in Perl. It just scans through all the mails, decodes the Subject and From headers (currently leaves the others), and transcodes all ISO-2022-JP (or anything that’s not UTF-8), so all the messages use the same encoding, and I just configure Apache to use UTF-8 for all of them. The best part is that, now that the actual content and the server-specified character encoding agree consistently, I can use online tools like Hiragana Megane to process this web mail archive, and automatically provide pronunciations for words I’m not very familiar with.

The script requires the following Perl modules: MIME::Tools (for parsing email format), Mail::Mbox::MessageParser (for parsing mbox), and Text::Iconv (to handle the transcoding).

Slides from Wget Talk

I’ve made the slides available from my Wget talk here in PDF form. You can also get the OpenOffice “Impress” file here, but note that it uses non-standard/non-free fonts, so most likely won’t display properly for you.

As part of my talk, I also used some recorded terminal sessions demonstrating wget usage, including prompts to wait for keypresses. I created these sessions using GNU Teseq (a project I wrote for debugging terminal sessions); but the sessions were then edited by hand and played using a specially-modified version of the reseq program (part of the Teseq project); those changes have not yet been pushed to the development sources. When I have time, I’ll push the final changes to the official development sources at savannah.gnu.org, and then make the automated terminal demos available as well.

Please note that full documentation on wget can be had at http://www.gnu.org/software/wget/manual/

Update 2011-01-04: The modifications to Teseq were pushed a few weeks ago, so I’ll go ahead and put the script files up; there’s a script demonstrating Wget’s automatic retry capabilities, and one demonstrating the --continue option, for continuing downloads across different sessions. These scripts are unlikely to be useful to anyone who isn’t familiar with the points I was making during the script’s run, but there you go. You’ll probably want to right-click the links and do a “Save as…”.

In order to process them, you need to go grab the development sources of Teseq; you do this by obtaining the Mercurial revision control software and the GNU autotools (autoconf, automake, etc), and then running “hg clone http://hg.savannah.gnu.org/hgweb/teseq/” somewhere suitable. Within the source directory, run ./autogen.sh and then do the typical ./configure && make && make install (all that’s actually needed to generate the reseq program is the ./configure step).

Update 2014-12-15: In order to process them, you need to install GNU Teseq (another project I maintain), at least version 1.1. It’s available on a variety of Unixy platforms, including Debian, Ubuntu, and Fedora. It’s available on Fink, but not version 1.1, so you’d need to follow the “get from repository” instructions in the struck-through paragraph above. On Debian/Ubuntu, you can install the teseq package by running apt-get install teseq; just ensure that it’s at least version 1.1, or it won’t support the interactive files.

Once you have reseq installed, you run the scripts like: reseq --replay --halts auto-retries.seq. Press a key whenever prompted to do so by a green-colored message. You’ll want to do this replay in an xterm-compatible emulator, at least 80 columns wide. You can also play it without the prompts/pauses by removing the --halts option from the invocation.

Upcoming Wget Talks

If you’re interested in GNU Wget, and live in California within a reasonable distance of either Davis (semi-near Sacramento, in Yolo County) or Mountain View (Silicon Valley), then you may be interested in coming to one of the talks I’m scheduled to give at those two locations about GNU Wget. More information can be found here (Davis) and here (Mountain View). The Davis one will take place Novemeber 15th (a week from Monday), and the Mountain View one is January 5th.

Here’s the blurb.

GNU Wget is a computer program that retrieves content from web servers. It supports downloading via HTTP, HTTPS, and FTP protocols, the most popular TCP/IP-based protocols used for web browsing.

Its features include recursive download, conversion of links for offline viewing of local HTML, support for proxies, and much more. It appeared in 1996, coinciding with the boom of popularity of the Web, causing its wide use among Unix users and distribution with most major Linux distributions. Written in portable C, Wget can be easily installed on any Unix-like system and has been ported to many environments, including Mac OS X, Microsoft Windows, OpenVMS, MorphOS and AmigaOS.

It has been used as the basis for graphical programs such as GWget for the GNOME Desktop and KGet for the KDE Desktop and VisualWget for Windows. [From Wikipedia].

Micah Cowan was maintainer of Wget between mid-2007 and early 2010. His talk will discuss:

  • What is Wget?
  • My history with Wget
  • How to use Wget
    • Restartable downloads
    • Website archiving/recursive downloads
    • Fine-grained controls over which links to follow
    • Content conversions for local browsing
  • Wget shortcomings
  • Lessons learned while maintaining Wget
  • Issues unique to maintaining a GNU project
  • Git vs Mercurial

    I just got out of a point-by-point comparison of Git and Mercurial, presented by one developer from each project (simultaneously). Both are DVCSs that are popular among Free and Open Source Software projects.

    The presentation pretty much confirmed my existing convictions about the two projects: they are extremely comparable in terms of the features provided, though of course there are differences in the approach that can make a big difference for people’s preferences. But the major difference seems to be that Mercurial has significantly better documentation, is easier to use overall (at least for the most common tasks), and runs better on Windows.

    The better performance on Windows was the main reason I chose to migrate GNU Wget to Mercurial rather than Git. The ease-of-use, and similarity to Subversion, is why I prefer to use it in my own personal projects, as well.

    When I had to choose a DVCS for GNU Screen, on the other hand, I went with Git. This was mainly because I believed it to be essentially equal in power to Mercurial (which I feel was confirmed by this presentation), was of no use to Windows users (at least, other than ones running a Unix environment such as Cygwin), and would be the more familiar DVCS to the folks most likely to be interested in developing it (several other important GNU projects, such as coreutils, gnulib, autoconf and automake, are managed with Git).

    Adventures in Haskell, Part 2: Kewlness

    This is a continuation on part one, which was a very basic introduction to functional programming. In this portion I’m pointing out some aspects of Haskell which I find to be fairly kick-ass. 🙂

    My apologies if 80-90% of this post is gibberish to you. I try to be clear, but I’m also trying not to spend a copious amount of time writing this post. Think of it as a rather detailed “best things about Haskell” list, rather than anything remotely like a tutorial, or something that otherwise gives you an actual decent understanding of Haskell as a language. The idea is to try to whet your appetite to investigate further.

    Death to Parentheses

    Well, the first thing that impressed me is going to be small hat to folks whose primary experience with functional programming is with languages other than Lisp or Scheme, or to folks who have no FP experience at all; but Lisp is notorious for its excessive use of parentheses. Every function call gets its own pair, and since invoking functions is a frequent occurrence in functional programming, you get quite a lot of them. Even better, when you’re done with all your nested function calls, you’re left to figure out how to place the dozen or so closing parentheses. Here’s an example of the crappy powers function from part one of this miniseries, in Lisp:

    For this example, it’s not too terrible; but it’s a simple function, and already we end up with five successive closing parentheses at the end. Imagine how more complex code might look! Still, you get used to it; and at least there’s never a question of what’s an argument of what.

    The parens tend to bunch up when you nest function calls:

    The example above includes non-standard functions (3* and filter) whose definitions I have not provided (they’re trivial); it takes a list of numbers from 1 to 10, then filters the list so it’s just the odd numbers in that range (1 3 5 9), multiplies each element of the list by three, and then reverses the list (largest to smallest). You may note that it’s sometimes easier to read expressions like the above from right to left.

    In Lisp there’s no way to avoid these parens in a series of nested function calls. In Haskell, you could also write something similar to the above…

    However, Haskell provides a “dot” operator, which applies the function on the left to the output of the function on the right, so you can also write this snippet like:

    The outer set of parentheses are necessary in the above example, because the dot operator takes a function on either end; and the result of filter odd [1..10] (which binds more tightly than the dot operator; function applications always bind more tightly than operators) is not a function, it’s a list result. You might wonder how filter odd and map (3*) can be functions; they look more like the result of a function applied to an argument. And they are: one of Haskell’s great features is partial function application: you can pass one argument to a function that takes two, and the result is a function that takes the “second” argument. The (3*) is actually a partial application too: in the Lisp example, I would have had to define a function named 3* to make it work; but in the Haskell example (3*) is a partial application of the * operator.

    I like to read the dot operator as “of”. So you can read (reverse . map (3*) . filter odd) as” reverse the order of mapping the multiply-by-three operation across the elements of the result of filtering for just the odd-numbered values of…”

    We can even get rid of those outer parentheses, too, if we want:

    The dollar operator takes a function on the left and applies it to everything on the right as a single argument. It’s right-associative, so we actually could also write this expression as:

    The only set of parentheses we can’t get rid of is the ones in (3*), where they are used in the partial application of the * operator, called a section. Parentheses are part of the syntax required to designate a section. Of course, if we really wanted, to, we could just write a function that does the same thing, and name it mulThree or what have you.

    Personally, I like this one best:

    Function Definitions and Pattern Matching

    In part one of this overview, we presented this version of our powers function in Haskell:

    The truth is, though, we don’t need both i and v. One value could be deduced from the other. I just kept both parameters because they’d been needed in C (they could each be deduced from the other in C as well, but at a performance penalty; there are other ways to write the example in C without using i; but the example was written for readability).

    A common way to write function definitions in Haskell, is to first write a definition of the function for a concrete value, and then define the function for other values in terms of that first definition. For instance, we could’ve written powers like:

    Note that this assumes no one ever calls powers 9; and also that powers now takes one argument instead of two (we dropped v).

    This series of definitions means that

    • “powers 0” is a list whose first element has the value 1, and whose remaining elements are defined by “powers 1”
    • “powers 8” is the empty list.
    • “powers n” for any other value n is a list whose first element is twice the value of the first element of “powers (n-1)”, and whose remaining values are defined by “powers (n+1)”.

    This winds up giving us the same list result for powers 0 as we previously got for powers 0 1 in our original definition of the function.

    Note that there’s just one function powers, but there are three definitions for that function. These definitions combine to provide the complete description of what powers means.

    Pattern Matching

    We’ve just made a slight use of a very powerful Haskell feature, pattern matching. In this case, we’re matching arguments to powers against various possibilities of 0, 1, or n (where n can be any value). But pattern matching can actually be performed against any sort of value or partial value.

    For instance, if we wanted to write a very simplistic function to convert a string into 1337-speak, transforming

    to

    we could write it like:

    The above could have been written somewhat more tersely, but it’s a good demonstration of using pattern-matched function definitions. The (x:xs) patterns describe a list (: is the list-construction operator), whose first element is x and whose remaining elements are the (possibly empty) list xs. The first definition in this example simply says that applying leetize to an empty string will yield an empty string as the result. The following definition means that if the leetize function is applied to a string whose first character is the letter ell, then the result is the one digit, followed by the application of leetize to the remaining characters of the string. The final definition matches (x:xs), for any string beginning with an arbitrary character x that doesn’t match any of the previous patterns; it just leaves that character as-is, and continues to transform transformations on the remaining characters.

    Roll your own operators

    Another thing that I think is really cool in Haskell is the ability to invent or redefine your own operators. Languages like C++ let you override the definition of existing operators for new types, but Haskell lets you invent entirely new operators. Want to define a new operator *!? It’s as easy as writing

    You can also use normal functions as if they were operators. This is common with functions like elem, where elem x mylist evaluates to True if x may be found as an element in mylist. While elem x mylist is fine, the meaning of x elem mylist seems just a little clearer.

    For both operators and functions-applied-as-operators, you can specify associativity and precedence. (You can only define binary operators; all operators in Haskell are binary operators, except for unary -, as used in expressions such as -3.)

    Lazy Evaluation

    One of the things that many people find really cool about Haskell (not just me!), is that it follows a “lazy” or “non-strict” evaluation scheme. The value of an expression is only calculated when it is required. This has some rather interesting consequences.

    One consequence is that Haskell allows you to define lists that don’t end. Among other things, this means I’ve been making our running powers example way more difficult than it needs to be. Instead of defining a function that takes some starting value and keeps doing calculations until it reaches a maximum, I can have it just keep calculating that result forever:

    Note that this latest powers doesn’t need any arguments at all. It’s just bound to the set of all 2n, n≥0. That’s a list without a limit; but that doesn’t matter, so long as we never try to evalute the entire list. But if I ask Haskell to give me just the first eight elements of that list (with take 8 powers), it works just fine. Haskell performs the necessary calculations to bring me the first eight elements, and doesn’t continue trying to calculate any further elements.

    Infinite-length lists are fun, but when you come down to it, they’re really just neat sugar. One area where Haskell’s lazy evaluation really comes in handy is in code like:

    This is a complete program to convert the standard input to all-uppercase, and print the results to the standard output. It binds the variable slurp to the full contents from standard input as a string, then binds the variable upperCased to a string which holds the result of mapping the toUpper function against each character in slurp. Finally, the string held in upperCased is printed to the standard output using putStr.

    Similar code in Perl might look like:

    The difference is that the Perl example will happily try to take a 5GB file and shove it all into memory (or, much more likely, make the attempt and choke), whereas the Haskell version, which looks nearly identical, will process input a bit at a time—how much of a bit depends on the buffering scheme, which depends on the implementation—but the point is, it’s not going to try to slurp it all into memory. It’ll process data a bit at a time, converting it to uppercase, and spitting it out. It only grabs input when it’s ready to print out the next few characters. Then it looks at what it’s printing—upperCased, and sees that it’s supposed to hold the uppercasing of slurp. It then sees that slurp is supposed to be a string holding the content from standard input, so it goes ahead and reads the next bit from standard input, uppercases it like it’s supposed to… rinse, lather, repeat.

    What’s neat about this is that it frees you from having to think about the details. Your code doesn’t need to “know” that it’s dealing with input piecemeal: as far as it’s concerned, it’s just transforming a string. Naturally, Perl is just as capable as Haskell at processing input a piece at a time; the difference is that the Perl code will be very aware that it’s doing things one bit at a time, while the Haskell code “thinks” it’s just working on the whole data.

    Note, the Haskell example is for illustration. A much simpler program that does the same thing is:

    Monads

    You may have noticed that the original version of the uppercasing program looked somewhat procedural-ish. The do keyword introduces a block of code that is essentially procedural; but it never for a moment violates the rules of a pure functional language. The behavior of do is defined in terms of a series of function applications and nested name-bindings. The important thing is that Haskell preserves the principle of non-destructive, immutable data, and do doesn’t let you do anything you can’t do without it; it just looks much cleaner than its equivalent.

    What’s cool about the do block, though, is that it actually provides a mechanism that many people refer to as overloading the semicolon. The do block is defined in terms of a couple of operators, >> and >>=, that the user can overload in the types phe creates. All types that can be used as the types of the expression-statements in a do block, and as a do block’s own type, belong to a class called Monad.

    For instance, take the Maybe parameterized type (parameterized means it takes another type as a parameter; for instance, it can be a Maybe String or a Maybe Integer). This is a type provided by the Haskell standard library, that provides a way to represent a “null”-like value. For instance, an expression that returns a Maybe String can construct return values such as Just "hello" or Just "what is your name?"; it can also use the constructor Nothing, to represent no value (often used to indicate an error condition). For instance, Haskell provides a find function that takes two arguments: a predicate function, and a list. If it finds an element a that matches the predicate in the list, it will return Just a; if it doesn’t find anything, it returns Nothing. So, find (< 2) [1 5 3 9 8] will return Just 1; but find (> 10) [1 5 3 9 8] would return Nothing.

    So, what happens if you need to use the result from several successive finds? Say, to find the customer ID corresponding to a name, and then use that to find a sales record? You could end up with successive, nested if expressions to check for Nothing before proceeding. Rough pseudo-Haskell:

    (There are severe glitches in the above, but it was written for human readability, and not actual compilability.) Note that it’s safe to bind record as above, even though we haven’t yet checked isJust record, because it’s not actually evaluated until we use it, and we don’t use it until after we check.

    Imagine if there were even more dependent lookups we needed to perform! The nested if expressions would quickly become unweildly. However, since the Maybe type is a Monad, we can write it with the do block, like this:

    The Maybe type overloads the meaning of the name bindings done with <- (actually, it overloads >>=, but <- is defined in terms of >>=). If the result of the right-hand side is a Just x (for any x), it binds custId to x (stripping off the Just) for the remaining statements. If either of the two <- statements result in a Nothing value, then the remaining statements are not processed, and the result of the entire do block is Nothing.

    I was at first dismayed when I discovered that Haskell’s exception mechanisms are not as powerful as one might hope. However, as it turns out, it doesn’t actually pose a problem, because you can just use Maybe and do blocks, as in the example above. The first <- binding or statement that evalutes to Nothing will short-circuit the rest of the code, and eliminates the need to manually check for error-status results after each statement!

    This “semicolon-overloading” mechanism is really very powerful. The do block provides a mechanism for compactly and abstractly representing a sequence-of-statements, where a statement is either a Monad-returning expression or a name-binding; but the monadic type gets to decide what “sequencing” these things actually means.

    A great example of the power of Monads is that simple lists are also Monads. But with list types, the name-binding <- notation is overloaded so that x <- someList will result in x being set to each element of someList in turn, and then all the remaining statements that appear after the <- binding statement in the same do block will be evaluated in turn, for each x. The result of the do block will be the list of each of those results. As an example, here’s a simple permuation of two lists:

    As perhaps you’re beginning to see, Monads are an incredibly powerful feature of Haskell. In fact, Monads combined with Haskell’s lazy, evaluate-as-needed behavior is what enables Haskell to be capable of elegantly dealing with interactive with user input, without sacrificing its status as a pure functional programming language. Interactivity (such as might be found in a typical text-editor) is not possible in languages that are purely functional (immutable data), and use strict (non-lazy) evaluation, because such languages require that they have all the input at the beginning before processing can even begin.

    String Literals

    I haven’t seen this emphasized anywhere, and it’s a small thing, but one more aspect of Haskell that strikes me as particularly elegant is the ways in which Haskell deals with string literals.

    One of the things I really like, is that Haskell’s string literals can include spans of whitespace that are ignored, when they are enclosed within backslashes; this whitespace can include newlines. So you can write something like:

    The large “Hello, World!” will be printed flush-left, but we were able to align the literal rather nicely in the Haskell source. This sort of thing can be a little more difficult in some other languages.

    Also, in addition to the usual C-style \r, \n, \b, etc, and hexadecimal and octal notations (which differ slightly from C’s), Haskell also provides a \^C notation (where '\^C' represents “Control-C”), and a \ESC notation (representing the escape character, using its ASCII acronym). Thus, the ASCII backspace character can be represented as any of '\b', '\BS', '\^H, '\o33', or '\x1B'. Every one of those representations can be useful in different circumstances, especially for a terminal enthusiast and GNU Screen co-maintainer such as myself.


    Well, that pretty much sums it all up. This took a lot longer to write than I really wanted it to, so maybe I’m finished talking about Haskell for a while. In presenting all the things that really impress me about Haskell, I’m perhaps painting an unrealistically rosy picture of the language. I do believe that Haskell, and its current best implementations, have some significant shortcomings (but none that actually render Haskell impractical, as is sometimes suggested); if I get around to writing a “part 3”, I’ll probably dwell on those, to counter-balance the wondrous things I covered here. However, that one might be a while in coming, as I think I’ve spent quite enough time writing about Haskell for a limited audience, for now. 😉

    Adventures in Haskell

    So, the last few weeks I’ve been learning the Haskell programming language, and thought I’d share my thoughts on it. Haskell is a pure functional programming language that supports lazy evaluation (I’ll explain all that in a moment), which has been gaining popularity in some circles (and particularly among academia and computer lingusts). It has (semi-)famously been used to write an implementation for the Perl 6 programming language, and also a distributed revision control system (DaRCS).

    I’d heard of the language a couple years ago, and have been wanting to learn it since then, but the freely available resources I’d been able to find for learning about the language were frustratingly poor, and printed books were in the range of $80 and up, which is a bit steep for learning a language as a hobby. I finally found an upcoming O’Reilly book, Real World Haskell, whose full content is available for browsing online. Amazon says the printed version’s going to be $50, which is a marked improvement over other books I’d been considering buying. The book’s not perfect—there are several minor beefs I have with it—but the fact remains that it is the highest quality resource for learning Haskell that is freely available. It also has the major advantage that it focuses heavily on real-world applications (as the name implies), including coverage of network programming, and concurrency (multi-threaded or clustered computing), writing language parsers; it even guides the reader through writing software to retrieve ISBN numbers from cell-phone photos of book backs! This probably makes it the most practical resource for learning Haskell, too—pricey or free.

    I’m still working through the book, but I’ve supplemented my understanding so far by reading through the official language specification, The Haskell 98 Report, so at this point I’ve gained a pretty solid understanding of the core language and minimum libraries.

    Intro to Functional Programming

    (If you’re fairly familiar with functional programming, might as well skip this article and wait for part 2.)

    I’ve had previous experience with functional programming languages, mainly Lisp and XSLT, which helped in trying to learn Haskell.

    Lisp is one of the more popular functional programming languages , and probably the oldest. My experience with it is primarily through the Emacs-Lisp dialect. Emacs is a very powerful text editor that is popular in the Unix world (and particularly on GNU/Linux). It is mostly implemented in Emacs-Lisp, and you can alter the program’s behavior while it’s running by editing the Lisp code. It’s the first Unix editor I learned to use, and though I primarily prefer Vim these days, I’m very comfortable with Emacs (these days I run it in a vi-emulating mode called viper), and have written code in Emacs-Lisp.  I’ve also dabbled in Scheme, a Lisp dialect.

    XSLT is a language for performing transformations on XML documents, and like Haskell but unlike Lisp is a “pure” functional programming language (more on that in a moment). Unfortunately, it doesn’t really provide a complete set of useful tools (version 1.0, anyway: I’m not very familiar with the XSLT 2.0 spec), so my experience with it was that writing XSLT can be an exercise in frustration.

    In imperative programming languages like C, C++, BASIC, Java, and many others, the focus is on performing a series of actions: do A, then do B, etc. In imperative programming, one often deals with data in memory that can be modified through a series of actions (mutable data), and the behavior of bits of code in the program may depend on the current value of some particular piece of data in memory, which may be changed by other bits of code.

    Here’s a simple example of imperative programming in C code:

    This snippet produces an eight-element array a whose elements hold successive powers of 2 (1,2,4,8,16,32,64,128). The code amounts to a series of instructions like:

    1. set the value of v and i to 0
    2. set the ith element of the array a to the current value of v
    3. set the value of v to twice its previous value.
    4. set the value of i to one more than its previous value.
    5. if the value of i is less than 8, repeat from step 2.

    Notice the heavy focus on performing a series of steps, and a dependency on modifying data.

    In contrast to imperative programming, functional programming focuses on transforming input data into output data, by defining the function that performs the transformation. In a “pure” functional programming language, it is not possible to change the value of some object of data from one thing to another; in fact, there are no objects—there are only values (and functions; immutable data). This tends to place a higher focus on what the desired result is, rather than how we want to arrive at a given result.

    Here’s an example of some code in Haskell that produces a result similar to the imperative snippet above:

    If you invoke the function above as powers 0 1, its result will be a list of successive powers-of-two from 1 to 128, just like for our C code. However, you may notice that the method by which we’ve arrived at that list is somewhat different. Translated to English, this code says:

    • Define the function powers(i,v) (for any v, and where 0 ≤ i < 8) to be a list whose first element has the value v, and whose remaining elements are the result of powers(i+1, v*2), or, for any i not matching the qualification just given, the empty list.

    …and that’s it. When you invoke powers 0 1, it constructs the element with value 1, then invokes itself again as powers 1 2, which constructs an element whose value is 2 and invokes powers 2 4 to construct the third element with a value of 4. When it finally reaches powers 8 256, it sees that i=8 doesn’t meet the criterion that 0 ≤ i < 8, and so it caps it off with an empty list result (which, since it doesn’t invoke itself recursively again, terminates the list, and the evaluation of powers).

    Functional code like the above can look cryptic to imperative programmers who aren’t used to seeing it, but it’s learned fairly quickly. It happens to correspond fairly closely to how a mathematician would formulate a function definition, so folks who are comfortable with math will tend to feel right at home.

    Note that it’s just as easy to do recursion like this in C:

    However, in real-world C implementations, each invocation of powers will consume additional memory. It works fine for our example, where we’re limited to a total of nine calls to powers, but for longer recursions you could run out of stack space, which is not a good thing.

    Note that our recursive C example still isn’t pure functional programming, as we’re modifying the values of an existing array, and not producing new values.

    Disclaimer: none of the C or Haskell examples are particularly idiomatic; they’re for rough example purposes only. I’d do them differently in real life; but I felt that these examples, as written, may be a little easier to discuss, and more accessible for folks that might not be terribly familiar with either language.

    To be continued (very soon)… Part 2 will discuss some of the things in Haskell that I think are really cool. 🙂

    Cool Little Maze

    A friend from #atheists at irc.freenode.net created this neat little interactively-revealed maze. The neat thing about it is that it is created using only HTML & CSS. No JavaScript, no programming, nothing. Just static HTML & CSS. (Note: works on most mainstream browsers except Internet Explorer.)

    The only real irritation is that if the pointer slips off the trail, you have to start over. Move slowly. 🙂

    Find the maze start at the upper left.

    (See also my own maze experiment.)