Category Archives: Software Development

General matters of designing and implementing software

Lying system calls

Only Debian hackers are likely to grok this.

I just spent a week, maybe more, trying to look into a variety of very bizarre build failures for some packages, after building and installing newer versions of coreutils, dpkg and debhelper. Things like “install -d” dying because its destination wasn’t a directory (I’d check afterwards, it was), tar complaining about a file changing out from under it (always about symlink files, where the block of code in tar that could generate that error could never be evaluated for symlink files), and dpkg-deb –build complaining that a pre-installation script wasn’t executable, when it clearly was.

Of course, I was mostly treating these as separate, bizarre problems that happened to start up within a similar timeframe. Tonight it suddenly dawned on me that maybe all these system calls appear to be lying to me, because they are. They are, and building a package in fact typically involves running a program whose job is specifically to make system calls lie: “fakeroot”.

Fakeroot is of course only intended to lie just enough so that programs think they’re running as root, even though they really aren’t. But fakeroot’s documentation clearly states that it will not work correctly if the programs running underneath it are linked against a different version of the system libraries. The seeds of this problem were sown some years ago when we built and installed a newer version of libc6, but the problem didn’t manifest itself until now, because the most commonly-used utilities for building packages are dpkg, debhelper, and coreutils, all or most of which were not updated until I updated them just a week or so ago. Gah!

Announcement: Niwt (Nifty Integrated Web Tools)

Alright, I’ve been working my ass off lately on this project, which I haven’t wanted to say a lot about until i had something reasonable to show for it. I now feel that I’ve reached this point (barely, and depending on your point of view).

Niwt is a project that aims to (eventually) reproduce most of the functionality of GNU Wget (and some additional), but with a radically different design philosophy—namely, that it is built entirely around Unix pipelines, and facilities to easily swap out or extend every existing piece of functionality with an alternative (or additional) program that offers equivalent (or improved) functionality. It is expected that this will result in a big trade-off between, the relative efficiency, lower resource consumption, and portability to other systems that Wget enjoys (which Niwt will certainly not), versus extreme and relatively easy customization. If a highly customizable tool is what you need, Niwt may (when it’s finished) fit your needs; if efficiency and general leanness are what is called for, it most likely will not.

In terms of functionality, Niwt has virtually nothing to offer at the moment. It can download files. It doesn’t have Wget’s automatic connection recovery (yet), nor does it have timestamping or recursion (yet). The point of this pre-pre-pre-prerelease is not to demonstrate what Niwt does, but what it could eventually do, and how it will allow you to do it. Every bit of Niwt’s operation is open and transparent to the user, and modifiable in every way.

To find out more about this project, please visit, and especially to get an idea of how it works (though that page is best enjoyed with your copy of niwt already installed, which you can get from

I’ve set up an IRC channel, #niwt @, where I’ll try to be available when I can, and a users’ discussion mailing list at .

Try it out and let me know what you think!

Niwt’s source code is free and open source software, and is available under the MIT (simple BSD-style) license.

Japanese Mail Archive Character-Set Transcoder

I have no idea if this will be helpful to anyone else, but I’ll just throw this out there for the search engines to pick up, just in case. I wrote a Perl script to take a Sendmail mbox archive of email messages, and transcode the text of all their bodies, and the Subject and From headers, to UTF-8. This script may be had here. It reads in the archive on standard input, and spits the transcoded archive on standard output.

I’m subscribed to a few Japanese-language mailing lists (well, more accurately, one Japanese-language mailing list, a daily mail of the Slashdot Japan headlines, and Google Alerts for wget and tmux). The idea for this was for me to get Japanese practice by reading regular Japanese content on subjects I’m interested in.

The problem is, I just don’t read it when it comes in. The best time for me to practice Japanese is on the train, during my work commute. Which is why I have a Kindle 3, so I can browse Japanese websites on the train. So, I need these mails web-accessible. No problem, I can use a mail web archive tool, like hypermail.

But hypermail doesn’t like dealing with mbox files that consist of messages that are in various incompatible encodings; some of my mail arrives in UTF-8 (unicode), and others in ISO-2022-JP (a popular encoding for Japanese-language text). Hypermail doesn’t deal with encoded characters in the mail headers, and also I didn’t know what character encoding to configure Apache to tell the browser, because it differed from one mail to the next.

So I wrote this transcoder tool in Perl. It just scans through all the mails, decodes the Subject and From headers (currently leaves the others), and transcodes all ISO-2022-JP (or anything that’s not UTF-8), so all the messages use the same encoding, and I just configure Apache to use UTF-8 for all of them. The best part is that, now that the actual content and the server-specified character encoding agree consistently, I can use online tools like Hiragana Megane to process this web mail archive, and automatically provide pronunciations for words I’m not very familiar with.

The script requires the following Perl modules: MIME::Tools (for parsing email format), Mail::Mbox::MessageParser (for parsing mbox), and Text::Iconv (to handle the transcoding).

Git vs Mercurial

I just got out of a point-by-point comparison of Git and Mercurial, presented by one developer from each project (simultaneously). Both are DVCSs that are popular among Free and Open Source Software projects.

The presentation pretty much confirmed my existing convictions about the two projects: they are extremely comparable in terms of the features provided, though of course there are differences in the approach that can make a big difference for people’s preferences. But the major difference seems to be that Mercurial has significantly better documentation, is easier to use overall (at least for the most common tasks), and runs better on Windows.

The better performance on Windows was the main reason I chose to migrate GNU Wget to Mercurial rather than Git. The ease-of-use, and similarity to Subversion, is why I prefer to use it in my own personal projects, as well.

When I had to choose a DVCS for GNU Screen, on the other hand, I went with Git. This was mainly because I believed it to be essentially equal in power to Mercurial (which I feel was confirmed by this presentation), was of no use to Windows users (at least, other than ones running a Unix environment such as Cygwin), and would be the more familiar DVCS to the folks most likely to be interested in developing it (several other important GNU projects, such as coreutils, gnulib, autoconf and automake, are managed with Git).

Adventures in Haskell, Part 2: Kewlness

This is a continuation on part one, which was a very basic introduction to functional programming. In this portion I’m pointing out some aspects of Haskell which I find to be fairly kick-ass. 🙂

My apologies if 80-90% of this post is gibberish to you. I try to be clear, but I’m also trying not to spend a copious amount of time writing this post. Think of it as a rather detailed “best things about Haskell” list, rather than anything remotely like a tutorial, or something that otherwise gives you an actual decent understanding of Haskell as a language. The idea is to try to whet your appetite to investigate further.

Death to Parentheses

Well, the first thing that impressed me is going to be small hat to folks whose primary experience with functional programming is with languages other than Lisp or Scheme, or to folks who have no FP experience at all; but Lisp is notorious for its excessive use of parentheses. Every function call gets its own pair, and since invoking functions is a frequent occurrence in functional programming, you get quite a lot of them. Even better, when you’re done with all your nested function calls, you’re left to figure out how to place the dozen or so closing parentheses. Here’s an example of the crappy powers function from part one of this miniseries, in Lisp:

For this example, it’s not too terrible; but it’s a simple function, and already we end up with five successive closing parentheses at the end. Imagine how more complex code might look! Still, you get used to it; and at least there’s never a question of what’s an argument of what.

The parens tend to bunch up when you nest function calls:

The example above includes non-standard functions (3* and filter) whose definitions I have not provided (they’re trivial); it takes a list of numbers from 1 to 10, then filters the list so it’s just the odd numbers in that range (1 3 5 9), multiplies each element of the list by three, and then reverses the list (largest to smallest). You may note that it’s sometimes easier to read expressions like the above from right to left.

In Lisp there’s no way to avoid these parens in a series of nested function calls. In Haskell, you could also write something similar to the above…

However, Haskell provides a “dot” operator, which applies the function on the left to the output of the function on the right, so you can also write this snippet like:

The outer set of parentheses are necessary in the above example, because the dot operator takes a function on either end; and the result of filter odd [1..10] (which binds more tightly than the dot operator; function applications always bind more tightly than operators) is not a function, it’s a list result. You might wonder how filter odd and map (3*) can be functions; they look more like the result of a function applied to an argument. And they are: one of Haskell’s great features is partial function application: you can pass one argument to a function that takes two, and the result is a function that takes the “second” argument. The (3*) is actually a partial application too: in the Lisp example, I would have had to define a function named 3* to make it work; but in the Haskell example (3*) is a partial application of the * operator.

I like to read the dot operator as “of”. So you can read (reverse . map (3*) . filter odd) as” reverse the order of mapping the multiply-by-three operation across the elements of the result of filtering for just the odd-numbered values of…”

We can even get rid of those outer parentheses, too, if we want:

The dollar operator takes a function on the left and applies it to everything on the right as a single argument. It’s right-associative, so we actually could also write this expression as:

The only set of parentheses we can’t get rid of is the ones in (3*), where they are used in the partial application of the * operator, called a section. Parentheses are part of the syntax required to designate a section. Of course, if we really wanted, to, we could just write a function that does the same thing, and name it mulThree or what have you.

Personally, I like this one best:

Function Definitions and Pattern Matching

In part one of this overview, we presented this version of our powers function in Haskell:

The truth is, though, we don’t need both i and v. One value could be deduced from the other. I just kept both parameters because they’d been needed in C (they could each be deduced from the other in C as well, but at a performance penalty; there are other ways to write the example in C without using i; but the example was written for readability).

A common way to write function definitions in Haskell, is to first write a definition of the function for a concrete value, and then define the function for other values in terms of that first definition. For instance, we could’ve written powers like:

Note that this assumes no one ever calls powers 9; and also that powers now takes one argument instead of two (we dropped v).

This series of definitions means that

  • “powers 0” is a list whose first element has the value 1, and whose remaining elements are defined by “powers 1”
  • “powers 8” is the empty list.
  • “powers n” for any other value n is a list whose first element is twice the value of the first element of “powers (n-1)”, and whose remaining values are defined by “powers (n+1)”.

This winds up giving us the same list result for powers 0 as we previously got for powers 0 1 in our original definition of the function.

Note that there’s just one function powers, but there are three definitions for that function. These definitions combine to provide the complete description of what powers means.

Pattern Matching

We’ve just made a slight use of a very powerful Haskell feature, pattern matching. In this case, we’re matching arguments to powers against various possibilities of 0, 1, or n (where n can be any value). But pattern matching can actually be performed against any sort of value or partial value.

For instance, if we wanted to write a very simplistic function to convert a string into 1337-speak, transforming


we could write it like:

The above could have been written somewhat more tersely, but it’s a good demonstration of using pattern-matched function definitions. The (x:xs) patterns describe a list (: is the list-construction operator), whose first element is x and whose remaining elements are the (possibly empty) list xs. The first definition in this example simply says that applying leetize to an empty string will yield an empty string as the result. The following definition means that if the leetize function is applied to a string whose first character is the letter ell, then the result is the one digit, followed by the application of leetize to the remaining characters of the string. The final definition matches (x:xs), for any string beginning with an arbitrary character x that doesn’t match any of the previous patterns; it just leaves that character as-is, and continues to transform transformations on the remaining characters.

Roll your own operators

Another thing that I think is really cool in Haskell is the ability to invent or redefine your own operators. Languages like C++ let you override the definition of existing operators for new types, but Haskell lets you invent entirely new operators. Want to define a new operator *!? It’s as easy as writing

You can also use normal functions as if they were operators. This is common with functions like elem, where elem x mylist evaluates to True if x may be found as an element in mylist. While elem x mylist is fine, the meaning of x elem mylist seems just a little clearer.

For both operators and functions-applied-as-operators, you can specify associativity and precedence. (You can only define binary operators; all operators in Haskell are binary operators, except for unary -, as used in expressions such as -3.)

Lazy Evaluation

One of the things that many people find really cool about Haskell (not just me!), is that it follows a “lazy” or “non-strict” evaluation scheme. The value of an expression is only calculated when it is required. This has some rather interesting consequences.

One consequence is that Haskell allows you to define lists that don’t end. Among other things, this means I’ve been making our running powers example way more difficult than it needs to be. Instead of defining a function that takes some starting value and keeps doing calculations until it reaches a maximum, I can have it just keep calculating that result forever:

Note that this latest powers doesn’t need any arguments at all. It’s just bound to the set of all 2n, n≥0. That’s a list without a limit; but that doesn’t matter, so long as we never try to evalute the entire list. But if I ask Haskell to give me just the first eight elements of that list (with take 8 powers), it works just fine. Haskell performs the necessary calculations to bring me the first eight elements, and doesn’t continue trying to calculate any further elements.

Infinite-length lists are fun, but when you come down to it, they’re really just neat sugar. One area where Haskell’s lazy evaluation really comes in handy is in code like:

This is a complete program to convert the standard input to all-uppercase, and print the results to the standard output. It binds the variable slurp to the full contents from standard input as a string, then binds the variable upperCased to a string which holds the result of mapping the toUpper function against each character in slurp. Finally, the string held in upperCased is printed to the standard output using putStr.

Similar code in Perl might look like:

The difference is that the Perl example will happily try to take a 5GB file and shove it all into memory (or, much more likely, make the attempt and choke), whereas the Haskell version, which looks nearly identical, will process input a bit at a time—how much of a bit depends on the buffering scheme, which depends on the implementation—but the point is, it’s not going to try to slurp it all into memory. It’ll process data a bit at a time, converting it to uppercase, and spitting it out. It only grabs input when it’s ready to print out the next few characters. Then it looks at what it’s printing—upperCased, and sees that it’s supposed to hold the uppercasing of slurp. It then sees that slurp is supposed to be a string holding the content from standard input, so it goes ahead and reads the next bit from standard input, uppercases it like it’s supposed to… rinse, lather, repeat.

What’s neat about this is that it frees you from having to think about the details. Your code doesn’t need to “know” that it’s dealing with input piecemeal: as far as it’s concerned, it’s just transforming a string. Naturally, Perl is just as capable as Haskell at processing input a piece at a time; the difference is that the Perl code will be very aware that it’s doing things one bit at a time, while the Haskell code “thinks” it’s just working on the whole data.

Note, the Haskell example is for illustration. A much simpler program that does the same thing is:


You may have noticed that the original version of the uppercasing program looked somewhat procedural-ish. The do keyword introduces a block of code that is essentially procedural; but it never for a moment violates the rules of a pure functional language. The behavior of do is defined in terms of a series of function applications and nested name-bindings. The important thing is that Haskell preserves the principle of non-destructive, immutable data, and do doesn’t let you do anything you can’t do without it; it just looks much cleaner than its equivalent.

What’s cool about the do block, though, is that it actually provides a mechanism that many people refer to as overloading the semicolon. The do block is defined in terms of a couple of operators, >> and >>=, that the user can overload in the types phe creates. All types that can be used as the types of the expression-statements in a do block, and as a do block’s own type, belong to a class called Monad.

For instance, take the Maybe parameterized type (parameterized means it takes another type as a parameter; for instance, it can be a Maybe String or a Maybe Integer). This is a type provided by the Haskell standard library, that provides a way to represent a “null”-like value. For instance, an expression that returns a Maybe String can construct return values such as Just "hello" or Just "what is your name?"; it can also use the constructor Nothing, to represent no value (often used to indicate an error condition). For instance, Haskell provides a find function that takes two arguments: a predicate function, and a list. If it finds an element a that matches the predicate in the list, it will return Just a; if it doesn’t find anything, it returns Nothing. So, find (< 2) [1 5 3 9 8] will return Just 1; but find (> 10) [1 5 3 9 8] would return Nothing.

So, what happens if you need to use the result from several successive finds? Say, to find the customer ID corresponding to a name, and then use that to find a sales record? You could end up with successive, nested if expressions to check for Nothing before proceeding. Rough pseudo-Haskell:

(There are severe glitches in the above, but it was written for human readability, and not actual compilability.) Note that it’s safe to bind record as above, even though we haven’t yet checked isJust record, because it’s not actually evaluated until we use it, and we don’t use it until after we check.

Imagine if there were even more dependent lookups we needed to perform! The nested if expressions would quickly become unweildly. However, since the Maybe type is a Monad, we can write it with the do block, like this:

The Maybe type overloads the meaning of the name bindings done with <- (actually, it overloads >>=, but <- is defined in terms of >>=). If the result of the right-hand side is a Just x (for any x), it binds custId to x (stripping off the Just) for the remaining statements. If either of the two <- statements result in a Nothing value, then the remaining statements are not processed, and the result of the entire do block is Nothing.

I was at first dismayed when I discovered that Haskell’s exception mechanisms are not as powerful as one might hope. However, as it turns out, it doesn’t actually pose a problem, because you can just use Maybe and do blocks, as in the example above. The first <- binding or statement that evalutes to Nothing will short-circuit the rest of the code, and eliminates the need to manually check for error-status results after each statement!

This “semicolon-overloading” mechanism is really very powerful. The do block provides a mechanism for compactly and abstractly representing a sequence-of-statements, where a statement is either a Monad-returning expression or a name-binding; but the monadic type gets to decide what “sequencing” these things actually means.

A great example of the power of Monads is that simple lists are also Monads. But with list types, the name-binding <- notation is overloaded so that x <- someList will result in x being set to each element of someList in turn, and then all the remaining statements that appear after the <- binding statement in the same do block will be evaluated in turn, for each x. The result of the do block will be the list of each of those results. As an example, here’s a simple permuation of two lists:

As perhaps you’re beginning to see, Monads are an incredibly powerful feature of Haskell. In fact, Monads combined with Haskell’s lazy, evaluate-as-needed behavior is what enables Haskell to be capable of elegantly dealing with interactive with user input, without sacrificing its status as a pure functional programming language. Interactivity (such as might be found in a typical text-editor) is not possible in languages that are purely functional (immutable data), and use strict (non-lazy) evaluation, because such languages require that they have all the input at the beginning before processing can even begin.

String Literals

I haven’t seen this emphasized anywhere, and it’s a small thing, but one more aspect of Haskell that strikes me as particularly elegant is the ways in which Haskell deals with string literals.

One of the things I really like, is that Haskell’s string literals can include spans of whitespace that are ignored, when they are enclosed within backslashes; this whitespace can include newlines. So you can write something like:

The large “Hello, World!” will be printed flush-left, but we were able to align the literal rather nicely in the Haskell source. This sort of thing can be a little more difficult in some other languages.

Also, in addition to the usual C-style \r, \n, \b, etc, and hexadecimal and octal notations (which differ slightly from C’s), Haskell also provides a \^C notation (where '\^C' represents “Control-C”), and a \ESC notation (representing the escape character, using its ASCII acronym). Thus, the ASCII backspace character can be represented as any of '\b', '\BS', '\^H, '\o33', or '\x1B'. Every one of those representations can be useful in different circumstances, especially for a terminal enthusiast and GNU Screen co-maintainer such as myself.

Well, that pretty much sums it all up. This took a lot longer to write than I really wanted it to, so maybe I’m finished talking about Haskell for a while. In presenting all the things that really impress me about Haskell, I’m perhaps painting an unrealistically rosy picture of the language. I do believe that Haskell, and its current best implementations, have some significant shortcomings (but none that actually render Haskell impractical, as is sometimes suggested); if I get around to writing a “part 3”, I’ll probably dwell on those, to counter-balance the wondrous things I covered here. However, that one might be a while in coming, as I think I’ve spent quite enough time writing about Haskell for a limited audience, for now. 😉

Adventures in Haskell

So, the last few weeks I’ve been learning the Haskell programming language, and thought I’d share my thoughts on it. Haskell is a pure functional programming language that supports lazy evaluation (I’ll explain all that in a moment), which has been gaining popularity in some circles (and particularly among academia and computer lingusts). It has (semi-)famously been used to write an implementation for the Perl 6 programming language, and also a distributed revision control system (DaRCS).

I’d heard of the language a couple years ago, and have been wanting to learn it since then, but the freely available resources I’d been able to find for learning about the language were frustratingly poor, and printed books were in the range of $80 and up, which is a bit steep for learning a language as a hobby. I finally found an upcoming O’Reilly book, Real World Haskell, whose full content is available for browsing online. Amazon says the printed version’s going to be $50, which is a marked improvement over other books I’d been considering buying. The book’s not perfect—there are several minor beefs I have with it—but the fact remains that it is the highest quality resource for learning Haskell that is freely available. It also has the major advantage that it focuses heavily on real-world applications (as the name implies), including coverage of network programming, and concurrency (multi-threaded or clustered computing), writing language parsers; it even guides the reader through writing software to retrieve ISBN numbers from cell-phone photos of book backs! This probably makes it the most practical resource for learning Haskell, too—pricey or free.

I’m still working through the book, but I’ve supplemented my understanding so far by reading through the official language specification, The Haskell 98 Report, so at this point I’ve gained a pretty solid understanding of the core language and minimum libraries.

Intro to Functional Programming

(If you’re fairly familiar with functional programming, might as well skip this article and wait for part 2.)

I’ve had previous experience with functional programming languages, mainly Lisp and XSLT, which helped in trying to learn Haskell.

Lisp is one of the more popular functional programming languages , and probably the oldest. My experience with it is primarily through the Emacs-Lisp dialect. Emacs is a very powerful text editor that is popular in the Unix world (and particularly on GNU/Linux). It is mostly implemented in Emacs-Lisp, and you can alter the program’s behavior while it’s running by editing the Lisp code. It’s the first Unix editor I learned to use, and though I primarily prefer Vim these days, I’m very comfortable with Emacs (these days I run it in a vi-emulating mode called viper), and have written code in Emacs-Lisp.  I’ve also dabbled in Scheme, a Lisp dialect.

XSLT is a language for performing transformations on XML documents, and like Haskell but unlike Lisp is a “pure” functional programming language (more on that in a moment). Unfortunately, it doesn’t really provide a complete set of useful tools (version 1.0, anyway: I’m not very familiar with the XSLT 2.0 spec), so my experience with it was that writing XSLT can be an exercise in frustration.

In imperative programming languages like C, C++, BASIC, Java, and many others, the focus is on performing a series of actions: do A, then do B, etc. In imperative programming, one often deals with data in memory that can be modified through a series of actions (mutable data), and the behavior of bits of code in the program may depend on the current value of some particular piece of data in memory, which may be changed by other bits of code.

Here’s a simple example of imperative programming in C code:

This snippet produces an eight-element array a whose elements hold successive powers of 2 (1,2,4,8,16,32,64,128). The code amounts to a series of instructions like:

  1. set the value of v and i to 0
  2. set the ith element of the array a to the current value of v
  3. set the value of v to twice its previous value.
  4. set the value of i to one more than its previous value.
  5. if the value of i is less than 8, repeat from step 2.

Notice the heavy focus on performing a series of steps, and a dependency on modifying data.

In contrast to imperative programming, functional programming focuses on transforming input data into output data, by defining the function that performs the transformation. In a “pure” functional programming language, it is not possible to change the value of some object of data from one thing to another; in fact, there are no objects—there are only values (and functions; immutable data). This tends to place a higher focus on what the desired result is, rather than how we want to arrive at a given result.

Here’s an example of some code in Haskell that produces a result similar to the imperative snippet above:

If you invoke the function above as powers 0 1, its result will be a list of successive powers-of-two from 1 to 128, just like for our C code. However, you may notice that the method by which we’ve arrived at that list is somewhat different. Translated to English, this code says:

  • Define the function powers(i,v) (for any v, and where 0 ≤ i < 8) to be a list whose first element has the value v, and whose remaining elements are the result of powers(i+1, v*2), or, for any i not matching the qualification just given, the empty list.

…and that’s it. When you invoke powers 0 1, it constructs the element with value 1, then invokes itself again as powers 1 2, which constructs an element whose value is 2 and invokes powers 2 4 to construct the third element with a value of 4. When it finally reaches powers 8 256, it sees that i=8 doesn’t meet the criterion that 0 ≤ i < 8, and so it caps it off with an empty list result (which, since it doesn’t invoke itself recursively again, terminates the list, and the evaluation of powers).

Functional code like the above can look cryptic to imperative programmers who aren’t used to seeing it, but it’s learned fairly quickly. It happens to correspond fairly closely to how a mathematician would formulate a function definition, so folks who are comfortable with math will tend to feel right at home.

Note that it’s just as easy to do recursion like this in C:

However, in real-world C implementations, each invocation of powers will consume additional memory. It works fine for our example, where we’re limited to a total of nine calls to powers, but for longer recursions you could run out of stack space, which is not a good thing.

Note that our recursive C example still isn’t pure functional programming, as we’re modifying the values of an existing array, and not producing new values.

Disclaimer: none of the C or Haskell examples are particularly idiomatic; they’re for rough example purposes only. I’d do them differently in real life; but I felt that these examples, as written, may be a little easier to discuss, and more accessible for folks that might not be terribly familiar with either language.

To be continued (very soon)… Part 2 will discuss some of the things in Haskell that I think are really cool. 🙂

Wget, Screen, Eseq

So, things are going pretty well with Wget. We just had our mid-term evaluations for the Google Summer of Code project. Our two GSoC students are right on-schedule with where they’d promised to be at this stage. Both of them had exams during the first portion, so the level of work they were supposed to get done was somewhat scaled down from what it would otherwise have been; still, it’s nice that there haven’t been any real difficulties, and things are coming along alright.

Also, the copyright assignment paperwork came through for a batch of changes that adds CSS support to Wget, so I’m excited about that. I haven’t gotten much done for my part, though. Been pretty busy.

I recently joined the GNU Screen project, as a co-maintainer, to help get things moving towards a next release. If you haven’t heard of Screen, I can’t really explain it in detail here: it’s sort of a special-interest thing. But, if you spend a lot of time using command-line/terminal programs, screen is a huge asset. Especially if you use them remotely.

Basically, what Screen does is act as a sort of reattachable, “pretend” terminal. I say “pretend” because, while it is a fairly full-featured vt100 terminal emulator, it’s missing a crucial component that most terminal emulators have: it doesn’t actually draw text to a screen. It interfaces with whatever “real” terminal emulator you’re running, and tells it what it should draw.

The nice thing about screen is that you can detach from it, and later reattach to it. If you lose your ssh connection, say, then you can simply log in again and attach to the same screen session you were running; none of your programs have to get killed due to a terminal hangup. Also, you can be attached to the same screen session from multiple terminals simultaneously. I do everything at work in screen; then when I come home, I can just ssh in and keep working on the same session. I can leave a build running, and then come home and check on its progress. You can even do more exotic things, like allow multiple users to use the same terminal session (mostly good for demonstrating how to do something).

My priority’s still with Wget, though, and I’ve made it clear to folks that while I’m happy to help out with organization and patch-integration, etc, I need to dedicate most of my free-time coding to Wget. I expect to be able to handle small patches and the like; but I have too much to do already with Wget. However, my work on Wget has ended up taking a sort of hiatus for a couple weeks, while I’ve been organizing some things (bug lists, mainly) at Screen; also, I’ve been spending most of that time coding a program I’ve always meant to write, but which has now become incredibly useful as a tool for debugging issues with Screen.

It’s a program that analyzes terminal escape sequences. These are special commands that are sent as part of the text stream to the terminal, to tell it to do something special. For instance, in Unix-like systems you could print a red-and-green “Merry Christmas” to the screen by issuing the command:

The “gibberish” in there are the special commands that tell the terminal how to color the output. Each bit of gibberish starts with \033, which is a numeric code representing the escape character. If you run the above command, you’ll get output like:

It’s not too bad to look at and analyze a string like the one I gave to printf above; but when you have a whole bunch of funky gobbledy-gook like that to sift through, it helps to break it down. The tool I’ve been working on lately would take that same string, and give this output:

It gives a breakdown of the text it sees, the escape sequences, what control function they represent, and what the actual effect they produce is. This makes it a lot handier to see what’s being sent to the terminal when not everything’s working as it should.

Wget, GSoC, …

It’s been quite a while since I’ve posted last. I’ve got quite a few posts I’ve been wanting to write, but I’ve been pretty insanely busy lately.

Aside from my new Wii (which I plan to post about again, later), my free time has been completely monopolized by my role as maintainer for GNU Wget. The main reason for my the upsurge in my activity related to this project, is that we’re participating in Google’s Summer of Code program this year.

For those who aren’t familiar with it, the Summer of Code (SoC or GSoC) program is an avenue whereby Google spends large chunks of money on fostering an interest in Free and Open Source Software among university student software developers. (See Google’s stated goals for GSoC.)

What happens is, a number of organizations and FOSS projects apply to join the program, and present lists of ideas for projects that students could take on during the summer. A number of students apply to join the program, and submit proposals for projects that they would like to do over the course of the summer. The organizations choose the project proposals they like best, and rank them in order of preference.

Google then decides how many students each organization will get, the organization communicates which students have the most interesting proposals, and—here’s the fun part—Google then pays each student a stipend of $4,500 to work full-time on the project over the summer. (The organization is also given $500 per project on which it has coached a student.)

I had been hearing of GSoC for some time, but had never really understood what it was. An interested student, however (Julien Buty), strongly encouraged me to participate in the program this year (as he wanted to apply for a project with Wget), and in fact got the ball rolling for me. I’m pleased to say that he wound up being accepted, and will now be paid by Google to work on Wget, to improve its handling of authentication over HTTP. One additional student was accepted, Saint Xavier, has also been accepted, and will be working on functionality related to internationalized domain names and web addresses.

This has brought in a real surge of developer interest in Wget, which is very welcome indeed. Up until now, the only active developer on Wget has been myself. Despite Wget being very ubiquitous in the Unix world, and used on millions of installations, it has recently had no real community to speak of. The mailing list has had only a handful of participants, and there are no active developers (sometimes to include myself—I have a day job, ya know!), only occasional patch submitters. But, even though we posted up our “ideas” page less than a week before the start of the student application process, we quickly began to see an influx of interested developers. In fact, along with GRUB, the GNU system bootloader, Wget proposals dominated the applications submitted for the GNU Project.

This ended up translating into a lot of work for me, though, because suddenly a lot of my time was being taken up responding to student questions, critiquing student proposals and giving advice on how to improve them.… Several projects needed to be specified in much greater detail before they could become a useful target for students to apply for, so I wound up spending a lot of time typing up rough specs, and discussing implementation approaches, as well.

While we were only able to choose two to participate through GSoC (which, in itself, was a happy surprise, as through most of the process we expected to get only one), several of the students whose proposals didn’t make the cut have continued on with the project anyway, because they’re interested in contributing and eager to learn and gain experience in the Free Software community.

An approximately equal number of contributors have also recently joined up outside of GSoC, thanks to Saint Xavier’s encouragement that I post a “help wanted” ad through GNU’s Savannah software development portal. I didn’t really think it’d grab much attention, especially as I knew that these ads were automatically closed after two weeks. Boy was I wrong! I got a new developer every day for the first four or five days after I posted.

Assuming it doesn’t fizzle out (it’s early to tell whether everyone will keep their enthusiasm for Wget over the long term), all this additional help means that I can actually realistically think about releasing version 1.12 before the end of the year, which otherwise would have been unlikely. I’m very excited about this, because there are a lot of features I’m going to be very happy to have. Julien and Saint Xavier are both working on pieces that are very high prorities for me for the 1.12 release, and I’m excited that updating Ted Mielczarek’s addition of CSS support to Wget was much easier than I’d hoped.

Perhaps soon, I’ll post an article that gives a better idea of what my pet project actually is, and why it’s so durn useful (as well as what its current shortcomings are, and what I hope to do in the future).