Category Archives: Linux

Pertaining to Linux-based operating systems

Lying system calls

Only Debian hackers are likely to grok this.

I just spent a week, maybe more, trying to look into a variety of very bizarre build failures for some packages, after building and installing newer versions of coreutils, dpkg and debhelper. Things like “install -d” dying because its destination wasn’t a directory (I’d check afterwards, it was), tar complaining about a file changing out from under it (always about symlink files, where the block of code in tar that could generate that error could never be evaluated for symlink files), and dpkg-deb –build complaining that a pre-installation script wasn’t executable, when it clearly was.

Of course, I was mostly treating these as separate, bizarre problems that happened to start up within a similar timeframe. Tonight it suddenly dawned on me that maybe all these system calls appear to be lying to me, because they are. They are, and building a package in fact typically involves running a program whose job is specifically to make system calls lie: “fakeroot”.

Fakeroot is of course only intended to lie just enough so that programs think they’re running as root, even though they really aren’t. But fakeroot’s documentation clearly states that it will not work correctly if the programs running underneath it are linked against a different version of the system libraries. The seeds of this problem were sown some years ago when we built and installed a newer version of libc6, but the problem didn’t manifest itself until now, because the most commonly-used utilities for building packages are dpkg, debhelper, and coreutils, all or most of which were not updated until I updated them just a week or so ago. Gah!

Slides from Wget Talk

I’ve made the slides available from my Wget talk here in PDF form. You can also get the OpenOffice “Impress” file here, but note that it uses non-standard/non-free fonts, so most likely won’t display properly for you.

As part of my talk, I also used some recorded terminal sessions demonstrating wget usage, including prompts to wait for keypresses. I created these sessions using GNU Teseq (a project I wrote for debugging terminal sessions); but the sessions were then edited by hand and played using a specially-modified version of the reseq program (part of the Teseq project); those changes have not yet been pushed to the development sources. When I have time, I’ll push the final changes to the official development sources at savannah.gnu.org, and then make the automated terminal demos available as well.

Please note that full documentation on wget can be had at http://www.gnu.org/software/wget/manual/

Update 2011-01-04: The modifications to Teseq were pushed a few weeks ago, so I’ll go ahead and put the script files up; there’s a script demonstrating Wget’s automatic retry capabilities, and one demonstrating the --continue option, for continuing downloads across different sessions. These scripts are unlikely to be useful to anyone who isn’t familiar with the points I was making during the script’s run, but there you go. You’ll probably want to right-click the links and do a “Save as…”.

In order to process them, you need to go grab the development sources of Teseq; you do this by obtaining the Mercurial revision control software and the GNU autotools (autoconf, automake, etc), and then running “hg clone http://hg.savannah.gnu.org/hgweb/teseq/” somewhere suitable. Within the source directory, run ./autogen.sh and then do the typical ./configure && make && make install (all that’s actually needed to generate the reseq program is the ./configure step).

Update 2014-12-15: In order to process them, you need to install GNU Teseq (another project I maintain), at least version 1.1. It’s available on a variety of Unixy platforms, including Debian, Ubuntu, and Fedora. It’s available on Fink, but not version 1.1, so you’d need to follow the “get from repository” instructions in the struck-through paragraph above. On Debian/Ubuntu, you can install the teseq package by running apt-get install teseq; just ensure that it’s at least version 1.1, or it won’t support the interactive files.

Once you have reseq installed, you run the scripts like: reseq --replay --halts auto-retries.seq. Press a key whenever prompted to do so by a green-colored message. You’ll want to do this replay in an xterm-compatible emulator, at least 80 columns wide. You can also play it without the prompts/pauses by removing the --halts option from the invocation.

Upcoming Wget Talks

If you’re interested in GNU Wget, and live in California within a reasonable distance of either Davis (semi-near Sacramento, in Yolo County) or Mountain View (Silicon Valley), then you may be interested in coming to one of the talks I’m scheduled to give at those two locations about GNU Wget. More information can be found here (Davis) and here (Mountain View). The Davis one will take place Novemeber 15th (a week from Monday), and the Mountain View one is January 5th.

Here’s the blurb.

GNU Wget is a computer program that retrieves content from web servers. It supports downloading via HTTP, HTTPS, and FTP protocols, the most popular TCP/IP-based protocols used for web browsing.

Its features include recursive download, conversion of links for offline viewing of local HTML, support for proxies, and much more. It appeared in 1996, coinciding with the boom of popularity of the Web, causing its wide use among Unix users and distribution with most major Linux distributions. Written in portable C, Wget can be easily installed on any Unix-like system and has been ported to many environments, including Mac OS X, Microsoft Windows, OpenVMS, MorphOS and AmigaOS.

It has been used as the basis for graphical programs such as GWget for the GNOME Desktop and KGet for the KDE Desktop and VisualWget for Windows. [From Wikipedia].

Micah Cowan was maintainer of Wget between mid-2007 and early 2010. His talk will discuss:

  • What is Wget?
  • My history with Wget
  • How to use Wget
    • Restartable downloads
    • Website archiving/recursive downloads
    • Fine-grained controls over which links to follow
    • Content conversions for local browsing
  • Wget shortcomings
  • Lessons learned while maintaining Wget
  • Issues unique to maintaining a GNU project