Lessons Learned

I learn something new every day. Some days I don't even get hurt.

Gene McCulley
mcculley@stackframe.com

• X11 and Terminal.app (2006-11-14T21:38:29Z)

  I routinely use X11 under OS X on my laptop to run programs on various Linux machines in my office and elsewhere. The two terminal choices on a standard OS X install are Terminal.app and xterm. Rather than switching between Terminal.app when doing local work and xterm when doing work involving X11 and ssh, I prefer to stay in Terminal.app as it is generally more pleasant. The problem is that Terminal.app has no connection to X11 and thus has no DISPLAY variable set. My solution is to have X11 automatically start Terminal.app and to autostart X11 when I log in. To do this, I have the following in my ~/.xinitrc:

  #!/bin/sh
  # This is just a copy of /etc/X11/xinit/xinitrc except for the lines that
  # start up the Apple terminal and allow connections from localhost.
  
  userresources=$HOME/.Xresources
  usermodmap=$HOME/.Xmodmap
  sysresources=/etc/X11/xinit/.Xresources
  sysmodmap=/etc/X11/xinit/.Xmodmap
  
  # merge in defaults and keymaps
  
  if [ -f $sysresources ]; then
      xrdb -merge $sysresources
  fi
  
  if [ -f $sysmodmap ]; then
      xmodmap $sysmodmap
  fi
  
  if [ -f $userresources ]; then
      xrdb -merge $userresources
  fi
  
  if [ -f $usermodmap ]; then
      xmodmap $usermodmap
  fi
  
  # Start up Terminal.app.  Starting it up from here ensures that it will get a
  # good value for the DISPLAY variable.
  /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal &
  
  xhost +localhost
  
  # start the window manager
  exec quartz-wm
      

  
  
• RSS export (2006-07-23T12:25:33Z)

  I added RSS export functionality to this blog using an XSL transform from the Atom source. That was no fun, but only because I don't know enough about XSL.

  I started with this transform written by porneL which is available with other Atom related XSL utilities.

  I used the Feed Validator to test the resulting RSS and it complained about the date formats being in ISO 8601 instead of RFC 822. I looked over the RSS 2.0 specification and found via Google some XSL code by Kimbro Staken to do the date conversion. I was initially concerned because this code requires the EXSLT date functions but it looks like xsltproc and the XSL transformer built into JDK 1.4.2 support these. That's one of several things that would be more obvious to me if I was more familiar with XSL. I had to add a little bit of code to zero-pad the number values.

  Feel free to copy and use my version of the transform. Let me know if you have problems with it or have ideas on how to improve it.

  
  
• Squirrelmail and LDAP problems (2006-07-22T21:17:59Z)

  If you are trying to configure SquirrelMail to talk to an LDAP server in order to make the address book functionality work and you are getting mysterious protocol errors, you probably just need to enable v2 protocol support, which is disabled by default in favor of only supporting v3 if you take the Debian defaults.

  
  
• Backing up (2006-07-08T21:15:01Z)

  I basically live out of my laptop, which is currently a 12" PowerBook G4 (1.33 GHz, 1.25 GB RAM) running OS X 10.4.7. My home directory contains not only all of my current mail, documents, and code, but recursively, the home directories of every other machine that has been my "main" machine for the last 10 years. That is a lot of data, about 36GB.

  I've used various backup strategies over the years and nothing beats rsync for backing up servers and workstations. However, on my laptop my home directory is stored in FileVault, which makes me a lot less anxious about it getting lost or stolen. This makes backup a lot easier as FileVault stores your home directory in a single encrypted sparse image. To do my backup, I logout and log back in as root so that my home directory will get flushed out to the sparse image and unmounted, plug in an external FireWire or USB 2.0 drive, and execute cp /Users/mcculley/mcculley.sparseimage /Volumes/backup/mcculley.sparseimage.`date` (I never have to actually type this as it is the only command I ever run as root, so I just type control-r cp . That command creates a file with a timestamp in the name. Every few backups I delete the oldest version.

  My backup today was 37,378,859,008 bytes and took 40 minutes and 50.508 seconds to a USB 2.0 drive. That works out to 122Mb/s. It was faster on my old FireWire drive but it died, so I guess I've got to get another one.

  
  
• java.lang.StackOverflowError (2005-11-04T00:11:03Z)

  I got a crash in some Java code today with a java.lang.StackOverflowError as the only line of output. Of course, I've seen this before. It is usually caused by infinite recursion and you only get the single line instead of the complete stack trace because the HotSpot JIT does not preserve the stack. I was temporarily mystified because the code that had been modified didn't have any recursion.

  So I ran again, adding -Xint to the invocation of the JVM. This tells it to run using the interpreter, which preserves at least a chunk of the stack. This time the stack showed infinite recursion in java.util.AbstractSet.hashCode(). The offending line of my code was subclasses.add(subclasses); where subclasses is a java.util.HashSet. The line should have been subclasses.add(subclass);. This was a simple typo that was causing the Set to be added to itself.

  Two lessons today for the gentle reader:

  1. Sometimes the interpreter is useful.
  2. Adding a set to itself causes infinite recursion because of the internal use of hashCode().
  
  
• Debian/IMAP gotcha (2005-11-03T04:08:11Z)

  This is the nugget that actually inspired me to set up this blog: When you install uw-imapd under Debian to provide an IMAP server, you might find that it does not allow any clients to login. If so, just try turning on SSL. The Debian package of uw-imapd is compiled with paranoid configuration options such that it does not allow logins with cleartext passwords if encryption is not provided via SSL. This is a good thing overall, but is a bit of a hassle when it is forced upon you.

  
  
• Java/XSLT/Tomcat annoyances (2005-11-03T02:32:02Z)

  Having set up the Atom feed, I thought it would be straightforward to use XSL to render the feed as HTML. I found an XSL stylesheet for Atom 0.3 written by Rich Manalang at http://manalang.com/wp-content/atom03.xsl and modified it to do something reasonable for Atom 1.0 and my feed. It doesn't make compliant XHTML yet, but I can fix that later. You can download my version of the stylesheet at http://www.stackframe.com/people/mcculley/atom.xsl (this URL may change if I get around to making this blog better). Having tested this locally with xsltproc and a web browser, I thought the hard part was done and went about plugging that stylesheet into our application server.

  We are using Tomcat 4.1.31 for our application server. We are using this version because it easily installs on Debian 3.1, which is what we are using for our servers right now. I was surprised to discover that this pretty recent version of the J2EE stack does not have any simple way to do XSL. After flailing about with various solutions that would require me to upgrade just about everything, I fell back to writing a simple servlet that just does an XSL transform on a given URL. Googling around, a lot of folks seem to have done that, but none of the ones I found would work with the versions of the software I was running. That was only 30 lines of Java code, so it was not a big deal.

  I ran into two non-obvious problems, hence this post:

  1. The javax.xml.transform.Transformer class failed complaining "Exception: stylesheet requires attribute: version". One would think that the stylesheet lacks a version attribute. But my stylesheet has a perfectly valid version attribute. It turns out this is an obscure known bug where the Transformer does not work correctly when dealing with documents already parsed into a DOM. Pointing the Transformer at files and letting it do the parsing makes it happy.
  2. Once I got past that, I got an error that it could not load the XMLEntities.res resource during the execution of the transform. This turned out to be because the Tomcat security manager sandboxes the servlet and it cannot go out to the w3.org website to download the definition of entities in XML. I temporarily fixed this by setting TOMCAT4_SECURITY in /etc/default/tomcat to no. This is not a real risk as that server only serves that domain, so we are not worried about multiple web applications stepping on each other. It makes one wonder what the real benefit of using classloaders to sandbox web applications is when one has to fall back to the OS doing the sandboxing. Another way to fix this is to create a local copy of the XMLEntities.res file and point to that. I've had to do that before and I seem to remember having to get all of the files it depends on as well.
  
  
• Hello World (2005-11-03T01:07:16Z)

  I used to be pretty dismissive about the value of blogging. Though there are a handful of blogs I get value from, it seems far too many people have a blog these days containing whatever random thoughts pass through their heads. A few days ago, I decided I should start documenting those little things I run across when doing software development and system/network administration that are not obvious and frustrating. This, I thought, would be of benefit to my colleagues and customers. After writing my first few emails implementing this resolution, I realized that I really should be putting it out on the web in a form that other folks could benefit from via Google.

  The next logical step was to give in and create a blog to make that publishing as easy as possible. Hence, this blog, which will contain just those things of a technical nature that I think are not obvious. No politics. I promise.

  This blog is implemented in Atom. Right now that means Atom 1.0 (RFC 4287). I'm not interested in supporting any of the flavors of RSS. The reason that Atom appeals to me is that it is designed from the ground up to be a good XML citizen and support real markup in the form of XHTML. Choosing Atom seems to be the easiest way for me to quickly get it rendered into HTML, support feed readers, and ensure that I can still work with this text portably in the future.

  Another reason for choosing Atom is that I've already written some Java code for dealing with it, so that gives me a warm fuzzy feeling that I can bend it to my will in the future. Right now this whole feed is just a full-text feed driven from a single XML file. I would like to replace this with an Atom database eventually and keep the URL the same. I picked the URLs and URIs with that in mind. We'll see how it goes.