Pulling back from the archives this is a repost of a previous blog post. This time ripped from a guest spot at The Nubby Admin, a fantastic blog from a fellow tech nerd.

At my old job I had a bottom of the line box sitting on my desk that I used for some testing. It was a hardware clone of the oldest Snort sensors I had deployed, and by old I mean corporate desktop grade vintage 2003. I kept it running and set up so that I can test configuration changes, new rules, software updates, etc. This all makes the system pretty mundane and cookie cutter. I'd often get a hankering to start from a blank slate, tell this thing to go koan itself, and come back to a clean install. This is all a rather long and drawn out way of saying that I didn't really care about the health of this system and didn't pay very much attention to it.

One morning, in the email containing the output from the auto-update script for this system, I saw gobs of errors of the form:

Error unpacking rpm package 4:perl-5.8.8-38.el5_8.i386
error: unpacking of archive failed on file /usr/bin/a2p;509097e4: cpio: open failed - Read-only file system

If there's one way to catch my interest, it is to tell me that system partitions are read-only. Some quick research showed me that yes, every single file-system was in fact read only. SMART also was showing errors out the wazoo. Over the years I have learned that SMART's false-negative rate was astronomical, but its false-positive rate was approaching zero. That is, a "healthy" report from SMART was meaningless, but a "failed" report is completely trustworthy. This was really no big deal, since the system wasn't used for anything time sensitive, I could just pull a hard drive off the shelf and reprovision.

Since one thing I had used this system for was to do some performance profiling of snort, I had mocked up a couple of analysis and test scripts on it. The scripts themselves were easily rebuildable, but to save me the effort I used the old tar+ssh trick to archive the home directories for myself and root for later extraction:

tar -zc /home/packs /root | ssh packs@node1 'cat - > snort-test_homes.tar.gz'

This is where things got hairy. Since the last time I had used ssh to go to node1 it had also been rebuilt, resulting in a host key change. With StrictHostKeyChecking enabled ssh refuses to connect if there is a host key mismatch. Ordinarily, I would just delete the key from the known_hosts file and move on. With the file-system being read only....

I worked around this by changing my known_hosts file on the command line. Since all the file-systems were read-only I couldn't actually write any files, nor did I need to save the information. This left me with the perfect choice of /dev/null. Adding in this option made my final command look something like this:

tar -zc /home/packs /root | ssh packs@node1 -o UserKnownHostsFile=/dev/null 'cat - > snort-test_homes.tar.gz'

I liked this because it worked and was easy. I don't like it because it felt skeezy.