Recent Entries

After an unclean system shutdown I always have these vim tmp files laying around that cause annoying messages to come up when I edit files that I had previously edited while my system went down. These things are spread out all over the file-system. I had a cron job set up that would use locate to hunt them down, but this caused two problems: it required more than a day for the files to be removed, because updatedb had to be updated, then I usually gave it 7 days of grace time in case I was editing something for multiple days...

The worst problem was a subtle bug in this code which caused me to have to recover my home directory from backups. This is why RAID is not a backup plan. :-)

I've recently switched to a new mechanism though, which works much, much better. Read on for more...
(read more | 0 Comments)

It sure would be nice if any of the installers had a "clone partition layout from another drive" option. We often end up setting up multiple drives with similar layouts, either because of RAID or LVM, and using the partitioning GUI can be painful during the install.

We were doing this the other day on a system with 4 drives, and I used something that I discovered in some other automated partitioning work we had recently done to automate the setup. Read on for more details.
(read more | 3 Comments)

Just an update on the Hitachi drive issues (see links to previous related posts if you need more information). Kevin tested a test kernel for CentOS, and that went well so the next CentOS 5 kernel should have that fixed. I also tested the current Hardy kernel, and it is fine.

So, the installers still have issues, but if you install either in the first 120GB or the last space beyond 140GB, then do updates and allocate the remaining space, you should be fine as far as working around it. Of course, if you boot from the CDs for rescue, you may run into this problem. Hopefully, the CentOS 5.3 media when they release it should resolve this. No idea if Hardy is going to roll new install media.
(go to article | 0 Comments)

Tonight at our LUG we had a great presentation about building a large scale LTSP network. As part of this they needed to spread the load out across a number of different machines. But they didn't have the ability to deploy a traditional stand-alone load-balancer.

I mentioned options of using CLUSTERIP or unifying the load-balancer with the application machines, so that a dedicated load-balancer isn't needed. I wanted to give some more information about these options because they aren't as well known as the more traditional methods. Read on for more information.
(read more | 0 Comments)

Just a quick post to note a few happenings in Fedora land for those interested:

This weekend will be the first set of IRC Classroom sessions in #fedora-classroom on irc.freenode.net. Take a look at https://fedoraproject.org/wiki/Communicate/IRC/Classroom for a list of classes and times. I'm teaching one on Firewall Basics on Sunday. Hopefully we will get a good turnout. Students (and Teachers) welcome.

It's also election season in Fedora land. There are 4 groups holding elections in December. See: http://fedoraproject.org/wiki/Elections for more info. In particular I would like to urge those folks that have disliked recent FESCo decisions to run for FESCo or perhaps see about putting forward someone who shares their views.

Finally, Fedora 10 is just around the corner. The Preview Release just went out this week. Testing is very welcome. I've upgraded my laptop from Fedora 9 with preupgrade and it went very smoothly. Congrats on all the great work for this release everyone!

The other day I had two different clients asking about processing e-mail from a python program. In particular, each e-mail message that comes in gets handed off to be processed by this program. Setting up the mail server to call the program is fairly easy, by configuring local delivery and using the .forward file or similar. However, the program which processes the message needs to do a number of things to do it's job reliably.

I pulled together some pieces of existing code I had to handle these issues. Read on for an example of my code for processing e-mail.
(read more | 0 Comments)

After a month of working with Hitachi on this, and digging through some kernel changes, it looks like the issue I previously reported (part number 0A35415) is a kernel bug.

This seems to be a change in how the firmware handles addressing one specific block on the drive, which other Hitachi drives were more tolerant of. However, this does conform to the ATA specification, so the problem is definitely in the Linux ATA driver.

A fix has been committed to kernel 2.6.27-rc7. However, until this change is rolled into the distributions install media there will be the opportunity for this change to cause grief, particularly for Red Hat Enterprise, CentOS, and Ubuntu LTS users who may not get a re-roll of the install media for quite some time...

Read on for more details and work-arounds.
(read more | 3 Comments)

All the cool kids are doing it.

Instructions: Take a picture of yourself right now. Don't change your clothes, don't fix your hair - just take a picture. Post that picture with NO editing. Post these instructions with the picture.
(go to article | 0 Comments)

Supermicro doesn't seem to have one of those whizzy BIOSes that will update directly from a non-bootable media or Internet or Linux (even with the remote management/KVM cards). In the past, most BIOS updates I've been able to just squeeze into a DOS bootable floppy, but it's been getting increasingly hard. However, the latest systems BIOS file alone is 2.2MB, so I had no hope of getting it on the 1.44MB disc I've been using the past. So, I spent some time and figured out how to make a FreeDOS bootable image. Read on for the details.
(read more | 0 Comments)

I haven't seen this exact situation before, so I wanted to write it down for helping others via google. I was installing Hardy on a system with 9 drives, and the installer was detecting the drive I wanted to have the root partition on as /dev/sdi rather than /dev/sda. The boot drive was properly set in the BIOS. Read on for information on how to fix it.
(read more | 0 Comments)

In short, we started seeing dramatically fewer drive errors after running read/write testing of drives before putting them into production.

As I mentioned in my previous post (linked above), we run fairly long-running tests (around a week with 500GB drives) before using them. However, in my experience this is fairly unusual -- it's rare that I run into others that do similar levels of testing. Even for RAID systems (all the systems we deploy these days are using RAID), reduced errors in production save time and attention.

We started doing these tests because we were finding that maybe one in 5 or 10 drives would report sporadic read errors, but when we replaced the drive and ran a badblocks on the removed drive it was showing no problems.

I theorized that there might be some marginal areas of the discs, perhaps from production, perhaps from shipping, I don't know. These areas of the disc were good enough to be written to, but then after some fairly short time the data there could not be read again because of bit-rot.

Once I started doing the read/write badblocks testing, writing and then verifying the full disc 20 to 40 times, we found that these issues basically stopped showing up in production.
(go to article | 3 Comments)

As many of you who know me are aware, I've been a pretty big advocate or IBM/Hitachi drives for a very long time. I've probably deployed around 500 of them over the years, and had a very low failure rate, and when I have had problems IBM and more recently Hitachi have been extremely helpful. Even the drives that brought about the "Death Star" nick-name I had almost no problems.

However, over the last 45 days we've been trying to work with Hitachi to try to resolve some problems and have had basically zero luck making any progress. We've given Hitachi quite a bit of time to provide some information on this, but so far have basically made no progress with them. I wanted to open up the discussion to other to see if others were seeing similar results, or perhaps it's just the level of testing we do, and nobody else is doing this testing.

In short, we test all drives before we put them into production by using a read/write badblocks test. We've done this for probably 5 to 7 years as part of our standard hardware burn-in. No hard drive goes into our managed hosting facility without going through 4 to 7 days of full read/write testing.

With our most recent batch of 500GB drives (part number 0A35415), the labels say "May 2008", we started seeing errors during the burn in. And I'm not talking about one drive, I'm talking about 22 tested drives out of 2 different 20-packs of drives, tested on 4 different systems with different controllers, all of which did not report errors when testing drives from older batches.

Have you seen similar problems with recent Hitachi drives? What sort of testing do you do of new drives, would you have even noticed? Read on for more details of what we've been seeing, including information on how to do your own testing.
(read more | 3 Comments)

Usually when I insert my Sprint cellular wireless card it shows up on /dev/ttyUSB3, but the other day it showed up on ttyUSB6. When you insert the card it creates 10 ttyUSB files, and I just want my card to work. So, I wrote a small python program (based off code I have previously written for talking to serial devices). Read on for how I solved it.
(read more | 3 Comments)

I've been using enigmail inside Thunderbird for mail for a while now. However, the default setting was such that mutt wasn't auto-detecting the messages as encrypted when I was sending them, because the content-type wasn't set -- programs had to look for the "BEGIN PGP MESSAGE" line.

It took a bit of hunting, but I was finally able to figure out how to change this. Since it's non-obvious, here's what you need to do:

• Click on "Write" to start writing a new message.
• Click on the OpenPGP menu.
• Select the "Default Composition Options" item.
• Select the "Signing/Encryption Options" sub-item.
• Check the "Always use PGP/MIME" box.
• Click "OK".

You can't change this option from the main "Preferences" window for Enigmail, you have to do it from the Composition window. That's what tripped me up.

Encrypted-ly yrs,
Sean
(go to article | 0 Comments)

One of the things I really miss about bash is the help for the "test" command (for shell conditionals). There's the test man page, but for some reason I just always want to type "help test" to see what I can use in shell conditionals. I finally decided to fix this and came up with the following line which can be added to ~/.zshrc:

function help { bash -c "help $@" }

When I'm coding, and sometimes when I'm writing, I like to see a lot of code. Usually I just maximize my window vertically, which gives me around 51 lines, and to the side of it I have a couple of windows I can use for a Python shell, regular shell, or another editing session. Sometimes I just want to see more code... Read on for a nifty trick I found in vim.
(read more | 1 Comment)

I've been doing some internal automation which is basically turning a workflow that Evelyn regularly does into a script that takes care of most of the "light manual labor", so she can concentrate on the higher value part of that work. Part of this involves prompting about what to do next...

One of the first libraries I wrote in Python back in 1997 when I started using it was a port of a previous C library I had which would set the terminal in raw mode and read a response to a prompt. I decided to put this on steroids and make the task a bit easier. Martin Blais has made hos if this code in his SVN helper, for example. Read on for more information...
(read more | 0 Comments)

It's been a month since I set up and started heavily using ZFS under Linux on my storage box, It's been working quite well. So well in fact that I set up a new machine to migrate 3TB worth of ZFS snapshots from an OpenSolaris system that we've been having problems with. That didn't go at all well. Read on for more details.
(read more | 5 Comments)

Steve Holden has posted a blog entry titled "Where's Sean Reifschneider When You Need Him?" I feel so super!

Fear not, citizen! I'll be in Chicago in 2009! Now where are my tights?

(With apologies to the WearPython folks :-)
(go to article | 0 Comments)

Enabling automatic upgrades on Ubuntu Hardy is fairly cryptic. There's a nice "unattended-upgrades" package, but installing it is (far less than) half the battle... I've created a helper script, and ended up getting fairly fancy with it, but if you want to make use of it the short form is:

wget ftp://ftp.tummy.com/pub/tummy/ubuntuenableautoupdate/ubuntuenableautoupdate
sh ubuntuenableautoupdate

It will prompt you for a couple of options you can tune (though command-line arguments are available, for example "ubuntuenableautoupdate -yu updatemaster@example.com". Read on for details on what exactly this does.
(read more | 0 Comments)

Seems like every time you turn around there's a story about whether "Linux is ready for the desktop". As someone who has been using a Linux desktop for over a decade, I just ignore them and keep typing. As an aside, I would like to say that we recently had an administrative assistant join the company and she was able to transition from Windows to Linux quickly and with no problems at all.

Anyway, tonight at the NCLUG meeting, I'm going to be one of a few people who are demonstrating our "desktop habits" -- how we get things done. Read on for a list of the things I do to help me get work done.
(read more | 2 Comments)

Putting together the last few posts I made, I've written up an article with detailed information about the hardware and software configuration for a 6TB encrypted Linux-based ZFS file storage system.
(go to article | 2 Comments)

As was pointed out by Daniel Webb in a comment to my previous post, under Linux you have to use FUSE to use ZFS. He just replied before I had a chance to get the next post in this series out. :-)

We've been using ZFS under Open Solaris for the last year or two in our hosting business for backup servers. It has some really compelling features (beyond what I mentioned in my last post) when used for backups. While it has worked well, it hasn't been entirely trouble-free. For a home backup/storage server I wanted to use ZFS but I absolutely have to keep the data encrypted.

ZFS under OpenSolaris doesn't currently support on disc encryption, though they are working on it. Linux has very mature disc encryption support, it's in the stock kernels and many installers support it now. That plus me being very familiar with Linux prompted me to look at ZFS under Linux again. Read on for my user report.
(read more | 5 Comments)

The data on computer systems is what makes them valuable. Most file-systems and RAID designs will go through all sorts of work to make sure that their own meta-data are correct, but very little about the user data that they contain. ZFS, in contrast, checksums everything that's written to disc -- meta-data as well as file contents. It can detect if the disc has silently been corrected, and recover from it. Read on for more of the benefits of ZFS.
(read more | 1 Comment)

I've been curious about SATA Port Multipliers because of my home storage server. SATA is great stuff, and not that bad even when dealing with 10 drives in a single relatively small case. However when you outgrow that case, or just as likely the power supply, you need to start adding drives externally. But do I really want 5 or 10 normal SATA cables routing out of my case? While it's easy to get 8 internal SATA ports, 8 eSATA ports is quite unusual.

I recently found that SATA II supports Port Multipliers, allowing multiple drives to be connected to a single SATA port. Sounds like just the trick, but how is support for them? Read on for more information.
(read more | 3 Comments)