October 31st, 2005

Sheepskin Back Up After Disk Crash

Sheepskin, the CS Alumni server, was back up at 11:35am EST after a disk failure. The user home directories were restored from backups. Here is the timeframe of the failure relative to the backups.

10/30/2005 6:28am - The backup started
10/30/2005 6:32am - The backup finished
10/30/2005 6:39am - The disk failed and was taken off-line

What this means is that any files created during the 7 minute period from 6:32am to 6:39am would not have been recoverable. Files created during the 4 minute period from 6:28am to 6:32am may or may not have been restored depending on when the backup reached a particular directory.

Please note that that it was only the home directory disk that failed. The disk housing email was not affected so no email in the system inbox would have been lost. Remote mail servers typically will queue email for up to 4 days if a server is down. Any email that was queued on a remote server should be delivered but it may take several hours for all the queued email to reach sheepskin. If a mail server is configured to bounce email in less than 1 day it is possible that email may have bounced. Such a short timeout would be somewhat unusual so this should not affect many users. What is common, however, is for mail servers to warn the sender that a message is being delayed if the message cannot be sent in 4 hours. The message will still be delivered but the sender just gets a notice letting them know that there is a delay.