March 11, 2003

The Beginning-of-the-Alphabet Conspiracy Strikes!

I knew I should have chosen a name closer to the start of the alphabet!

Date: Mon, 10 Mar 2003 09:02:52 -0800 (PST)
From: UCLink Consulting <consult@uclink4.berkeley.edu>
To: jbdelong@uclink4.berkeley.edu
Subject: lost mail for jbdelong


jbdelong:

On Friday morning, the inbox and folders for your account and all other accounts beginning with j, k, l, m, n, and o, were lost. We are recovering from backups the inboxes and folders that were in your account as of Thursday night. This is unfortunately proceeding extremely slowly. When done, the recovered mail will be merged into your current inbox and folders.

However, all mail delivered to your account between Thursday, 6 p.m. and Friday, 10 a.m. was lost and can not be recovered.

I had thought that modern email systems had redundancy redundancy so that a problem with a single disk would not fry six letters' worth of inboxes. Just naive, I guess...

Posted by DeLong at March 11, 2003 08:22 PM | TrackBack

Comments

Ah, but you forget. Redundant disk drives, &c. cost money. And as this money is spent for no apparent reason ("what, it's a second disk that does the same thing as the first one? That's like having two coffee machines, what a waste. We don't need that!") it's usually much harder to convince people to spend the money on this.

Until, of course, there's a failure. At this time, the poor sysadmin who's been getting budget requests for redundant systems knocked back year after year will, if he's lucky, cop abuse and sneering from the very same people who knocked back the budget requests. If he's unlucky, he might even get sacked for not having redundancy built-in.

This is something like reason #236 why I am no longer a sysadmin. No, I didn't get sacked. I just got sick of being held responsible for failures that I was not allowed to prevent.

So if you get into this situation, be nice to the poor sysadmin. He or she is probably getting 15 kinds of grief from their boss. Go kick the people with budgetary control up the backside, hard.

Posted by: anthony baxter on March 12, 2003 06:19 AM

So you're saying its time to confiscate this month's salary from the Associate Vice Chancellor for Information Whatever and give it to the sysadmins to use to buy disk drives?

Posted by: Brad DeLong on March 12, 2003 09:52 AM

"Redundundancy" is the word you're looking for.

Posted by: David Lloyd-Jones on March 12, 2003 10:23 AM

What's sadder is the attempts at PR spin by the UClink people. Bearmail (bearmail.berkeley.edu) lists the number 642.4774 for 'status updates.' 4/5 times, the status update is a busy signal. The fifth time gets the message 'Uclink is currently not working' along with a new number. Calling this new number yields a confused receptionist who isn't sure why they listed her number.

Chancellor Berdahl has stated that he intends to do all important notifications over e-mail, and all students are required to get one. He'll probably feel some need to bulk up reliability if this is really the University's first choice of communication.

It also might be a nice chance to rationalize Berkeley's blizzard of subdomains, or do away with it entirely. For example, there's xxx@uclink.berkeley.edu for most, socrates,berkeley.edu for faculty/GSIs, haas.berkeley.edu, and econ.berkeley.edu. Which one would Professor Varian get? All of the above?

-Kevin
calstuff.blogspot.com

Posted by: Kevin D on March 12, 2003 10:24 AM

What's sadder is the attempts at PR spin by the UClink people. Bearmail (bearmail.berkeley.edu) lists the number 642.4774 for 'status updates.' 4/5 times, the status update is a busy signal. The fifth time gets the message 'Uclink is currently not working' along with a new number. Calling this new number yields a confused receptionist who isn't sure why they listed her number.

Chancellor Berdahl has stated that he intends to do all important notifications over e-mail, and all students are required to get one. He'll probably feel some need to bulk up reliability if this is really the University's first choice of communication.

It also might be a nice chance to rationalize Berkeley's blizzard of subdomains, or do away with it entirely. For example, there's xxx@uclink.berkeley.edu for most, socrates,berkeley.edu for faculty/GSIs, haas.berkeley.edu, and econ.berkeley.edu. Which one would Professor Varian get? All of the above?

-Kevin
calstuff.blogspot.com

Posted by: Kevin D on March 12, 2003 10:28 AM

Sound like it is that time, yes.

But there are levels of redundany, they could have actually had a redundant RAID system, but all of the drives, on one SCSI bus, could have gotten fried at the same time (seen it, it's ugly).

More complex, additional, layers of live redundancy CAN be done, but they are wildly more expensive, and normally only seen as justified in the eyes of the very largest email systems, banks, large accounting systems, etc.

Now we'll see how well there backup restore system works. I was at a technical conference once where a speaker asked a room of 1000+ sysadmins, "how many of you have backup systems?"

All but 2 or 3 hands went up.
Then he asked "how many of you have tested the restore feature?"

All but 2 or 3 hands went down (mine stayed up).

So, here's hoping your mail came back from the grave!

Posted by: David Mercer on March 12, 2003 10:30 AM
Post a comment