6:00pm - Opening flag burning ceremony.
6:30pm - Anti-war rally no. 1.
6:40pm - Ted Kennedy proposes a toast.
7:00pm - Tribute theme to France.
7:10pm - Collect offerings for al-Zawahri defense fund.
7:20pm - Ted Kennedy proposes a toast
7:25pm - Tribute theme to Spain.
7:45pm - Anti-war rally no. 2. (Moderated by Michael Moore)
8:00pm - John Kerry presents one side of the issues
8:25pm - Ted Kennedy proposes a toast.
8:30pm - Terrorist appeasement workshop.
9:00pm - Gay marriage ceremony.

9:30pm - * Intermission *

10:00pm - Flag burning ceremony no. 2.
10:15pm - Re-enactment of Kerry's fake medal toss.
10:30pm - Cameo by Dean 'Yeeearrrrrrrg!'
10:40pm - Ted Kennedy proposes a toast.
10:50pm - Pledge of allegiance to the UN.
11:00pm - Double gay marriage ceremony.
11:15pm - Maximizing Welfare workshop.
11:20pm - John Kerry presents the other side of the issues
11:30pm - 'Free Saddam' pep rally.
11:59pm - Ted Kennedy proposes a toast.
12:00am - Nomination of Democrat candidate.

It's nice to see that Nigeria has mandatory leash laws for hyenas and baboons.

I took the basic motorcycle rider course again this weekend and this time passed with flying colors: four points off the written and three points off the skills test. I was stressing pretty hard when we first starting practicing for the skills test, but once I decided each stage of the test didn't have to be perfect I was able to relax and ace the rest of the afternoon. Add in the V-22 tests, and Todd's wedding, and this turned into a pretty good, but very tiring weekend.

In a continued commitment to customer service, Nielson//NetRatings recently conducted a customer satisfaction survey on behalf of Kernow Internet Solutions with startling results. While a vast majority of our customers were satisfied with the quality of service they're receiving, there was a small but vocal group that felt we could do better. Well, really only one person complained, but at KIS if one customer is unhappy with the free service they're receiving then, by God!, we're going to fix that!

The chief concern expressed was that our online album did not display photos in reverse-chronological order, meaning each time an album is opened it displayed the oldest photos first. We wanted to resolve this issue as quickly as possible, so rather than contract with our usual partners in Bangalore, we contacted the folks at Primate Programming. Their trained staff immediately swung into action and before you can say "Let go, that's not a banana!" they had the new album software up and running. You want an album to show in chronological order? OK, but it could already do that. Want it in reverse order? Yup, it can do now as well. Happy now?

London Mayor Ken Livingston was recently quoted in the Guardian Unlimited (UK) as saying:

I just long for the day I wake up and find that the Saudi royal family are swinging from lampposts and they've got a proper government that represents the people of Saudi Arabia.

London Mayor Ken Living

The truth is slowly coming out about the extent of the damage at C I Host. While my service was restored in a little over 159 hours, it looks like I was one of the lucky ones. Scott Dudley reports that the situation with his server is far worse that C I Host's CEO Christopher Faulkner is admitting publicly.

After 16 days of downtime and many hours on hold in the previous two weeks, I once again got in touch with a support representative at CIhost today. She tells me that I've been on hold for too long for her to sugar-coat the story:

Our server, "ares", was completely destroyed by the worm. Their tape backups failed. Their technicians have tried to restore the site, but they weren't able to retrieve anything. Nothing. Nada. Zilch. All gone.

Anything that we had posted there is irretrievably lost. The only way I found this out was by calling their tech support. They did not appear to have any plans to notify the clients on the impacted servers.

Is this some sort of joke? What about their daily, weekly or monthly backups?

What about the "100% of data restored" that their CEO claimed? How long have they been sitting on this tidbit of information?

An interesting note is that our server was running Windows NT 4.0. According to ISS, it appears to me that the BlackICE firewall product reached its end-of-life in AUGUST 2003 for the NT 4.0 platform [PDF document]

Scott Dudley on WebHostingTalk

This of course contradicts what Faulkner said in an email to customers on March 25th. I'd previously only quoted portions of that email but I think its important to document the full text:

We, at C I Host, are respectful of the serious nature of attacks and the overall current state of the Internet as malicious attacks increase. Our concerns are for our customers, our business of course, and the economy in general.

When will it all stop?

On March 20, a premeditated and criminal "worm" attack struck with unprecedented malevolence, seriously affecting C I Host, its partners and many other Internet companies. In recent months, together, we have gone untouched by literally dozens of widespread attacks. Others were not so fortunate in those earlier attacks.

If there is a silver lining, it is that each attack alerts all of us to be more prepared. We are thankful beyond description that because of contingencies, the data from all of our hosted sites was backed up on magnetic tape - 100 percent. We are saddened to hear reports that other companies, using other backup systems did not fare as well.

When the attack first struck at or near midnight, our administrators in several locations worldwide, detected problems and within minutes located the traffic and stopped the invasion. Unfortunately those crucial minutes were enough to deliver a deadly payload. In a sordid twist, the witty worm thrived on a vulnerability of a widely used Internet security firewall. "Witty" attacked and obliterated systems running on Windows NT, Windows 2000 and Windows 2003 leaving nothing in its wake.

Today, most of our servers have been rebuilt from the ground up. The vast majority of our customers' Web sites have been restored to full operation. Hosted accounts have been moved or revived. At first e-mail, and then Internet commerce began to flow, albeit at a measured rate. Then our experts made an important, innovative discovery to quicken the pace. (We will tell you more about that when it is appropriate.) Our goal is to have all shared hosting, Windows clients (minus the 7 servers that are fully destroyed) fully online and functional by midnight, March 25. The last "Magic 7" will be back online before Sunday March 28, 2004 at the very latest. Those servers were so badly damaged all new hardware had to be pulled from inventory and built completely from scratch with Windows OS reloaded, IMail and every piece of software individually installed by hand and then pulling the 120-200GB of client data from tape backup for each server. It has been one of the most time consuming and frustrating projects I have every undertaken.

We apologize sincerely and humbly to all clients who were so adversely victimized, as were we. Some of you already know there are a few servers still restoring and some backup content is still loading. It is a drawn out, meticulous process.

There is no way to comfort a business person who has been shut down for several days, due to a calculated and anonymous saboteur. We can empathize because we all were hit together. It is little consequence, but you should know that quick action avoided an even larger catastrophe.

The healing process began immediately. As soon as the attack was stopped, administrators begin rebuilding servers . that very hour. This attack was like nothing we had experienced before. The company went into emergency mode, reassigning all technical personnel to positions that would create the fastest recovery time. If this created longer waits in customer service, we apologize, but made the decision to reallocate manpower to best and most efficiently benefit our customers.

When the dust settles, we will double our efforts to provide faster, more accessible answers from technical experts via telephones and Internet chat. But until every site for every customer is up and running, our resolve will remain unchanged.

Rest assured my team and myself personally are working has hard as any human can work for you and your organization.

Sincerely,

Christopher Faulkner
CEO
C I Host
www.cihost.com

PS: All clients affected will be credited AT LEAST 1 month of free service and your account has been marked so that you will receive unlimited free account upgrades and addons for the entire life of your hosting account. You will NEVER pay us for another addon or additional feature again. Add what you like and choose as many of our addons and features for your account - they are now free.

Christopher Faulkner, CEO, C I Host - 3/25/2004 @ 20:18

So why am I continuing to ride C I Host like this? After all, I moved and had service restored in just four days. Larry Wall, the creator of Perl, once said that the three greatest virtues of a programmer are "laziness, impatience, and hubris." While these are fine for a programmer, they're a recipe for disaster for a network service provider. A responsible service provider must be ever vigilant, constantly testing their network for vulnerabilities and developing new strategies for protection and enhancement. This starts by hiring engaged network and system administrators who think of security and accountability first, and the time clock second. C I Host has the mark of a company that built a nice little hosting business and spent the last five years increasing their public profile and adding servers without spending additional money on software, security, and personnel. Hubris brought them down, and their customers' laziness will keep them in business.

• CaffeDerm®
• Enabling Cuneiform Support in InfoPath 2003 SP1
• Google Copernicus Center is hiring
• How Hydro-Ordnance Works
• PC EZ-Bake Oven
• Sony Ericsson LH-Z200 for Lefties from Virgin Mobile