The Graphical User Interface Gallery includes screenshots of dozens of operating systems and hundreds of system components. One of the most interesting charts shows the icons used by different operating systems for drives and devices

Sunday was the two month anniversary of the MyDoom virus. Since then, we've seen 112 new viruses, including 8 versions of MyDoom, 7 versions of Nachi, 17 versions of NetSky, and 21 versions of Bagle. The most recent of these, NetSky-Q, takes advantage of a flaw in Windows that was patched three years ago.

So why would a virus writer take advantage of such an old vulnerability? Simply put, people don't patch. Specifically, dial-up users with older Windows operating systems (pre-XP) don't patch. The latest patches usually require the latest service packs, and at a couple hundred megs a piece, most dial-up users don't have time to download the service packs. Add to that the fact that patching can be a scary and confusing process. Do I install patch X1248 before X3429, or is it the other way around? Microsoft greatly improved the process when it introduced Windows Update, but not everyone can use WU. Users of Windows ME with Internet Explorer 4.0 installed must, you guessed it, update and patch before they can run Windows Update.

Fortunately, Microsoft has come to the rescue with the Windows Security Update CD. It contains all of the critical updates and patches through October, 2003. Certainly not the latest updates, but it is a huge step in the right direction. Pop this CD into your computer and in a short time you're up and running with most of what you'll need. You'll then be able to access the Windows Update service to grab the latest updates. The CD is free, so order your copy today.

Oh, the plural of virus is viruses, not virii as some have suggested. If you must refer to 2 viruses as virii please carry that out to it's natural conclusion: 3 viriii, 4 viriv, 5 virv, and so on.

I ran across this quote yesterday:

Learning perl-style regular expressions, while it has been extremely worthwhile, has also been about as comfortable as a surprise circumcision.

Chaingun in NANAE on Usenet

The WTO has ruled that the United States ban on online gambling violates international trade laws. The Black Pot of Stupidity award goes to US Representative Bob Goodlatte (R-Va) who said:

It cannot be allowed to stand that another nation can impose its values on the U.S. and make it a trade issue.

US Representative Bob Goodlatte (R-Va)

How do you learn to ride a bike? Practice, fall, dust yourself off, and try again. Turns out learning to ride a motorcycle is similar. I spent two hours today with John, the instructor I had a couple of weeks ago, and this time I really got the hang of it. I don't know if it was the one-on-one training, or my mental attitude, or what, but not only did I master shifting, but I conquer the u-turn box, and had a hell of a lot of fun to boot. The next scheduled class is this weekend, but I'll be at the Lubbock County Republican Party convention, so I'll have to wait a few weeks before I can finish the class.

Update: I've started updating Connor's photo album.

Around 11pm (5:00am GMT) my website lost contact with it's SQL server. A few minutes after that, the website disappeared. At 9am the next morning I confirmed my worst fears: my site was down... hard. My hosting provider, C I Host, here-after referred to as Ass Clowns, lost their Windows servers to the Witty virus. This post is being written offline, and will be updated frequently as I monitor the situation. Some updates will be in real-time, others will be pre-dated. I'll update it online just as soon as I can.

Quick Recap
C I Host restored service to my server in 159h 31m
I retrieved and removed by data from their server in 31m
C I Host processed my cancellation in 234h 18m

Links:
Trend Micro's information on Witty
Witty worm infects, dies quickly
CAIDA Analysis: The Spread of the Witty Worm
C I Host: A Host To Avoid
Steamed! Review: C I Host / CI Dial / Creative Innovations

Timeline:

3/19/2004 23:00
Noticed database for www.kernow.com was down

3/19/2004 23:03 (T+ 0h 03m)
Website stopped responding. Verified that POP3, SMTP, FTP, and webmail were also down. Verified that I could still see C I Host's network. Past experience (12 outages) told me that their tech support people would tell me the problem was on my end (it never has been). So, rather than get treated like an AOL user, and because it's late, I called it a night. If the website was still down I'll submit a ticket.

3/20/2004 05:18 (T+ 6h 18m)
Website still down. By this point I was concerned that the cause might be Witty.

3/20/2004 09:00 (T+ 10h 00m)
An unknown tech (you'll see why he's unknown in a minute) replied that a virus has swept through C I Host's network and that all Windows servers were down. He went on to say they'd email me with more information.

3/20/2004 09:33 (T+ 10h 33m)
I asked for the virus' name and pointed out emailing me was stupid since their email servers were down

3/20/2004 14:55 (T+ 15h 55m)
"MikeM" confirmed that the virus was Witty. He then removed the previous tech's remarks from my trouble ticket.

3/20/2004 15:34 (T+ 16h 34m)
I made a progress inquiry

3/20/2004 16:14 (T+ 17h 14m)
"TonyL" replied that their admins "were working around the clock to attempt to assess and repair the damage to the servers. Due to the nature of the damage, the admins handle each server on a case by case basis, and as such, there is no ETR."

3/21/2004 08:30 (T+ 33h 30m)
Confirmed in a chat session with Tech Support that there was still no ETR. Was told to contact Billing on Monday (the next day) about a credit

3/21/2004 08:40 (T+ 33h 40m)
Started researching a new hosting company

3/21/2004 15:47 (T+ 40h 47m)
I made a progress inquiry

3/21/2004 21:13 (T+ 46h 13m)
C I Host posted the following notice: "A quickly spreading Internet worm destroyed or damaged tens of thousands of personal computers worldwide Saturday morning by exploiting a security flaw in a firewall program designed to protect PCs from online threats.

In response to this serious threat, we are in the process of isolating and examining all Windows servers to ensure that no client data is at risk. Windows hosting clients may experience intermittent connectivity issues until this matter is fully resolved."

3/22/2004 09:03 (T+ 58h 03m)
Entered a chat session with "jennifert" in their Billing department. Asked how C I Host was going to credit their customers. "Jennifert" said absolutely nothing for the next 22 minutes.

3/22/2004 09:26 (T+ 58h 26m)
Re-entered the chat session with "jennifert". Again, she remained quiet for the next 20 minutes.

3/22/2004 09:45 (T+ 58h 45m)
Entered a chat session with "Tech_-_Jess" in the Tech Support department. Was told the database servers have been restored and they're working on the web servers. Still no ETR.

3/22/2004 09:57 (T+ 58h 57m)
C I Host is not answering their phones

3/22/2004 10:02 (T+ 59h 02m)
Entered a chat session with "CSR-DamienB" in their Customer Service department. Asked about credit. Was given a link 30 minutes later. Turns out C I Host employees chat with customers while talking with other customers on the phone.

3/22/2004 14:00 (T+ 63h 00m)
Connor is born. This is not the time for my family to be without email.

3/22/2004 20:00 (T+ 69h 00m)
Signed up with a new host. Uploaded the website from my most recent backup (2/11/2004) and started the DNS transfer process

3/22/2004 20:45 (T+ 69h 45m)
Called Tech Support and learned that the database servers had been rebuilt but none of the databases had been restored. Was sent to an online form to request that my database be restored

3/23/2004 09:00 (T+ 82h 00m)
Database still offline. Website still offline

3/23/2004 17:45 (T+ 90h 45m)
Configured email with the new hosting company. DNS changes are propagating. Family back online with email. From this point on my site is up, but I still need C I Host to restore everything so I can grab the database and the most recent changes.

3/23/2004 21:30 (T+ 94h 30m)
Called Tech Support and was told the websites were up. I asked why I couldn't access the site (by IP address, but the tech didn't need to know that) and was told that the web servers were up, but the websites were still being restored. Asked about the database. Was told the website and database would be back up in the morning.

3/23/2004 (time unknown)
Sometime on the 23rd C I Host posted the following notice:

Many Internet companies were struck by one of the most damaging, malevolent worms in Internet history, which affects Windows servers. A worm known as "witty" attacked the Internet early Saturday morning, preying on vulnerabilities in one of the most well-respected and widely used firewall security products.

"Witty" thoroughly destroyed servers and their data. Servers that were knocked out by the worm must be manually rebuilt and restored from backups. Please be advised, it is a labor-intensive and arduous process which may take days to complete.

C I Host: Witty Worm Information

It's difficult to know exactly when this was posted. The original notice is not dated, and can only be seen if you use one of the three "doors" into their support chat system.

3/24/2004 08:45 (T+ 105h 45m)
Website and database still down

3/24/2004 11:08 (T+ 108h 8m)
Tech-JerrodH told me the restore was "coming along". He went on to say that "at this point the server [with my website] along with another 37 have the OS re-installed, we have done disk checks on all of them and are now currently working on the restores." Jerrod added that the database server is up, but he's not certain of the state of the restore.

I've resubmitted the form to have my database restored

3/24/2004 17:46 (T+ 114h 46m)
Signs of life. I can access the webserver via FTP and HTTP but I can not login and there is no content. So, the server's been rebuilt, IIS has been configured, but no users or content have been restored.

3/24/2004 22:57 (T+ 119h 46m)
Tech-Jess6 told me in a chat that my webserver was serving mail and webpages and that they were still restoring the database server. I informed him that it wasn't serving my webpages. Chat was abruptly terminated

3/24/2004 23:14 (T+ 120h 14m)
Tech-Jess (is this guy working multiple chat sessions?) told me they were still restoring websites and that none of the FTP accounts had been set up yet.

3/25/2004 01:52 (T+ 122h 52m)
Posted on C I Host's Network Status page: "All NTSQL## shared server boxes are up and online without any data loss from the "WITTY" worm. The last box, NTSQL05 came back up this evening. IF you are still having issues with your databases you might need to request a DSN to be recreated to attach back to your database. The form for this is located at: http://hostingsupport.com/sql/" I resubmitted the form (third time now).

3/25/2004 02:00 (T+ 123h)
Posted on C I Host's Network Status page: "Many shared Windows servers have been returned to service or are in the queue to be returned to service by midnight on March 25 (Thursday). If your site is currently up without its SQL database, please use this form (http://www.hostingsupport.com/sql) to have your db re-attached. Otherwise the vast majority of our shared Windows hosting customers are up and running without any issue or data loss."

3/25/2004 11:21 (T+ 132h 21m)
I received a call from C I Host. The caller, a man with a very strong Indian accent, informed me that the Internet suffered from a massive virus attach Saturday morning and that my website has been down. He went on to say that C I Host appreciates my patience and is committed to resolving this situation.

What? OMG, my website's down?!? Holy shi... oh, never mind, I moved it to another host and it's running just fine. It's still down at C I Host though, so I still don't have the last month's updates.

The call came from 866-557-0643. That is not a C I Host number, and it is not answered as C I Host when called.

3/25/2004 16:53 (T+ 137h 53m)
The CEO of C I Host, Christopher Faulkner, hosts a "CEO Chat" every Thursday between 3pm and 6pm. I stuck around for 20 minutes but he never showed. Imagine that.

3/25/2004 20:18 (T+ 141h 18m)

...Today, most of our servers have been rebuilt from the ground up. The vast majority of our customers' Web sites have been restored to full operation...

...Our goal is to have all shared hosting, Windows clients (minus the 7 servers that are fully destroyed) fully online and functional by midnight, March 25. The last "Magic 7" will be back online before Sunday March 28, 2004 at the very latest. Those servers were so badly damaged all new hardware had to be pulled from inventory and built completely from scratch...

Christopher Faulkner, CEO, C I Host

I may post the full text of the email at a later date. Update: I've posted the full email here

3/25/2004 23:00 (T+ 144h)
I had three chat sessions with Tech Support over a 49 minute period (their chat server kept crashing).'Rachel_M' put my database online for me (it wasn't working immediately prior to chatting with her). 'Jenn_P' told me that the websites had been restored, but access to them would not be enabled until sometime tomorrow (Friday). I used the downtime during the chats to transfer my database from the C I Host to Atlantic.net, the new host.

3/26/2004 14:31 (T+ 159h 31m)
Spoke with JohnB via a chat (initiated at 10:50am, but I had the window in the background.) Minutes after giving him my site name and IP address it was working again and I could access it using FTP. It's almost as if they're moving the complainers to the head of the line. Least they could do after 6 1/2 days of downtime.

3/26/2004 14:55
I've retrieved and removed all kernow.com data from C I Host

3/26/2004 15:26
Account cancellation submitted and refund for remaining 4 months requested. C I Host responded: "We are sorry to see you leave. Your account cancellation is now being processed. However, your account will NOT be cancelled until you receive an email or telephone call confirmation back from a Customer Relations employee. The turnaround time on a cancellation is approximately 48 hours." Great, more waiting.

3/29/2004 12:52
I received the following email from C I Host. I guess they haven't cancelled my account yet.

To Our Valued Customers:

The battle to restore your sites continued for the entire 48 hours of the weekend, around the clock, with an expanded staff.

Many of you have expressed gratitude and there have been many complaints as well. We take both very seriously.

Both "camps" have asked for a better understanding of exactly what the rebuilding process entails. While we continue working to get our customers' sites back up, this checklist will give you an idea of part of the process. Typically, these actions would be a gradual process, over weeks or even months during the life and updates of a single site. In this crisis, these installations and repairs are running concurrently when possible ... multiplied by the number of sites adversely affected by the worm attack. By working around the clock over the last 7 days our NOC Engineers have completed the following steps on EVERY shared server affected by the worm:

• Total install: new hard drives
• Total install: operating system
• "Harden" operating system
• Install Microsoft patches
• Install IIS Web server
• Install IMAIL
• Configure users
• Install FTP users an
• Configure passwords
• Install content from tape
• Sort, install information from hard drives damaged by Witty
• Map FTP users to INETPUB content folders
• Install Perl
• Install PHP
• Install Cold Fusion
• Install SSL
• Install Miva Engine
• Install Miva Merchant
• Install Miva Order
• Install NetShield
• Set user permissions
• Install FrontPage extensions
• Configure FrontPage for each and every domain
• Install ASP Objects
• Perform clean up on file system and box
• Determine if any domains are missing from server
• Pull information from tape, if original drive restored

Gina Sanchez, Director of Customer Service, C I Host

Just think, all of this could have been prevented by installing a solid firewall between the Internet and their Windows hosting servers (Web and SQL).

04/02/2004 11:30
I've called C I Host's Billing department three times this morning attempting to get written confirmation that my account has been closed. C I Host's policies state that no account is closed until they've sent an email or fax confirmation. I was hung up on all three times.

04/02/2004 19:38
I received the following email from C I Host.

Subject: Following up on your shared windows account - want to discuss credit

Dear Windows Customer:

I wanted to follow up with you regarding your Windows Shared hosting account us.

I have you on my list of email addresses to contact back to discuss downtime credit for your account. I want to restore your faith in our hosting service and make sure that every portion of your hosting account is up and working to your complete satisfaction.

For the down time - even though we all understand it was beyond our control - we will offer you a generous contract extensions and account credit to help you as your budget absorbs the effects. In addition, I want to offer you free account accounts and upgrades for as long as you remain a customer. You can choose any number of account addons and we would more than happy to setup those for free and waive all fees associated with them. This includes SQL databases which are a $25/month value alone! We do this as a gesture of our commitment and our desire to continue to serve you and your businesses.

When you get a moment, and I know you have been busy getting through all of this, please email me back so we can open a dialog of discussion on your account credits and get you taken care of. I am available at ginas@supportteam.net.

I will do whatever it takes to build or continue long-term relationships.

Thank you for your business.

Gina Sanchez, Director of Customer Service, C I Host

04/04/2004 21:28
My reply to Gina's email (her quoted comments are in emphasized below):

I want to restore your faith in our hosting service and make sure that every portion of your hosting account is up and working to your complete satisfaction.

Witty was the 14th outage over two hours in length that C I Host has experienced in the last 20 months. There is nothing you can do to restore my faith in your company.

For the down time - even though we all understand it was beyond our control

No. Your boss may tell you that, but the fact remains this outage was very preventable. Any network professional could have configured your routers to prevent this attach. I'd be happy to give you a few names if you like.

When you get a moment, and I know you have been busy getting through all of this, please email me back so we can open a dialog of discussion on your account credits and get you taken care of.

Why does your billing department hang up on me every time I call to request written verification that my account has been cancelled?

When will I receive a refund on the unused portion of my pre-paid hosting?

04/05/2004 09:44
I think I've just received Gina's final answer:

C I HOST
"Your E-Business Solution."
------------------------------------------------------
C I HOST
1851 Central Drive #110
Bedford, Texas 76021
USA
(888) 868-9931 - USA
(817) 868-6999 - OUTSIDE USA
(888) 242-7554 - FAX USA
(817) 485-6119 - FAX OUTSIDE USA
------------------------------------------------------
YOUR ACCOUNT CANCELLATION IS COMPLETE
------------------------------------------------------

Domain: kernow.com

Additional Comments:

SPECIAL OFFER: Should you ever want to return to C I Host and host this account with us again, we will waive the setup fees and set you back up instantly!

Additionally, if you have any other domains you want to move over from a competitor, we will waive the first month's hosting charges on EVERY domain you transfer.

We already miss you :-(

C I Host

C I Host Cancellation Department

Well, that cancellation only took 234h 18m.