5 Day Nightmare
By Vic on Oct 2, 2009 in Blogger Unleashed
As most of you by now noticed BU Blogging Zoom and my other main large websites where down for 5 days. Let m e say it has been a real nightmare it seems Murphy has become a prick in the past few months and maybe it is time for me to always think of what Murphy might do and anticipate it. We just moved our main sites to probably one of the biggest Hosting companies in the world PEER1 as they scale up veyr easy. We have two specific projects that we look forward to having huge traffic as such we needed the space to grow and grow quick after much research we decided to go tih the company that manages YouTube we figured if they do YouTube why not us.
Here comes the nightmare. On order processing I had in front of me a detailed server order form with all the details my account manager and me decided we needed it. In the actuall ordering online I missed one simple dropdown box but one of the most important ones the Raid configuration. Once the server was up and live my account manager noticed we had not selected the raid configuration and placed a call to me. In the conversation we agreed that we would probably in a week have the server taken ofline and do the hardware modifications to make sure we had our raid. Well that was almost two months ago I kept putting it on the side because it was not that big of a deal. Well Murphy came into play and we had a catastrophic hardrive failure putting the server offline now normally this would just be a few hour thing of having the server online while they did the full backup restore.
It turns out that if you are so fucking stupid that your backups are stored on the same hardrive and not off site this might create a severe problem.
Ending Result.
Blogger Unleashed lost post and code fixes.
Blogging Zoom is totally destroyed and I am waiting for a hardrive to be shipped from Puerto Rico that should ahve a full backup with a few months missing.
IQbizz is still beign restored and we will end up losing sales data for about 5 days meaning we will have to contact our customers to re-download there sales so we can get them in our system again.
TAA is fully destroyed and only have a backup sql file with about 2 weeks missing of data but we lost the full coding of the site.
My new personal blog VictorFranqui.com lost post and coding which we are currently fixing.
And here comes the kicker, I had moved about 350 money sites into the rack until I had time to switch them to reseller accounts in MidPhase all gone totally about 6k worth of my monthly income. Currently have one of my employees going site by site and downloading the cache from Google to rebuild the sites.
Folks can not stress enough how important it is to always have full current backups of all your sites and to never under estimate Murphy just cuz it has never happened does not mean it will not happen to you.
As sated before I will rarely post here anymore after many emails I decided to start posting on my personal site at VictorFranqui.com
Popularity: 3% [?]
Vic, Nice to see your sites slowly reappearing. So sorry you had to go through this. Wish I was a coding expert so I could be of some help.