VPSVille – When hosting goes wrong
I have a funny story regarding a VPS hosting company called VPSVille, actually it’s not a funny story… it’s pretty annoying.
I changed my hosting provider when I moved over to Canada to a company called VPSVille:
I was originally with a great little company called Bytemark in the UK:
but wanted to move because I wanted to a VPS this side of the “pond”. VPSVille looked to be good from what I saw of their website and reviews about them. This is what VPSVille have on their front page:
“Welcome to VPSVille, one of the most advanced VPS service providers on the internet. Virtual Private Servers (VPS) are the future of web hosting. A VPS performs and executes exactly like a stand-alone server; Your VPS can be rebooted independently and have root access, users, IP addresses, memory, processes, files, applications, system libraries and configuration files. This website is being hosted on one of our VPS’s.”
I was a little concerned about the lack of automated backups – in particular seeing as I’d previously had a backup server with Bytemark. The VPSVille FAQ stated though:
“We perform regular backups of all our servers and also allow for all users to make their own backups from our custom control panel. You can choose when to backup your server and when to restore it.”
So all good, First impressions were pretty good, cheap, lots of bandwidth for what I wanted, full root access and an installation of Debian Etch, Centos, Fedora whichever I wanted. I migrated all my websites over along with my MySQL database onto a fresh new Centos install. I hosted Subversion and Mercurial there, used it to backup my thesis etc…
Gargh
This was back in May, all was good for the next six months or so… until two days ago (Tuesday 7th October to be exact).
All my websites went down that evening…
I figured it was a glitch…
They didn’t come back up the next morning…
I figured something was drastically wrong so I posted a support ticket. No reply, in fact an hour later the entire support portal was removed from their website along with the user forum. I decided to have a look to see if anybody else was having similar problem finding this thread on webhostingtalk.com:
http://www.webhostingtalk.com/showthread.php?t=727603
I was in the same position as a lot of other people. It would appear that VPSVille had pulled the plug on their customers.
VPSVille Announcement
A while later the following announcement appeared on their website, not on the front page I might add, and in very broken English without the use of a spell checker:
“On October the 7th we exeprienced what looks like total failure of some storage devices. This has resulted in data loss for many customers that have been migrated to these new systems. Data recovery is not possible, and we are unable to allocate space for these customers on other nodes. This is a terrible situation and if we could salvage the data in some way we would, but unfortunately its not possible. This data is lost permanently. We also lack the capacity to move affected customers to other nodes.
We are working hard to add more capacity but it will be some time until its online. We are therefore forced to cancel these accounts and starting next week will be refunding all affected customers. We have also suspended all new sign-ups until this situation is resolved and VPSVille returns to normal operations.
Due to the volume of support requests we are unable to respond to affected users individually at this time. We will be in contact with you in the future. Your patience in this regard is appreciated”
so in short we were all pretty screwed and very very annoyed.
The Response
A while later vpsville posted an entry on the forum:
“Hello, we have an announcement on our site now explaining our predicament.
Unfortunately, the data that was lost is permanently gone, and we can’t recover it. We will be canceling all affected accounts and refunding users.
The reason for the outage isn’t power related as we at first thought, it turns out there was a malicious attack on our servers aided by an internal leak. We won’t be accepting new customers until we rebuild all of our servers and templates to ensure they were not tampered with.
This was devastating blow to our business and we understand that our valuable customers are also victims.
When we begin accepting new customers again, rest assured we will be better than ever, and more focused than ever on providing good value hosting for the Canadian market.”
It doesn’t really make things any better does it? As well as admitting that you didn’t have a sufficient redundant setup in place you’re now telling us that you don’t have the required security measures in place. In short I lost six websites I’d been working on for two years, a whole Subversion repository (which contained months worth of my thesis work) and lots of other important documents – thanks vpsville!
Moral of the Story
I would advise a number of things here when choosing a Virtual Private Server (VPS) Hosting Company:
1. Research your VPS hosting provider before you invest money, time and effort. Look at reviews on the internet. Look for well written reviews from people who clearly explain the issues they had with the hosting provider and what they did about them.
2. Choose an established company not amateurs who have purchased a dedicated server and are looking to divide it up in order to make some quick money.
3. Always, always, always backup your server – even if your VPS provider claims to do this. You can look at rsync as the simplest method, but there is also rsnapshot or rdiff-backup.
4. Never ever host with VPSVille, full bloody stop.
Update From VPSville (Thanksgiving – ironically)
I had a fairly decent update from VPSville, after a week and a half:
“We here at VPSVille have had a few days of frantic phone calls, hair-pulling, yelling, crying, screaming, sector-recovery tools and angry accusations. We’ve also looked closely at our logs and have pieced together what happened with our recent data loss incident.
Initially we thought it might be a case of gross negligence and some harsh words and harsher threats were exchanged, but we now believe it was just an honest, stupid mistake. Inexperience sunk the ship.
One of our new employee’s made some crucial errors regarding security, and this was compounded by one of our techs travelling abroad and requiring some relaxation in our usually
strict access policy. One firewall in front of another and both temporarily disabled = no firewall at all.
This unfortunately resulted in some malicious SOB with no real life to waltz into an almost unprotected segment of the network through a newly installed appliance with a default password
and start deleting things. Several servers actually and a NAS unit assigned to back them up. One of the servers was our control node, which interfaces between the website and the other servers. Even our mail server was affected.
Some servers were running even after their files were deleted, through some strange quirk of Linux buffering that we still don’t fully understand.
Its easy to blame a hacker of course, but we do feel that an unfortunate series of events combined to make their hack particularly easy. Far to easy for a network of this magnitude and importance to so many people.
This looked like a total loss, with no way to recover the data, so we sent out a somewhat panicked email to affected users that their accounts would just be canceled and refunded. Shortly afterwords our outgoing email died.
This resulted in a torrent of email, most of it surprising friendly, and much of it understandably angry and dismayed, telling us they wished to remain customers even if their data was not recovered.
Happily we were able to restore many servers with sector recovery tools. In light of this we have not arbitrarily canceled any accounts.
All affected accounts are re-enabled and have been credited. If we were able to restore your data we have sent you an email about it (hardware nodes rapier, cutlass and spear).
We sincerely express our condolences to affected customers for this unfortunate series of events, nobody should lose data at VPSVille, its an awful thing and we feel your pain.
Things do happen in life, but this won’t happen again.
- Staff”
So, just to add salt to the wound they admitted poor infrastructure, poor security measures, giving junior employees work to do without providing sufficient training and a complete lack of communication on the issue. They also claimed they sent the server disks away for recovery. In fact what they did was to attempt to recover the data using their own tools.
The final reply I got from them on their forums stated this:
“We used several tools, including one given to us by adaptec support. None of the tools were able get the files on Katana. The tools let you view the bytes on the drive in various ways. Its quite interesting actually, if you’d like to know how sector recovery works read up on it or try it yourself.
You need to understand that interrogating us will not get your data back. Your data stored on Katana is gone. We tried our best.”
to which I responded:
“You said you sent the drives off for sector recovery – that’s not quite the same as using some tool you found via Google.”
I understand that interrogating will not get my data back…
I was just trying to ascertain exactly what measures you went to in order to recover the data.
I’m still incredibly annoyed about the lack of communication here, the explanations, the complete lack of professionalism you’ve demonstrated.
You’re obviously not willing to go to any further measures to recover the data. I’d like a refund on my complete hosting charges with you, that amounts to $135.00 in total which is nowhere near how much it will cost me to reimburse the clients I have and the data they have lost.”
I didn’t get a reply….
and most recently:
“I noticed you haven’t applied for a refund, yet you seem very angry. I’m angry about losing data too, but I blame the people that deleted it, not the people who spent days trying to recover it, at no charge. Please submit your transaction details to refunds@vpsville.ca and I’ll expedite the process for you.”
I had sent my refund, but due to their normal unorganized ways they’d obviously missed it – doh.
You have been warned about hosting with VPSVille.