Before I begin, I want to be completely transparant with this post and let you know the full details. We hope that you can understand the severity of this issue and stay with us as we get through this.
This is a post I was hoping to never have to write and is difficult for me to do so. Honestly, it is just embarrassing, but please read the entire post. Around 5:30 cst this evening our server crashed. We have all experienced these and commonly the server would just restart. This time it didn’t.
Once this crash was brought to my attention I quickly wrote our server host about this issue and they responded with this.
We have some good news and some bad news for you, and I’m not going to try to “sugar coat” it and make it sound better than it really is. We’re not one of those companies.
Unfortunately, the node you are on (USA Node 1) had a catastrophic Multicraft control panel malfunction that caused it to irrecoverably delete all customer data on that node.
There’s 13 other customers in the same place as you right now.
Because of this, we’re offering 2 free months of service, if you choose to stay with us.
If you choose to not stay with us, I completely understand.
We’re implementing a backup system very soon to prevent this from ever happening again.
I am sure that your reactions were about the same as mine when I read that. From this email it appears that ALL of our data from the last 9 months is gone.
Out of anger and frustration, I responded with this message:
Let me begin this email to say that I am royally pissed.
Okay. How in the world can you not have a backup system for customer files? This is completely irresponsible for a hosting company, as it should be basic systems installed.
We have well over 100 members spending hours working on projects that they have invested time into and are proud of the work. How am I supposed to tell them that all that is just GONE??
I would be considering changing hosts, but at this point I can as I have just recently spent all the work changing over. I may have to if this is the kind of hosting company you guys are! Two months free is it for losing nearly 9 months of work?!
Is all the data corrupted? Is there anything that can be recovered? I have finals this week and have no time to deal with fixing this, especially since we have many users who have donated and expect good uptime. I would be contacting Multicraft and finding out what happened, letting them know of this.
Please let me know more information soon so I can piss off all of our members.
We can not say sorry enough as a staff about this, and I personally wished this would never happen. I know there is nothing I can say to make to make this situation better.
Now, here is where we will pray we have your help with this. At this time, our staff has not made a decision as to how we will move forward with this, but we do want your input. I can tell you that we do have an entire server backup from March 16, 2012. This would lose about 1 months worth of work, and we know that that time was well spent. We want your opinion on what you think we should do to move forward. I will be moderating the comments in this post closely, so PLEASE DO NOT POST WITH MASSIVE ANGER. We want you to post your concerns about this situation and let us know your suggestions. Mainly we have two choices.
- Upload backup from March 16, 2012 and begin reconstruction.
- Start fresh with an entirely new world.
My guess would be that the 2nd of these may take more time, but some may want to do that.
Also, for donators. I know you have personally invested funds into this server, as you know it does cost money to run this. If you want a refund, please email me and we will discuss that. (email@example.com) I can not promise a full refund, but I will do what I can to get a good majority of it. (this depends on when you donated) I can promise you that we will keep this server running though and with whatever decision we make, we will be up and running (hopefully) within a week or so. (could be much sooner) Uploading from a backup would be a quicker turn around time, but some may want to consider a new map.
Either way, our staff will make the final decision and will be letting you know the decision we make.
We will keep you up to date on this situation here. Pray that somehow our data can be recovered from the server malfunction.
Yes, we understand completely as we all come from hosting popular Minecraft servers ourselves.
We had already begun working on a new control panel and backup system before this ever happened in an effort to make sure it never did happen, but obviously we weren’t able to get it done fast enough.
We have already spoken with the creator of Multicraft and he’s aware of the issue and assured us that in the latest version, 1.6, which has just recently been released, the problem is fixed.
We had already made plans for upgrading to 1.6 tonight or tomorrow night, so the possibility of this ever happening again is literally nil.
In addition, as Devin has said, we are currently getting a remote backup system in place which again, we had already planned to do before this happened.
As for the data being recovered, we’re working on it right now but more than likely, we won’t be able to get much if anything back.
When files are deleted on linux, it’s pretty difficult, if not impossible, to get them back, especially hours after it has happened.
Also, if you have that many active members and have put that amount of work into your server, you could have created/downloaded backups of your map at any time.
I’m sure your loyal members will understand and stick with you; there’s simply nothing more we can do at this point other than apologize, give you free service, and keep trying to recover the data.
If you are deeply concerned about this happening again and decide to stay with us, simply wait until we roll out our central backup service which should be out within a week.
Update 10:15pm cst:
There have been several messages between our host and myself. I can say that the work they are putting into this situation is phonomal and I am happy with how they are treating this situation. I will let you know of any major developments that arise in our conversations. I know that their owner will hopefully be calling me here within an hour or so with a personal update on this situation.
Update 11:59pm cst
I just got off the phone with our host‚??s owner. Here are a few notes.
-Attempted recovery has not begun as they are waiting for things from the data center.
-He would be able to set us up a temporary server for the time being on another node to allow us to reconfigure things. (may be good if we restart fresh and I have to redo all our plugins). Thankfully I have those backed up from 3/15.
-He really wants to help us out as he is a crafter himself. This guy knows his stuff and knows how big if a problem this is.
-There were a few other things but am tired an cant remember. Those are the more important notes.
Again I will keep you up to date here on what the latest is.