EVE-KILL is arguably the largest killboard in EVE, and we daily serve several thousand killmail hungry pilots, who both want to see who their friends and enemies killed while they were away.
We are currently hosting 4663 active killboards (Pilot: 1599 Corp: 2579 Alliance: 485) and 14.515.048 killmails.
Lately however, this community of murderers have been disturbed, and downright plagued by terrible luck, and downright shitty hardware.
The reason behind the downtimes we have had as of late, has been multiple, it started out with a hard drive dying, and a RAID controller failing to use the mirror drive, which it should have.
Then became a tale of the worlds slowest hard drive replacement, and finally a success story, after Beansman spent hours trying to figure out how to turn the working mirror, into a single drive, that would mount.
All the while this was happening, we was busy with smacking ourselves in the head over the fact our backup scripts had failed about two weeks before.
Luckily it all came back up, without any data lost, but the downtime had already cost us a weekend.
However, the bad luck didn’t end there, no, not even close. Now the controller decided to start a new war, something it liked to call “The timeout war”. Simply put, at a random time it will suddenly stop responding to the requests of the operating system, few moments later the operating system dumps the mounted array, and bam, all virtual machines grind to a halt.
At which point we have to rescan the system to get it to remount the drives, so the virtual machines returns to a functional state. The problem however, is that this requires direct intervention, and can’t easily be scripted away.
So this leaves us with three options.
First option is living with it, and staying where we are for another 2-3 months till we scrape together enough money to buy our own hardware.
Second option is switching to a different host, we already have one lined up which has made us a great offer for a server, but we will be bound to them for the next 12 months, which isn’t ideal.
Third option is to do a donation drive, and hope we can get the money that way.
We obviously picked the third option.
This said, we are not gonna put the burden of funding this entirely on you, our users – we are throwing all the money we have saved, plus money from our own pockets into it. Which nets us just shy of 3000 USD, sadly however, the server cost is upwards of 8000 USD, depending on the hardware we pick. However, for now we have set the donation goal at 4000 USD, should we blow past that, then we’re golden in terms of all the extras we are considering.
Now you might have some questions.
But why would you host it yourself, when you can get others to do the work for you?
Both for the fun of it, but also so we can better control what can and can’t fail. Currently we are relying too much on RAID to protect us, since we are limited in terms of amount of bandwidth we have available, but also in terms of the amount of drives we can throw at the system. With us owning and running the hardware, we won’t have any such problems. We can add many layers of redundancy, such as running RAID 6 (two drives can fail without compromising the array), hot spares, mirror RAID 6 and so forth. There are a myriad of possibilities with us owning the hardware.
Ok, but what kind of hardware do you want then? with it costing upwards of 8000 USD in total, it must be some pretty bleeep beefy shit!
You are completely right, it is going to be some beefy shit. Right now we are thinking of dual socket 2011, meaning Intel Xeon E5’s.
To be more specific, we are thinking of E5 2620s (6 cores at 2 GHz each, with HT, VT-d, VT-x etc.)
Two of those together with 128 GB memory (or more), +4 Seagate Constellation ES and +4 Intel 520 SSDs (or possibly their SLC variant) – i will drop a comprehensive list at the bottom with all the information you’ll want.
Sounds like nice hardware, but what about the hosting?
Beansman has talked with a good friend of his, who has graciously allowed us to setup hosting at his company for a fair price, only downside is we have to provide our own UPS incase the power goes. But in terms of bandwidth, we will have plenty, and peering is also excellent (2 jumps from the DIX (Danish Internet Exchange) who peers directly with AMS IX (Amsterdam Internet Exchange) which in turn peers with everyone else in the world. That said, DIX peers with a lot of others too, you can check all of that out on peerdb)
We will get all the space we want, so we can expand as much as we want (several +40 disk SANs ? boom, done..)
Wait, non-professional hosting? NOPE!
We had the same thought, that we would only go with professional hosting centers doing co-location. But then we saw the prices for co-lo in Denmark near Copenhagen.
Then we changed our mind, and decided that whatever benefit a hosting center would provide, we could do ourselves.
Granted, diesel generator is probably out of our reach, unless we suddenly get a lot of money. But UPS and proper cooling is perfectly doable.
This said, if at any point hosting becomes terrible, we always have the option of packing our things, and going somewhere else. Seeing as we own the hardware!
But, what will you do with the spare computing power? A simple killboard doesn’t eat all that.
The spare computing power, we give away to other 3rd parties who are in the same situation as ourselves. We have already given away virtualized hardware, to EVEFit and Kingboard – whom both enjoy free hosting (albeit, spotty the past few weeks).
And we have thought about continuing doing this, and even expanding on it by getting more hardware as time goes.
This could be for that guy who has a great idea to do a public tool telling everyone what you ate for dinner, or. maybe not. Regardless, it would be almost entirely free. (By that i mean, you’d have to say where you got hosting at the very least)
Now, you might have more questions than these, but these should at least cover the majority, if there are more, feel free to ask below – and we’ll try and answer asap.
Lastly there isn’t much else to say, other than i hope you’ll decide to donate us some of your hard earned money, so we can continue doing what we do best (or worst, if you’re a hater).
And actually deliver a reliable product that we can be proud of.
Hardware
CPU: Xeon E5-2620 (6 core at 2GHz each)
Memory: 128 GB DDR3 Registered ECC
HDD: Seagate Constellation ES 1TB drives
SSD: Intel SSD 520 180GB drives
And then we are looking at two possibilities for rack
Asus RS720-X7/RS8 and Dell R720
Both will be upgraded with PIKE/PERC RAID controllers, and in the case of the Asus server, iKVM module for remote management.
And we will of course post images if the donation is successful. Every nerd loves hardware, esp. beefy hardware!
If you have any other questions, feel free to drop by
Mentions:
Comments (6)
Niels Brinkø said:
As a note, i have added $79 from earlier donations.
Edit Comment
Robert Böhm said:
why dont just put google add on your page and get rich like every one else… i just dont get it.
Edit Comment
Alex Smith said:
@Robert: They already have ad’s on the site… Might want to whitelist them in your ad-blocker.
Anyway: as a suggestion, take a look at the Supermicro gear. I use them for a number of larger projects and they are really really nice to use. If your going to build them yourself you’d get much more bang for your buck using their super-server setups.
Not to mention the KVMoIP is much better than the Dell DRAC (and a lot cheaper than the Dell one that allows Virtual Media…). For example, with the basic spec’s you listed take a glance at something like the 6027R-TDARF then add CPU, Memory, Disks :)
Also, instead of worrying about bandwidth costs to backup stuff – use rSync.
Edit Comment
Niels Brinkø said:
@Alex: Yeh, the list is more of a “suggestions” to get an idea of the type of hardware we’re going after.
@Robert: if only it made us rich. It pays the servers and a bit more (which we’re saving up for times like this).
Edit Comment
@atonewell said:
Have you considered hosting on something like Amazon EC2 instead?
If you’re spending 8k on hardware, to allow for catastrophic failures you should probably double your budget.
Reserved instances on Amazon EC2 might work out more cost effective in the longer term.
Edit Comment
Niels Brinkø said:
@atonewell: Yes, we have considered “cloud” hosting over and over again, but the budget simply doesn’t have room for it. Cloud is expensive when you have a VERY slow database that uses this much memory.
/Niels
Edit Comment
You must be signed in to make a comment.