Welcome to the servers behind series. In this series we will be looking at how some of the largest persistent universes. In this first article we will be looking at the EVE cluster known as Tranquility.
What is Tranquility?
Tranquility is the beating heart of EVE Online. A single shard/Server with a large amount of nodes doing all sorts of jobs from tracking chat channels to selling.
Tranquility is known as a super computer due to the factors of having a huge amount of readily available nodes. These “nodes” are running on hardware provided by IBM (The Business Division which has been taken over by Lenovo). The beauty of the setup provided by CCP Games is it’s ability to change the servers to different job roles/types on the fly. As CCP Explorer said in a recent tweet “We think of TQ as a ‘single cluster of nodes in an ecosystem of services’ including login, store, voice etc.” He went on to say “The cluster proper handles solarsystem, market, chat, inventory, etc.” Overall it seems since the previous PC Gamer article there has been more cores added to the super computer cluster.
Hardware Vendor: IBM
Storage Capacity: 4.7TB
CPU Cores: 294 (plus 1 core for developer workloads.)
CPU GHZ: 2,574
Network Ports: 8gb/s Fiber Chan x 4
Transfer Speed 32gb/s Local SAN
CPU Type: x5260 @ 3.33 GH
Main “Workers”: HS-21 x5260 @ 3.33 GHz blades, dual-CPU / dual-core 4 cores
Disk Storage system: Storwize v700 (Running Enterprise SSDs since 2007)
DB Cluster Capacity: 200,000 IOPS capacity
The server system is housed in the UK with it’s own cages rented via collocation. On the network level CCP Games uses Level3 as it’s global transit provider meaning that the network/server can be accessed faster with smaller hops. The server’s actual location is not known however we suspect the location is in “the city”/london telehouse which provides global connectivity.
Ordinarily a company would host servers at multiple geolocations however CCP Games decided a single shard would work in what could be classed as a “private cloud”. By having all the services running in a single cluster, By putting all the eggs in a single basket CCP has been able to create the largest gaming super computer cluster at the moment of this publication.
CCP decided to install SSDs back in 2007 before the technology was commercialized and required that the company reps had clearance to gain access to the contractors involved. The full commitment to pushing there systems to the absolute full capacity means that you have upto 2125 players in a single battle. EVE Online does use some cheats. For example Time Dilation helps provide a much more stable connection to the server while allowing the server to handle a multitude of requests in clusters. It’s well known that Jita (the main trading hub) has around 1,700 users connected for-filling orders, chatting to fellow capsulars and much more.
Instances by Design
When a user connects to EVE Online they are using multiple clusters for certain jobs E.g: Voice, Market, Inventory, Solar System.
When a user connects to a system they can navigate around the Universe clicking on certain areas or landmarks to jump into a warp drive around the world. finally if the user switches to a different system, They do so by going thought a star gate. This time is used to process the requests over to a new cluster. This ensures that the solar systems can work with all types of traffic flows with a single CPU to optimize upon. It’s one huge cluster with a multitude of sub-nodes controlling every aspect of the universe.
In 2013 the Dust 514 server merged with the Tranquility shard. This means the developers can focus more on a single setup for instances in a private cloud instance interconnected with the EVE Universe without having to bridge accross multiple datacenters and setups. This simplification will also help with EVE Online’s upcoming project Legion which will be using the technology gained from EVE Online, Dust 514 and EVE Valkyrie
EVE Online has a daily downtime of around (depending on deployments this can vary from 4-8 minutes upto around 30 minutes), this is to ensure anything that needs replacing in the cluster is done so during one of these outages however outages have been known to be increased due to software and hardware bugs.
CCP is currently in the stages of replacing some of the server side hardware, It seems that some of the main servers are using processes from 2007.
This will be coming we will be covering in a future article.
On a side note
There are two other servers apart from Tranquality
-Serenity Asia Server uses similar version of Tranquility’s Hardware
-Singularity Test server (to deploy test updates/patches which the community report back on)
Thanks for this articles which helped inspire us to create this series
Many Thanks to CCP Explorer for providing additional input on the specs.