We're Back! So What Happened?

Hi everyone,

You may have noticed that Wynncraft took a nap last night:



So what happened?

A number of things went super wrong to allow Wynncraft to stay down for so long.

1. Our storage servers failed, and that failure cascaded

Wynncraft has a lot of data that we need to store. We have over 20 dedicated servers that all work together to provide over 50 terabytes (50,000 GB) of storage that we can use to store player data, logs, etc. These servers use special software to provide storage in a redundant way. This means that even if a server catches on fire, other servers will automatically recover any lost data.

Last night at around midnight EST, one of our services crashed and began to use exponentially more storage than usual. This service usually uses around 60GB of storage, but as of midnight last night, it was using over 1000 GB of storage. Because it was using so much storage, certain servers began to run out of disk space. Our special storage software recognized that those servers were running out of space, so it automatically decided to move the large files to different servers, which quickly also ran out of disk space since the files were so large (and growing). This cascaded until all of our storage servers were out of space, which caused our network to crash.

2. Our monitoring systems crashed, so it did not pick up those failures
We have a sophisticated system of monitoring which should prevent issues like this. Once storage servers began to fail, it should have alerted our monitoring system, which would automatically alert (slash wake up) admins and let us resolve the issue.

Unfortunately, because this issue was related to our storage servers, our monitoring servers couldn't store any telemetry, so our monitoring systems crashed as well. This meant that we had no systems telling us that anything was wrong.

3. Our servers did not fail gracefully
While all of this was going on, our game servers tried to stay up even though the storage servers were down. For around 2-3 hours, a small number of players were still able to log in to our network, but the game servers had nowhere to store data. Regretfully, this caused a small number of players to lose 2-3 hours of progress.

How we are preventing future issues

1. Upgrading our storage servers and adding fail-safes
In the next few days, we'll be upgrading our storage servers and adding new guards to ensure that no single service can use so much data. This will hopefully prevent issues from cascading and taking down our entire storage cluster.

2. Adding a watchdog to our monitoring systems
When our monitoring systems crashed, there was nothing out there that could alert admins that something is wrong. In fact, according to our monitoring system, everything was handy dandy all night.

Starting today, we'll be adding something called a "Watchdog" alert which will always be firing (ie always telling us that an issue is ongoing). This alert will serve as a dead man's switch. If it ever stops alerting, it will be treated as if the network went down and automatically wake up the appropriate people.

3. We're working on making our game servers fail more gracefully
In the next few weeks, we'll be working on systems to make our game servers fail more gracefully. This will make it so that when something is wrong with the network which will impact player experience (ie data not saving), game servers automatically turn off (instead of trying to stay up in a broken state), which will hopefully prevent any future rollbacks.

We're very, very sorry!

We strive to have 100% uptime and provide a stable and reliable experience for yall. We know that you are investing a lot into Wynncraft, and we are absolutely devastated that this issue had such a large impact. We will be looking for ways in the next few days to recover any time lost and to make up for any lost time on the network.

 

Worth noting that I literally had to DM a mod on the Wynncraft Discord that it was happening, alongside someone breaking the rules, because the Discord server was on fire while this was going on.

I'm guessing the alarm for the watchdog alert will play Yahya's theme, but bass-boosted and deep fried to unrecognizability?

It goes without saying that we are definitely grateful for having a great admin team. Kudos to the entire team!

 

It was a service we use to save local data snapshots that allow for rapid data rollbacks when necessary. Still investigating why it crashed.