Hi everyone!

We've been burning the midnight oil (well, weekend oil) to get to the bottom of the recent crash on World 1 that caused a rollback of character progress.

The good news is that we've found the problem(s)!

The bad news is that we won't have a fix ready for you until Tuesday.

We will definitely be compensating everyone that's affected by this problem. Thankfully, we have a log of who lost data (although we don't know what you lost!). As we deploy this fix, we will also be giving substantial gold and king's coin awards to those affected.

I know this doesn't give back what was lost, but it's the least we can do when we've lost your data! Shame on us!

Log out to save critical progress! The system that saves your character when you log out works perfectly. So, you can use that to save any critical progress that you've made. Characters that were made using Build 91 should log out to save progress. Characters that were made before Build 91 still save properly every 15 minutes or when a level up occurs.

steve

Quote:

---

There are actually two problems here:

Periodic Saves Fail on Newly Made Characters

The server is designed to save your character often. It's supposed to save your character whenever you level up, every fifteen minutes of game play and when you log out.

Characters created after the release of Build 91 no longer save on level up or the 15 minute timer. They only save on log out. We didn't catch this in our testing. We will adjust our test plan to cover this system more carefully in the future.

Crash-time Saving May Be Incomplete

In the unlikely event of a server crash, the game will save every character to the central database. This is the final method that saves your character progress. This system has been found to fail saving all characters in some cases.

Our architecture includes a central server where we store all characters. It's inventively called the DBCache Server. Each game server connects to the cache server over the internet and uses that network connection to save and load characters.

Well, on crashing, a lot of character data has to be sent over that network connection. A subtle change in our socket code exposed a bug in the system that detects when all character data were saved. As a result, the saving of characters on crash may stop at about 50% completion.

This problem bypassed our testing because we don't crash the server as part of our test plan. We will begin doing so under load to ensure that this system works properly before every release.

---

Just to make it superclear: everyone affected will receive compensation, including those who experienced this problem prior to the Sept 1st incident. Thanks for being patient with us

The hotfix is currently running through its paces in QA. It looks to be given the all-clear for deployment to live shortly. I'll post further downtime information as soon as I have it.

Each affected player will receive: 100,000 gold and 40 King's Coins for their loss. This should be enough to get you back up to speed (minus the levels).

This hotfix will also resolve the issues with periodic saves and crash-time saving.

steve