On the last day of my recent vacation, I noticed a bunch of errors from the site about the database being unreachable. I thought this was unusual, but it seemed to fix itself so I decided to look into it when I got back.

Sure enough, the problem persisted, but for short bursts. Every 12-24 hours or so the database would crash, then restart itself and continue serving requests fine. What I didn’t notice was that it was also causing our nightly backups to fail! This morning I noticed it while going through the logs and decided to fix it ASAP.

Unfortunately, as I delved into it deeper, I discovered that two tables in the database had become corrupt. I’d always heard of MySQL corrupting data, but never had the pleasure of experiencing it until now. I’m not sure what caused it — I probably never will — but I set out to fix it.

Every solution I googled for failed, which sucked. So I was forced to restore the two missing tables from the last successful backup of the site, which is about 2 days old.

What this means is that you may be missing tubmails or upgrades to your characters if they came in the last two days. I’m awfully sorry about this, but I did my best to recover the data and couldn’t.

If upgrades are missing, a simple respec will fix the problem by visiting Haxlrose. If you notice one or more of your stats is negative, try this page and it’ll fix you right up.

Finally, as our way of saying sorry for the downtime, I’ve given 3 extra visits to all players. Enjoy them!

– The Evilest of Trouts