Major Outage 10-July-2013

[MagnetON]

Today Tenths suffered its longest unplanned outage ever.

At the moment I've not attempted a root cause analysis as the focus has been on getting Tenths back and running.

At approximately 2:30pm we experienced a major server failure that could only be resolved by re-booting the server. This caused a large database corruption and also a software corruption.

Unfortunately as I was in a meeting for the "day job" (yes, Tenths is a hobby for me too!) and was unable to give the problem my full attention.

Data was restored from the backup replication server however re-starting the forum software also caused the backup to corrupt.

Using the replication server as the main database server ALSO caused the backup to corrupt.

At approximately 7pm the decision was made to abandon attempting to repair the primary server and switch to our secondary server in the US. This involved shortening the DNS cache time from 1 hour to 30 minutes in order to allow a quicker change over when the site was brought back online.

This migration work also involved checking and configuring the secondary server to act as "primary".

This work was completed around 8:30pm and a further hour was spent testing functionality from here at Ten-Tenths Towers!

DNS migration took place at approximately 10pm and by the time of this post we are seeing traffic levels increase as the DNS change propagates.

[johngee]

Having had to manage similar problems before I retired, I feel your pain! Well done

[NaBUru38]

Great job! I'm happy that we're back on track

[JimW]

Thanks to Grant and Terri. Sounds like your disaster planning has worked well.

Regards

Jim

[GORDON STREETER]

Come off it Grant we all know it's just a case off flicking a swich !!!

[Tim the Grey]

Nice to have you back.
Have you traced what was causing ALL the backups to fail???

[porman]

Thank you for all that you do. 10-Tenths is far and away the best motorsport forum.

[MagnetON]

Originally Posted by Tim the Grey

Nice to have you back.
Have you traced what was causing ALL the backups to fail???

Backups were all good, what was happening was that when a copy from the "replication" site (which is the live/realtime backup) was loaded on to the primary server it would be pretty instantly corrupted. The same data was fine on the replication server. Pointing the primary web server to use the replication server also caused the data to be corrupted, so it was at that point the decision was made to switch the whole site to run on the secondary server.