Site Outage Friday 3rd February 2023

[FIRE]

Thanks

[skycafe]

Thanks for all the effort!

[Tourer]

Thanks for getting us all back on the air so quickly - 19 hours sounds like a damn fine benchmark!

[MagnetON]

I'm sure I can improve on that time.

[275 GTB-4]

Originally Posted by MagnetON

I'm sure I can improve on that time.

Its NOT a contest! DON'T even think about improving your time!!!!

[MagnetON]

Well, I do have to move everything back to the original hardware at some point...

[MagnetON]

I've been asked if there's anything can be done to guard against this happening again.

In this specific attack, yes. The underlying vulnerability (which was the hosting platform itself) has been patched.

However it's impossible to protect against everything as new exploits are found every day.

I have a robust backup strategy that lets me recover the site in around 12-16hrs to a new server with the loss of, at most, a day worth of posts. Going forward I hope to reduce the time to recover, and the data loss range. This strategy will be tested very shortly when there will be an outage to move the site back to the original hardware.

Tenths doesn't make any money and is entirely funded by myself. To provision the temporary server it's currently running on will cost around £100/month for 6 months (minimum contract) in addition to the normal £150/month hosting bill.

It's always possible to build in more resilience, have multiple servers on multiple locations, or move the whole lot to cloud based infrastructure such as AWS but the money isn't there.

Follow up question:
"What about putting advertising back?"
Well, at the point the advertising was removed it was generating about a tenth (ironically!) of the monthly hosting bill. I figured if I was paying 90% anyway then the extra wasn't going to bankrupt me and you folks could have an increasingly rare thing.... A popular website without adverts.

So unfortunately there may be occasionally outages when major things happen.

[Richard Casto]

Thanks for all the hard work. I think your strategy is the right one. Right level of cost vs. risk. You can't protect against everything and having a good recovery strategy allows you to recover from things that slip through the protection model.

Originally Posted by MagnetON

Follow up question:
"What about putting advertising back?"
Well, at the point the advertising was removed it was generating about a tenth (ironically!) of the monthly hosting bill. I figured if I was paying 90% anyway then the extra wasn't going to bankrupt me and you folks could have an increasingly rare thing.... A popular website without adverts.

Funny, I had been subscribing for years, so I didn't notice the advertisements had gone for everyone! While you might be able to self fund this, have you considered opening up something like Patreon for those who might just want to contribute even if they don't get anything "special" back in return. Then again, I guess if more wanted to do help fund the site, you would have been more than 10% funded on the subscription model!

Richard

[Mike Harte]

May I add my thanks to Grant for all his hard work getting us back online almost before my withdrawal symptoms had kicked in.

As a previous subscriber, I would be more than happy to carry on the £10.00 subscription, and I would like to think that many more of us keyboard warriors would do so also.

[Aysedasi]

Originally Posted by Mike Harte

As a previous subscriber, I would be more than happy to carry on the £10.00 subscription, and I would like to think that many more of us keyboard warriors would do so also.

Absolutely. I contributed for a number of years when I didn't actually have to and a tenner a year is an incredibly small price to pay for everything I get out of Tenths.

[broadrun96]

We need to start a tab at his local for MagnetOn

[MagnetON]

Just a quick note, servers were restarted to add additional security patches around 3:15pm today (Thursday)