|
||||||||||
|
||||||||||
4 Feb 2023, 07:10 (Ref:4142577) | #1 | ||
Admin
Veteran
Join Date: Mar 2002
Posts: 12,063
|
Site Outage Friday 3rd February 2023
At approximately 10:40am on Friday morning the platform providing all the servers for TenTenths, along with other projects was subjected to a "ransomware" attack.
Hundreds of servers around the world have been impacted, not just Tenths. You can read more about it here. This attack effectively wiped out the Tenths web and database servers. Shortly after analysing the attack I commissioned new hardware for Tenths and started the process of re-creating the servers. We have a very robust backup / disaster recovery plan for Tenths with multiple generations of backup stored off-site from our hosting provider (including a 20Tb NAS in my attic!) It takes a bit of time to create new database and web servers, once these are created the backups have to be uploaded to them, the database restored, and all the rest of the web server files put back in to place. It's time consuming because there's a fair bit of data to ship around. From experience it takes around 24 hours to get everything moved and done. In this case we returned the site to functional in around 19! There's more work to be done, but hopefully it won't involve any long interruptions. |
||
|
4 Feb 2023, 07:55 (Ref:4142581) | #2 | ||
Veteran
Join Date: Feb 2010
Posts: 6,538
|
Wondered what the issue was when regardless of browser or device I couldnt access, but knew the return to life was in good hands and would happen as soon as it could.
Thanks "Magneton" and any others involved for your effort and time involved returning access and seemingly normal navigation to all us site users. Much appreciated. |
||
|
4 Feb 2023, 07:56 (Ref:4142582) | #3 | ||
Veteran
Join Date: Aug 2007
Posts: 4,479
|
Thank you!
|
||
|
4 Feb 2023, 07:56 (Ref:4142583) | #4 | ||
Racer
Join Date: Aug 2003
Posts: 110
|
Excellent work getting back online so quickly!
|
||
|
4 Feb 2023, 08:18 (Ref:4142588) | #5 | ||
Veteran
Join Date: Nov 2007
Posts: 2,197
|
Well done to all concerned in restoring service - obviously a well drilled back-up plan. Great work.
|
||
__________________
Gary Clarke |
4 Feb 2023, 09:17 (Ref:4142604) | #6 | |
Veteran
Join Date: Jul 2013
Posts: 18,795
|
I knew something was up when I came on here just now and found a lot of my posts (and other people's) had disappeared.
|
|
__________________
He who dares wins! He who hesitates is lost! |
4 Feb 2023, 09:26 (Ref:4142613) | #7 | ||
Team Crouton
20KPINAL
Join Date: Oct 2001
Posts: 39,995
|
Fantastic work Grant, let's hope you don't get any chance to break the 19 hour record!
|
||
__________________
280 days...... |
4 Feb 2023, 09:40 (Ref:4142616) | #8 | ||
Veteran
Join Date: Nov 2006
Posts: 9,446
|
Well done, I suppose that all you had to do was switch it off and back on
|
||
__________________
Balls of steel (knob of butter) They're Asking For Larkins. ( Proper beer) not you're Eurofizz crap. Hace más calor en España. Me han conocido a hablar un montón cojones! Send any cheques and cash to PO box 1 Lagos Nigeria Africa ! |
4 Feb 2023, 10:05 (Ref:4142621) | #9 | ||
Veteran
Join Date: Apr 2005
Posts: 5,721
|
Well done & thank you. (I actually got quite bored at work yesterday without this distraction!)
|
||
__________________
Incognito: An Italian phrase meaning Nice Gearchange! |
4 Feb 2023, 10:24 (Ref:4142625) | #10 | ||
Veteran
Join Date: Sep 2009
Posts: 854
|
Many thanks for all your efforts, I'm sure everyone appreciates them.
|
||
|
4 Feb 2023, 12:38 (Ref:4142637) | #11 | ||
Veteran
Join Date: May 2013
Posts: 11,392
|
Thank you and well done, very much appreciated.
|
||
|
4 Feb 2023, 12:45 (Ref:4142638) | #12 | ||
14th
1% Club
Join Date: Jul 2001
Posts: 44,191
|
Well done and thank you
|
||
__________________
Brum brum |
4 Feb 2023, 13:33 (Ref:4142639) | #13 | ||
Veteran
Join Date: Aug 2022
Posts: 509
|
Very thankful for all the hard work involved. You folks did a great job.
|
||
|
4 Feb 2023, 13:47 (Ref:4142642) | #14 | ||
Veteran
Join Date: Jun 2009
Posts: 4,150
|
Busy yesterday so didn't notice the outage, but thanks for your quick and effective work - sure it was a stressful time.
|
||
__________________
Richard Murtha: You don't stop racing because you are too old, you get old when you stop racing! But its looking increasingly likely that I've stopped.....have to go back to rallying ;) |
4 Feb 2023, 13:58 (Ref:4142645) | #15 | ||
Veteran
Join Date: Mar 2005
Posts: 10,026
|
Missed everyone yesterday. Thanks for fixing!
|
||
__________________
Home, is where I want to be but I guess I'm already there I come home, she lifted up her wings guess that this must be the place |
4 Feb 2023, 15:39 (Ref:4142649) | #16 | |
Race Official
Veteran
Join Date: Jan 2003
Posts: 18,896
|
Thanks
|
|
|
4 Feb 2023, 16:06 (Ref:4142654) | #17 | ||
Race Official
Veteran
Join Date: May 2003
Posts: 4,154
|
Thanks for all the effort!
|
||
__________________
You live and learn. At any rate, you live. Douglas Adams |
4 Feb 2023, 21:14 (Ref:4142713) | #18 | ||
Veteran
Join Date: Aug 2003
Posts: 4,697
|
Thanks for getting us all back on the air so quickly - 19 hours sounds like a damn fine benchmark!
|
||
__________________
“We’re far from having too much horsepower…[m]y definition of too much horsepower is when all four wheels are spinning in every gear.” ― Mark Donohue |
4 Feb 2023, 21:38 (Ref:4142719) | #19 | ||
Admin
Veteran
Join Date: Mar 2002
Posts: 12,063
|
I'm sure I can improve on that time.
|
||
|
4 Feb 2023, 22:51 (Ref:4142732) | #20 | ||
Veteran
Join Date: Jul 2002
Posts: 3,499
|
|||
__________________
The good old days sure seem like a long time ago!! |
4 Feb 2023, 23:00 (Ref:4142733) | #21 | ||
Admin
Veteran
Join Date: Mar 2002
Posts: 12,063
|
Well, I do have to move everything back to the original hardware at some point...
|
||
|
6 Feb 2023, 09:37 (Ref:4142844) | #22 | ||
Admin
Veteran
Join Date: Mar 2002
Posts: 12,063
|
I've been asked if there's anything can be done to guard against this happening again.
In this specific attack, yes. The underlying vulnerability (which was the hosting platform itself) has been patched. However it's impossible to protect against everything as new exploits are found every day. I have a robust backup strategy that lets me recover the site in around 12-16hrs to a new server with the loss of, at most, a day worth of posts. Going forward I hope to reduce the time to recover, and the data loss range. This strategy will be tested very shortly when there will be an outage to move the site back to the original hardware. Tenths doesn't make any money and is entirely funded by myself. To provision the temporary server it's currently running on will cost around £100/month for 6 months (minimum contract) in addition to the normal £150/month hosting bill. It's always possible to build in more resilience, have multiple servers on multiple locations, or move the whole lot to cloud based infrastructure such as AWS but the money isn't there. Follow up question: "What about putting advertising back?" Well, at the point the advertising was removed it was generating about a tenth (ironically!) of the monthly hosting bill. I figured if I was paying 90% anyway then the extra wasn't going to bankrupt me and you folks could have an increasingly rare thing.... A popular website without adverts. So unfortunately there may be occasionally outages when major things happen. |
||
|
6 Feb 2023, 16:13 (Ref:4142875) | #23 | ||
Veteran
Join Date: Sep 2009
Posts: 6,194
|
Thanks for all the hard work. I think your strategy is the right one. Right level of cost vs. risk. You can't protect against everything and having a good recovery strategy allows you to recover from things that slip through the protection model.
Quote:
Richard |
||
__________________
To paraphrase Mark Twain... "I'm sorry I wrote such a long post; I didn't have time to write a short one." |
6 Feb 2023, 18:39 (Ref:4142904) | #24 | ||
Veteran
Join Date: Oct 2011
Posts: 5,986
|
May I add my thanks to Grant for all his hard work getting us back online almost before my withdrawal symptoms had kicked in.
As a previous subscriber, I would be more than happy to carry on the £10.00 subscription, and I would like to think that many more of us keyboard warriors would do so also. |
||
|
6 Feb 2023, 23:21 (Ref:4142928) | #25 | ||
Team Crouton
20KPINAL
Join Date: Oct 2001
Posts: 39,995
|
Absolutely. I contributed for a number of years when I didn't actually have to and a tenner a year is an incredibly small price to pay for everything I get out of Tenths.
|
||
__________________
280 days...... |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Site outage 7th March | MagnetON | Announcements and Feedback | 1 | 7 Mar 2016 09:17 |
Scheduled Outage For Upgrades Saturday 28th February | MagnetON | Announcements and Feedback | 9 | 2 Mar 2009 07:36 |