Hi All
It doesn't help that I am on hols on a rubbish internet connection, but for now I think I have solved some of the issues making this forum go crazy slow and give terrible messages.
In short the disk ran out because someone is running a load of scripts through mulitple proxies in the US against another forum hosted on this server. Not my software and all about bicycles.
Clearly malicious as the forum is for local cycling groups and of no interest to people in the US.
The massive load is killing the server and generating huge logs which eventually filled the disk so it ran out of space. It has only taken less than a day to fill a few Gb up!
For now I have disabled their forum, but it still doesn't stop the millions of requests arriving at this server.
It is one of our smallest machines as it doesn't host much, so it the usual tactic of just having enough power to overcome such attacks isn't there.
We will see tomorrow if they give up and maybe turn them back on.
In the meantime - thank you to those who emailed through to let us know there was an issue (Fullchat to name just one)
|
|
Thank you. Enjoy your holiday and hope the sun is shining.
|
Thanks. Down in New Forest. Not quite so sunny today!
Was actually up to 2am this morning trying to sort out the attack. Never seen anything quite like this before. Clearly whoever was doing it had control of a massive bot farm as there were literally millions of requests coming from IP addresses all around the world. I ended up blocking entire ranges that are not in the UK, but the implication is that anyone also trying to access C4P from somewhere else could have been caught up in the firewall.
I'll look at a smarter firewal way of doing it that basically adds entries to the firewall from the logs.
|
|
Thanks and wishing you a wonderful holiday!
|
|
Thanks for sorting it out.
|
Down again today with similar symptoms to earlier.
Do we think it's sorted now?
|
Yes, they came back again - having banned whole IP ranges.
It's just not practical to add IPs to a list and block them as there are too many. So we used fail2ban to autoblock any that are causing issues. It registered around 100,000 in a few minutes to give you the scale of the problem. So even if it blocks them, there are still loads and loads that have to hit at least once.
What is worth realising is that someone out there has control of a huge bot farm. i.e. a whole load of thousands of machines attached to the internet that can directed at will to get pages from a website if they so desire. That's all they have to do, and the sheer number overwhelms servers.
Where do all these machines come from - basically compromised devices - not necessarily PCs. Think webcams, where a user buys one, sticks it on their network, connects to a remote server so they can get access to it when they are away and then leaves it thinking all is good, because as far as they are concerned it does what they want.... but they didn't change the master password. Same for routers etc. So these guys just run scanners through IPs and ports until they get in and then can install whatever they want on that device using some command line interface.
And they install a tool that can simply do a curl or wget from a website. Compromise thousands of these defvices in this way and you have yourself a botnet.
The thing I don't get is why they are attacking a cycling forum of all things. Bored maybe? Or maybe they have a grudge against cyclists? Who knows?
|
|
Err is cycling mickey on it? He has had a recent and very publicised altercation with a driver. He is all over the press, and the video of his altercation is trending on all media platforms - its very polarising
Last edited by: Zero on Tue 19 Aug 25 at 19:41
|
Do you know which cycling forum? Cycle chat is one of the biggest and there's no problem over there.
|
>> Do you know which cycling forum? Cycle chat is one of the biggest and there's
>> no problem over there.
As you know we're both on that one.
There are plenty of others though.
|
A club website?
Now that's a surprise, why would anyone target them?
|
I know. It’s rather odd isn’t it.
Maybe they’re just testing out how evil it can be before unleashing it on some large institution for ransom money.
|
Anything interesting in where the source IPs are/are-not located and who they are assigned to?
And I'm struggling to understand why they'd waste a botnet on a club website. Have they mistaken the IP for someone else's?
|
|
It's possibly the "cash" bit that's attracted attention.
|
|
Nice to see a website that provides members with information. Not blinged up by a 5 year old, and stuffed full of adverts..
|
Great things are back up and running but I would say that since this has happened I'm unable to view the forum on my work PC, on both Edge and Chrome that my session is 'timed out' or the site is insecure - no problem with it at home on my laptop or phone.
|
Yep I've been off most of today with that, I assumed it was still down, didn't think to try with a different device.
|
Forum Issues - Fullchat
