Search
- lazarus corporation
- Lord Protector
- Posts: 3426
- Joined: 09 May 2004, 17:42
- Location: out there on a darkened road
- Contact:
For the past couple of days I've been getting error messages on search saying 'the server is under a heavy load so you can't use search', and now I'm getting "Sorry but you are not permitted to use the search system."
Morning.
Barry asked me yesterday if I'd noticed anything amiss as he was getting messages about the CPU utilization.
I hadn't noticed anything out of the ordinary so thanks for pointing that out. I'll let him know
Barry asked me yesterday if I'd noticed anything amiss as he was getting messages about the CPU utilization.
I hadn't noticed anything out of the ordinary so thanks for pointing that out. I'll let him know
- Quiff Boy
- Herr Administrator
- Posts: 16768
- Joined: 25 Jan 2002, 00:00
- Location: Lurking and fixing
- Contact:
Hey folks.
Yep, so it seems that the webserver has been running at 70% cpu capacity since sometime around the 6th Jan. When the server is at high capacity, the search feature turns itself off to prevent causing any additional system load.
This kind of thing is usually temporary and resolves itself after a few mins, so mostly goes unnoticed.
However, this time its different and the server cpu has been running at 70-80% capacity constantly since 6th Jan. (I get emails every 30 mins to tell me this
I have f**king hundreds of them now!)
I'm looking into it, but one hypothesis is that the database behind the forum is now just over 1gb in size. It's possible the server run out of memory, and is having to write every single query to the disc instead. I don't know. Still chatting to a few tech support folks about possible causes.
Of that 1gb (actually about 1.2gb as of today) about 650mg is the search index table. If I delete that, search wont be able to find anything (which at this point is academic because search is disabled right now anyway!)
For the techie folks still following this, I am looking to switch how the search system indexes content - away from storing it all in a database table (which serves as a cache) and instead use real time mysql db indexing. This will mean we don't need the cache of search terms, so that 650mb table can be deleted, which might in turn resolve the cpu issues.
This alternative indexing may be more performant in the short term, but I'm unsure on the implications for load because there's no physical cache of words...
Anyways, I'm looking into it and playing with some configs - you may see some untoward behaviour over the next couple of days but that will likely just be me playing with configs, and occasionally restarting the server, so no need to panic
(at least, until I actually tell you all to panic!)
Thanks for your patience.
Yep, so it seems that the webserver has been running at 70% cpu capacity since sometime around the 6th Jan. When the server is at high capacity, the search feature turns itself off to prevent causing any additional system load.
This kind of thing is usually temporary and resolves itself after a few mins, so mostly goes unnoticed.
However, this time its different and the server cpu has been running at 70-80% capacity constantly since 6th Jan. (I get emails every 30 mins to tell me this
![Laughing :lol:](./images/smilies/lolol.gif)
I'm looking into it, but one hypothesis is that the database behind the forum is now just over 1gb in size. It's possible the server run out of memory, and is having to write every single query to the disc instead. I don't know. Still chatting to a few tech support folks about possible causes.
Of that 1gb (actually about 1.2gb as of today) about 650mg is the search index table. If I delete that, search wont be able to find anything (which at this point is academic because search is disabled right now anyway!)
For the techie folks still following this, I am looking to switch how the search system indexes content - away from storing it all in a database table (which serves as a cache) and instead use real time mysql db indexing. This will mean we don't need the cache of search terms, so that 650mb table can be deleted, which might in turn resolve the cpu issues.
This alternative indexing may be more performant in the short term, but I'm unsure on the implications for load because there's no physical cache of words...
Anyways, I'm looking into it and playing with some configs - you may see some untoward behaviour over the next couple of days but that will likely just be me playing with configs, and occasionally restarting the server, so no need to panic
![Smile :)](./images/smilies/icon_smile.gif)
![Wink ;)](./images/smilies/icon_wink.gif)
Thanks for your patience.
What’s the difference between a buffalo and a bison?
- alanm
- Gonzoid Amphetamine Filth
- Posts: 254
- Joined: 03 Mar 2020, 13:11
- Location: this is not my map at all
I haven't had much success with search here ever, vs say searching on Google with "site:". That could be an alternative too?
(no offense intended, really appreciate all the pro bono work on this)
(no offense intended, really appreciate all the pro bono work on this)
Last edited by alanm on 12 Jan 2024, 11:21, edited 1 time in total.
Do you remember a time when angels... do you remember a time when fear?
In the days when I was stronger, in the days when you were here?
In the days when I was stronger, in the days when you were here?
- Quiff Boy
- Herr Administrator
- Posts: 16768
- Joined: 25 Jan 2002, 00:00
- Location: Lurking and fixing
- Contact:
No offense taken, and you're absolutely right.
Google do a far better job of indexing this site than we could ever do (their budget is a little higher than ours
)
https://www.google.com/search?q=site%3A ... k+eldritch
as does duckduckgo, for the more privacy minded:
https://duckduckgo.com/?q=site%3Amyhear ... k+eldritch
Google do a far better job of indexing this site than we could ever do (their budget is a little higher than ours
![Wink ;)](./images/smilies/icon_wink.gif)
https://www.google.com/search?q=site%3A ... k+eldritch
as does duckduckgo, for the more privacy minded:
https://duckduckgo.com/?q=site%3Amyhear ... k+eldritch
What’s the difference between a buffalo and a bison?
- alanm
- Gonzoid Amphetamine Filth
- Posts: 254
- Joined: 03 Mar 2020, 13:11
- Location: this is not my map at all
Yeah. Lotta smaller sites these days just take send your search form submission over to the big G with a site prefix and display an embedded list of results.
Do you remember a time when angels... do you remember a time when fear?
In the days when I was stronger, in the days when you were here?
In the days when I was stronger, in the days when you were here?
- Quiff Boy
- Herr Administrator
- Posts: 16768
- Joined: 25 Jan 2002, 00:00
- Location: Lurking and fixing
- Contact:
The main advantage of us doing search is that if you;re logged in and have permissions, we can also show you results from the private areas (such as trading/sharing etc) that 3rd parties can't access.
All of which is a moot point if our search is broken though![Laughing :lol:](./images/smilies/lolol.gif)
All of which is a moot point if our search is broken though
![Laughing :lol:](./images/smilies/lolol.gif)
What’s the difference between a buffalo and a bison?
I'd be in favour of just dropping the search index for the greater good; if it's affecting the site then it seems obvious that the site itself has to take priority.
If I told them once, I told them a hundred times to put 'Spinal Tap' first and 'Puppet Show' last.
- lazarus corporation
- Lord Protector
- Posts: 3426
- Joined: 09 May 2004, 17:42
- Location: out there on a darkened road
- Contact:
I'd suggest adding a robots.txt file with the following:
Code: Select all
# Block unauthorised use of this site as training material for AI
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: OmgiliBot
Disallow: /
User-agent: The Knowledge AI
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: cohere-ai
Disallow: /
# Everyone else, you're ok
User-agent: *
Disallow:
# Wait 5 seconds between successive requests
Crawl-delay: 5
I've no idea if it will make any material difference to this site - it depends what is causing the processor usage - but it can't hurt.
- Quiff Boy
- Herr Administrator
- Posts: 16768
- Joined: 25 Jan 2002, 00:00
- Location: Lurking and fixing
- Contact:
Done, thanks.lazarus corporation wrote: ↑15 Jan 2024, 21:42I'd suggest adding a robots.txt file with the following:
That should block a lot of the new AI bots that are currently aggressively crawling sites and extracting the text for LLM training, and it also enforces a crawl delay (for compliant bots) of 5 seconds between the crawling of each page.Code: Select all
# Block unauthorised use of this site as training material for AI User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: OmgiliBot Disallow: / User-agent: The Knowledge AI Disallow: / User-agent: FacebookBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / # Everyone else, you're ok User-agent: * Disallow: # Wait 5 seconds between successive requests Crawl-delay: 5
I've no idea if it will make any material difference to this site - it depends what is causing the processor usage - but it can't hurt.
What’s the difference between a buffalo and a bison?
-
- Black, black, black & even blacker
- Posts: 4955
- Joined: 11 Jul 2002, 01:00
Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen. ![😭](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f62d.svg)
Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.
Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.
Goths have feelings too
- ribbons69
- Slight Overbomber
- Posts: 1599
- Joined: 24 Jan 2009, 12:57
- Location: Somewhere, terrified of dying.
I'm also having an issue, both on my work pc and my android phone where every couple of days I have to relog in to the site despite having ticked the "remember me" box
"I've seen Andrew Eldritch in an ice hockey shirt onstage, and I've given him the benefit of the doubt"
Tom G Warrior of Celtic Frost
we fall to rise
Tom G Warrior of Celtic Frost
we fall to rise
- Quiff Boy
- Herr Administrator
- Posts: 16768
- Joined: 25 Jan 2002, 00:00
- Location: Lurking and fixing
- Contact:
afraid so.paint it black wrote: ↑16 Jan 2024, 14:48 Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen.
Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.
the "latest posts" feature is basically a shortcut for running a search, sorted by most recent first
it basically IS a search
![Sad :(](./images/smilies/icon_sad.gif)
What’s the difference between a buffalo and a bison?
- Quiff Boy
- Herr Administrator
- Posts: 16768
- Joined: 25 Jan 2002, 00:00
- Location: Lurking and fixing
- Contact:
that might be my fault actually
i've been doing some ad-hoc maintenance on the database in an attempt to get cpu usage under control, and one of the things i have done a couple of times this week is clear the SESSIONS database table, which is where the forum remembers who's logged in.
clearing down the SESSIONS table basically logs everyone out.
What’s the difference between a buffalo and a bison?
- Quiff Boy
- Herr Administrator
- Posts: 16768
- Joined: 25 Jan 2002, 00:00
- Location: Lurking and fixing
- Contact:
The good (ish) news is that a number of remedial measures have brought CPU usage down from a scary, server-melting, >80% to an average of 30%
Graphs showing CPU and server load over the past 14 days
![Image](https://i.ibb.co/X3pPmyY/image.png)
Graphs showing CPU and server load over the past 7 days
![Image](https://i.ibb.co/kKM9Ndp/image.png)
its still too high, and before January it has been averaging around 10% for years, so I still need to look into whats going on, but its a lot better than it was.
There's also one or two parts of the forum's admin section that are currently broken, due to a tangentially related issue (I had to upgrade the version of PHP on the server from v7 to v8, and while the forum itself is fine with PHP8, the admin area doesn't like it).
My short-term fix is to look at how to get the server to have 2 parallel versions of PHP - v7 for the forum and v8 for the other sites running on this box
The longer term solution is to rebuild the server from scratch on new (virtual) hardware, a more up to date version of the OS (Ubuntu), with PHP v8 and powered by an updated/patched version of the forum software that can handle PHP v8 properly. That is likely a couple of weeks' worth of effort so unfortunately will have to happen when I have more time...
In the meantime, I'm working to restore the forum to as near to full functionality as possible, as described above... but during that time, there may be the occasional server reboot, or few mins of downtime while I take a snapshot to do some tests on.
Apologies for an inconvenience, and thanks for your ongoing patience.![Smile :)](./images/smilies/icon_smile.gif)
Graphs showing CPU and server load over the past 14 days
![Image](https://i.ibb.co/X3pPmyY/image.png)
Graphs showing CPU and server load over the past 7 days
![Image](https://i.ibb.co/kKM9Ndp/image.png)
its still too high, and before January it has been averaging around 10% for years, so I still need to look into whats going on, but its a lot better than it was.
There's also one or two parts of the forum's admin section that are currently broken, due to a tangentially related issue (I had to upgrade the version of PHP on the server from v7 to v8, and while the forum itself is fine with PHP8, the admin area doesn't like it).
My short-term fix is to look at how to get the server to have 2 parallel versions of PHP - v7 for the forum and v8 for the other sites running on this box
The longer term solution is to rebuild the server from scratch on new (virtual) hardware, a more up to date version of the OS (Ubuntu), with PHP v8 and powered by an updated/patched version of the forum software that can handle PHP v8 properly. That is likely a couple of weeks' worth of effort so unfortunately will have to happen when I have more time...
In the meantime, I'm working to restore the forum to as near to full functionality as possible, as described above... but during that time, there may be the occasional server reboot, or few mins of downtime while I take a snapshot to do some tests on.
Apologies for an inconvenience, and thanks for your ongoing patience.
![Smile :)](./images/smilies/icon_smile.gif)
What’s the difference between a buffalo and a bison?
- eastmidswhizzkid
- Faster Than The Light Of Speed
- Posts: 9818
- Joined: 24 Mar 2005, 00:01
- Location: WhizzWorld
- Contact:
new posts link has been buggered for weeks.
![Image](https://i.ibb.co/mvfNj0k/Screenshot-from-2024-01-17-17-38-11.png)
![Image](https://i.ibb.co/mvfNj0k/Screenshot-from-2024-01-17-17-38-11.png)
Well I was handsome and I was strong
And I knew the words to every song.
"Did my singing please you?"
"No! The words you sang were wrong!"
![Flying Bat! :bat:](./images/smilies/icon_flyingbat.gif)
And I knew the words to every song.
"Did my singing please you?"
"No! The words you sang were wrong!"
![Flying Bat! :bat:](./images/smilies/icon_flyingbat.gif)
- Quiff Boy
- Herr Administrator
- Posts: 16768
- Joined: 25 Jan 2002, 00:00
- Location: Lurking and fixing
- Contact:
yep
Quiff Boy wrote: ↑17 Jan 2024, 00:46afraid so.paint it black wrote: ↑16 Jan 2024, 14:48 Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen.
Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.
the "latest posts" feature is basically a shortcut for running a search, sorted by most recent first
it basically IS a search![]()
What’s the difference between a buffalo and a bison?
- lazarus corporation
- Lord Protector
- Posts: 3426
- Joined: 09 May 2004, 17:42
- Location: out there on a darkened road
- Contact:
It looks promising!
What are the server access logs looking like over periods of high CPU? (I'm wondering if bots are triggering multiple searches per second, which in turn puts a high load on mysql)
What are the server access logs looking like over periods of high CPU? (I'm wondering if bots are triggering multiple searches per second, which in turn puts a high load on mysql)