Search

[lazarus corporation]

For the past couple of days I've been getting error messages on search saying 'the server is under a heavy load so you can't use search', and now I'm getting "Sorry but you are not permitted to use the search system."

[Pista]

Morning.
Barry asked me yesterday if I'd noticed anything amiss as he was getting messages about the CPU utilization.
I hadn't noticed anything out of the ordinary so thanks for pointing that out. I'll let him know

[Quiff Boy]

Hey folks.

Yep, so it seems that the webserver has been running at 70% cpu capacity since sometime around the 6th Jan. When the server is at high capacity, the search feature turns itself off to prevent causing any additional system load.

This kind of thing is usually temporary and resolves itself after a few mins, so mostly goes unnoticed.

However, this time its different and the server cpu has been running at 70-80% capacity constantly since 6th Jan. (I get emails every 30 mins to tell me this I have f**king hundreds of them now!)

I'm looking into it, but one hypothesis is that the database behind the forum is now just over 1gb in size. It's possible the server run out of memory, and is having to write every single query to the disc instead. I don't know. Still chatting to a few tech support folks about possible causes.

Of that 1gb (actually about 1.2gb as of today) about 650mg is the search index table. If I delete that, search wont be able to find anything (which at this point is academic because search is disabled right now anyway!)

For the techie folks still following this, I am looking to switch how the search system indexes content - away from storing it all in a database table (which serves as a cache) and instead use real time mysql db indexing. This will mean we don't need the cache of search terms, so that 650mb table can be deleted, which might in turn resolve the cpu issues.

This alternative indexing may be more performant in the short term, but I'm unsure on the implications for load because there's no physical cache of words...

Anyways, I'm looking into it and playing with some configs - you may see some untoward behaviour over the next couple of days but that will likely just be me playing with configs, and occasionally restarting the server, so no need to panic (at least, until I actually tell you all to panic!)

Thanks for your patience.

[alanm]

I haven't had much success with search here ever, vs say searching on Google with "site:". That could be an alternative too?
(no offense intended, really appreciate all the pro bono work on this)

[Quiff Boy]

No offense taken, and you're absolutely right.

Google do a far better job of indexing this site than we could ever do (their budget is a little higher than ours )

https://www.google.com/search?q=site%3A ... k+eldritch

as does duckduckgo, for the more privacy minded:

https://duckduckgo.com/?q=site%3Amyhear ... k+eldritch

[alanm]

Yeah. Lotta smaller sites these days just take send your search form submission over to the big G with a site prefix and display an embedded list of results.

[Quiff Boy]

The main advantage of us doing search is that if you;re logged in and have permissions, we can also show you results from the private areas (such as trading/sharing etc) that 3rd parties can't access.

All of which is a moot point if our search is broken though

[mh]

I'd be in favour of just dropping the search index for the greater good; if it's affecting the site then it seems obvious that the site itself has to take priority.

[Quiff Boy]

Fair

[lazarus corporation]

Fair

I'd suggest adding a robots.txt file with the following:

Code: Select all

# Block unauthorised use of this site as training material for AI
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: OmgiliBot
Disallow: /

User-agent: The Knowledge AI
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: cohere-ai
Disallow: /

# Everyone else, you're ok
User-agent: *
Disallow:

# Wait 5 seconds between successive requests
Crawl-delay: 5

That should block a lot of the new AI bots that are currently aggressively crawling sites and extracting the text for LLM training, and it also enforces a crawl delay (for compliant bots) of 5 seconds between the crawling of each page.

I've no idea if it will make any material difference to this site - it depends what is causing the processor usage - but it can't hurt.

[Quiff Boy]

Fair

I'd suggest adding a robots.txt file with the following:

Code: Select all

# Block unauthorised use of this site as training material for AI
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: OmgiliBot
Disallow: /

User-agent: The Knowledge AI
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: cohere-ai
Disallow: /

# Everyone else, you're ok
User-agent: *
Disallow:

# Wait 5 seconds between successive requests
Crawl-delay: 5

That should block a lot of the new AI bots that are currently aggressively crawling sites and extracting the text for LLM training, and it also enforces a crawl delay (for compliant bots) of 5 seconds between the crawling of each page.

I've no idea if it will make any material difference to this site - it depends what is causing the processor usage - but it can't hurt.

Done, thanks.

[paint it black]

Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen.

Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.

[ribbons69]

I'm also having an issue, both on my work pc and my android phone where every couple of days I have to relog in to the site despite having ticked the "remember me" box

[Quiff Boy]

Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen.

Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.

afraid so.

the "latest posts" feature is basically a shortcut for running a search, sorted by most recent first

it basically IS a search

[Quiff Boy]

I'm also having an issue, both on my work pc and my android phone where every couple of days I have to relog in to the site despite having ticked the "remember me" box

that might be my fault actually

i've been doing some ad-hoc maintenance on the database in an attempt to get cpu usage under control, and one of the things i have done a couple of times this week is clear the SESSIONS database table, which is where the forum remembers who's logged in.

clearing down the SESSIONS table basically logs everyone out.

[Quiff Boy]

The good (ish) news is that a number of remedial measures have brought CPU usage down from a scary, server-melting, >80% to an average of 30%

Graphs showing CPU and server load over the past 14 days


Graphs showing CPU and server load over the past 7 days


its still too high, and before January it has been averaging around 10% for years, so I still need to look into whats going on, but its a lot better than it was.

There's also one or two parts of the forum's admin section that are currently broken, due to a tangentially related issue (I had to upgrade the version of PHP on the server from v7 to v8, and while the forum itself is fine with PHP8, the admin area doesn't like it).

My short-term fix is to look at how to get the server to have 2 parallel versions of PHP - v7 for the forum and v8 for the other sites running on this box

The longer term solution is to rebuild the server from scratch on new (virtual) hardware, a more up to date version of the OS (Ubuntu), with PHP v8 and powered by an updated/patched version of the forum software that can handle PHP v8 properly. That is likely a couple of weeks' worth of effort so unfortunately will have to happen when I have more time...

In the meantime, I'm working to restore the forum to as near to full functionality as possible, as described above... but during that time, there may be the occasional server reboot, or few mins of downtime while I take a snapshot to do some tests on.

Apologies for an inconvenience, and thanks for your ongoing patience.

[mh]

I manage servers myself, so you have my full tolerance and understanding in all of this.

[eastmidswhizzkid]

new posts link has been buggered for weeks.

[Quiff Boy]

new posts link has been buggered for weeks.

yep

Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen.

Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.

afraid so.

the "latest posts" feature is basically a shortcut for running a search, sorted by most recent first

it basically IS a search

[Quiff Boy]

Looks like I spoke too soon



back up to >70%

dammit

[Quiff Boy]

in case anyone wants to get super nerdy:



i haven't got a scooby why mysql has suddenly starting using so much cpu

[Quiff Boy]

i might have to take the forum offline for a couple of hours (probably best done overnight) and see if that takes any of the load off

[Quiff Boy]

genuinely not got a scooby doo why it's doing this

[Quiff Boy]

this is a good sign.... i hope?



and

[lazarus corporation]

It looks promising!

What are the server access logs looking like over periods of high CPU? (I'm wondering if bots are triggering multiple searches per second, which in turn puts a high load on mysql)