Search

For you to give feedback to the Heartland moderators. Wish lists, fault reports, etc. You may get an answer, you may not. As in all things, it largely depends on how nicely you ask the people in charge... You're welcome.
User avatar
lazarus corporation
Lord Protector
Posts: 3426
Joined: 09 May 2004, 17:42
Location: out there on a darkened road
Contact:

For the past couple of days I've been getting error messages on search saying 'the server is under a heavy load so you can't use search', and now I'm getting "Sorry but you are not permitted to use the search system."
User avatar
Pista
Cureboi
Posts: 17480
Joined: 25 Jun 2006, 15:03
Location: Lost In A Forest
Contact:

Morning.
Barry asked me yesterday if I'd noticed anything amiss as he was getting messages about the CPU utilization.
I hadn't noticed anything out of the ordinary so thanks for pointing that out. I'll let him know
Cheers.
Steve
Just like the old days

TheCureCommunity
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

Hey folks.

Yep, so it seems that the webserver has been running at 70% cpu capacity since sometime around the 6th Jan. When the server is at high capacity, the search feature turns itself off to prevent causing any additional system load.

This kind of thing is usually temporary and resolves itself after a few mins, so mostly goes unnoticed.

However, this time its different and the server cpu has been running at 70-80% capacity constantly since 6th Jan. (I get emails every 30 mins to tell me this :lol: I have f**king hundreds of them now!)

I'm looking into it, but one hypothesis is that the database behind the forum is now just over 1gb in size. It's possible the server run out of memory, and is having to write every single query to the disc instead. I don't know. Still chatting to a few tech support folks about possible causes.

Of that 1gb (actually about 1.2gb as of today) about 650mg is the search index table. If I delete that, search wont be able to find anything (which at this point is academic because search is disabled right now anyway!)

For the techie folks still following this, I am looking to switch how the search system indexes content - away from storing it all in a database table (which serves as a cache) and instead use real time mysql db indexing. This will mean we don't need the cache of search terms, so that 650mb table can be deleted, which might in turn resolve the cpu issues.

This alternative indexing may be more performant in the short term, but I'm unsure on the implications for load because there's no physical cache of words...

Anyways, I'm looking into it and playing with some configs - you may see some untoward behaviour over the next couple of days but that will likely just be me playing with configs, and occasionally restarting the server, so no need to panic :) (at least, until I actually tell you all to panic!) ;)

Thanks for your patience.
What’s the difference between a buffalo and a bison?
User avatar
alanm
Amphetamine Filth
Posts: 245
Joined: 03 Mar 2020, 13:11
Location: this is not my map at all

I haven't had much success with search here ever, vs say searching on Google with "site:". That could be an alternative too?
(no offense intended, really appreciate all the pro bono work on this)
Last edited by alanm on 12 Jan 2024, 11:21, edited 1 time in total.
Do you remember a time when angels... do you remember a time when fear?
In the days when I was stronger, in the days when you were here?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

No offense taken, and you're absolutely right.

Google do a far better job of indexing this site than we could ever do (their budget is a little higher than ours ;) )

https://www.google.com/search?q=site%3A ... k+eldritch

as does duckduckgo, for the more privacy minded:

https://duckduckgo.com/?q=site%3Amyhear ... k+eldritch
What’s the difference between a buffalo and a bison?
User avatar
alanm
Amphetamine Filth
Posts: 245
Joined: 03 Mar 2020, 13:11
Location: this is not my map at all

Yeah. Lotta smaller sites these days just take send your search form submission over to the big G with a site prefix and display an embedded list of results.
Do you remember a time when angels... do you remember a time when fear?
In the days when I was stronger, in the days when you were here?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

The main advantage of us doing search is that if you;re logged in and have permissions, we can also show you results from the private areas (such as trading/sharing etc) that 3rd parties can't access.

All of which is a moot point if our search is broken though :lol:
What’s the difference between a buffalo and a bison?
User avatar
mh
Above the Chemist
Posts: 8059
Joined: 23 Jun 2003, 14:41
Location: A city built on rock 'n' roll

I'd be in favour of just dropping the search index for the greater good; if it's affecting the site then it seems obvious that the site itself has to take priority.
If I told them once, I told them a hundred times to put 'Spinal Tap' first and 'Puppet Show' last.
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

Fair :)
What’s the difference between a buffalo and a bison?
User avatar
lazarus corporation
Lord Protector
Posts: 3426
Joined: 09 May 2004, 17:42
Location: out there on a darkened road
Contact:

Quiff Boy wrote: 12 Jan 2024, 22:04Fair :)
I'd suggest adding a robots.txt file with the following:

Code: Select all

# Block unauthorised use of this site as training material for AI
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: OmgiliBot
Disallow: /

User-agent: The Knowledge AI
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: cohere-ai
Disallow: /

# Everyone else, you're ok
User-agent: *
Disallow:

# Wait 5 seconds between successive requests
Crawl-delay: 5
That should block a lot of the new AI bots that are currently aggressively crawling sites and extracting the text for LLM training, and it also enforces a crawl delay (for compliant bots) of 5 seconds between the crawling of each page.

I've no idea if it will make any material difference to this site - it depends what is causing the processor usage - but it can't hurt.
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

lazarus corporation wrote: 15 Jan 2024, 21:42
Quiff Boy wrote: 12 Jan 2024, 22:04Fair :)
I'd suggest adding a robots.txt file with the following:

Code: Select all

# Block unauthorised use of this site as training material for AI
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: OmgiliBot
Disallow: /

User-agent: The Knowledge AI
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: cohere-ai
Disallow: /

# Everyone else, you're ok
User-agent: *
Disallow:

# Wait 5 seconds between successive requests
Crawl-delay: 5
That should block a lot of the new AI bots that are currently aggressively crawling sites and extracting the text for LLM training, and it also enforces a crawl delay (for compliant bots) of 5 seconds between the crawling of each page.

I've no idea if it will make any material difference to this site - it depends what is causing the processor usage - but it can't hurt.
Done, thanks.
What’s the difference between a buffalo and a bison?
paint it black
Black, black, black & even blacker
Posts: 4949
Joined: 11 Jul 2002, 01:00

Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen. 😭

Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.
Goths have feelings too
User avatar
ribbons69
Slight Overbomber
Posts: 1566
Joined: 24 Jan 2009, 12:57
Location: Somewhere, terrified of dying.

I'm also having an issue, both on my work pc and my android phone where every couple of days I have to relog in to the site despite having ticked the "remember me" box
"I've seen Andrew Eldritch in an ice hockey shirt onstage, and I've given him the benefit of the doubt"
Tom G Warrior of Celtic Frost




we fall to rise
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

paint it black wrote: 16 Jan 2024, 14:48 Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen. 😭

Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.
afraid so.

the "latest posts" feature is basically a shortcut for running a search, sorted by most recent first

it basically IS a search :(
What’s the difference between a buffalo and a bison?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

ribbons69 wrote: 17 Jan 2024, 00:03 I'm also having an issue, both on my work pc and my android phone where every couple of days I have to relog in to the site despite having ticked the "remember me" box
that might be my fault actually

i've been doing some ad-hoc maintenance on the database in an attempt to get cpu usage under control, and one of the things i have done a couple of times this week is clear the SESSIONS database table, which is where the forum remembers who's logged in.

clearing down the SESSIONS table basically logs everyone out.
What’s the difference between a buffalo and a bison?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

The good (ish) news is that a number of remedial measures have brought CPU usage down from a scary, server-melting, >80% to an average of 30%

Graphs showing CPU and server load over the past 14 days
Image

Graphs showing CPU and server load over the past 7 days
Image

its still too high, and before January it has been averaging around 10% for years, so I still need to look into whats going on, but its a lot better than it was.

There's also one or two parts of the forum's admin section that are currently broken, due to a tangentially related issue (I had to upgrade the version of PHP on the server from v7 to v8, and while the forum itself is fine with PHP8, the admin area doesn't like it).

My short-term fix is to look at how to get the server to have 2 parallel versions of PHP - v7 for the forum and v8 for the other sites running on this box

The longer term solution is to rebuild the server from scratch on new (virtual) hardware, a more up to date version of the OS (Ubuntu), with PHP v8 and powered by an updated/patched version of the forum software that can handle PHP v8 properly. That is likely a couple of weeks' worth of effort so unfortunately will have to happen when I have more time...

In the meantime, I'm working to restore the forum to as near to full functionality as possible, as described above... but during that time, there may be the occasional server reboot, or few mins of downtime while I take a snapshot to do some tests on.

Apologies for an inconvenience, and thanks for your ongoing patience. :)
What’s the difference between a buffalo and a bison?
User avatar
mh
Above the Chemist
Posts: 8059
Joined: 23 Jun 2003, 14:41
Location: A city built on rock 'n' roll

I manage servers myself, so you have my full tolerance and understanding in all of this. :notworthy:
If I told them once, I told them a hundred times to put 'Spinal Tap' first and 'Puppet Show' last.
User avatar
eastmidswhizzkid
Faster Than The Light Of Speed
Posts: 9753
Joined: 24 Mar 2005, 00:01
Location: WhizzWorld
Contact:

new posts link has been buggered for weeks.


Image
Well I was handsome and I was strong
And I knew the words to every song.
"Did my singing please you?"
"No! The words you sang were wrong!"

:bat:
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

eastmidswhizzkid wrote: 17 Jan 2024, 18:40 new posts link has been buggered for weeks.
yep 👇
Quiff Boy wrote: 17 Jan 2024, 00:46
paint it black wrote: 16 Jan 2024, 14:48 Hi QB. Is the search issue related to the latest posts issue? It's not been working for around the same period of time, and makes it more difficult to track forum activity. Especially with aging eyes on a small screen. 😭

Ps. It seems to reinstate itself after you post. So for now at least I can see this thread.
afraid so.

the "latest posts" feature is basically a shortcut for running a search, sorted by most recent first

it basically IS a search :(
What’s the difference between a buffalo and a bison?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

Looks like I spoke too soon

Image

back up to >70%

dammit
What’s the difference between a buffalo and a bison?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

in case anyone wants to get super nerdy:

Image

i haven't got a scooby why mysql has suddenly starting using so much cpu
What’s the difference between a buffalo and a bison?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

i might have to take the forum offline for a couple of hours (probably best done overnight) and see if that takes any of the load off
What’s the difference between a buffalo and a bison?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

genuinely not got a scooby doo why it's doing this :urff:

Image
What’s the difference between a buffalo and a bison?
User avatar
Quiff Boy
Herr Administrator
Posts: 16762
Joined: 25 Jan 2002, 00:00
Location: Lurking and fixing
Contact:

this is a good sign.... i hope?

Image

and

Image
What’s the difference between a buffalo and a bison?
User avatar
lazarus corporation
Lord Protector
Posts: 3426
Joined: 09 May 2004, 17:42
Location: out there on a darkened road
Contact:

It looks promising!

What are the server access logs looking like over periods of high CPU? (I'm wondering if bots are triggering multiple searches per second, which in turn puts a high load on mysql)
Post Reply