Sporadic 'Connection refused' errors in 4.2.4

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Locked
kernow5000
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Sporadic 'Connection refused' errors in 4.2.4

Post by kernow5000 »

Hi there,

First post so be gentle with me.

I have a Nagios 4.x install which I've been running for a few years. It is currently upgraded to 4.2.4 which I believe is current.

Every so often I get false positives via check_http plugin which are usually 'connection refused' - this 'mainly' seems to happen on this plugin only.

I'm also getting these errors in /var/log/messages which from reading have been changed to warnings and not errors in the current version. These still go to /var/log/messages however.

Below:

Jan 8 08:51:21 backupserver nagios: job 6328 (pid=30393): read() returned error 11
Jan 8 08:53:21 backupserver nagios: job 6333 (pid=30501): read() returned error 11
Jan 8 13:45:41 backupserver nagios: job 7103 (pid=18374): read() returned error 11
Jan 8 13:47:41 backupserver nagios: job 7108 (pid=19397): read() returned error 11
Jan 9 11:05:31 backupserver nagios: job 179 (pid=30032): read() returned error 11
Jan 9 11:07:31 backupserver nagios: job 184 (pid=31363): read() returned error 11



These errors match my alerts exactly.
Nothing changes on the servers to cause these connection refused errors. And these errors and resulting alerts only happen perhaps a burst of 10, a couple of times a week.

It's driving me nuts! Especially when I know it's on the Nagios side and not the boxes being monitored.

Please send me in the right direction!
kernow5000
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by kernow5000 »

https://support.nagios.com/forum/viewto ... =7&t=39856

Seems related to this. But from what I can read these error 11's are informational rather than anything. I just cannot understand why I get these 'connection refused' errors 1% of the time at most.

I can't see anything changing on the boxes being monitored. The Nagios server has plenty of resource (quad core, 32GB RAM) and isn't doing much else except Nagios really.
kernow5000
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by kernow5000 »

Another thing. use_syslog is off and it's still logging to /var/log/messages after a service restart. It's always been off however.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by dwhitfield »

kernow5000 wrote:isn't doing much else except Nagios really.
What else is it doing? It doesn't need to be doing much, it just needs to be doing something at the right time.

Also, what OS are you running this on? That will help us provide appropriate file paths.
kernow5000
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by kernow5000 »

Hi,

It's running bacula as well which runs in the evening. But only for about 480GB of data.

I believe it's CentOS 6.8, updated with yum-cron, nagios is built from source each time.

I've just increased some timeouts on the few commands I've noticed that sometimes fail, and changed a few from check_tcp!22 to check_ssh and things - leftovers from learning days I guess.

Will run it for a bit and see what happens.

Funny though, I swear I was getting a lot of check entries in /usr/local/nagios/var/nagios.log - but now I've just got details about the service starting and initialising. All working correctly though.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by dwhitfield »

kernow5000 wrote: Funny though, I swear I was getting a lot of check entries in /usr/local/nagios/var/nagios.log - but now I've just got details about the service starting and initialising. All working correctly though.
Check out https://assets.nagios.com/downloads/nag ... ebug_level

It's possible your old setting got replaced in an upgrade. That anchor is not the only place that might have changed, but it should at least point you in the right direction.
kernow5000
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by kernow5000 »

Debug level is set to 0, I did have it enabled yesterday while I was doing some diagnostics. I could see the failure in there, but not the cause why, just the informational string really.

Will have another play with it. Strange though, I'm sure nagios.log would move a bit too.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by dwhitfield »

kernow5000 wrote: I'm sure nagios.log would move a bit too.
Higher up on that page, starting more-or-less with "Log Current States" there are a lot of log options.

As for the syslog issue, you might want to post on our github. I will note https://github.com/NagiosEnterprises/na ... issues/304, so syslog stuff should already be getting some attention.
kernow5000
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by kernow5000 »

Thanks, I'll check this out and try contribute to the github issues.

Will report back tomorrow and see how Nagios does.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Sporadic 'Connection refused' errors in 4.2.4

Post by dwhitfield »

kernow5000 wrote:Thanks, I'll check this out and try contribute to the github issues.


Thanks! That will be much-appreciated!
Will report back tomorrow and see how Nagios does.
We'll be here! Our day starts at 9am US Central, but of course there are community members from across the globe. :)
Locked