Sporadic 'Connection refused' errors in 4.2.4

An open discussion forum for obtaining help with Nagios Core. Nagios Core users of all experience levels are welcome here. Subforum have been created for the discussion of Nagios Core and Nagios Plugin development.

NOTE: The SourceForge.net mailing lists have been deprecated in favor of this forum in order to expedite support and provide additional features not available on the old mailing list.

Sporadic 'Connection refused' errors in 4.2.4

Postby kernow5000 » Mon Jan 09, 2017 9:12 am

Hi there,

First post so be gentle with me.

I have a Nagios 4.x install which I've been running for a few years. It is currently upgraded to 4.2.4 which I believe is current.

Every so often I get false positives via check_http plugin which are usually 'connection refused' - this 'mainly' seems to happen on this plugin only.

I'm also getting these errors in /var/log/messages which from reading have been changed to warnings and not errors in the current version. These still go to /var/log/messages however.

Below:

Jan 8 08:51:21 backupserver nagios: job 6328 (pid=30393): read() returned error 11
Jan 8 08:53:21 backupserver nagios: job 6333 (pid=30501): read() returned error 11
Jan 8 13:45:41 backupserver nagios: job 7103 (pid=18374): read() returned error 11
Jan 8 13:47:41 backupserver nagios: job 7108 (pid=19397): read() returned error 11
Jan 9 11:05:31 backupserver nagios: job 179 (pid=30032): read() returned error 11
Jan 9 11:07:31 backupserver nagios: job 184 (pid=31363): read() returned error 11



These errors match my alerts exactly.
Nothing changes on the servers to cause these connection refused errors. And these errors and resulting alerts only happen perhaps a burst of 10, a couple of times a week.

It's driving me nuts! Especially when I know it's on the Nagios side and not the boxes being monitored.

Please send me in the right direction!
kernow5000
 
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby kernow5000 » Mon Jan 09, 2017 10:10 am

viewtopic.php?f=7&t=39856

Seems related to this. But from what I can read these error 11's are informational rather than anything. I just cannot understand why I get these 'connection refused' errors 1% of the time at most.

I can't see anything changing on the boxes being monitored. The Nagios server has plenty of resource (quad core, 32GB RAM) and isn't doing much else except Nagios really.
kernow5000
 
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby kernow5000 » Mon Jan 09, 2017 10:18 am

Another thing. use_syslog is off and it's still logging to /var/log/messages after a service restart. It's always been off however.
kernow5000
 
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby dwhitfield » Mon Jan 09, 2017 11:47 am

kernow5000 wrote:isn't doing much else except Nagios really.


What else is it doing? It doesn't need to be doing much, it just needs to be doing something at the right time.

Also, what OS are you running this on? That will help us provide appropriate file paths.
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 3859
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby kernow5000 » Mon Jan 09, 2017 11:51 am

Hi,

It's running bacula as well which runs in the evening. But only for about 480GB of data.

I believe it's CentOS 6.8, updated with yum-cron, nagios is built from source each time.

I've just increased some timeouts on the few commands I've noticed that sometimes fail, and changed a few from check_tcp!22 to check_ssh and things - leftovers from learning days I guess.

Will run it for a bit and see what happens.

Funny though, I swear I was getting a lot of check entries in /usr/local/nagios/var/nagios.log - but now I've just got details about the service starting and initialising. All working correctly though.
kernow5000
 
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby dwhitfield » Mon Jan 09, 2017 11:58 am

kernow5000 wrote:Funny though, I swear I was getting a lot of check entries in /usr/local/nagios/var/nagios.log - but now I've just got details about the service starting and initialising. All working correctly though.


Check out https://assets.nagios.com/downloads/nag ... ebug_level

It's possible your old setting got replaced in an upgrade. That anchor is not the only place that might have changed, but it should at least point you in the right direction.
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 3859
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby kernow5000 » Mon Jan 09, 2017 12:00 pm

Debug level is set to 0, I did have it enabled yesterday while I was doing some diagnostics. I could see the failure in there, but not the cause why, just the informational string really.

Will have another play with it. Strange though, I'm sure nagios.log would move a bit too.
kernow5000
 
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby dwhitfield » Mon Jan 09, 2017 12:13 pm

kernow5000 wrote: I'm sure nagios.log would move a bit too.


Higher up on that page, starting more-or-less with "Log Current States" there are a lot of log options.

As for the syslog issue, you might want to post on our github. I will note https://github.com/NagiosEnterprises/na ... issues/304, so syslog stuff should already be getting some attention.
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 3859
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby kernow5000 » Mon Jan 09, 2017 12:18 pm

Thanks, I'll check this out and try contribute to the github issues.

Will report back tomorrow and see how Nagios does.
kernow5000
 
Posts: 58
Joined: Mon Jan 09, 2017 9:06 am

Re: Sporadic 'Connection refused' errors in 4.2.4

Postby dwhitfield » Mon Jan 09, 2017 12:29 pm

kernow5000 wrote:Thanks, I'll check this out and try contribute to the github issues.


Thanks! That will be much-appreciated!

Will report back tomorrow and see how Nagios does.


We'll be here! Our day starts at 9am US Central, but of course there are community members from across the globe. :)
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 3859
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Next

Return to Nagios Core

Who is online

Users browsing this forum: Google [Bot] and 18 guests