Sporadic 'Connection refused' errors in 4.2.4
-
- Posts: 58
- Joined: Mon Jan 09, 2017 9:06 am
Sporadic 'Connection refused' errors in 4.2.4
Hi there,
First post so be gentle with me.
I have a Nagios 4.x install which I've been running for a few years. It is currently upgraded to 4.2.4 which I believe is current.
Every so often I get false positives via check_http plugin which are usually 'connection refused' - this 'mainly' seems to happen on this plugin only.
I'm also getting these errors in /var/log/messages which from reading have been changed to warnings and not errors in the current version. These still go to /var/log/messages however.
Below:
Jan 8 08:51:21 backupserver nagios: job 6328 (pid=30393): read() returned error 11
Jan 8 08:53:21 backupserver nagios: job 6333 (pid=30501): read() returned error 11
Jan 8 13:45:41 backupserver nagios: job 7103 (pid=18374): read() returned error 11
Jan 8 13:47:41 backupserver nagios: job 7108 (pid=19397): read() returned error 11
Jan 9 11:05:31 backupserver nagios: job 179 (pid=30032): read() returned error 11
Jan 9 11:07:31 backupserver nagios: job 184 (pid=31363): read() returned error 11
These errors match my alerts exactly.
Nothing changes on the servers to cause these connection refused errors. And these errors and resulting alerts only happen perhaps a burst of 10, a couple of times a week.
It's driving me nuts! Especially when I know it's on the Nagios side and not the boxes being monitored.
Please send me in the right direction!
First post so be gentle with me.
I have a Nagios 4.x install which I've been running for a few years. It is currently upgraded to 4.2.4 which I believe is current.
Every so often I get false positives via check_http plugin which are usually 'connection refused' - this 'mainly' seems to happen on this plugin only.
I'm also getting these errors in /var/log/messages which from reading have been changed to warnings and not errors in the current version. These still go to /var/log/messages however.
Below:
Jan 8 08:51:21 backupserver nagios: job 6328 (pid=30393): read() returned error 11
Jan 8 08:53:21 backupserver nagios: job 6333 (pid=30501): read() returned error 11
Jan 8 13:45:41 backupserver nagios: job 7103 (pid=18374): read() returned error 11
Jan 8 13:47:41 backupserver nagios: job 7108 (pid=19397): read() returned error 11
Jan 9 11:05:31 backupserver nagios: job 179 (pid=30032): read() returned error 11
Jan 9 11:07:31 backupserver nagios: job 184 (pid=31363): read() returned error 11
These errors match my alerts exactly.
Nothing changes on the servers to cause these connection refused errors. And these errors and resulting alerts only happen perhaps a burst of 10, a couple of times a week.
It's driving me nuts! Especially when I know it's on the Nagios side and not the boxes being monitored.
Please send me in the right direction!
-
- Posts: 58
- Joined: Mon Jan 09, 2017 9:06 am
Re: Sporadic 'Connection refused' errors in 4.2.4
https://support.nagios.com/forum/viewto ... =7&t=39856
Seems related to this. But from what I can read these error 11's are informational rather than anything. I just cannot understand why I get these 'connection refused' errors 1% of the time at most.
I can't see anything changing on the boxes being monitored. The Nagios server has plenty of resource (quad core, 32GB RAM) and isn't doing much else except Nagios really.
Seems related to this. But from what I can read these error 11's are informational rather than anything. I just cannot understand why I get these 'connection refused' errors 1% of the time at most.
I can't see anything changing on the boxes being monitored. The Nagios server has plenty of resource (quad core, 32GB RAM) and isn't doing much else except Nagios really.
-
- Posts: 58
- Joined: Mon Jan 09, 2017 9:06 am
Re: Sporadic 'Connection refused' errors in 4.2.4
Another thing. use_syslog is off and it's still logging to /var/log/messages after a service restart. It's always been off however.
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Sporadic 'Connection refused' errors in 4.2.4
What else is it doing? It doesn't need to be doing much, it just needs to be doing something at the right time.kernow5000 wrote:isn't doing much else except Nagios really.
Also, what OS are you running this on? That will help us provide appropriate file paths.
-
- Posts: 58
- Joined: Mon Jan 09, 2017 9:06 am
Re: Sporadic 'Connection refused' errors in 4.2.4
Hi,
It's running bacula as well which runs in the evening. But only for about 480GB of data.
I believe it's CentOS 6.8, updated with yum-cron, nagios is built from source each time.
I've just increased some timeouts on the few commands I've noticed that sometimes fail, and changed a few from check_tcp!22 to check_ssh and things - leftovers from learning days I guess.
Will run it for a bit and see what happens.
Funny though, I swear I was getting a lot of check entries in /usr/local/nagios/var/nagios.log - but now I've just got details about the service starting and initialising. All working correctly though.
It's running bacula as well which runs in the evening. But only for about 480GB of data.
I believe it's CentOS 6.8, updated with yum-cron, nagios is built from source each time.
I've just increased some timeouts on the few commands I've noticed that sometimes fail, and changed a few from check_tcp!22 to check_ssh and things - leftovers from learning days I guess.
Will run it for a bit and see what happens.
Funny though, I swear I was getting a lot of check entries in /usr/local/nagios/var/nagios.log - but now I've just got details about the service starting and initialising. All working correctly though.
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Sporadic 'Connection refused' errors in 4.2.4
Check out https://assets.nagios.com/downloads/nag ... ebug_levelkernow5000 wrote: Funny though, I swear I was getting a lot of check entries in /usr/local/nagios/var/nagios.log - but now I've just got details about the service starting and initialising. All working correctly though.
It's possible your old setting got replaced in an upgrade. That anchor is not the only place that might have changed, but it should at least point you in the right direction.
-
- Posts: 58
- Joined: Mon Jan 09, 2017 9:06 am
Re: Sporadic 'Connection refused' errors in 4.2.4
Debug level is set to 0, I did have it enabled yesterday while I was doing some diagnostics. I could see the failure in there, but not the cause why, just the informational string really.
Will have another play with it. Strange though, I'm sure nagios.log would move a bit too.
Will have another play with it. Strange though, I'm sure nagios.log would move a bit too.
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Sporadic 'Connection refused' errors in 4.2.4
Higher up on that page, starting more-or-less with "Log Current States" there are a lot of log options.kernow5000 wrote: I'm sure nagios.log would move a bit too.
As for the syslog issue, you might want to post on our github. I will note https://github.com/NagiosEnterprises/na ... issues/304, so syslog stuff should already be getting some attention.
-
- Posts: 58
- Joined: Mon Jan 09, 2017 9:06 am
Re: Sporadic 'Connection refused' errors in 4.2.4
Thanks, I'll check this out and try contribute to the github issues.
Will report back tomorrow and see how Nagios does.
Will report back tomorrow and see how Nagios does.
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Sporadic 'Connection refused' errors in 4.2.4
kernow5000 wrote:Thanks, I'll check this out and try contribute to the github issues.
Thanks! That will be much-appreciated!
We'll be here! Our day starts at 9am US Central, but of course there are community members from across the globe.Will report back tomorrow and see how Nagios does.
![Smile :)](./images/smilies/icon_e_smile.gif)