Upgraded Nagios XI to 5.6.2 and recevived many false alerts

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
MOHANREDDY
Posts: 81
Joined: Tue Apr 10, 2018 4:14 pm

Upgraded Nagios XI to 5.6.2 and recevived many false alerts

Post by MOHANREDDY »

Hi,

Last Friday we upgraded Nagios XI from 5.6.1 to 5.6.2 and the installation is successful. but after 5 mins users are bombarded with lots of email alerts saying Return code of 13 for service / Disk Usage on host was out of bounds , All the alerts says out of bounds, we rolled back to previous version(5.6.1) from snapshot. The alerts were no more and no notifications nothing, everything is fine.

I would to like to know the reason for those alerts and what do they mean?
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by scottwilkerson »

What type of checks were these "/ Disk Usage" ?

Unfortunately we may need some information when updated to 5.6.2 to debug the problem such as permissions in this directory

Code: Select all

ls -al /usr/local/nagios/libexec/
What I'm afraid of is that you have a custom umask or something that is changing the permissions of the files when changed.
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
User avatar
smblair
Posts: 31
Joined: Thu Nov 16, 2017 3:49 pm

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by smblair »

We had the same experience yesterday but were able to revert the RPMS and not restore a snapshot. Our update was from 5.6.1 to 5.6.2. Running database recovery had no effect. Immediately after downgrading, the system stabilized and started sending recovery alerts.
MOHANREDDY
Posts: 81
Joined: Tue Apr 10, 2018 4:14 pm

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by MOHANREDDY »

scottwilkerson wrote:What type of checks were these "/ Disk Usage" ?

Unfortunately we may need some information when updated to 5.6.2 to debug the problem such as permissions in this directory

Code: Select all

ls -al /usr/local/nagios/libexec/
What I'm afraid of is that you have a custom umask or something that is changing the permissions of the files when changed.
Currently, we have a default mask settings as below,
]# umask
0022
we didn't change any mask before or upgrade, a small correction I did restore from the backups not from snapshot. Today now again I upgraded on another server from 5.6.1 to 5.6.2, it again started alerting a lot of emails.

see the attached doc of the output you asked.

Thanks,
You do not have the required permissions to view the files attached to this post.
User avatar
smblair
Posts: 31
Joined: Thu Nov 16, 2017 3:49 pm

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by smblair »

Here is our listing.
You do not have the required permissions to view the files attached to this post.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by scottwilkerson »

I'm confused, you showed 2 different things
the libexec_files.docx has completely wrong permissions

lib-perms.gz has permissions that should work fine..

Are you working together on the same server?
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
User avatar
smblair
Posts: 31
Joined: Thu Nov 16, 2017 3:49 pm

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by smblair »

scottwilkerson wrote:I'm confused, you showed 2 different things
the libexec_files.docx has completely wrong permissions

lib-perms.gz has permissions that should work fine..

Are you working together on the same server?
No, no. Completely separate. They were just having the same issue we were.
Would you prefer our issue in a separate post? We do have paid support.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by scottwilkerson »

smblair wrote:No, no. Completely separate. They were just having the same issue we were.
Would you prefer our issue in a separate post? We do have paid support.
Yes please
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by scottwilkerson »

MOHANREDDY wrote:
scottwilkerson wrote:What type of checks were these "/ Disk Usage" ?

Unfortunately we may need some information when updated to 5.6.2 to debug the problem such as permissions in this directory

Code: Select all

ls -al /usr/local/nagios/libexec/
What I'm afraid of is that you have a custom umask or something that is changing the permissions of the files when changed.
Currently, we have a default mask settings as below,
]# umask
0022
we didn't change any mask before or upgrade, a small correction I did restore from the backups not from snapshot. Today now again I upgraded on another server from 5.6.1 to 5.6.2, it again started alerting a lot of emails.

see the attached doc of the output you asked.

Thanks,
Somehow your permissions are changing on the libexec directory

Run the following after upgrade if they still look the same:

Code: Select all

chmod ug+x -R /usr/local/nagios/libexec/
chmod u+s -R /usr/local/nagios/libexec/check_dhcp
chmod u+s -R /usr/local/nagios/libexec/check_icmp
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
MOHANREDDY
Posts: 81
Joined: Tue Apr 10, 2018 4:14 pm

Re: Upgraded Nagios XI to 5.6.2 and recevived many false ale

Post by MOHANREDDY »

why does permissions change after each and every release, we have such type of issues when we upgraded from 5.5.11 to 5.6.1 and again the same repeated from 5.6.1 to 5.6.2?
Locked