Upgrade Issues

[CameronWP]

Hi:

Since upgrading to 5.5.7 we have been having some strange issues. First, we deleted a number of services from the configuration and yet they aren't removed from the dashboard. If you click on any of them the report that the object is not found. I have attached screenshots. The second issue is that notifications are being sent even if they are disabled or the event is acknowledged.

I had reported this issue before and as a possible fix run the repair_database.sh script. I thought it was resolved but all of the deleted hosts have come back again and notifications aren't behaving.

Thanks!

[lmiltchev]

I have attached screenshots.

Did you forget to upload the screenshots on the forum?

Run the following commands, and show the output:

Code: Select all

cat /etc/*release
service nagios status
service ndo2db status
ps -ef | grep nagios.cfg | grep -v grep
grep 'NagiosRunFile=' /etc/init.d/nagios
grep 'nagios.lock' /usr/local/nagios/etc/nagios.cfg
ls -la /usr/local/nagios/var/nagios.lock
ls -la /var/run/nagios.lock

[CameronWP]

Sorry, I had re-opened the issue and forgot the original screenshots. I have attached them to this reply along with the info you asked for in your reply. Thanks!

[lmiltchev]

Are you using ModGearman? I see a few issues - multiple nagios processes running, two nagios.lock files in different locations, etc.

Before we proceed with troubleshooting the issue, run the following command and show the output:

Code: Select all

/usr/local/nagios/bin/nagios | head -3

The resolution of the problem would be different, depending on the version of Nagios Core that you are currently using.

[CameronWP]

We aren't using ModGearman at all. I have attached a screenshot of the command you requested I run as well. This was a clean installation of Nagios installed about 5 years ago that has been consistently upgraded to the newest versions.

[lmiltchev]

With CentOS 7 (systemd) you should be starting nagios service via the nagios.service unit file. Ideally, you should get rid of the "old" nagios init file that is left over from the old Nagios XI installs.

Let's check a few more things though. Run the following commands, and show the output:

# This is to check if you are using ramdisk (old location)

Code: Select all

cat /etc/sysconfig/nagios

# Viewing the nagios.service file contents

Code: Select all

cat /usr/lib/systemd/system/nagios.service

# This is to check if you are using ramdisk (new location)

Code: Select all

cat /usr/lib/systemd/system/ramdisk.service

[CameronWP]

Hi:

I have attached the results of those commands.

Thanks!

[lmiltchev]

OK, so you don't have the nagios.service unit file in the expected location. Run the following command just to make sure it's not somewhere else on your system:

Code: Select all

find / -name nagios.service

If you don't find it anywhere, do the following:

1. Stop the nagios and ndo2db service:

Code: Select all

service nagios stop
service ndo2db stop

Double check to make sure that there is no nagios process running:

Code: Select all

ps -ef | grep nagios.cfg | grep -v grep

If there is a nagios process running yet - kill it by running:

Code: Select all

kill -9 <PID>

2. Move the /etc/init.d/nagios file to the /tmp directory. Don't delete it yet (just in case).

Code: Select all

mv /etc/init.d/nagios /tmp

3. Remove nagios.lock file (if it exists):

Code: Select all

rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/nagios.lock

4. Create a file named "nagios.service" in the "/usr/lib/systemd/system/" directory with contents:

Code: Select all

[Unit]
Description=Nagios Core 4.4.2
Documentation=https://www.nagios.org/documentation
After=network.target local-fs.target

[Service]
Type=forking
ExecStartPre=/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
ExecStart=/usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
ExecStop=/usr/bin/kill -s TERM ${MAINPID}
ExecStopPost=/usr/bin/rm -f /usr/local/nagios/var/rw/nagios.cmd
ExecReload=/usr/bin/kill -s HUP ${MAINPID}

[Install]
WantedBy=multi-user.target

Save and exit.

5. Run the following commands:

Code: Select all

systemctl daemon-reload
systemctl enable nagios.service
systemctl start ndo2db.service
systemctl start nagios.service

6. Check the status of nagios and ndo2db service.

Code: Select all

systemctl status ndo2db.service
systemctl status nagios.service

Let us know if this helped.

[CameronWP]

Hi:

I performed all of those steps and it seems things started as expected although when checking the status of the nagios service it returned the following:

NagResp4.jpg

The web site is available but it continues to show deleted hosts and services.

Thanks!

[lmiltchev]

Did you kill the multiple nagios processes running?

Run this:

Code: Select all

service nagios stop
killall nagios
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/nagios.lock
systemctl start nagios.service
systemctl status nagios.service

and show the output.