Apply configuration is taking long time and not completing

[deek]

Hi

I tried apply configuration on one of our nagios instances and it is taking so much of time and not completing. Couldn't identify the issue. I found /usr/local filesystem was full and addded more space. But still the same issue. Attached the snapshot for the same.

Thanks,
Deepan.

[lmiltchev]

First off, verify that you have enough space on the system:

Code: Select all

df -h

After running out of disk space, quite often there are crashed tables in the database. Run the database repair script:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

then check to see if you still have crashed tables (error messages in the log):

Code: Select all

tail /var/log/mysqld.log

or

Code: Select all

tail /var/log/mariadb/mariadb.log

Sometimes you need to run the repair script more than once in order to fix all of the errors.

There are few more things that you need to check when apply configuration never completes. They are described in our KB article below:

https://support.nagios.com/kb/article/n ... es-34.html

I suspect that you need to increase the resource limits in the php.ini file, but it could be some other reason for failing, .e.g. old reconfigure_nagios.lock, sudoers, etc.

Hope this helps.

[deek]

I also wanna mention here that eventhandler.log is generating on a rapid pace and getting filled immediately after deleting the log. Total space for /usr/local/ is 4GB and in just 10 mins eventhandler.log is getting filled and 50% space has been occupied already. Attached the snapshot.

[deek]

In the next 10 mins /usr/local is full. since eventhandler.log is full and the event manager went down. Attached the snapshot.

[lmiltchev]

Have you performed any of the steps I recommended in my previous post?

Run the following command and show the output:

Code: Select all

cat /etc/cron.d/nagiosxi

Also run:

Code: Select all

tail -2000 /usr/local/nagiosxi/var/event_handler.log > /tmp/eventhandler.log

and upload the eventhandler.log file that was generated in the /tmp directory from running the above command. We would need to review the file to find out why so many events are firing.

[deek]

yes i had performed the previous steps. We are still seeing the same issue. I will share the eventhandler.log data here shortly.

[deek]

I have attached the information that you have asked for

[lmiltchev]

The eventhandler.log has some "old" entries, but nothing recent. The events are not being processed for some reason. Can you PM me your latest profile?

Admin > System Profile > Download Profile

[deek]

Tried downloading the profile. and its throwing up error.

[lmiltchev]

Hmm, try resolving the issue by following the KB article below:
https://support.nagios.com/kb/article/n ... d-533.html