Nagios XI Report Generating Issues.

[madhukarneelaiahgari]

Hi,

I've been observing issues with Nagios XI reports and performance graphs.

1. Nagios XI is taking forever to generate a report and the reports which are generated doesn't haven any data on them.
2. Performance Graphs on few services/servers are not displayed on the GUI.

We are running nagios XI on a VM with 16 CPU's and 24GB of Memory and monitoring 3000 Servers and 25000 services.

Regards,
Madhu.

[benjaminsmith]

Hi Madhu,

We have a general guide on how to maximize the performance of Nagios XI ( see link below). However, we'll want to review your system profile to a help diagnose your issues.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket.

Thanks.

Maximizing Performance In Nagios XI

[madhukarneelaiahgari]

Sent via private message

[benjaminsmith]

Hi,

Thanks for sending over the system profile. There are quite a few processes open for the availability report, and the system load is high. If you have those as scheduled reports, you may want to space them apart as the availability report is resource intensive.

Please send me the full output of the following commands:

Code: Select all

ls -lh /usr/local/nagios/var/nagios.log
ls -lh /usr/local/nagios/var/archives

Additionally, you have an offloaded database and I would like to check the table sizes and test for corrupted tables. Please post the output of the following. NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password.

Code: Select all

# Check Size
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

#Check Tables
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table | grep NULL

Next, we'll increase the timeout and load thresholds for the performance data daemon as it's frequently hitting the max settings and halting the processing of performance data.

Then edit your /usr/local/nagios/etc/pnp/npcd.cfg and change this:
load_threshold = 10.0

To this:
load_threshold = 60.0

Then edit this file:
/usr/local/nagios/etc/pnp/process_perfdata.cfg and change this:
TIMEOUT = 5

To:
TIMEOUT = 20

After you make those changes, re-start the entire Nagios stack by running the following commands:

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
rm -rf /usr/local/nagios/var/ndo2db.lock
rm -rf /usr/local/nagios/var/ndo2db.pid
rm -rf /var/run/nagios.lock
rm -rf /usr/local/nagios/var/nagios.lock
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl restart httpd
systemctl restart snmptt

Then wait 15 minutes and send us a FRESH copy of your profile so that we can review the logs along with the other requested information.

Thanks.

[madhukarneelaiahgari]

We are planning to reboot the nagios server next weekend and will send you all the requested data

[benjaminsmith]

Sounds good. We'll wait for your update.

Thanks.

[madhukarneelaiahgari]

sent you the details in PM after the config updates.
FYI we still see the report generation issues

[benjaminsmith]

Hello,

Thanks for sending that over. Just to confirm, are you still having issues with performance graphs for some of the services (did the changes to npcd.cfg help resolve that issue) ?

With respect to the reporting issues, the slowness is related to the Availability Reports. This report is resource intensive and the top command is showing 18 processes open for avail.cgi resulting in a high CPU load.

1. If you have setup scheduled availability reports, can you space them out to prevent them from running simultaneously?

2. Increasing the PHP limits in php.ini can help mitigate any timeouts.

Code: Select all

max_execution_time = 90     ; Maximum execution time of each script, in  seconds
max_input_time = 90     ; Maximum amount of time each script may spend parsing request data
memory_limit = 512M      ; Maximum amount of memory a script may consume
# Then restart Apache
service httpd restart

3. Even running the availability report for a short period of time requires Nagios to parse past log files to check state history. Removing older log data not required in /usr/local/nagios/var/archives will help boost performance ( make a backup before making any changes).

Also, as a reference here's the link to our general guide for increasing system performance.

Maximizing Performance In Nagios XI

Let me know if you have any questions.

[madhukarneelaiahgari]

Thanks for your reply. I'll look into these things.
I also have one more question, is there an official docs explaining how to migrate Mysql to Postgres for NagiosXI.

[npolovenko]

@madhukarneelaiahgari, No, we only have a doc on how to migrate Postgres to MySQL. We moved away from PostgresQL and only very old XI versions still have nagiosxi database in Postgres. Nagios and NagiosQL databases were never supported in Postgres.