Getting frequent Swap and Memory warning/critical message

[pratikmehta003]

server stats.PNG

free cmd.PNG

Hi Team,
Getting frequent Swap and Memory warning/critical message for Nagios XI VM. And often the monitoring engine status shows in Red color. When we click start in the console for that Monitoring engine status, then it turns to green.. but not sure why this is happening...

attached output of free command..

we are monitoring 33 SAN switches and services are 3400... attached screenshot servers statistics too

[benjaminsmith]

Hi,

Thanks for the screenshot. A couple of issues here to work out, the monitoring engine stopping and they swap space. Let's resolve the error with the monitoring engine stopping before resolving the swap space as that is more critical.

1. How often does this occur? Is there any correlation between memory usage and the monitoring engine stopping?

2. If you haven't done so already, please run the Nagios XI Server Wizard so you'll receive a notification if there's an issue with any of the required services.

3. Lastly, please send over a system profile and I can check the logs for errors. Thanks, Benjamin

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

[pratikmehta003]

sure, will send the details soon..

[benjaminsmith]

sure, will send the details soon..

Sounds good. Please reply to the thread once you send over the profile.

Thanks,
Benjamin

[pratikmehta003]

Hi Benjamin,

Have sent the info privately.. i am also seeing some other errors now for other components.. we had got the server rebooted and i have restarted again nagios service today.... something seems to be really wrong...

[benjaminsmith]

Hi,

I got your message. Can you send the entire profile.zip file over? It has the full set of logs and configurations for troubleshooting system issues. If you're not able to download it from the web interface using the following steps:

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

Then go run the following commands to generate it from the command line.

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile.zip
/usr/local/nagiosxi/scripts/components/getprofile.sh​ SUPPORT

Then send or attach the resulting /usr/local/nagiosxi/var/components/profile.zip​ file.

Lastly, if you're experiencing critical issues with your production server, then I would recommend opening a support ticket for faster resolution.

https://support.nagios.com/tickets/

In the meantime time, please try doing a full restart of the system and let us know if you notice any improvement.

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl restart httpd
systemctl start nagios
systemctl start npcd
systemctl start crond

[pratikmehta003]

Hi Benjamin,
From the console its not allowing to download the profile..

do i need to run below 2 commands to get the same output? I hope it doesnt make any changes as this is a Live instance and devices are getting monitored:
rm -rf /usr/local/nagiosxi/var/components/profile.zip
/usr/local/nagiosxi/scripts/components/getprofile.sh​ SUPPORT

I will check with customer on full restart part since its in Prod..

[benjaminsmith]

Hi,

From the console its not allowing to download the profile..

What error message are your getting? Typically, this is the result of an incorrect sudoers file, which can cause other issues as well, take a look at the following KB article, and let me know if that's the issue.

Nagios XI - Profile Build Failed

Those commands will just remove the old profile(s) saved on the system and then generate a new one. It won't make any changes to the server. You can just run this one command as well and get the latest profile from the directory (usr/local/nagiosxi/var/components/)

Code: Select all

/usr/local/nagiosxi/scripts/components/getprofile.sh​ SUPPORT

[pratikmehta003]

HI Benjamin,

Yes m getting same error as shown in KB...

This customer where we have this running, has pretty strict rules... so is it possible to know what edits will be needed in sudoers?
If i give an extract and if u can have a look at it...

[benjaminsmith]

Hi,

This customer where we have this running, has pretty strict rules... so is it possible to know what edits will be needed in sudoers?
If i give an extract and if u can have a look at it.

Sounds good. You can post it to the thread or send it over in a PM.

The default sudoers should be as follows:

Code: Select all

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d
User_Alias      NAGIOSXI=nagios
User_Alias      NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/send_to_nls.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_ssl_config.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *