Nagios stopped sending alerts after rebooting nagios server

[[email protected]]

Nagios running in AWS instance which is restarted from then nagios stopped sending alerts. I can send test email. but automatic alerts stopped . I see notifications enabled in nagios.cfg.

[dwhitfield]

First off, what's the output of the following? Please note that you will need to remove the "#" before each command line for the commands to work.
# df -i
# df -h

It's possible you are out of space.

Please run through https://assets.nagios.com/downloads/nag ... tabase.pdf and report any errors. If you stop at any point, please know at which point you stop.

If the repair script and other instructions in the document do not resolve the issue, please continue.

Regarding the instructions below, if you do not have killall, you can install it via the following command:
# yum install psmisc

If psmisc is not in your repos, then instead you can check to make sure nagios is not running with
# ps -aef | grep nagios

If that document does not resolve your issue, please run the following commands in order and report any errors. You ***must*** use mariadb instead of mysqld in the commands below, ***if*** you have mariadb.
# service nagios stop
# service ndo2db stop
# service mysqld stop
# service crond stop
# service httpd stop
# killall -9 nagios
# killall -9 ndo2db
# rm -rf /usr/local/nagios/var/rw/nagios.cmd
# rm -rf /usr/local/nagios/var/nagios.lock
# rm -f /usr/local/nagios/var/ndo.sock
# rm -f /usr/local/nagios/var/ndo2db.lock
# rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
# for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
# service ndo2db start
# service nagios start
# service mysqld start
# service crond start
# service httpd start

Can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

[[email protected]]

I have checked file system which is perfectly fine and not able to download profile. May I please know how to get output of View System Info.

[lmiltchev]

I have checked file system which is perfectly fine and not able to download profile. May I please know how to get output of View System Info.

What is the issue that you are having with downloading the profile? Do you see any errors in the web UI?

You need to click on the Admin menu first, then click on System Profile menu on the left. If you can, click on Download Profile button and PM us the profile.zip file (preferable). If you can't, click on View System Info, copy/paste the output on the forum.

example01.PNG

[[email protected]]

Please attached screen shot to see what I can get after clicking system profile. Nay how I am copying below info which I get after clicking system profile

Nagios XI Installation Profile
Download Profile
System:
Nagios XI Version : 5.2.3
ip-172-31-21-217.us-west-1.compute.internal 2.6.32-220.7.1.el6.x86_64 x86_64
CentOS release 6.7 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0
Server Name: ec2-54-215-253-149.us-west-1.compute.amazonaws.com
Server Address: 172.31.21.217
Server Port: 80
Date/Time
PHP Timezone: US/Eastern
PHP Time: Fri, 23 Jun 2017 15:36:24 -0400
System Time: Fri, 23 Jun 2017 15:36:24 -0400
Nagios XI Data
nagios (pid 11382) is running...
NPCD running (pid 1239).
ndo2db (pid 11314) is running...
CPU Load 15: 1.16
Total Hosts: 12
Total Services: 71
Function 'get_base_uri' returns: http://ec2-54-215-253-149.us-west-1.com ... /nagiosxi/
Function 'get_base_url' returns: http://ec2-54-215-253-149.us-west-1.com ... /nagiosxi/
Function 'get_backend_url(internal_call=false)' returns: http://ec2-54-215-253-149.us-west-1.com ... rofile.php
Function 'get_backend_url(internal_call=true)' returns: http://localhost/nagiosxi/backend/
Ping Test localhost
Running:

/bin/ping -c 3 localhost 2>&1

PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=2 ttl=64 time=0.032 ms
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=3 ttl=64 time=0.029 ms

--- localhost.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.023/0.028/0.032/0.003 ms
Test wget To locahost
WGET From URL: http://localhost/nagiosql/index.php
Running:

/usr/bin/wget http://localhost/nagiosql/index.php

--2017-06-23 15:36:26-- http://localhost/nagiosql/index.php
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5259 (5.1K) [text/html]
Saving to: "/tmp/nagiosql_index.tmp"

0K ..... 100% 348M=0s

2017-06-23 15:36:26 (348 MB/s) - "/tmp/nagiosql_index.tmp" saved [5259/5259]



and below is the error I get when trying to download profile

PROFILE BUILD FAILED

Array
(
)

CODE: 110

[lmiltchev]

OK, so profile failed to build for some reason. Have you modified/hardened the server in any way? What is the output of the following commands?

Code: Select all

sestatus
chage nagios -l
chage apache -l

1. Start the following command in the CLI:

Code: Select all

tail -f /var/log/httpd/error_log &> /tmp/error_log.txt

then go to the web UI, and try to download the profile again. When you see the "PROFILE BUILD FAILED" error message, go back to the CLI, stop the tail command by pressing Ctrl+c, then upload the error_log.txt that was created in the /tmp directory.

2. Post the /etc/sudoers file from your Nagios XI system.

[[email protected]]

Hi There is nothing wrong with config we are able to get the email alerts but we are not getting alerts to our mobile phones which we use to get previously. We have updated below handlers in contact.cfg file for all users to receive email alerts. So we are having issues with handlers which sends alerts to our phones. Can you please send us correct handlers for phone alerts

host_notification_commands notify-host-by-email,xi_host_notification_handler
service_notification_commands notify-service-by-email,xi_service_notification_handler

define contact {
contact_name [email protected]
alias Sai Kuppam
host_notifications_enabled 1
service_notifications_enabled 1
host_notification_period 24x7
service_notification_period 24x7
host_notification_options d,u,r,f,s
service_notification_options w,u,c,r,f,s
host_notification_commands notify-host-by-email,xi_host_notification_handler
service_notification_commands notify-service-by-email,xi_service_notification_handler
email [email protected]
use xi_contact_generic
}

[dwhitfield]

Please post or PM your /etc/sudoers file. That's a common issue with profiles, and will likely cause other issues as well. If you PM, please make sure you update this thread so it comes back up on our support dashboard.

[[email protected]]

Please find /etc/sudoers file content below

Defaults !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files.
#
Defaults always_set_home

Defaults env_reset
Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLO RS"
Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES "
Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY "

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults env_keep += "HOME"

Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Allow root to run any commands anywhere
root ALL=(ALL) ALL
sysd ALL=(ALL) ALL

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d
ec2-user ALL=(ALL) NOPASSWD: ALL
User_Alias NAGIOSXI=nagios
User_Alias NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/component s/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/get profile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/compon ents/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/ getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

[tgriep]

To get emails sent you your mobile phones, the settings for the Mail Method for the Nagios XI server has to be set to use a SMPT server.
Login to the XI GUI and go to the Admin > Manage Email Settings and set the Mail Method to SMTP and update the SMTP Settings with a valid server, port, etc...

The Notification Handlers that work with the Mobile Phone settings are as follows.
xi_host_notification_handler
xi_service_notification_handler

Edit that contact in the Core Config Manager > Contacts menu and remove the following notification handlers.
notify-host-by-email
notify-service-by-email

Save and Apply the Config.

Test it out and see if the changes make the mobile phone alerts work.

If not, try the following.

Can you login to the XI GUI and click on the username in the top right corner.
Then click of the Notification Preferences and the Notification Methods menus and verify that the settings are correct under the Mobile Text tab.

Please run the following commands as root and PM the /tmp/info.txt file back to me.

Code: Select all

echo 'select * from xi_users;' | mysql -u root -pnagiosxi nagiosxi -t >/tmp/info.txt
echo 'select * from xi_usermeta;' | mysql -u root -pnagiosxi nagiosxi -t >>/tmp/info.txt

What is the username that you are trying to get the mobile phone setup to receive emails?