Creating snapshots never completes

[mike4vr]

Whenever I try to create snapshots in the web interface, it will never complete. The wheel keeps spinning with to no end. I am very new to this setup. What steps can I take to figure out what the problem is? See screen shot below:



Same thing happens when I try to do "Apply Configuration."

[Box293]

Go and create a snapshot and then run these commands

Code: Select all

tail /var/log/messages -n 100 > /tmp/messages_log.txt
tail /var/log/httpd/error_log -n 100 > /tmp/httpd_log.txt

Send us these files:

Code: Select all

/tmp/messages_log.txt
/tmp/httpd_log.txt

[mike4vr]

Go and create a snapshot and then run these commands

Code: Select all

tail /var/log/messages -n 100 > /tmp/messages_log.txt
tail /var/log/httpd/error_log -n 100 > /tmp/httpd_log.txt

Send us these files:

Code: Select all

/tmp/messages_log.txt
/tmp/httpd_log.txt

Thank you for the response. Unfortunately, nothing is being logged when I submit the snapshot. Anything else I can look at?

[jolson]

Hello,

It's possible that Nagios Log Server is not accepting commands from the GUI. If this is the case, the following file will be missing:

Code: Select all

/usr/local/nagioslogserver/var/jobs.log

If that file does exist, please tail it on all nodes:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log

Once tailed, please attempt to create the snapshot again. Report the logs that appear in jobs.log here.

If jobs.log does not exist, please tail your cron log for any errors:

Code: Select all

tail -n20 /var/log/cron

Report the output to us.

[mike4vr]

Log from /usr/local/nagioslogserver/var/jobs.log

Code: Select all

# tail /usr/local/nagioslogserver/var/jobs.log
Running command run_alerts with args ' ' for job id: 3qRJYN20Sr27QKUtWg21wg
SUCCESS
Running command run_alerts with args ' ' for job id: 3qRJYN20Sr27QKUtWg21wg
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.

Not very helpful.

However, the log from cron yields a permission error every minute:

Code: Select all

Mar 23 11:22:01 loghost01 crond[5969]: (nagios) FAILED to authorize user with PAM (Permission denied)
Mar 23 11:22:01 loghost01 crond[5970]: (nagios) FAILED to authorize user with PAM (Permission denied)

Here's my /etc/sudoers.d/nagioslogserver file:

Code: Select all

ser_Alias      NAGIOSLOGSERVER=nagios
User_Alias      NAGIOSLOGSERVERWEB=apache
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/change_timezone.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/get_logstash_ports.sh

And, finally, this is the entry in /etc/cron.d/

Code: Select all

# cat nagioslogserver
# /etc/cron.d/nagioslogserver: crontab fragment for nagioslogserver

* * * * * nagios /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
* * * * * nagios /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1

[jolson]

In your sudoers file:

Code: Select all

ser_Alias      NAGIOSLOGSERVER=nagios
User_Alias      NAGIOSLOGSERVERWEB=apache

The top line needs a 'U' to be added:

Code: Select all

User_Alias      NAGIOSLOGSERVER=nagios

If that was a pasting mistake, check the age of the nagios account:

Code: Select all

chage -l nagios

It's possible that the nagios account is expired. Let me know! Thanks!

[mike4vr]

I apologize. That, indeed, was a pasting mistake. The sudoers file is complete.

Here's the age of the nagios account:

Code: Select all

# chage -l nagios
Last password change					: Feb 04, 2015
Password expires					: never
Password inactive					: never
Account expires						: never
Minimum number of days between password change		: 0
Maximum number of days between password change		: 99999
Number of days of warning before password expires	: 7

[jolson]

I suspect this is a problem with your sudoers permissions. In /etc/sudoers, do you have the following line?

Code: Select all

#includedir /etc/sudoers.d

If not, your sub-sudo directories will not be read. If you do have that line, could you attempt to put the Nagios sudo information at the bottom of your /etc/sudoers file instead of in a sub-directory?

Code: Select all

vi /etc/sudoers

Add:

Code: Select all

User_Alias      NAGIOSLOGSERVER=nagios
User_Alias      NAGIOSLOGSERVERWEB=apache
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/change_timezone.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/get_logstash_ports.sh

After this is added, you may have to restart elasticsearch and logstash:

Code: Select all

service elasticsearch restart
service logstash restart

Please let me know if this works for you.

[mike4vr]

I did have the includedir entry in /etc/sudoers. I added the entries to the bottom of the file and restarted. Same problem.

[jolson]

Please post your PAM configuration file for cron.

Code: Select all

cat /etc/pam.d/crond

Also, check whether or not SELinux is on:

Code: Select all

sestatus