Instance status different in console and GUI

[jolson]

Thank you for that, much appreciated. Any chance that selinux is running on either server?

Code: Select all

sestatus

If so, let's try turning it off temporarily and seeing if that helps:

Code: Select all

setenforce 0

Let us know. Thanks!

[pccwglobalit]

hi any updates on this? thanks

[tmcdonald]

hi any updates on this? thanks

Did you check the things in jolson's post?

[pccwglobalit]

all selinuxs are disabled.

[jolson]

Okay, it seems we're at a point where basic troubleshooting isn't leading us down the correct path. I'll need some more information so I better understand your setup.
Some questions I'd like you to answer in a detailed way are as follows:
When was this NLS cluster installed and setup? Was it working previously?
When did this stop working and were there any events that led up to it?

It looks like commands from the WebGUI are not reaching the cluster. This can mean a variety of things, but I'd like you to try the following:

Make sure cron is running on every server:

Code: Select all

systemctl status crond

Ensure that the nagios account isn't expired:

Code: Select all

chage -l nagios

Report cron logs to ensure it's functioning:

Code: Select all

tail -n20 /var/log/cron.d/log

Let me know. Thank you.

[pccwglobalit]

Okay, it seems we're at a point where basic troubleshooting isn't leading us down the correct path. I'll need some more information so I better understand your setup.
Some questions I'd like you to answer in a detailed way are as follows:
When was this NLS cluster installed and setup? Was it working previously?
When did this stop working and were there any events that led up to it?

we setup the NLS cluster on dec. it was working previously until we add nfs to server and change uid of nagios.

It looks like commands from the WebGUI are not reaching the cluster. This can mean a variety of things, but I'd like you to try the following:

Make sure cron is running on every server:

the crond is running

Code: Select all

systemctl status crond

Ensure that the nagios account isn't expired:
crond.service - Command Scheduler
Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled)
Active: active (running) since Wed 2015-03-11 11:10:11 GMT; 1 weeks 4 days ago
Main PID: 8115 (crond)
CGroup: /system.slice/crond.service
__8115 /usr/sbin/crond -n

Mar 11 11:10:11 nls1-tmh2.it.pccwglobal.com systemd[1]: Started Command Sched...
Mar 11 11:10:11 nls1-tmh2.it.pccwglobal.com crond[8115]: (CRON) INFO (RANDOM_...
Mar 11 11:10:11 nls1-tmh2.it.pccwglobal.com crond[8115]: (CRON) INFO (running...
Mar 11 11:10:11 nls1-tmh2.it.pccwglobal.com crond[8115]: (CRON) INFO (@reboot...
Hint: Some lines were ellipsized, use -l to show in full.[/color]

Code: Select all

chage -l nagios

Report cron logs to ensure it's functioning:

the password is setup never expired

Code: Select all

Mar 23 01:48:01 nls1-tmh2 CROND[31296]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 23 01:48:01 nls1-tmh2 CROND[31297]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 23 01:49:01 nls1-tmh2 CROND[31366]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 23 01:49:01 nls1-tmh2 CROND[31367]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 23 01:50:01 nls1-tmh2 CROND[31540]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 23 01:50:01 nls1-tmh2 CROND[31541]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 23 01:50:01 nls1-tmh2 CROND[31542]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 23 01:51:01 nls1-tmh2 CROND[31584]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 23 01:51:01 nls1-tmh2 CROND[31585]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 23 01:52:01 nls1-tmh2 CROND[31654]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 23 01:52:01 nls1-tmh2 CROND[31655]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 23 01:53:01 nls1-tmh2 CROND[31794]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 23 01:53:01 nls1-tmh2 CROND[31795]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 23 01:54:01 nls1-tmh2 CROND[31935]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 23 01:54:01 nls1-tmh2 CROND[31936]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)

Updating Cluster Hosts File
Updating Elasticsearch with instance...
sudo: unknown uid 8005: who are you?
sudo: unknown uid 8005: who are you?
Updating Cluster Hosts File
Updating Elasticsearch with instance...
sudo: unknown uid 8005: who are you?
sudo: unknown uid 8005: who are you?

Let me know. Thank you.

[jolson]

Can you change the UID of nagios back to what it was originally as a test, and then restart elasticsearch and logstash?

Code: Select all

usermod -u <UID> nagios

Code: Select all

systemctl restart elasticsearch

Code: Select all

systemctl restart logstash

[pccwglobalit]

After change usermod, now seems working. will change another one and confirm the status. thanks for your help.

[jolson]

No problem - I will wait for your confirmation before locking this thread. Thanks!