Nagios Support Forum

What changes have been made on this machine? The reason I ask, is first we had to adjust the sudoers file, and now even in your profile we are missing quite a few log files. I suspect this could be related.

The files for configuration do exist in your /usr/local/nagios/etc/hosts/ /usr/local/nagios/etc/services/ file - what happens if you run /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Also, please attach your SQL logs, and cron, and other information as @lmiltchev mentioned, because with your profile we are missing quite a bit -

lmiltchev wrote:Run the following commands, and show the output:

Code: Select all

grep tty /etc/sudoers
grep nag /etc/group
chage -l nagios
chage -l apache
service nagios status
service ndo2db status
tail -50 /var/log/cron

Also show the output of:

Code: Select all

tail -50 /var/log/mysqld.log

or

Code: Select all

tail -50 /var/log/mariadb/mariadb.log

depending on the OS.

1. > The issue started when the inode on the “/” partition of our Nagios server became 100% full, which further corrupted 2 tables in Nagios Database.

MySQL logs on 14th Nov:-
161114 21:05:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed
161114 21:05:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_statehistory' is marked as crashed and last (automatic?) repair failed

In order to fix the tables we implemented force repair of these 2 tables, the details we explain in the below forum post.

https://support.nagios.com/kb/article.php?id=24

-bash-4.1$ myisamchk -r -f nagios_logentries
- recovering (with sort) MyISAM-table 'nagios_logentries'
Data records: 2242983
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
-bash-4.1$

-bash-4.1$ myisamchk -r -f nagios_statehistory
- recovering (with sort) MyISAM-table 'nagios_statehistory'
Data records: 4165809
- Fixing index 1
- Fixing index 2
- Fixing index 3
-bash-4.1$

2.> Running Processes:
[nagios@eems-monitor ~]/etc/init.d/nagios status
nagios (pid 15804) is running...
[nagios@eems-monitor ~]/etc/init.d/ndo2db status
ndo2db (pid 10125) is running...

3.> The Nagios user on this host is not a local user (Can’t have local user as per company policy), rather it’s a generic NIS user.

[nagios@eems-monitor ~]getent passwd Nagios
nagios:VAS:900986:25:nagios:/users/nagios:/bin/bash

[xxxx@eems-monitor log]# chage -l apache
Last password change : Aug 11, 2015
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : -1
Maximum number of days between password change : -1
Number of days of warning before password expires : -1

eems-monitor:156> sudo cat /etc/group | grep nag
nagios56049:nagios,apache
nagcmd56050:nagios,apache

eems-monitor:157> sudo cat /etc/group | grep apache
apache48:
nagios56049:nagios,apache
nagcmd56050:nagios,apache

4.> Nagios Config Check. (1181 Warnings, have hosts with no contacts/contact groups defined.)

Warning: Host 'uwsa-app-dev-001' has no default contacts or contactgroups defined!
Warning: Host 'uwsa-app-lt-001' has no default contacts or contactgroups defined!
Warning: Host 'uwsa-app-stg-001' has no default contacts or contactgroups defined!
Warning: Host 'vm-piranha' has no default contacts or contactgroups defined!
Warning: Host 'vm-sdt-um-005' has no default contacts or contactgroups defined!
Warning: Host 'vm18-piranha' has no default contacts or contactgroups defined!
Warning: Host 'vm2-piranha' has no default contacts or contactgroups defined!
Checked 834 hosts.
Checked 40 host groups.
Checked 4 service groups.
Checked 42 contacts.
Checked 29 contact groups.
Checked 218 commands.
Checked 47 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 834 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 47 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 1181
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check

I looked in your System Profile and is looks like there are multiple nagios processes running and that could cause the issue you are having.
To stop the extra processes, login as root and run the following commands. Wait a few minutes and check the XI GUI and see if screen populates with data.

If not, can you run the following commands and post the output? Thanks.

We executed the suggested commands, but that did not solve the issue..

[root@eems-monitor log]# service nagios stop && service ndo2db stop
Stopping nagios:. done.
Stopping ndo2db: done.

[root@eems-monitor log]# killall -9 nagios
nagios: no process killed

[root@eems-monitor log]# killall -9 ndo2db
ndo2db: no process killed

[root@eems-monitor log]# service ndo2db start
Starting ndo2db: done.

[root@eems-monitor log]# service nagios start
Starting nagios: done.

[root@eems-monitor log]# ipcs -q

------ Message Queues --------
key msqid owner perms used-bytes messages

[root@eems-monitor log]# grep NAGIOS /etc/sudoers
User_Alias NAGIOS1_USERS=ttaleric,nagios1,gibabu,yvij,hlavana
User_Alias NAGIOSXI=nagios
User_Alias NAGIOSXIWEB=apache
User_Alias NAGIOS_USERS=fas
Cmnd_Alias NAGIOSXIWEB_CMDS=/usr/local/nagiosxi/scripts/manage_services.sh *,/etc/init.d/nagios *,/etc/init.d/ndo2db *,/etc/init.d/npcd *,/etc/init.d/snmptt restart,/usr/bin/nmap *,/usr/local/nagiosxi/scripts/change_timezone.sh,/usr/local/nagiosxi/scripts/repair_databases.sh,/usr/local/nagiosxi/scripts/upgrade_to_latest.sh,/usr/bin/tail -100 /var/log/httpd/error_log,/usr/bin/tail -100 /var/log/messages,/usr/bin/tail -100 /var/log/mysqld.log,/usr/local/nagiosxi/scripts/repairmysql.sh *,/usr/local/nagiosxi/scripts/reset_config_perms.sh,/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
Cmnd_Alias NAGIOSXI_CMDS=/usr/local/nagiosxi/scripts/upgrade_to_latest.sh,/usr/bin/tail -100 /var/log/httpd/error_log,/usr/bin/tail -100 /var/log/messages,/usr/bin/tail -100 /var/log/mysqld.log,/usr/local/nagiosxi/scripts/manage_services.sh *,/etc/init.d/nagios *,/etc/init.d/ndo2db *,/etc/init.d/npcd *,/etc/init.d/snmptt restart,/usr/bin/nmap *,/usr/local/nagiosxi/scripts/change_timezone.sh,/usr/local/nagiosxi/scripts/repair_databases.sh,/usr/local/nagiosxi/scripts/reset_config_perms.sh,/usr/local/nagiosxi/scripts/repairmysql.sh *
Host_Alias NAGIOS1_HOSTS=earms-app-4-vm,earms-app-10-vm,earms-dcsw,earms-dev-2-vm,bvm-dcsw,earms-e-app-8,earms-e-app-7,earms-t-dev,earms-app-11-vm,earms-exec-vm11,earms-exec-vm10,earms-exec-vm1,earms-exec-vm5,earms-exec-vm7,earms-exec-vm8,earms-exec-vm9,earms-app-t-vm,earms-app-vm,earms-app-2-vm,earms-app-3-vm,earms-app-6-vm,earms-e-lnx5,earms-e-lnx6,earms-e-lnx7,earms-e-lnx8,earms-e-lnx9
Host_Alias NAGIOS_HOSTS=sfo-no-active-members-remaining
NAGIOS_USERS NAGIOS_HOSTS=/bin/su nagios
NAGIOS1_USERS NAGIOS1_HOSTS=/bin/su nagios1,/bin/su - nagios1
NAGIOSXI EEMS_HOSTS=NOPASSWD:NAGIOSXI_CMDS
NAGIOSXIWEB EEMS_HOSTS=NOPASSWD:NAGIOSXIWEB_CMDS
[root@eems-monitor log]#

Can you login to the XI server, as root, run the following commands and post the output? You also have some missing entries in the /etc/sudoers file. You should add the following.

Will get back soon..

We'll keep this open, but please note our upcoming holiday office hours:

https://support.nagios.com/forum/viewto ... 39&t=41605

We are requesting you to please keep this thread open until we respond back. Happy Holidays!

Will do, and you as well!

We have started our operations, and will be back with the details shortly.