Some check are not showing via the Xi UI

[nthaler]

Hello Nagios!

We discovered today that some of our checks are being run, and alerting via email and pagerduty, but are not showing up in the UI. We noticed this with our disk space check that runs on all of our Linux machines. There are serveral hosts that the service is not showing for.

It picks up this service via a hostgroup. The hostgroup, linux-server, adds checks for CPU, Disk Space and Memory. The CPU and Memory services seem to be appearing on all hosts in the UI but not the disk space check.

We have run complete stops of the monitoring engine and DB with from both the UI and the command line.

We have killed the apache service to restart the UI.

Any idea why some of our checks are running but not showing up in the UI?

[rkennedy]

That is pretty odd, usually the culprit is multiple processes running. What is the full output of ps -ef | grep nagios.cfg?

As for the configuration part of it, could you post your /usr/local/nagios/var/objects.cache for us to review, and also name off a host / specific name of the disk check that isn't appearing in the GUI?

[nthaler]

Out put of ps -ef | grep nagios.cfg
[root@nagios-02 ~]# ps -ef | grep nagios.cfg
nagios 24930 1 1 11:57 ? 00:01:05 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 24944 24930 0 11:57 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 29408 23871 0 12:56 pts/1 00:00:00 grep nagios.cfg

Looked like 2 instances were running so I ran
service nagios stop
pkill -u nagios
service nagios start

output of ps -ef | grep nagios.cfg after wards.
[root@nagios-02 var]# ps -ef | grep nagios.cfg
nagios 7594 1 13 13:12 ? 00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 7602 7594 0 13:12 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 7838 23871 0 13:13 pts/1 00:00:00 grep nagios.cfg

Seems like there are still 2 running.

I tired uploading the object.cache here but it seems to fail.
I did upload a copy of the check that is not showing up for all hosts in the UI.

ONe host we know the check is running on but not showing in the UI is sassnfs04

[rkennedy]

That's actually fine since they are running under the same parent process. (24930 1) (24944 24930)

Could you try PMing over your profile instead? This should have the information. (Admin -> System Profile -> Download Profile)

With the issue we're dealing with, we'll need to see all of the information and not just the one configuration file.

EDIT: profile received.

[nthaler]

I sent you the requested files via PM.

[tgriep]

Can you login to the Nagios Core interface and see if the checks are displayed there?
To login to the core interface, use the example below, replacing the xxx.xxx.xxx.xxx with the IP address of the server.

Code: Select all

http://xxx.xxx.xxx.xxx/nagios/

Use the nagiosadmin account to login.

If they are displayed there, it could be a corrupt mysql database, so can you run the following as root on the XI server to repair them and see if the checks show up in the XI interface?

Code: Select all

mysqlcheck -f -r -u nagiosxi -pnagios --databases nagiosxi -h 10.220.50.1
mysqlcheck -f -r -u nagios -pnagios --databases nagios  -h 10.220.50.1
mysqlcheck -f -r -u nagiosql -pnagios --databases nagiosql -h 10.220.50.1

[nthaler]

I logged in via IP as nagiosadming and the check is showing there.

When I ran the commands I get this output.
[root@nagios-02 ~]# mysqlcheck -f -r -u nagiosxi -pnagios --databases nagiosxi -h 10.220.50.1
mysqlcheck: Got error: 1045: Access denied for user 'nagiosxi'@'nagios-02.HighWire.org' (using password: YES) when trying to connect
[root@nagios-02 ~]# mysqlcheck -f -r -u nagios -pnagios --databases nagios -h 10.220.50.1
mysqlcheck: Got error: 1045: Access denied for user 'nagios'@'nagios-02.HighWire.org' (using password: YES) when trying to connect
[root@nagios-02 ~]# mysqlcheck -f -r -u nagiosql -pnagios --databases nagiosql -h 10.220.50.1
mysqlcheck: Got error: 1045: Access denied for user 'nagiosql'@'nagios-02.HighWire.org' (using password: YES) when trying to connect

We have previously offloaded the nagios mysql DB and changed the default passwords. I am not sure if that matters.

[tgriep]

In my examples for the mysqlcheck, I used the default username and passwords, if yours are not default, please use them instead.
I did find another error in your profile and it looks like one of the tables in the MYSQL database is not large enough for some of the commands and that is probable the issue.
Run the following to expand the MYSQL table replacing the <username> and <password> with the correct information.

Code: Select all

echo "use nagios;alter table nagios_servicestatus modify check_command varchar(65535) not null;" |mysql -u <username> -p<password> --databases nagios  -h 10.220.50.1

You can find the username and password in the following file.

Code: Select all

/usr/local/nagiosxi/html/config.inc.php

[nthaler]

I reran those 3 commands with the proper password.

[root@nagios-02 ~]# mysqlcheck -f -r -u nagiosxi -p###### --databases nagiosxi -h 10.220.50.1
nagiosxi.xi_auditlog OK
nagiosxi.xi_commands OK
nagiosxi.xi_events OK
nagiosxi.xi_incidents OK
nagiosxi.xi_meta OK
nagiosxi.xi_options OK
nagiosxi.xi_sysstat OK
nagiosxi.xi_usermeta OK
nagiosxi.xi_users OK

[root@nagios-02 ~]# mysqlcheck -f -r -u nagios -p##### --databases nagios -h 10.220.50.1
nagios.nagios_acknowledgements OK
nagios.nagios_commands OK
nagios.nagios_commenthistory OK
nagios.nagios_comments OK
nagios.nagios_configfiles OK
nagios.nagios_configfilevariables OK
nagios.nagios_conninfo OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands OK
nagios.nagios_contactgroup_members OK
nagios.nagios_contactgroups OK
nagios.nagios_contactnotificationmethods OK
nagios.nagios_contactnotifications OK
nagios.nagios_contacts OK
nagios.nagios_contactstatus OK
nagios.nagios_customvariables OK
nagios.nagios_customvariablestatus OK
nagios.nagios_dbversion OK
nagios.nagios_downtimehistory OK
nagios.nagios_eventhandlers OK
nagios.nagios_externalcommands OK
nagios.nagios_flappinghistory OK
nagios.nagios_host_contactgroups OK
nagios.nagios_host_contacts OK
nagios.nagios_host_parenthosts OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members OK
nagios.nagios_hostgroups OK
nagios.nagios_hosts OK
nagios.nagios_hoststatus OK
nagios.nagios_instances OK
nagios.nagios_logentries OK
nagios.nagios_notifications OK
nagios.nagios_objects OK
nagios.nagios_processevents OK
nagios.nagios_programstatus OK
nagios.nagios_runtimevariables OK
nagios.nagios_scheduleddowntime OK
nagios.nagios_service_contactgroups OK
nagios.nagios_service_contacts OK
nagios.nagios_service_parentservices OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members OK
nagios.nagios_servicegroups OK
nagios.nagios_services OK
nagios.nagios_servicestatus OK
nagios.nagios_statehistory OK
nagios.nagios_systemcommands OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges OK
nagios.nagios_timeperiods OK
[root@nagios-02 ~]# mysqlcheck -f -r -u nagiosql --p#####--databases nagiosql -h 10.220.50.1
nagiosql.tbl_command OK
nagiosql.tbl_contact OK
nagiosql.tbl_contactgroup OK
nagiosql.tbl_contacttemplate OK
nagiosql.tbl_domain OK
nagiosql.tbl_host OK
nagiosql.tbl_hostdependency OK
nagiosql.tbl_hostescalation OK
nagiosql.tbl_hostextinfo OK
nagiosql.tbl_hostgroup OK
nagiosql.tbl_hosttemplate OK
nagiosql.tbl_info OK
nagiosql.tbl_lnkContactToCommandHost OK
nagiosql.tbl_lnkContactToCommandService OK
nagiosql.tbl_lnkContactToContactgroup OK
nagiosql.tbl_lnkContactToContacttemplate OK
nagiosql.tbl_lnkContactToVariabledefinition OK
nagiosql.tbl_lnkContactgroupToContact OK
nagiosql.tbl_lnkContactgroupToContactgroup OK
nagiosql.tbl_lnkContacttemplateToCommandHost OK
nagiosql.tbl_lnkContacttemplateToCommandService OK
nagiosql.tbl_lnkContacttemplateToContactgroup OK
nagiosql.tbl_lnkContacttemplateToContacttemplate OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact OK
nagiosql.tbl_lnkHostToContactgroup OK
nagiosql.tbl_lnkHostToHost OK
nagiosql.tbl_lnkHostToHostgroup OK
nagiosql.tbl_lnkHostToHosttemplate OK
nagiosql.tbl_lnkHostToVariabledefinition OK
nagiosql.tbl_lnkHostdependencyToHost_DH OK
nagiosql.tbl_lnkHostdependencyToHost_H OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H OK
nagiosql.tbl_lnkHostescalationToContact OK
nagiosql.tbl_lnkHostescalationToContactgroup OK
nagiosql.tbl_lnkHostescalationToHost OK
nagiosql.tbl_lnkHostescalationToHostgroup OK
nagiosql.tbl_lnkHostgroupToHost OK
nagiosql.tbl_lnkHostgroupToHostgroup OK
nagiosql.tbl_lnkHosttemplateToContact OK
nagiosql.tbl_lnkHosttemplateToContactgroup OK
nagiosql.tbl_lnkHosttemplateToHost OK
nagiosql.tbl_lnkHosttemplateToHostgroup OK
nagiosql.tbl_lnkHosttemplateToHosttemplate OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition OK
nagiosql.tbl_lnkServiceToContact OK
nagiosql.tbl_lnkServiceToContactgroup OK
nagiosql.tbl_lnkServiceToHost OK
nagiosql.tbl_lnkServiceToHostgroup OK
nagiosql.tbl_lnkServiceToServicegroup OK
nagiosql.tbl_lnkServiceToServicetemplate OK
nagiosql.tbl_lnkServiceToVariabledefinition OK
nagiosql.tbl_lnkServicedependencyToHost_DH OK
nagiosql.tbl_lnkServicedependencyToHost_H OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H OK
nagiosql.tbl_lnkServicedependencyToService_DS OK
nagiosql.tbl_lnkServicedependencyToService_S OK
nagiosql.tbl_lnkServiceescalationToContact OK
nagiosql.tbl_lnkServiceescalationToContactgroup OK
nagiosql.tbl_lnkServiceescalationToHost OK
nagiosql.tbl_lnkServiceescalationToHostgroup OK
nagiosql.tbl_lnkServiceescalationToService OK
nagiosql.tbl_lnkServicegroupToService OK
nagiosql.tbl_lnkServicegroupToServicegroup OK
nagiosql.tbl_lnkServicetemplateToContact OK
nagiosql.tbl_lnkServicetemplateToContactgroup OK
nagiosql.tbl_lnkServicetemplateToHost OK
nagiosql.tbl_lnkServicetemplateToHostgroup OK
nagiosql.tbl_lnkServicetemplateToServicegroup OK
nagiosql.tbl_lnkServicetemplateToServicetemplate OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod OK
nagiosql.tbl_logbook OK
nagiosql.tbl_mainmenu OK
nagiosql.tbl_service OK
nagiosql.tbl_servicedependency OK
nagiosql.tbl_serviceescalation OK
nagiosql.tbl_serviceextinfo OK
nagiosql.tbl_servicegroup OK
nagiosql.tbl_servicetemplate OK
nagiosql.tbl_settings OK
nagiosql.tbl_submenu OK
nagiosql.tbl_timedefinition OK
nagiosql.tbl_timeperiod OK
nagiosql.tbl_user OK
nagiosql.tbl_variabledefinition OK

I then ran the new command and get the below response.
[root@nagios-02 ~]# echo "use nagios;alter table nagios_servicestatus modify check_command varchar(65535) not null;" |mysql -u nagios -p##### --databases nagios -h 10.220.50.1
mysql: unknown option '--databases'

[tgriep]

I forgot to remove that option.
Run this instead

Code: Select all

echo "use nagios;alter table nagios_servicestatus modify check_command varchar(65535) not null;" |mysql -u nagios -p#####  nagios -h 10.220.50.1