Nagios cmdsubsys.log Growing Excessively

[tgriep]

OK, thanks for the details on the filename differences.

In the ps output, it looks like there is a stuck cron job that is causing the issue so we need to stop / start some processes.
Run the following as root.

Code: Select all

service nagios stop
service ndo2db stop
service crond stop
pkill -9 -u nagios

service crond start
service ndo2db start
service nagios start
service npcd restart

Then check the cmdsubsys.log-20180821 file to see that it is not increasing in size anymore.

[spperry]

I've run these commands and will keep an eye on it. It was filling the drive about once very 1.5-2 hrs, so it should be obvious pretty quickly.

I'll update once I know more. Thanks.

[spperry]

Okay, it's now been two hours and we're not getting any growth at all on that file. Is that cron job that was hung something that we would have caused somehow, or is it indicative of a bigger problem? I just want to have a root cause for it, if that's possible. Either way, I'm ecstatic it's working again and I can stop babysitting it.

Thank you!

Scott

[tgriep]

Your welcome. That is good news that it fixed the issue.
The stuck cron job happened on August 20th but without having the data for that day to look at, it is hard to guess what caused it.

[nortonhealthcare]

I'm having this same issue, only the monitoring engine will no longer start. I've followed the troubleshooting steps in this thread and the file is no longer growing. What troubleshooting steps should I take?

[tgriep]

Can you run a verification of the nagios config files and post the output here so we can see if there are any errors on why the nagios process is not starting?
Run this as root and post the output here.

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

[nortonhealthcare]

Here is the output:
--------------------------------------------------------------------------------
[root@oplp1521 ~]# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.4.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2018-06-25
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 12 services.
Checked 1 hosts.
Checked 1 host groups.
Checked 0 service groups.
Checked 84 contacts.
Checked 8 contact groups.
Checked 148 commands.
Checked 92 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 1 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 92 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
--------------------------------------------------------------------------------

I will mention that I was attempting to edit the /usr/local/nagiosxi/html/config.inc.php file when I didn't realize the drive was full, and /usr/local/nagiosxi/html/config.inc.php ended up blank, so I copied /usr/local/nagiosxi/html/config.inc.php from another server to this one. If that was not sufficient to effectively restore the file, please let me know if there is another method. This is a new server that does not have backups configured. I can reinstall Nagios from scratch if need be.

[tgriep]

Copying the config.inc.php from the other server should be OK.
The only thing to look at is if you setup the Nagios RAM Disk of SSL access to the web interface, the file should be OK.

If the server ran out of space, that typically corrupts the MYSQL tables so if you have cleaned up the drive, run this as root to clean up the system a bit.

Code: Select all

service npcd stop
service nagios stop
service ndo2db stop
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
mysqlcheck -f -r -u root -pnagiosxi --all-databases
service mysqld restart
rm -rf /usr/local/nagios/var/rw/nagios.cmd
rm -rf /usr/local/nagios/var/nagios.lock
rm -rf /usr/local/nagios/var/ndo.sock
rm -rf /usr/local/nagios/var/ndo2db.lock
rm -rf /usr/local/nagiosxi/var/reconfigure_nagios.lock
rm -rf /var/lib/mrtg/mrtg_l
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
pkill -9 -u nagios
service httpd restart
service ndo2db start
service nagios start
service npcd start
service crond restart

The configs did not generate an error so it should start up.
Try running this as root to check the status of the nagios proces.

Code: Select all

service nagios status

Let us know what the status output is.

[nortonhealthcare]

Nagios still will not start. Some of those commands didn't work as expected:

Code: Select all

[root@oplp1521 init.d]# mysqlcheck -f -r -u root -pnagiosxi --all-databases
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.servers                                      OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
nagios.nagios_acknowledgements                     OK
nagios.nagios_commands                             OK
nagios.nagios_commenthistory                       OK
nagios.nagios_comments                             OK
nagios.nagios_configfiles                          OK
nagios.nagios_configfilevariables                  OK
nagios.nagios_conninfo                             OK
nagios.nagios_contact_addresses                    OK
nagios.nagios_contact_notificationcommands         OK
nagios.nagios_contactgroup_members                 OK
nagios.nagios_contactgroups                        OK
nagios.nagios_contactnotificationmethods           OK
nagios.nagios_contactnotifications                 OK
nagios.nagios_contacts                             OK
nagios.nagios_contactstatus                        OK
nagios.nagios_customvariables                      OK
nagios.nagios_customvariablestatus                 OK
nagios.nagios_dbversion                            OK
nagios.nagios_downtimehistory                      OK
nagios.nagios_eventhandlers                        OK
nagios.nagios_externalcommands                     OK
nagios.nagios_flappinghistory                      OK
nagios.nagios_host_contactgroups                   OK
nagios.nagios_host_contacts                        OK
nagios.nagios_host_parenthosts                     OK
nagios.nagios_hostchecks                           OK
nagios.nagios_hostdependencies                     OK
nagios.nagios_hostescalation_contactgroups         OK
nagios.nagios_hostescalation_contacts              OK
nagios.nagios_hostescalations                      OK
nagios.nagios_hostgroup_members                    OK
nagios.nagios_hostgroups                           OK
nagios.nagios_hosts                                OK
nagios.nagios_hoststatus                           OK
nagios.nagios_instances                            OK
nagios.nagios_logentries                           OK
nagios.nagios_notifications                        OK
nagios.nagios_objects                              OK
nagios.nagios_processevents                        OK
nagios.nagios_programstatus                        OK
nagios.nagios_runtimevariables                     OK
nagios.nagios_scheduleddowntime                    OK
nagios.nagios_service_contactgroups                OK
nagios.nagios_service_contacts                     OK
nagios.nagios_service_parentservices               OK
nagios.nagios_servicechecks                        OK
nagios.nagios_servicedependencies                  OK
nagios.nagios_serviceescalation_contactgroups      OK
nagios.nagios_serviceescalation_contacts           OK
nagios.nagios_serviceescalations                   OK
nagios.nagios_servicegroup_members                 OK
nagios.nagios_servicegroups                        OK
nagios.nagios_services                             OK
nagios.nagios_servicestatus                        OK
nagios.nagios_statehistory                         OK
nagios.nagios_systemcommands                       OK
nagios.nagios_timedeventqueue                      OK
nagios.nagios_timedevents                          OK
nagios.nagios_timeperiod_timeranges                OK
nagios.nagios_timeperiods                          OK
nagiosql.tbl_command
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_contact
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_contactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_contacttemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_domain
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_host
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_hostdependency
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_hostescalation
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_hostextinfo
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_hostgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_hosttemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_info
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContactToCommandHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContactToCommandService
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContactToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContactToContacttemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContactToVariabledefinition
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContactgroupToContact
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContactgroupToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContacttemplateToCommandHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContacttemplateToCommandService
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContacttemplateToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContacttemplateToContacttemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkContacttemplateToVariabledefinition
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostToContact
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostToHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostToHostgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostToHosttemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostToVariabledefinition
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostdependencyToHost_DH
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostdependencyToHost_H
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostdependencyToHostgroup_DH
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostdependencyToHostgroup_H
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostescalationToContact
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostescalationToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostescalationToHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostescalationToHostgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostgroupToHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHostgroupToHostgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHosttemplateToContact
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHosttemplateToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHosttemplateToHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHosttemplateToHostgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHosttemplateToHosttemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkHosttemplateToVariabledefinition
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceToContact
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceToHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceToHostgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceToServicegroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceToServicetemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceToVariabledefinition
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicedependencyToHost_DH
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicedependencyToHost_H
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicedependencyToHostgroup_DH
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicedependencyToHostgroup_H
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicedependencyToService_DS
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicedependencyToService_S
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceescalationToContact
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceescalationToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceescalationToHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceescalationToHostgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServiceescalationToService
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicegroupToService
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicegroupToServicegroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicetemplateToContact
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicetemplateToContactgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicetemplateToHost
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicetemplateToHostgroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicetemplateToServicegroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicetemplateToServicetemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkServicetemplateToVariabledefinition
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_lnkTimeperiodToTimeperiod
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_logbook
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_mainmenu
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_permission
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_permission_inactive
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_service
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_servicedependency
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_serviceescalation
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_serviceextinfo
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_servicegroup
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_servicetemplate
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_session
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_session_locks
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_settings
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_submenu
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_timedefinition
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_timeperiod
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_user
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_variabledefinition
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_auditlog
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_auth_tokens
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_cmp_trapdata
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_cmp_trapdata_log
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_commands
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_eventqueue
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_events
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_incidents
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_meta
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_options
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_sessions
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_sysstat
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_usermeta
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_users
note     : The storage engine for the table doesn't support repair

Code: Select all

[root@oplp1521 init.d]# service mysqld restart
Redirecting to /bin/systemctl restart mysqld.service
Failed to restart mysqld.service: Unit not found.
[root@oplp1521 init.d]# service mariadb restart
Redirecting to /bin/systemctl restart mariadb.service

Code: Select all

[root@oplp1521 init.d]# service nagios start
Starting nagios (via systemctl):                           [  OK  ]
[root@oplp1521 init.d]# service nagios status
nagios is not running

I'm running the following version of CentOS:

Code: Select all

[root@oplp1521 init.d]# cat /etc/*release
CentOS Linux release 7.5.1804 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.5.1804 (Core)
CentOS Linux release 7.5.1804 (Core)

Any help is greatly appreciated.

[tgriep]

Can you run the following commands as root and post the output here?

Code: Select all

df -h
df -i
service nagios restart
tail -50 /var/log/messages
tail /usr/local/nagios/var/nagios.log
ps -ef --cols=300
chage -l nagios

Thanks