XI System Component Status - Multiple Flags

[Bionic___]

Every morning I come in only to find that there are multiple errors showing in the XI System Component Status. All red are:

• Monitoring Engine
  Database Maintenance
  Command Subsystem
  Event Manger
  Cleaner
  System Statistics

Attempts to restart Monitoring engine from the wheel button get:

Please Wait
Your request was not processed in a timely manner. It may still execute, as the server may be temporarily busy.

I have found that if I stop Nagios and the database and do a repair on the database they sometimes will come back green for about 12 to 14 hours. I often have to reboot the server. Here is a copy of my repair and restart:

Code: Select all

root@baza ~> service nagios stop
Stopping nagios (via systemctl):                           [  OK  ]
root@baza ~> service mysqld stop
Redirecting to /bin/systemctl stop  mysqld.service
Failed to stop mysqld.service: Unit mysqld.service not loaded.
root@baza ~> /usr/local/nagiosxi/scripts/repairmysql.sh nagios
â mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-06-14 06:19:44 CDT; 2 days ago
  Process: 7596 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
  Process: 7567 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 7595 (mysqld_safe)
   CGroup: /system.slice/mariadb.service
           ââ7595 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
           ââ7752 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log...

Jun 14 06:19:42 baza systemd[1]: Starting MariaDB database server...
Jun 14 06:19:43 baza mysqld_safe[7595]: 160614 06:19:43 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Jun 14 06:19:43 baza mysqld_safe[7595]: 160614 06:19:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jun 14 06:19:44 baza systemd[1]: Started MariaDB database server.
DATABASE: nagios
TABLE:
/var/lib/mysql/nagios ~
- recovering (with sort) MyISAM-table 'nagios_acknowledgements.MYI'
Data records: 43
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_commands.MYI'
Data records: 146
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_commenthistory.MYI'
Data records: 126
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_comments.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_configfiles.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_configfilevariables.MYI'
Data records: 131
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_conninfo.MYI'
Data records: 102
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_contact_addresses.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactgroup_members.MYI'
Data records: 54
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactgroups.MYI'
Data records: 12
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contact_notificationcommands.MYI'
Data records: 300
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactnotificationmethods.MYI'
Data records: 1449
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_contactnotifications.MYI'
Data records: 1371
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4

---------

- recovering (with sort) MyISAM-table 'nagios_contacts.MYI'
Data records: 25
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactstatus.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_customvariables.MYI'
Data records: 29
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_customvariablestatus.MYI'
Data records: 196
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with keycache) MyISAM-table 'nagios_dbversion.MYI'
Data records: 1

---------

- recovering (with sort) MyISAM-table 'nagios_downtimehistory.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_eventhandlers.MYI'
Data records: 240
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_externalcommands.MYI'
Data records: 63
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_flappinghistory.MYI'
Data records: 137
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_hostchecks.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_host_contactgroups.MYI'
Data records: 70
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_host_contacts.MYI'
Data records: 24
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostdependencies.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalation_contactgroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalation_contacts.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalations.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostgroup_members.MYI'
Data records: 52
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostgroups.MYI'
Data records: 11
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_host_parenthosts.MYI'
Data records: 16
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hosts.MYI'
Data records: 55
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_hoststatus.MYI'
Data records: 55
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
- Fixing index 7
- Fixing index 8
- Fixing index 9
- Fixing index 10
- Fixing index 11
- Fixing index 12
- Fixing index 13
- Fixing index 14
- Fixing index 15
- Fixing index 16
- Fixing index 17
- Fixing index 18
- Fixing index 19

---------

- recovering (with sort) MyISAM-table 'nagios_instances.MYI'
Data records: 1
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_logentries.MYI'
Data records: 212406
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4

---------

- recovering (with sort) MyISAM-table 'nagios_notifications.MYI'
Data records: 392
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4

---------

- recovering (with sort) MyISAM-table 'nagios_objects.MYI'
Data records: 617
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5

---------

- recovering (with sort) MyISAM-table 'nagios_processevents.MYI'
Data records: 453
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_programstatus.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_runtimevariables.MYI'
Data records: 18
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_scheduleddowntime.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicechecks.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4

---------

- recovering (with sort) MyISAM-table 'nagios_service_contactgroups.MYI'
Data records: 8
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_service_contacts.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicedependencies.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contactgroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contacts.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalations.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroup_members.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroups.MYI'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_service_parentservices.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_services.MYI'
Data records: 6
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_servicestatus.MYI'
Data records: 267
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
- Fixing index 7
- Fixing index 8
- Fixing index 9
- Fixing index 10
- Fixing index 11
- Fixing index 12
- Fixing index 13
- Fixing index 14
- Fixing index 15
- Fixing index 16
- Fixing index 17
- Fixing index 18
- Fixing index 19

---------

- recovering (with sort) MyISAM-table 'nagios_statehistory.MYI'
Data records: 18426
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_systemcommands.MYI'
Data records: 109984
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_timedeventqueue.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

- recovering (with sort) MyISAM-table 'nagios_timedevents.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiods.MYI'
Data records: 28
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiod_timeranges.MYI'
Data records: 151
- Fixing index 1
- Fixing index 2
~

===============
REPAIR COMPLETE
===============
root@baza ~> service mysqld start
Redirecting to /bin/systemctl start  mysqld.service
Failed to start mysqld.service: Unit mysqld.service failed to load: No such file or directory.
root@baza ~> service nagios start
Starting nagios (via systemctl):                           [  OK  ]
root@baza ~>

I am not sure where to start looking for the problem.

Now I am finding that I cannot apply changes but "apply changes" does not generate a snapshot file to show me where the error is.
Bummer

[rkennedy]

What version of XI are you running? This is odd -- sounds almost like your system is shutting down services.

Could you PM over a profile for us to look at?

[Bionic___]

Multiple attempts to download profile got me "Failed to retrieve zip file!" even though I can display the profile on the screen.
I will do a copy of the screen and send it on.

[rkennedy]

Can you also post the full output of ps -aef and top|head -n20?

[Bionic___]

I found a log file that ate 50 GB of space and deleted it.
I will see how things look in the morning.

[lmiltchev]

I found a log file that ate 50 GB of space and deleted it.
I will see how things look in the morning.

Let us know if you continue experiencing issues.