Nagios XI Event Log Empty

[ahoward12]

https://support.nagios.com/forum/viewto ... 16&t=44632

Hey Gents,

Appears I am still having this issue. I missed a very important notification I should have gotten. I can't even see if it was generated because the mysqld log file has the usual:

[ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed

I went through the entire log file and it is marked as crashed as far as I can see back. I don't often look in unless something goes wrong...Anyone want to take another stab at this? I am quick to respond if you need any information Not provided in the other 2 threads..

Thanks

[dwhitfield]

I don't often look in unless something goes wrong

I would suggest using your XI test box (or a Core box) to monitor that db file for new errors. There are several ways you could do this, but we have this doc, so I'll present it as the first option: https://assets.nagios.com/downloads/nag ... Swatch.pdf


As far as the issue at hand, can you post or PM your /etc/my.cnf and /etc/sudoers? Also, what is the output of sestatus?

[ahoward12]

Here are the outputs. I don't really want to spin up another box for this. I''ll run the calender reminder route to check it every 6 hours or so. Also, as a side note, for a long time now I have had a crontab that reboots the box every morning at 2 am to reduce memory consumption and such.

Code: Select all

[root@NAGIOS etc]# cat my.cnf
[mysqld]
query_cache_size=16M
query_cache_limit=4M
tmp_table_size=64M
max_heap_table_size=64M
key_buffer_size=32M
table_open_cache=32

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Code: Select all

[root@NAGIOS etc]# cat sudoers
## Sudoers allows particular users to run various commands as
## the root user, without needing the root password.
##
## Examples are provided at the bottom of the file for collections
## of related commands, which can then be delegated out to particular
## users or groups.
##
## This file must be edited with the 'visudo' command.

## Host Aliases
## Groups of machines. You may prefer to use hostnames (perhaps using
## wildcards for entire domains) or IP addresses instead.
# Host_Alias     FILESERVERS = fs1, fs2
# Host_Alias     MAILSERVERS = smtp, smtp2

## User Aliases
## These aren't often necessary, as you can use regular groups
## (ie, from files, LDAP, NIS, etc) in this file - just use %groupname
## rather than USERALIAS
# User_Alias ADMINS = jsmith, mikem


## Command Aliases
## These are groups of related commands...

## Networking
# Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

## Installation and management of software
# Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum

## Services
# Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig

## Updating the locate database
# Cmnd_Alias LOCATE = /usr/bin/updatedb

## Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
# Cmnd_Alias DRIVERS = /sbin/modprobe

# Defaults specification

#
# Refuse to run if unable to disable echo on the tty.
#
Defaults   !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
#
Defaults    always_set_home

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults   env_keep += "HOME"

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
##      user    MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL

## Allows members of the 'sys' group to run networking, software,
## service management apps and more.
# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands
# %wheel        ALL=(ALL)       ALL

## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL

## Allows members of the users group to mount and unmount the
## cdrom as root
# %users  ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom

## Allows members of the users group to shutdown this system
# %users  localhost=/sbin/shutdown -h now

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d
User_Alias      NAGIOSXI=nagios
User_Alias              NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *



# NEEDED TO ALLOW NAGIOS TO CHECK SERVICE STATUS
Defaults:nagios !requiretty
nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_init_service

# ASTERISK-SPECIFIC CHECKS
# NOTE: You can uncomment the following line if you are monitoring Asterisk locally
#nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_asterisk_sip_peers.sh, /usr/local/nagios/libexec/nagisk.pl, /usr/sbin/asterisk

Code: Select all

[root@NAGIOS etc]# sestatus
SELinux status:                 disabled

[scottwilkerson]

https://support.nagios.com/forum/viewto ... 16&t=44632

Hey Gents,

Appears I am still having this issue. I missed a very important notification I should have gotten. I can't even see if it was generated because the mysqld log file has the usual:

[ERROR] /usr/libexec/mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed

Thanks

You should run this to repair the database.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Then, If you want to verify if the notification was sent you would look in the nagios.log or the archives/nagios*.log file in /usr/local/nagios/var

[ahoward12]

Hey Scott, I have since repaired the database. That is not the issue. The issue is how often it happens....

[scottwilkerson]

Hey Scott, I have since repaired the database. That is not the issue. The issue is how often it happens....

Is this a VM?

If so do you restart the host computer without shutting down the VM from the command line first?

This is often the cause the DB crashed errors because it is similar to pulling the plug out of the wall and the database is constantly being written to.

[ahoward12]

This is a VM yes, I don't restart the host at all, only the Nagios XI VM

[scottwilkerson]

This is a VM yes, I don't restart the host at all, only the Nagios XI VM

when you restart the VM, are you doing it from the command line or using the STOP or reset button in VMware ?

I ask because if you are not using the command line with a command like

Code: Select all

shutdown -h now

it will cause a hard HALT on the VM, giving the affect above and causing database corruption almost every time

[ahoward12]

I appreciate the help but I understand how to restart the machine the correct way. I have had this issue well before having a cron tab doing scheduled restarts. I put that comment in for dwhitfield just so he has situational awareness.

No, I am not using VMware for any of it.

[scottwilkerson]

I appreciate the help but I understand how to restart the machine the correct way. I have had this issue well before having a cron tab doing scheduled restarts. I put that comment in for dwhitfield just so he has situational awareness.

No, I am not using VMware for any of it.

No worries.

Hey Scott, I have since repaired the database. That is not the issue. The issue is how often it happens....

What frequency are you getting the crashed tables in the database?

Are there any similar events in your environment that seem to occur around this same schedule?