Nagios Support Forum

Posted: **Sat Jan 07, 2017 12:12 pm**

Yesterday we shut the server down and then later turned it back on. We are running xi 5.2.7. When it came back up we had a lot of incorrect file format errors on the main page. I followed the instructions in this doc:
https://assets.nagios.com/downloads/nag ... tabase.pdf

There were quite a few tables I ran the command on. I also ran the repair script as well. This fixed the actual monitoring. So at least I can now see what's broken. However I still am not getting notifications. Under incident management and notifications I had another file format error so I ran the repair on that table as well and it's fixed that view but notifications still did not go out out. Hosts were in scheduled down time when the corruption occurred. I've checked and they are not in down time now.

I'm fairly certain there are other problems with the database still as when I go to admin view and system status there is no system status available. Host group summary view shows no info either.

Any help would be fantastic! I just want this thing to work.

i really appreciate any help.

Posted: **Mon Jan 09, 2017 8:36 am**

Anyone?

Posted: **Mon Jan 09, 2017 10:27 am**

Please do something to trigger an alert and then run tail /var/log/maillog and post the output.

Also, can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the Download Profile button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info).

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

UPDATE: Profile received and shared with techs

Posted: **Mon Jan 09, 2017 11:12 am**

This is our log,

Also I've noticed today that all 3 host group and all 3 service group summary status views do not show any servers. I believe this to be par of the problem as well.

Code: Select all

Jan  9 10:00:03 NagiosXIMon postfix/pickup[55229]: 15C783DEB: uid=0 from=<root>
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 15C783DEB: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 15C783DEB: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.24, delays=0.12/0.05/0/0.07, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 443273DEE: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/bounce[58609]: 15C783DEB: sender non-delivery notification: 443273DEE
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: removed
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 443273DEE: to=<[email protected]>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: removed
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:00:03 NagiosXIMon postfix/pickup[55229]: 15C783DEB: uid=0 from=<root>
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 15C783DEB: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 15C783DEB: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.24, delays=0.12/0.05/0/0.07, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 443273DEE: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/bounce[58609]: 15C783DEB: sender non-delivery notification: 443273DEE
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: removed
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 443273DEE: to=<[email protected]>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: removed
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:00:03 NagiosXIMon postfix/pickup[55229]: 15C783DEB: uid=0 from=<root>
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 15C783DEB: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 15C783DEB: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.24, delays=0.12/0.05/0/0.07, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 443273DEE: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/bounce[58609]: 15C783DEB: sender non-delivery notification: 443273DEE
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: removed
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 443273DEE: to=<[email protected]>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: removed
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:00:03 NagiosXIMon postfix/pickup[55229]: 15C783DEB: uid=0 from=<root>
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 15C783DEB: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 15C783DEB: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.24, delays=0.12/0.05/0/0.07, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 443273DEE: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/bounce[58609]: 15C783DEB: sender non-delivery notification: 443273DEE
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: removed
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 443273DEE: to=<[email protected]>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: removed
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:00:03 NagiosXIMon postfix/pickup[55229]: 15C783DEB: uid=0 from=<root>
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 15C783DEB: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 15C783DEB: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.24, delays=0.12/0.05/0/0.07, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/cleanup[58548]: 443273DEE: message-id=<[email protected]>
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:00:03 NagiosXIMon postfix/bounce[58609]: 15C783DEB: sender non-delivery notification: 443273DEE
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 15C783DEB: removed
Jan  9 10:00:03 NagiosXIMon postfix/local[58586]: 443273DEE: to=<[email protected]>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:00:03 NagiosXIMon postfix/qmgr[1872]: 443273DEE: removed
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:05:02 NagiosXIMon postfix/pickup[55229]: 5A90F3E81: uid=0 from=<root>
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 5A90F3E81: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 5A90F3E81: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.26, delays=0.13/0.05/0/0.08, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 8752B3E86: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/bounce[15096]: 5A90F3E81: sender non-delivery notification: 8752B3E86
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: removed
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 8752B3E86: to=<[email protected]>, relay=local, delay=0.01, delays=0/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: removed
You have new mail in /var/spool/mail/root
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:05:02 NagiosXIMon postfix/pickup[55229]: 5A90F3E81: uid=0 from=<root>
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 5A90F3E81: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 5A90F3E81: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.26, delays=0.13/0.05/0/0.08, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 8752B3E86: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/bounce[15096]: 5A90F3E81: sender non-delivery notification: 8752B3E86
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: removed
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 8752B3E86: to=<[email protected]>, relay=local, delay=0.01, delays=0/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: removed
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:05:02 NagiosXIMon postfix/pickup[55229]: 5A90F3E81: uid=0 from=<root>
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 5A90F3E81: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 5A90F3E81: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.26, delays=0.13/0.05/0/0.08, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 8752B3E86: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/bounce[15096]: 5A90F3E81: sender non-delivery notification: 8752B3E86
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: removed
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 8752B3E86: to=<[email protected]>, relay=local, delay=0.01, delays=0/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: removed
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:05:02 NagiosXIMon postfix/pickup[55229]: 5A90F3E81: uid=0 from=<root>
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 5A90F3E81: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 5A90F3E81: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.26, delays=0.13/0.05/0/0.08, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 8752B3E86: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/bounce[15096]: 5A90F3E81: sender non-delivery notification: 8752B3E86
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: removed
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 8752B3E86: to=<[email protected]>, relay=local, delay=0.01, delays=0/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: removed
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan  9 10:05:02 NagiosXIMon postfix/pickup[55229]: 5A90F3E81: uid=0 from=<root>
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 5A90F3E81: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 5A90F3E81: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.26, delays=0.13/0.05/0/0.08, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/cleanup[15069]: 8752B3E86: message-id=<[email protected]>
Jan  9 10:05:02 NagiosXIMon postfix/bounce[15096]: 5A90F3E81: sender non-delivery notification: 8752B3E86
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: from=<>, size=2719, nrcpt=1 (queue active)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 5A90F3E81: removed
Jan  9 10:05:02 NagiosXIMon postfix/local[15081]: 8752B3E86: to=<[email protected]>, relay=local, delay=0.01, delays=0/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan  9 10:05:02 NagiosXIMon postfix/qmgr[1872]: 8752B3E86: removed

Posted: **Mon Jan 09, 2017 11:42 am**

OK I PM'd you the system config

Posted: **Mon Jan 09, 2017 1:32 pm**

What's the output of df -h? If you're low, you'll want to look at https://assets.nagios.com/downloads/nag ... M-Disk.pdf You might just need to clear out some logs though. You might be setting a log file size setting though and not an actual disk space issue. I'm just seeing error writing message: File too large a bunch of times.

Also, what's the output of tail /var/log/mysqld.log? Thanks!

Posted: **Mon Jan 09, 2017 2:16 pm**

Here you go!

df -h

Code: Select all

[root@NagiosXIMon ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      146G   75G   64G  54% /
tmpfs                 1.5G     0  1.5G   0% /dev/shm
/dev/sda1             477M  101M  351M  23% /boot
[root@NagiosXIMon ~]#

Code: Select all

[root@NagiosXIMon ~]# tail /var/log/mysqld.log
170108 19:20:25 [Note] /usr/libexec/mysqld: Shutdown complete

170108 19:20:25 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
170108 19:21:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170108 19:21:53  InnoDB: Initializing buffer pool, size = 8.0M
170108 19:21:53  InnoDB: Completed initialization of buffer pool
170108 19:21:53  InnoDB: Started; log sequence number 0 44233
170108 19:21:53 [Note] Event Scheduler: Loaded 0 events
170108 19:21:53 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
You have new mail in /var/spool/mail/root

Posted: **Mon Jan 09, 2017 2:27 pm**

in /etc/postfix/main.cf please set mailbox_size_limit = 0.

Then run service postfix reload.

Please let us know if that corrects your issue.

Posted: **Mon Jan 09, 2017 2:52 pm**

[root@NagiosXIMon ~]# nano /etc/postfix/main.cf
You have mail in /var/spool/mail/root
[root@NagiosXIMon ~]# service postfix reload
Reloading postfix: [ OK ]
[root@NagiosXIMon ~]# tail /var/log/maillog
Jan 9 13:45:02 NagiosXIMon postfix/qmgr[1872]: 6481914D5: from=<[email protected]>, size=766, nrcpt=1 (queue active)
Jan 9 13:45:02 NagiosXIMon postfix/local[62919]: 6481914D5: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.23, delays=0.11/0.05/0/0.07, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan 9 13:45:02 NagiosXIMon postfix/cleanup[62887]: 8D68965E2: message-id=<[email protected]>
Jan 9 13:45:02 NagiosXIMon postfix/qmgr[1872]: 8D68965E2: from=<>, size=2719, nrcpt=1 (queue active)
Jan 9 13:45:02 NagiosXIMon postfix/bounce[62945]: 6481914D5: sender non-delivery notification: 8D68965E2
Jan 9 13:45:02 NagiosXIMon postfix/qmgr[1872]: 6481914D5: removed
Jan 9 13:45:02 NagiosXIMon postfix/local[62919]: 8D68965E2: to=<[email protected]>, relay=local, delay=0.01, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan 9 13:45:02 NagiosXIMon postfix/qmgr[1872]: 8D68965E2: removed
Jan 9 13:45:16 NagiosXIMon postfix/postfix-script[64195]: refreshing the Postfix mail system
Jan 9 13:45:16 NagiosXIMon postfix/master[1860]: reload -- version 2.6.6, configuration /etc/postfix

I forced a failure and still no messages.

Also please see the attached about the other problem we are having. I'm sure they related in some way.

Posted: **Mon Jan 09, 2017 2:58 pm**

Please post the output of postconf -n. Thanks!

Nagios Support Forum

Corrupted tables now no notifications

Corrupted tables now no notifications

Re: Corrupted tables now no notifications

Re: Corrupted tables now no notifications

Re: Corrupted tables now no notifications

Re: Corrupted tables now no notifications

Re: Corrupted tables now no notifications

Re: Corrupted tables now no notifications

Re: Corrupted tables now no notifications

Re: Corrupted tables now no notifications

Re: Corrupted tables now no notifications