Nagios Support Forum

Posted: **Tue Oct 03, 2017 8:30 am**

scottwilkerson wrote:For some reason your system cannot run any of these crons, it is likely a permissions error

Lets send back the results of running the following
Code: Select all
sudo su -c '/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1'
ls -la /usr/bin/php
ls -la /usr/local/nagiosxi/cron
/usr/local/nagiosxi/var

ls -la /usr/bin/php
-rwxr-xr-x 1 root root 4054560 Nov 14 2014 /usr/bin/php

ls -la /usr/local/nagiosxi/cron
total 176
drwxr-xr-x 2 nagios nagios 4096 Apr 24 2015 .
drwxr-xr-x 10 nagios nagios 4096 Apr 24 2015 ..
-rwxr-xr-x 1 nagios nagios 1450 Oct 13 2015 cleaner.php
-rwxr-xr-x 1 nagios nagios 37519 Oct 13 2015 cmdsubsys.php
-rw-r--r-- 1 nagios nagios 216 Oct 13 2015 cookie.txt
-rwxr-xr-x 1 nagios nagios 14277 Oct 13 2015 dbmaint.php
-rw-r--r-- 1 nagios nagios 25700 Oct 13 2015 deadpool.php
-rwxr-xr-x 1 nagios nagios 4389 Oct 13 2015 eventman.php
-rwxr-xr-x 1 nagios nagios 1699 Oct 13 2015 feedproc.php
-rwxr-xr-x 1 nagios nagios 1911 Oct 13 2015 nom.php
-rwxr-xr-x 1 nagios nagios 21645 Oct 13 2015 perfdataproc.php
-rwxr-xr-x 1 nagios nagios 19995 Oct 13 2015 recurringdowntime.pl
-rwxr-xr-x 1 nagios nagios 1525 Oct 13 2015 reportengine.php
-rwxr-xr-x 1 nagios nagios 9188 Oct 13 2015 sysstat.php

ls -la /usr/local/nagiosxi/var
total 520
drwxr-xr-x 5 nagios nagios 4096 Oct 2 16:28 .
drwxr-xr-x 10 nagios nagios 4096 Apr 24 2015 ..
-rw-r--r-- 1 nagios nagios 2 Oct 2 15:46 cleaner.log
-rw-r--r-- 1 nagios nagios 302 Sep 30 17:37 cleaner.log.1
-rw-r--r-- 1 nagios nagios 302 Sep 24 03:23 cleaner.log.2
-rw-r--r-- 1 nagios nagios 302 Sep 17 03:32 cleaner.log.3
-rw-r--r-- 1 nagios nagios 302 Sep 10 03:21 cleaner.log.4
-rw-r--r-- 1 nagios nagios 302 Sep 3 03:15 cleaner.log.5
-rw-r--r-- 1 nagios nagios 302 Aug 27 03:33 cleaner.log.6
-rw-r--r-- 1 nagios nagios 302 Aug 20 03:43 cleaner.log.7
-rw-r--r-- 1 nagios nagios 302 Aug 13 03:42 cleaner.log.8
-rw-r--r-- 1 nagios nagios 302 Aug 6 03:22 cleaner.log.9
-rw-r--r-- 1 nagios nagios 0 Oct 1 03:13 cmdsubsys.log
-rw-r--r-- 1 nagios nagios 82 Sep 30 17:38 cmdsubsys.log.1
-rw-r--r-- 1 nagios nagios 82 Sep 24 03:24 cmdsubsys.log.2
-rw-r--r-- 1 nagios nagios 82 Sep 17 03:33 cmdsubsys.log.3
-rw-r--r-- 1 nagios nagios 82 Sep 10 03:22 cmdsubsys.log.4
-rw-r--r-- 1 nagios nagios 82 Sep 3 03:16 cmdsubsys.log.5
-rw-r--r-- 1 nagios nagios 82 Aug 27 03:34 cmdsubsys.log.6
-rw-r--r-- 1 nagios nagios 82 Aug 20 03:44 cmdsubsys.log.7
-rw-r--r-- 1 nagios nagios 82 Aug 13 03:43 cmdsubsys.log.8
-rw-r--r-- 1 nagios nagios 82 Aug 6 03:23 cmdsubsys.log.9
drwsrwsr-x 2 apache nagios 4096 Oct 13 2015 components
-rw-r--r-- 1 nagios nagios 8 Sep 30 17:37 corelog.data
-rw-r--r-- 1 nagios nagios 0 Sep 30 17:37 corelog.diff
-rw-r--r-- 1 nagios nagios 0 Oct 1 03:13 dbmaint.log
-rw-r--r-- 1 nagios nagios 2681 Sep 30 17:35 dbmaint.log.1
-rw-r--r-- 1 nagios nagios 10606 Sep 24 03:20 dbmaint.log.2
-rw-r--r-- 1 nagios nagios 2681 Sep 17 03:30 dbmaint.log.3
-rw-r--r-- 1 nagios nagios 2681 Sep 10 03:20 dbmaint.log.4
-rw-r--r-- 1 nagios nagios 2681 Sep 3 03:15 dbmaint.log.5
-rw-r--r-- 1 nagios nagios 2681 Aug 27 03:30 dbmaint.log.6
-rw-r--r-- 1 nagios nagios 2681 Aug 20 03:40 dbmaint.log.7
-rw-r--r-- 1 nagios nagios 2681 Aug 13 03:40 dbmaint.log.8
-rw-r--r-- 1 nagios nagios 10606 Aug 6 03:20 dbmaint.log.9
-rw-r--r-- 1 nagios nagios 0 Oct 1 03:13 deadpool.log
-rw-r--r-- 1 nagios nagios 75 Sep 30 17:35 deadpool.log.1
-rw-r--r-- 1 nagios nagios 75 Sep 24 03:20 deadpool.log.2
-rw-r--r-- 1 nagios nagios 75 Sep 17 03:30 deadpool.log.3
-rw-r--r-- 1 nagios nagios 75 Sep 10 03:20 deadpool.log.4
-rw-r--r-- 1 nagios nagios 75 Sep 5 03:05 deadpool.log.5
-rw-r--r-- 1 nagios nagios 75 Aug 27 03:30 deadpool.log.6
-rw-r--r-- 1 nagios nagios 75 Aug 20 03:40 deadpool.log.7
-rw-r--r-- 1 nagios nagios 75 Aug 13 03:40 deadpool.log.8
-rw-r--r-- 1 nagios nagios 75 Aug 6 03:20 deadpool.log.9
-rw-r--r-- 1 nagios nagios 0 Oct 1 03:13 eventman.log
-rw-r--r-- 1 nagios nagios 40 Sep 30 17:38 eventman.log.1
-rw-r--r-- 1 nagios nagios 40 Sep 24 03:24 eventman.log.2
-rw-r--r-- 1 nagios nagios 40 Sep 17 03:33 eventman.log.3
-rw-r--r-- 1 nagios nagios 40 Sep 10 03:22 eventman.log.4
-rw-r--r-- 1 nagios nagios 40 Sep 3 03:16 eventman.log.5
-rw-r--r-- 1 nagios nagios 40 Aug 27 03:34 eventman.log.6
-rw-r--r-- 1 nagios nagios 16516 Aug 20 03:44 eventman.log.7
-rw-r--r-- 1 nagios nagios 40 Aug 13 03:43 eventman.log.8
-rw-r--r-- 1 nagios nagios 40 Aug 6 03:23 eventman.log.9
-rw-r--r-- 1 nagios nagios 0 Oct 1 03:13 feedproc.log
-rw-r--r-- 1 nagios nagios 25 Sep 30 17:38 feedproc.log.1
-rw-r--r-- 1 nagios nagios 25 Sep 24 03:24 feedproc.log.2
-rw-r--r-- 1 nagios nagios 25 Sep 17 03:33 feedproc.log.3
-rw-r--r-- 1 nagios nagios 25 Sep 10 03:22 feedproc.log.4
-rw-r--r-- 1 nagios nagios 25 Sep 3 03:16 feedproc.log.5
-rw-r--r-- 1 nagios nagios 25 Aug 27 03:34 feedproc.log.6
-rw-r--r-- 1 nagios nagios 25 Aug 20 03:44 feedproc.log.7
-rw-r--r-- 1 nagios nagios 25 Aug 13 03:43 feedproc.log.8
-rw-r--r-- 1 nagios nagios 25 Aug 6 03:23 feedproc.log.9
-rw-r--r-- 1 nagios nagios 0 Jul 12 03:48 load_url.log
-rw-r--r-- 1 nagios nagios 922 Jul 11 08:16 load_url.log.1
-rw-r--r-- 1 nagios nagios 946 Jul 2 03:30 load_url.log.2
-rw-r--r-- 1 nagios nagios 947 Apr 2 2017 load_url.log.3
-rw-r--r-- 1 nagios nagios 945 Mar 5 2017 load_url.log.4
-rw-r--r-- 1 nagios nagios 934 Oct 6 2016 load_url.log.5
-rw-r--r-- 1 nagios nagios 949 Jul 31 2016 load_url.log.6
-rw-r--r-- 1 nagios nagios 940 Jun 18 2016 load_url.log.7
-rw-r--r-- 1 nagios nagios 948 May 8 2016 load_url.log.8
-rw-r--r-- 1 nagios nagios 947 Apr 3 2016 load_url.log.9
-rw-r--r-- 1 nagios nagios 0 Sep 30 17:37 nom.log
-rw-r--r-- 1 nagios nagios 0 Oct 1 03:13 perfdataproc.log
-rw-r--r-- 1 nagios nagios 81 Sep 30 17:38 perfdataproc.log.1
-rw-r--r-- 1 nagios nagios 81 Sep 24 03:24 perfdataproc.log.2
-rw-r--r-- 1 nagios nagios 81 Sep 17 03:33 perfdataproc.log.3
-rw-r--r-- 1 nagios nagios 81 Sep 10 03:22 perfdataproc.log.4
-rw-r--r-- 1 nagios nagios 81 Sep 3 03:16 perfdataproc.log.5
-rw-r--r-- 1 nagios nagios 81 Aug 27 03:34 perfdataproc.log.6
-rw-r--r-- 1 nagios nagios 81 Aug 20 03:44 perfdataproc.log.7
-rw-r--r-- 1 nagios nagios 81 Aug 13 03:43 perfdataproc.log.8
-rw-r--r-- 1 nagios nagios 81 Aug 6 03:23 perfdataproc.log.9
-rw-r--r-- 1 nagios nagios 0 Oct 1 03:13 recurringdowntime.log
-rw-r--r-- 1 nagios nagios 139 Sep 30 17:01 recurringdowntime.log.1
-rw-r--r-- 1 nagios nagios 139 Sep 24 03:01 recurringdowntime.log.2
-rw-r--r-- 1 nagios nagios 139 Sep 17 03:01 recurringdowntime.log.3
-rw-r--r-- 1 nagios nagios 139 Sep 10 03:01 recurringdowntime.log.4
-rw-r--r-- 1 nagios nagios 139 Sep 3 03:01 recurringdowntime.log.5
-rw-r--r-- 1 nagios nagios 139 Aug 27 03:01 recurringdowntime.log.6
-rw-r--r-- 1 nagios nagios 139 Aug 20 03:01 recurringdowntime.log.7
-rw-r--r-- 1 nagios nagios 139 Aug 13 03:01 recurringdowntime.log.8
-rw-r--r-- 1 nagios nagios 139 Aug 6 03:01 recurringdowntime.log.9
-rw-r--r-- 1 nagios nagios 0 Sep 30 17:37 reportengine.log
drwxr-xr-x 2 nagios nagios 4096 Oct 2 09:51 subsys
-rw-r--r-- 1 nagios nagios 8251 Oct 2 16:05 sysstat.log
-rw-r--r-- 1 nagios nagios 8239 Sep 30 17:38 sysstat.log.1
-rw-r--r-- 1 nagios nagios 8232 Sep 24 03:24 sysstat.log.2
-rw-r--r-- 1 nagios nagios 8236 Sep 17 03:33 sysstat.log.3
-rw-r--r-- 1 nagios nagios 8240 Sep 10 03:22 sysstat.log.4
-rw-r--r-- 1 nagios nagios 8235 Sep 3 03:16 sysstat.log.5
-rw-r--r-- 1 nagios nagios 8249 Aug 27 03:34 sysstat.log.6
-rw-r--r-- 1 nagios nagios 8249 Aug 20 03:44 sysstat.log.7
-rw-r--r-- 1 nagios nagios 8256 Aug 13 03:43 sysstat.log.8
-rw-r--r-- 1 nagios nagios 8260 Aug 6 03:23 sysstat.log.9
drwxr-xr-x 2 nagios nagios 4096 Oct 13 2015 upgrades
-rw-r--r-- 1 nagios nagios 12108 Oct 13 2015 xi-sys.cfg
-rw-r--r-- 1 nagios nagios 195 Oct 13 2015 xiversion

Trying to get permissions to run the first command...

Posted: **Tue Oct 03, 2017 9:20 am**

actually, I made an error in the first command, I need it to be

Code: Select all

sudo su nagios -c '/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1'

Thanks

Posted: **Tue Oct 03, 2017 11:36 am**

Figured out the issue... I had put in a req two weeks ago to change the cron permissions so we could edit it and I could take off the DB back ups as we won't need them.

Well Saturday it got done and changing the permissions from 664 I believe hosed everything up. Backed out that change and everything seems fine now but all the SNMP events got qued and are flying out like crazy... as are service email alerts.

There a good way to flush those out??? They haven't stopped yet... I turned off the trap sender for a bit on one of the boxes but curious if there's a way to flush it.

Posted: **Tue Oct 03, 2017 12:49 pm**

JakeHatMacys wrote:Figured out the issue... I had put in a req two weeks ago to change the cron permissions so we could edit it and I could take off the DB back ups as we won't need them.

Well Saturday it got done and changing the permissions from 664 I believe hosed everything up. Backed out that change and everything seems fine now but all the SNMP events got qued and are flying out like crazy... as are service email alerts.

There a good way to flush those out??? They haven't stopped yet... I turned off the trap sender for a bit on one of the boxes but curious if there's a way to flush it.

Not sure which you are referring to. Regular notifications emails?

Posted: **Tue Oct 03, 2017 12:58 pm**

scottwilkerson wrote:
JakeHatMacys wrote:Figured out the issue... I had put in a req two weeks ago to change the cron permissions so we could edit it and I could take off the DB back ups as we won't need them.

Well Saturday it got done and changing the permissions from 664 I believe hosed everything up. Backed out that change and everything seems fine now but all the SNMP events got qued and are flying out like crazy... as are service email alerts.

There a good way to flush those out??? They haven't stopped yet... I turned off the trap sender for a bit on one of the boxes but curious if there's a way to flush it.
Not sure which you are referring to. Regular notifications emails?

Yes

***** Nagios XI Alert *****

Nagios has detected a problem with this service.

Notification Type: PROBLEM

These guys... I think there's a persisting DB issue so running the repair now. Hopefully that fixes it. I was getting some MySQL connection errors in the report engine log and cycling MySQL didn't fix it.

Posted: **Tue Oct 03, 2017 1:10 pm**

This was the message in the log:

PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning: mysqli_real_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
<h3>Databse Error</h3>A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.<p>Run the following from the CLI as root to attempt to repair the DB:<br><pre>/usr/local/nagiosxi/scripts/repair_databases.sh</pre></p>

It's not moving anymore after the repair but still seem to be getting emails in intervals :/

Posted: **Tue Oct 03, 2017 1:16 pm**

Something is still up after the repair... still getting spammed with notification emails and the checks spamming time outs are green currently and the UI is updating fine now showing green...

Posted: **Tue Oct 03, 2017 1:19 pm**

It might make sense to turn this into a ticket via [email protected]

That said, you could try postfix flush

Posted: **Tue Oct 03, 2017 1:20 pm**

My guess is there were multiple issues.

Please run the following from the CLI to attempt to repair the database

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Additionally, if the system is really backed up, it could take a bit of time to catchup.

Posted: **Tue Oct 03, 2017 1:45 pm**

dwhitfield wrote:It might make sense to turn this into a ticket via [email protected]

That said, you could try postfix flush

Literally run that from the CLI??? "postfix flush"?

I'll give it a whirl... my cohort kicked off another repair despite telling him I did this server already (we have over 10) so will have to wait a bit.

Nagios Support Forum

ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!

Re: ndomod: Error writing to data sink!