Engine hosed after network disconnect

[abrist]

... Hey guys!

NDO is resilient enough to reconnect the data sink once the db is back. The issue here may be how long it is down. Checkresults will backup as they cannot be written to the historical db, and if the db is down long enough, the engine may have an uphill battle between new checks and cached old checks. It could take a considerable amount of time for everything to normalize if the outage was of a decently long duration (or the number of checks per 5 minutes is very high)..

[vAJ]

Where might I find indicators of such a backlog? Any known error messages and from which log files?

[slansing]

I would look in the following logs:

Code: Select all

/var/log/messages

Code: Select all

/var/log/mysqld.log

You may also find some useful information from what happened when the outage hit your network in:

Code: Select all

/usr/local/nagios/var/nagios.log

You can also find the nagios log archives in the above directory.

In addition, do you have logging set up on the remote mysql server? At the least a log showing when mysqld is initiated? Possibly the syslog?

I'd also check to see if you have an inordinate amount of temp check* files in /tmp/. Or in:

Code: Select all

/usr/local/nagios/var/spool/checkresults/

[vAJ]

Well, we just found an issue yesterday where logging was not rotating. /var/log/messages is about 146MB

[slansing]

Dear Lord that is a lot. Can you run the following and share the output:

Code: Select all

cat /var/lib/logrotate.status | grep messages

[vAJ]

2013-10-6

[sreinhardt]

So that 146mb after rotation and within the two days or prior to rotation and some extended period of time? As you said, thats a pretty big log! Hopefully clearing that up some should let you see ndo messages just a bit more clearly.

[vAJ]

Log still hadn't rotated this morning, LInux admin tells me issued a rotate command but it said it wasn't necessary. So he forced it.

Rotate schedule was set to weekly. we changed to daily. Will know more tomorrow.

On another note, we had another stab at a big L3 network change last night and the engine did not fall over. Only difference seems to be that I rebooted the box yesterday and it had very little buffer/cache used on the box. Usually it's taking up 19GB in cache on a box with 24GB physical mem.

[abrist]

I haven't noticed any big upstream kernel bugs with caching recently. I guess we will not have a good test until ram fills up with disk cache again. Let us know what happened with the log rotation today.

[vAJ]

OK. Log rotation is good at 24hrs now.