nagios.cmd changes

[onegative]

@npolovenko

So yesterday just before I left for the day the files looked as follows:

Code: Select all

[root@dcom-nagiosxi-p1 rw]# ll
total 0
prw-rw---- 1 nagios nagcmd 0 Mar  4 13:24 nagios.cmd
srw-rw---- 1 nagios nagcmd 0 Mar  4 12:12 nagios.qh

So I looked first thing this morning and found the following:

Code: Select all

[root@dcom-nagiosxi-p1 rw]# ll
total 4
-rw-r--r-- 1 nagios nagcmd 136 Mar  5 05:01 nagios.cmd
srw-rw---- 1 nagios nagcmd   0 Mar  5 05:00 nagios.qh

Seeing that the file now had changed and contained the last data sent to it which was a Recurring Downtime:

Code: Select all

[root@dcom-nagiosxi-p1 rw]# cat nagios.cmd
[1551790863] SCHEDULE_HOST_SVC_DOWNTIME;epic-obapp-t161;1551952500;1551960000;1;0;7500;wessauer;AUTO: Windows Updates 2 weeks per month

So the above Scheduled Downtime is a result of a Recurring Downtime for a Host Group:

epicOnBaseTestRecurringDowntime.png

I also checked and 05:00 is when my daily scheduled sftp backup runs:

nagiosxiBackup20190305.png

Here is the listing of the /usr/local/nagios/var directory:

Code: Select all

[root@dcom-nagiosxi-p1 rw]# ls -l /usr/local/nagios/var/
total 418088
drwxrwxr-x. 2 nagios nagios     24576 Mar  4 23:59 archives
-rw-r--r--  1 nagios nagios         0 Mar  5 06:49 host-perfdata
-rw-r--r--. 1 nagios nagios        34 Mar  5 06:39 nagios.configtest
-rw-r--r--  1 nagios nagios         6 Mar  5 06:39 nagios.lock
-rw-r-----  1 nagios nagios 143246850 Mar  5 06:49 nagios.log
-rw-r--r--  1 nagios nagios         5 Feb 27 12:37 ndo2db.lock
-rw-r--r--  1 nagios nagios         5 Nov  2 10:35 ndo2db.lock.bak
-rw-r--r--  1 nagios nagios         0 Mar  5 06:39 ndomod.tmp
srwxr-xr-x  1 nagios nagios         0 Feb 27 12:37 ndo.sock
srwxr-xr-x  1 nagios nagios         0 Nov  2 10:35 ndo.sock.bak
-rw-r--r--  1 nagios nagios   3907993 Mar  5 06:41 npcd.log
-rw-r--r--. 1 nagios nagios  10485827 Jan 11 03:27 npcd.log.old
-rw-r--r--. 1 nagios nagios  37550845 Mar  5 06:39 objects.cache
-rw-r--r--. 1 nagios nagios  37550845 Mar  5 06:39 objects.precache
-rw-rw-r--. 1 nagios nagios    594514 Mar  5 06:41 perfdata.log
-rw-------  1 nagios nagios  82280171 Mar  5 06:39 retention.dat
drwxrwsr-x. 2 nagios nagcmd        39 Mar  5 06:41 rw
-rw-r--r--  1 nagios nagios    102924 Mar  5 06:49 service-perfdata
drwxr-xr-x. 5 nagios nagios        52 Sep 12  2017 spool
drwxr-xr-x. 2 nagios nagios        21 Mar  5 06:49 stats
-rw-rw-r--  1 nagios nagios  82907755 Mar  5 06:49 status.dat

So when I hit Apply Configuration it immediately removes the nagios.cmd file and then takes approx. 1 and a half minutes to returned...please observe the small for i loop I created to show the exact time:

Code: Select all

Seconds: 03
prw-rw---- 1 nagios nagcmd 0 Mar  5 06:41 nagios.cmd
Seconds: 04
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 05
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 06
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 07
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 08
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 09
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 10
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 11
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 12
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 13
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 14
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 15
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 16
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 17
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 18
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 19
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 20
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 21
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 22
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 23
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 24
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 25
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 26
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 27
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 28
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 29
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 30
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 31
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 32
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 33
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 34
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 35
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 36
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 37
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 38
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 39
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 40
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 41
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 42
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 43
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 44
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 45
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 46
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 47
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 48
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 49
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 50
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 51
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 52
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 53
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 54
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 55
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 56
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 57
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 58
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 59
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 00
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 01
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 02
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 03
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 04
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 05
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 06
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 07
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 08
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 09
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 10
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 11
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 12
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 13
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 14
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 15
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 16
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 17
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 18
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 19
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 20
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 21
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 22
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 23
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 24
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh
Seconds: 26
prw-rw---- 1 nagios nagcmd 0 Mar  5 07:11 nagios.cmd
srw-rw---- 1 nagios nagcmd 0 Mar  5 07:10 nagios.qh

So it appears that the backup might be causing this issue or Scheduled Downtimes??? How is this possible?

Let me know and thanks,
Danny

[npolovenko]

@onegative, I talked to the devs and figured that most likely the scheduled downtime changed the cmd file. However, something might've removed the pipe even before the scheduled downtime. What version of XI are you running? We've had some bug fixes related to the nagios.cmd in Core 4.4.2. If you're not on the latest version of XI please consider upgrading.

[onegative]

Well I had planned on upgrading here in a week or so...perhaps I will go ahead and do it...
Currently on this version:

Nagios Core 4.2.4
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 12-07-2016
License: GPL

After upgrading I will determine if the problem continues...

Danny

[npolovenko]

@onegative, Sounds good, I will leave this thread open.

[onegative]

@npolovenko


So just FYI...I changed my Backup Schedule from 06:00 to 07:00 and when I looked this morning the file had changed during that same time period...it is following the backup process....how is that possible?

Code: Select all

[nagios@dcom-nagiosxi-p1 rw]$ ll
total 4
-rw-r--r-- 1 nagios nagcmd  0 Mar  6 07:01 nagios.cmd
srw-rw---- 1 nagios nagcmd  0 Mar  6 07:00 nagios.qh

Thanks,
Danny

[npolovenko]

@onegative, This is interesting. We looked into the backup component source code and it doesn't look like it's using or changing the cmd file. I would suggest disabling backups for 1 night and seeing if the cmd file stays with the right permissions. That way we could really narrow down this problem.
Also, upgrading XI could automatically fix this problem since there were a number of bug fixes in later releases.

[onegative]

@npolovenko

OK, I will disable the Scheduled Backup and look in the morning.

I will be upgrading the production system this weekend while no users will be utilizing the system...hopefully it will go well.

Thanks,
Danny

[npolovenko]

@onegative , Sounds good, I'm interested to hear an update on this tomorrow.
On the weekend, try to back-up the XI server or create a VM snapshot before upgrading. I also suggest upgrading from the command line instead of the GUI:
https://assets.nagios.com/downloads/nag ... ctions.pdf

[onegative]

@npolovenko

Well it did not get changed even though it timestamp updated at 02:00 hours...since I do not know/understand how all the components interact with this file I am unsure why/how it gets new timestamp...

Anyway will be upgrading the production instance this weekend...

Thanks,
Danny

Code: Select all

[nagios@dcom-nagiosxi-p1 rw]$ ll
total 0
prw-rw---- 1 nagios nagcmd 0 Mar  6 10:51 nagios.cmd
srw-rw---- 1 nagios nagcmd 0 Mar  6 07:51 nagios.qh
[nagios@dcom-nagiosxi-p1 rw]$ ll
total 0
prw-rw---- 1 nagios nagcmd 0 Mar  7 02:01 nagios.cmd
srw-rw---- 1 nagios nagcmd 0 Mar  6 15:23 nagios.qh

[npolovenko]

@onegative, cmd file can be used by multiple processes that send commands to the Nagios core The most important part is that the cmd file did not lose the pipe permissions this time. I believe XI upgrade should take of this backup component issue. I will leave this ticket opened till we hear from you on the result.