All files in var deleted on every reboot?

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
nuclear
Posts: 5
Joined: Mon Aug 29, 2016 11:28 am

All files in var deleted on every reboot?

Post by nuclear »

Centos 7
Nagios 4

Everything in /usr/local/nagios/var/ is deleted after every reboot? Anybody know what is going on here?

I have to do this after every reboot to get nagios restarted.

Code: Select all

root@nms/usr/local/nagios/var# mkdir -p spool/checkresults
root@nms/usr/local/nagios/var# mkdir -p rw
root@nms/usr/local/nagios/var# chmod -R 775 spool
root@nms/usr/local/nagios/var# chmod 775 rw
root@nms/usr/local/nagios/var# chown -R nagios:nagios *
root@nms/usr/local/nagios/var# service nagios restart
Restarting nagios (via systemctl):                         [  OK  ]
It makes all the right files when starting....

Code: Select all

root@nms/usr/local/nagios/var# ls
nagios.configtest  nagios.lock  nagios.log  objects.cache  objects.precache  retention.dat  rw/  spool/  status.dat
Then I reboot and find everything gone in var!

And this in the log telling me about the missing spool dir (among others)

Code: Select all

Aug 29 14:43:16 nms nagios: Starting nagios:
Aug 29 14:43:16 nms nagios: Nagios Core 4.2.0
Aug 29 14:43:16 nms nagios: Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Aug 29 14:43:16 nms nagios: Copyright (c) 1999-2009 Ethan Galstad
Aug 29 14:43:16 nms nagios: Last Modified: 08-01-2016
Aug 29 14:43:16 nms nagios: License: GPL
Aug 29 14:43:16 nms nagios: Website: https://www.nagios.org
Aug 29 14:43:16 nms nagios: Reading configuration data...
Aug 29 14:43:16 nms nagios: Error in configuration file '/usr/local/nagios/etc/nagios.cfg' - Line 457 (Check result path '/usr/local/nagios/var/spool/checkresults' is not a valid directory)
Aug 29 14:43:16 nms nagios: Error processing main config file!
Aug 29 14:43:16 nms systemd: nagios.service: control process exited, code=exited status=8
Aug 29 14:43:16 nms systemd: Failed to start LSB: Starts and stops the Nagios monitoring server.
Aug 29 14:43:16 nms systemd: Unit nagios.service entered failed state.
Aug 29 14:43:16 nms systemd: nagios.service failed.
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: All files in var deleted on every reboot?

Post by tmcdonald »

This is likely an issue with your init script, nagios config file, permissions, filesystem, or a combination of these factors.

Do you have .../var set up as a RAM disk by chance?
Former Nagios employee
nuclear
Posts: 5
Joined: Mon Aug 29, 2016 11:28 am

Re: All files in var deleted on every reboot?

Post by nuclear »

tmcdonald wrote:This is likely an issue with your init script, nagios config file, permissions, filesystem, or a combination of these factors.

Do you have .../var set up as a RAM disk by chance?

That's what I thought... but no.

Code: Select all

root@nms/usr/local/nagios# ls -alh
total 24K
drwxr-xr-x.  9 root   root   4.0K Aug 29 15:55 ./
drwxr-xr-x. 13 root   root   4.0K Aug  4 15:40 ../
drwxr-xr-x.  2 nagios nagios   68 Aug 23 09:58 bin/
drwxrwxr-x.  3 nagios nagios 4.0K Aug 23 09:58 etc/
drwxr-xr-x.  2 root   root      6 Aug  4 17:06 include/
drwxrwxr-x.  2 nagios nagios 4.0K Aug 23 09:58 libexec/
drwxrwxr-x.  2 nagios nagios 4.0K Aug  4 15:40 sbin/
drwxrwxr-x. 15 nagios nagios 4.0K Aug  4 17:06 share/
drwxr-xr-x.  4 nagios nagios   67 Aug 29 16:25 var/
root@nms/usr/local/nagios# df
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  1.9T  2.5G  1.9T   1% /
devtmpfs                 2.8G     0  2.8G   0% /dev
tmpfs                    2.9G     0  2.9G   0% /dev/shm
tmpfs                    2.9G  8.5M  2.8G   1% /run
tmpfs                    2.9G     0  2.9G   0% /sys/fs/cgroup
/dev/md126p1             497M  247M  251M  50% /boot
tmpfs                    575M     0  575M   0% /run/user/0
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: All files in var deleted on every reboot?

Post by ssax »

Please attach your /etc/init.d/nagios file so that we may review it.

Thank you
nuclear
Posts: 5
Joined: Mon Aug 29, 2016 11:28 am

Re: All files in var deleted on every reboot?

Post by nuclear »

Here is the init.d file. I have not modified since make.
Attachments
initd-nagios.txt
(9.69 KiB) Downloaded 278 times
nuclear
Posts: 5
Joined: Mon Aug 29, 2016 11:28 am

Re: All files in var deleted on every reboot?

Post by nuclear »

Ok, I did some additional testing....
1. Deleted /usr/local/nagios/var (what does it hurt since the contents were being deleted at reboot anyway?)
2. Reboot
3. Login and ll /usr/local/nagios and there is var back from the dead. Note time is the same as the reboot.

Code: Select all

root@nms~# ll /usr/local/nagios/
total 16
drwxr-xr-x.  2 nagios nagios   68 Aug 23 09:58 bin/
drwxrwxr-x.  3 nagios nagios 4096 Aug 23 09:58 etc/
drwxr-xr-x.  2 root   root      6 Aug  4 17:06 include/
drwxrwxr-x.  2 nagios nagios 4096 Aug 23 09:58 libexec/
drwxrwxr-x.  2 nagios nagios 4096 Aug  4 15:40 sbin/
drwxrwxr-x. 15 nagios nagios 4096 Aug  4 17:06 share/
drwxr-xr-x.  2 nagios nagios    6 Aug 29 18:41 var/
chkconfig shows nagios is not starting with init.d, so I don't think init.d/nagios is doing it....

Code: Select all

nagios          0:off   1:off   2:off   3:off   4:off   5:off   6:off
nuclear
Posts: 5
Joined: Mon Aug 29, 2016 11:28 am

Re: All files in var deleted on every reboot?

Post by nuclear »

More testing....

Recompiled nagios with --prefix=/opt/nagios/ and /opt/nagios/var/* are intact after reboots! So what part of the OS would be doing such a thing in /usr/local/nagios/var only?
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: All files in var deleted on every reboot?

Post by tmcdonald »

Misconfigured log rotation? Misconfigured back+restore? Misconfigured security tool? Gremlins?

Honestly nothing in a standard install should be doing that at all. You might look into something like auditd to track changes:

http://security.blogoverflow.com/2013/0 ... to-auditd/
http://www.tutorialspoint.com/unix_commands/auditd.htm
Former Nagios employee
frankfe
Posts: 2
Joined: Thu Jul 06, 2017 5:22 pm

Re: All files in var deleted on every reboot?

Post by frankfe »

I get the exact same behavior. I'm using Nagios 4.3.1 on CentOS 7.3 with SELinux disabled.
frankfe
Posts: 2
Joined: Thu Jul 06, 2017 5:22 pm

Re: All files in var deleted on every reboot?

Post by frankfe »

Solved it! The culprit is systemd.

Systemd will wipe out temporary files on exit. If you want the temporary file structure to be recreated on boot, then you need to have a configuration file in the /etc/tmpfiles.d directory. I created a nagios.service.conf file in this directory with the following contents:

d /usr/local/nagios/var/spool 0775 nagios nagios --
d /usr/local/nagios/var/spool/checkresults 0775 nagios nagios - -
d /usr/local/nagios/var/rw 0775 nagios nagios - -

The temporary file structure is recreated at boot. I even have re-enabled SELinux, and Nagios is having no problems. :D

Here are the links that helped me figure it out:
https://serverfault.com/questions/54696 ... ra-machine
https://www.freedesktop.org/software/sy ... les.d.html#
https://www.freedesktop.org/software/sy ... files.html
Locked