Page 1 of 2
All files in var deleted on every reboot?
Posted: Mon Aug 29, 2016 11:49 am
by nuclear
Centos 7
Nagios 4
Everything in /usr/local/nagios/var/ is deleted after every reboot? Anybody know what is going on here?
I have to do this after every reboot to get nagios restarted.
Code: Select all
root@nms/usr/local/nagios/var# mkdir -p spool/checkresults
root@nms/usr/local/nagios/var# mkdir -p rw
root@nms/usr/local/nagios/var# chmod -R 775 spool
root@nms/usr/local/nagios/var# chmod 775 rw
root@nms/usr/local/nagios/var# chown -R nagios:nagios *
root@nms/usr/local/nagios/var# service nagios restart
Restarting nagios (via systemctl): [ OK ]
It makes all the right files when starting....
Code: Select all
root@nms/usr/local/nagios/var# ls
nagios.configtest nagios.lock nagios.log objects.cache objects.precache retention.dat rw/ spool/ status.dat
Then I reboot and find everything gone in var!
And this in the log telling me about the missing spool dir (among others)
Code: Select all
Aug 29 14:43:16 nms nagios: Starting nagios:
Aug 29 14:43:16 nms nagios: Nagios Core 4.2.0
Aug 29 14:43:16 nms nagios: Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Aug 29 14:43:16 nms nagios: Copyright (c) 1999-2009 Ethan Galstad
Aug 29 14:43:16 nms nagios: Last Modified: 08-01-2016
Aug 29 14:43:16 nms nagios: License: GPL
Aug 29 14:43:16 nms nagios: Website: https://www.nagios.org
Aug 29 14:43:16 nms nagios: Reading configuration data...
Aug 29 14:43:16 nms nagios: Error in configuration file '/usr/local/nagios/etc/nagios.cfg' - Line 457 (Check result path '/usr/local/nagios/var/spool/checkresults' is not a valid directory)
Aug 29 14:43:16 nms nagios: Error processing main config file!
Aug 29 14:43:16 nms systemd: nagios.service: control process exited, code=exited status=8
Aug 29 14:43:16 nms systemd: Failed to start LSB: Starts and stops the Nagios monitoring server.
Aug 29 14:43:16 nms systemd: Unit nagios.service entered failed state.
Aug 29 14:43:16 nms systemd: nagios.service failed.
Re: All files in var deleted on every reboot?
Posted: Mon Aug 29, 2016 1:24 pm
by tmcdonald
This is likely an issue with your init script, nagios config file, permissions, filesystem, or a combination of these factors.
Do you have .../var set up as a RAM disk by chance?
Re: All files in var deleted on every reboot?
Posted: Mon Aug 29, 2016 1:31 pm
by nuclear
tmcdonald wrote:This is likely an issue with your init script, nagios config file, permissions, filesystem, or a combination of these factors.
Do you have .../var set up as a RAM disk by chance?
That's what I thought... but no.
Code: Select all
root@nms/usr/local/nagios# ls -alh
total 24K
drwxr-xr-x. 9 root root 4.0K Aug 29 15:55 ./
drwxr-xr-x. 13 root root 4.0K Aug 4 15:40 ../
drwxr-xr-x. 2 nagios nagios 68 Aug 23 09:58 bin/
drwxrwxr-x. 3 nagios nagios 4.0K Aug 23 09:58 etc/
drwxr-xr-x. 2 root root 6 Aug 4 17:06 include/
drwxrwxr-x. 2 nagios nagios 4.0K Aug 23 09:58 libexec/
drwxrwxr-x. 2 nagios nagios 4.0K Aug 4 15:40 sbin/
drwxrwxr-x. 15 nagios nagios 4.0K Aug 4 17:06 share/
drwxr-xr-x. 4 nagios nagios 67 Aug 29 16:25 var/
root@nms/usr/local/nagios# df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 1.9T 2.5G 1.9T 1% /
devtmpfs 2.8G 0 2.8G 0% /dev
tmpfs 2.9G 0 2.9G 0% /dev/shm
tmpfs 2.9G 8.5M 2.8G 1% /run
tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup
/dev/md126p1 497M 247M 251M 50% /boot
tmpfs 575M 0 575M 0% /run/user/0
Re: All files in var deleted on every reboot?
Posted: Mon Aug 29, 2016 1:48 pm
by ssax
Please attach your /etc/init.d/nagios file so that we may review it.
Thank you
Re: All files in var deleted on every reboot?
Posted: Mon Aug 29, 2016 3:39 pm
by nuclear
Here is the init.d file. I have not modified since make.
Re: All files in var deleted on every reboot?
Posted: Mon Aug 29, 2016 3:47 pm
by nuclear
Ok, I did some additional testing....
1. Deleted /usr/local/nagios/var (what does it hurt since the contents were being deleted at reboot anyway?)
2. Reboot
3. Login and ll /usr/local/nagios and there is var back from the dead. Note time is the same as the reboot.
Code: Select all
root@nms~# ll /usr/local/nagios/
total 16
drwxr-xr-x. 2 nagios nagios 68 Aug 23 09:58 bin/
drwxrwxr-x. 3 nagios nagios 4096 Aug 23 09:58 etc/
drwxr-xr-x. 2 root root 6 Aug 4 17:06 include/
drwxrwxr-x. 2 nagios nagios 4096 Aug 23 09:58 libexec/
drwxrwxr-x. 2 nagios nagios 4096 Aug 4 15:40 sbin/
drwxrwxr-x. 15 nagios nagios 4096 Aug 4 17:06 share/
drwxr-xr-x. 2 nagios nagios 6 Aug 29 18:41 var/
chkconfig shows nagios is not starting with init.d, so I don't think init.d/nagios is doing it....
Code: Select all
nagios 0:off 1:off 2:off 3:off 4:off 5:off 6:off
Re: All files in var deleted on every reboot?
Posted: Mon Aug 29, 2016 3:50 pm
by nuclear
More testing....
Recompiled nagios with --prefix=/opt/nagios/ and /opt/nagios/var/* are intact after reboots! So what part of the OS would be doing such a thing in /usr/local/nagios/var only?
Re: All files in var deleted on every reboot?
Posted: Mon Aug 29, 2016 4:42 pm
by tmcdonald
Misconfigured log rotation? Misconfigured back+restore? Misconfigured security tool? Gremlins?
Honestly nothing in a standard install should be doing that at all. You might look into something like
auditd to track changes:
http://security.blogoverflow.com/2013/0 ... to-auditd/
http://www.tutorialspoint.com/unix_commands/auditd.htm
Re: All files in var deleted on every reboot?
Posted: Thu Jul 06, 2017 5:26 pm
by frankfe
I get the exact same behavior. I'm using Nagios 4.3.1 on CentOS 7.3 with SELinux disabled.
Re: All files in var deleted on every reboot?
Posted: Thu Jul 06, 2017 5:49 pm
by frankfe
Solved it! The culprit is systemd.
Systemd will wipe out temporary files on exit. If you want the temporary file structure to be recreated on boot, then you need to have a configuration file in the /etc/tmpfiles.d directory. I created a nagios.service.conf file in this directory with the following contents:
d /usr/local/nagios/var/spool 0775 nagios nagios --
d /usr/local/nagios/var/spool/checkresults 0775 nagios nagios - -
d /usr/local/nagios/var/rw 0775 nagios nagios - -
The temporary file structure is recreated at boot. I even have re-enabled SELinux, and Nagios is having no problems.
Here are the links that helped me figure it out:
https://serverfault.com/questions/54696 ... ra-machine
https://www.freedesktop.org/software/sy ... les.d.html#
https://www.freedesktop.org/software/sy ... files.html