Perf data lost

[ctretelea]

Hello,
After I reboot the server I lost all my perf data. How do I do it persistent?

thanks.

[npolovenko]

Hello, @ctretelea, Could you clarify on what do you mean by the lost perfdata? Did Nagios start new graphs for all the service checks? Or did the graphs stop working all at once? Can you show us the output of:

Code: Select all

ls -l /usr/local/nagios/share/perfdata

[bolson]

Hello,

This document should provide you with exactly what you're looking for:

https://assets.nagios.com/downloads/nag ... fdata.html

See the section titled "Writing Performance Data To Files"

Let us know if this answers your question. Thank you!

[ctretelea]

Hello, @ctretelea, Could you clarify on what do you mean by the lost perfdata? Did Nagios start new graphs for all the service checks? Or did the graphs stop working all at once? Can you show us the output of:

Code: Select all

ls -l /usr/local/nagios/share/perfdata

So, what happen is: I lost all performance graph that I had before to reboot the server after reboot I see the graphs but I don't see anything before reboot.
Can I do it persistent to reboot?

Code: Select all

[user@ip ~]$ ls -l /usr/local/nagios/share/perfdata/
total 152580
drwxrwxr-x 2 nagios nagios      4096 Feb 19 12:46 132.147.160.53
drwxrwxr-x 2 nagios nagios      4096 Feb 19 12:46 192.168.2.82
drwxrwxr-x 2 nagios nagios     12288 Feb 21 13:49 lxablapp
drwxrwxr-x 2 nagios nagios      4096 Feb 21 13:49 lxabldb
drwxrwxr-x 2 nagios nagios      4096 Feb 21 13:49 ntablrep
drwxrwxr-x 2 nagios nagios      4096 Feb 21 13:49 ntablts
...

[npolovenko]

@ctretelea, Can you remember one of the services that lost a graph after the server restarted? Go to the /usr/local/nagios/share/perfdata folder, then find a subfolder that has a name of the host, and upload the rrd file inside it. RRD file will have the same name as a service check. We'll be able to look into it and see if the perfdata is still there if it somehow got deleted.

[ctretelea]

@npolovenko, I see that all services after reboot lost their perf data before reboot. Now all services a perf data starting with reboot day. I check rrd file but I cannot say if it's missing something in the file or not. I attached it to this message maybe you will understand.

[scottwilkerson]

The only logical reason I can think that this could happen is if you somehow created a ramdisk or tmpfs for /usr/local/nagios/share/perfdata

what is the output of

Code: Select all

df -h

[ctretelea]

@scottwilkerson, I have set up a ramdisk. Can that be the cause?

Code: Select all

[root@ip]# df -h
Filesystem                      Size  Used Avail Use% Mounted on
/dev/xvda1                      100G   42G   59G  42% /
devtmpfs                        7.8G     0  7.8G   0% /dev
tmpfs                           7.8G     0  7.8G   0% /dev/shm
tmpfs                           7.8G  585M  7.2G   8% /run
tmpfs                           7.8G     0  7.8G   0% /sys/fs/cgroup
s3fs                            256T     0  256T   0% /store/backups/nagiosxi1
/dev/mapper/data_vg1-mysql_lv1   20G  7.8G   11G  42% /var/lib/mysql
/dev/mapper/data_vg1-perf_lv1   4.8G  3.1G  1.6G  67% /apps/perf
tmpfs                           2.0G   38M  2.0G   2% /var/nagiosramdisk
tmpfs                           1.6G     0  1.6G   0% /run/user/1001
tmpfs                           1.6G     0  1.6G   0% /run/user/0
tmpfs                           1.6G     0  1.6G   0% /run/user/1000

Thanks.

[scottwilkerson]

via our normal directions it would not affect that directory. Can you run the following:

Code: Select all

ls -al /usr/local/nagios/share/

Also, a backup should restore those folders

Finally, you may want to also test to verify it does it every time, or was a tmpfs somehow created just one time

[ctretelea]

Hello @scottwilkerson,
I run and this is results:

Code: Select all

[root@ip]# ls -al /usr/local/nagios/share/
total 156
drwxrwxr-x 17 nagios nagios 4096 Mar  2 08:34 .
drwxr-xr-x 11 root   root    107 Jul 18  2017 ..
drwxr-xr-x  2 root   root   4096 Feb 17  2017 ajaxterm
drwxrwxr-x  4 nagios nagios   85 Apr  5  2017 angularjs
drwxrwxr-x  3 nagios nagios   16 Feb 17  2017 bootstrap-3.3.0
-rw-rw-r--  1 nagios nagios  576 Apr  5  2017 config.inc.php
drwxrwxr-x  2 nagios nagios 4096 Apr  5  2017 contexthelp
drwxrwxr-x  2 nagios nagios   22 Apr  5  2017 d3
drwxrwxr-x  3 nagios nagios   19 Feb 17  2017 docs
-rw-rw-r--  1 nagios nagios  879 Apr  5  2017 graph-header.html
-rw-rw-r--  1 nagios nagios 4136 Apr  5  2017 histogram-form.html
-rw-rw-r--  1 nagios nagios 5579 Apr  5  2017 histogram-graph.html
-rw-rw-r--  1 nagios nagios 2855 Apr  5  2017 histogram.html
-rw-rw-r--  1 nagios nagios 2372 Apr  5  2017 histogram-links.html
drwxr-xr-x  3 nagios nagios 4096 Apr  5  2017 images
drwxrwxr-x  3 nagios nagios   62 Apr  5  2017 includes
-rw-rw-r--  1 nagios nagios 1913 Apr  5  2017 index.php
-rw-rw-r--  1 nagios nagios 1660 Apr  5  2017 infobox.html
drwxrwxr-x  2 nagios nagios 4096 Apr  5  2017 js
-rw-rw-r--  1 nagios nagios 1177 Apr  5  2017 jsonquery.html
drwxr-xr-x  4 root   root     24 Feb 17  2017 locale
-rw-rw-r--  1 nagios nagios 8621 Apr  5  2017 main.php
-rw-rw-r--  1 nagios nagios 2015 Apr  5  2017 map-directive.html
-rw-rw-r--  1 nagios nagios 3824 Apr  5  2017 map-form.html
-rw-rw-r--  1 nagios nagios  647 Apr  5  2017 map-links.html
-rw-rw-r--  1 nagios nagios 3310 Apr  5  2017 map.php
-rw-rw-r--  1 nagios nagios 1723 Apr  5  2017 map-popup.html
drwxrwxr-x  2 nagios nagios    6 Feb 17  2017 media
lrwxrwxrwx  1 nagios nagios   20 Feb 19 10:39 perfdata -> /apps/perf/perfdata/
drwxr-xr-x 11 nagios nagios 4096 Sep 27 15:31 pnp
-rw-rw-r--  1 nagios nagios   26 Apr  5  2017 robots.txt
-rw-rw-r--  1 nagios nagios 6193 Apr  5  2017 side.php
drwxrwxr-x  2 nagios nagios   24 Apr  5  2017 spin
drwxrwxr-x  2 nagios nagios    6 Feb 17  2017 ssi
drwxr-xr-x  2 nagios nagios 4096 Apr  5  2017 stylesheets
-rw-r--r--  1 root   root   5612 Feb 19 10:40 tmp_xi_vars.cfg
-rw-rw-r--  1 nagios nagios 4243 Apr  5  2017 trends-form.html
-rw-rw-r--  1 nagios nagios 4256 Apr  5  2017 trends-graph.html
-rw-rw-r--  1 nagios nagios  852 Apr  5  2017 trends-host-yaxis.html
-rw-rw-r--  1 nagios nagios 2990 Apr  5  2017 trends.html
-rw-rw-r--  1 nagios nagios 3586 Apr  5  2017 trends-links.html
-rw-rw-r--  1 nagios nagios 1065 Apr  5  2017 trends-service-yaxis.html

Code: Select all

[root@ip-10-60-0-29 centos]# ls -al /usr/local/nagios/share/perfdata/
total 152656
drwxrwxr-x 307 nagios nagios     12288 Feb 22 11:16 .
drwxr-xr-x   4 root   root        4096 Feb 19 12:46 ..
drwxrwxr-x   2 nagios nagios      4096 Feb 19 12:46 132.147.160.53
drwxrwxr-x   2 nagios nagios      4096 Feb 19 12:46 192.168.2.82
drwxrwxr-x   2 nagios nagios     12288 Mar  2 08:38 lxablapp
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 lxabldb
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ntablrep
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ntablts
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 Acem
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 Acem_-_Cognos_TS
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 Acem_-_eliteapp
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 Acem_-_elitedb
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 Acem_-_Iguana
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 Acem_-_Optio_Loftware
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ACE-ntaceapp
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ACE-ntaceapp2
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ACE-ntacecnsprd
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:37 ACE-ntacecnstest
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ACE-ntacedb
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ACE-ntacedb1
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ACE-ntacedb2
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 ACE-ntacerep
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:37 ACE-ntacets
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 AMG
drwxrwxr-x   2 nagios nagios      4096 Feb 19 12:47 AMG-amfedi
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 AMG-amgedi
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:37 AMG-amgloftware
drwxrwxr-x   2 nagios nagios     12288 Mar  2 08:38 AMG-amgredhat
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 AWS-Central
drwxrwxr-x   2 nagios nagios      4096 Mar  2 08:38 AWS-OpenVPN
...

Yes I do backup but how to merge old one with new one?