Page 1 of 5

host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 1:13 pm
by pmithil7
Hi,

I'm facing a peculiar issue where my /var is increasing in an alrming manner. I found out that 'host-perfdata' & 'service-perdata' are increasing continuously in an alarming manner with 32G & 83G respectively and counting.

In the nagios.cfg file i even stopped processing performance data by putting the 'process_performance_data=0' and restarted nagios but still i see the two files growing. I'm not sure why is it still processing this data.

Can anyone help before it fills up my whole /var?

-Mithil

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 1:17 pm
by rkennedy
That's huge - it shouldn't be getting that big. Can you PM over a profile to myself, and dwhitfield for us to review? (Admin -> System profile -> Download profile)

Can you also show us the exact path / file name that seems to be filling up? What version of XI are you running?

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 1:33 pm
by pmithil7
Thank you for pitching in to help me. I have sent a PM to you as you requested. My files are in '/var/pnp4nagios/var/host-perfdata' & '/var/pnp4nagios/var/service=perfdata'.

[root@nagios-eqx-01 var]# du -sh *
3.5G host-perfdata
4.0K kohana
202M perfdata
3.7M perfdata.log
75G service-perfdata
4.0K spool
8.0K stats

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 1:34 pm
by dwhitfield
You posted in the XI forums, but you PMed a Core version. Could you send us a screenshot to clarify what you are running?

Thanks!

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 1:38 pm
by pmithil7
[root@nagios-eqx-01 bin]# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 10608 services.
Checked 635 hosts.
Checked 29 host groups.
Checked 1 service groups.
Checked 33 contacts.
Checked 52 contact groups.
Checked 77 commands.
Checked 1 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 635 hosts
Checked 374 service dependencies
Checked 0 host dependencies
Checked 1 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check


Yes. It is a core 4.0.8. Can you help here or should i post it again over there?

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 2:23 pm
by dwhitfield
Please wrap the output of the following four commands in separate code blocks:

Code: Select all

tail -50 /usr/local/nagios/var/perfdata.log
tail -50 /usr/local/nagios/var/npcd.log
cat /etc/*release
grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
I see you set processing to zero, but I wonder if that last one might help us understand how you got in this jam in the first place.

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 2:55 pm
by pmithil7
1) Attached is a file containing output of perfdata.log. I can explain that to you about why i have those errors in it. Some time back my /var got full due to a similar reason and my nagios server crashed and since then perfdata has stopped growing and only the 'host-perfdata' and 'service-perfdata' files were increasing. Also nothing would be going into the 'spool' directory. Since then i haven't been able to see data in the graphs. I haven't been able to figure out that yet and hence the errors are because of that.

2) I don't have a npcd.log file in my server.

3)[root@nagios-eqx-01 ~]# cat /etc/*release
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)

4) [root@nagios-eqx-01 ~]# grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
host_perfdata_file_processing_interval=15
service_perfdata_file_processing_interval=15

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 3:24 pm
by rkennedy
Can you also show us all your performance data related commands definitions?

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 4:30 pm
by pmithil7
The below are some of the definitions in the main configuration file for performance.
'''
# PROCESS PERFORMANCE DATA OPTION
# This determines whether or not Nagios will process performance
# data returned from service and host checks. If this option is
# enabled, host performance data will be processed using the
# host_perfdata_command (defined below) and service performance
# data will be processed using the service_perfdata_command (also
# defined below). Read the HTML docs for more information on
# performance data.
# Values: 1 = process performance data, 0 = do not process performance data

process_performance_data=0

# HOST AND SERVICE PERFORMANCE DATA PROCESSING COMMANDS
# These commands are run after every host and service check is
# performed. These commands are executed only if the
# enable_performance_data option (above) is set to 1. The command
# argument is the short name of a command definition that you
# define in your host configuration file. Read the HTML docs for
# more information on performance data.

#host_perfdata_command=process-host-perfdata
#service_perfdata_command=process-service-perfdata

# HOST AND SERVICE PERFORMANCE DATA FILES
# These files are used to store host and service performance data.
# Performance data is only written to these files if the
# enable_performance_data option (above) is set to 1.

host_perfdata_file=/usr/local/pnp4nagios/var/host-perfdata
service_perfdata_file=/usr/local/pnp4nagios/var/service-perfdata

# HOST AND SERVICE PERFORMANCE DATA FILE TEMPLATES
# These options determine what data is written (and how) to the
# performance data files. The templates may contain macros, special
# characters (\t for tab, \r for carriage return, \n for newline)
# and plain text. A newline is automatically added after each write
# to the performance data file. Some examples of what you can do are
# shown below.

host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$

# HOST AND SERVICE PERFORMANCE DATA FILE MODES
# This option determines whether or not the host and service
# performance data files are opened in write ("w") or append ("a")
# mode. If you want to use named pipes, you should use the special
# pipe ("p") mode which avoid blocking at startup, otherwise you will
# likely want the defult append ("a") mode.

host_perfdata_file_mode=a
service_perfdata_file_mode=a

# HOST AND SERVICE PERFORMANCE DATA FILE PROCESSING INTERVAL
# These options determine how often (in seconds) the host and service
# performance data files are processed using the commands defined
# below. A value of 0 indicates the files should not be periodically
# processed.

host_perfdata_file_processing_interval=15
service_perfdata_file_processing_interval=15

# HOST AND SERVICE PERFORMANCE DATA FILE PROCESSING COMMANDS
# These commands are used to periodically process the host and
# service performance data files. The interval at which the
# processing occurs is determined by the options above.

host_perfdata_file_processing_command=process-host-perfdata-file
service_perfdata_file_processing_command=process-service-perfdata-file

'''

Thanks.

Re: host & service perfdata increasing alarmingly

Posted: Thu Feb 09, 2017 5:18 pm
by rkennedy
Can you look up your command definitions for what's used to process the perf data? You may just be logging it to a file based on the name, which explains why it's filling up.

Code: Select all

process-host-perfdata-file
process-service-perfdata-file