host & service perfdata increasing alarmingly

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

host & service perfdata increasing alarmingly

Post by pmithil7 »

Hi,

I'm facing a peculiar issue where my /var is increasing in an alrming manner. I found out that 'host-perfdata' & 'service-perdata' are increasing continuously in an alarming manner with 32G & 83G respectively and counting.

In the nagios.cfg file i even stopped processing performance data by putting the 'process_performance_data=0' and restarted nagios but still i see the two files growing. I'm not sure why is it still processing this data.

Can anyone help before it fills up my whole /var?

-Mithil
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: host & service perfdata increasing alarmingly

Post by rkennedy »

That's huge - it shouldn't be getting that big. Can you PM over a profile to myself, and dwhitfield for us to review? (Admin -> System profile -> Download profile)

Can you also show us the exact path / file name that seems to be filling up? What version of XI are you running?
Former Nagios Employee
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

Thank you for pitching in to help me. I have sent a PM to you as you requested. My files are in '/var/pnp4nagios/var/host-perfdata' & '/var/pnp4nagios/var/service=perfdata'.

[root@nagios-eqx-01 var]# du -sh *
3.5G host-perfdata
4.0K kohana
202M perfdata
3.7M perfdata.log
75G service-perfdata
4.0K spool
8.0K stats
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: host & service perfdata increasing alarmingly

Post by dwhitfield »

You posted in the XI forums, but you PMed a Core version. Could you send us a screenshot to clarify what you are running?

Thanks!
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

[root@nagios-eqx-01 bin]# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 10608 services.
Checked 635 hosts.
Checked 29 host groups.
Checked 1 service groups.
Checked 33 contacts.
Checked 52 contact groups.
Checked 77 commands.
Checked 1 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 635 hosts
Checked 374 service dependencies
Checked 0 host dependencies
Checked 1 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check


Yes. It is a core 4.0.8. Can you help here or should i post it again over there?
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: host & service perfdata increasing alarmingly

Post by dwhitfield »

Please wrap the output of the following four commands in separate code blocks:

Code: Select all

tail -50 /usr/local/nagios/var/perfdata.log
tail -50 /usr/local/nagios/var/npcd.log
cat /etc/*release
grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
I see you set processing to zero, but I wonder if that last one might help us understand how you got in this jam in the first place.
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

1) Attached is a file containing output of perfdata.log. I can explain that to you about why i have those errors in it. Some time back my /var got full due to a similar reason and my nagios server crashed and since then perfdata has stopped growing and only the 'host-perfdata' and 'service-perfdata' files were increasing. Also nothing would be going into the 'spool' directory. Since then i haven't been able to see data in the graphs. I haven't been able to figure out that yet and hence the errors are because of that.

2) I don't have a npcd.log file in my server.

3)[root@nagios-eqx-01 ~]# cat /etc/*release
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)

4) [root@nagios-eqx-01 ~]# grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
host_perfdata_file_processing_interval=15
service_perfdata_file_processing_interval=15
Attachments
perfdata log.txt
perfdata.log output
(7.32 KiB) Downloaded 248 times
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: host & service perfdata increasing alarmingly

Post by rkennedy »

Can you also show us all your performance data related commands definitions?
Former Nagios Employee
pmithil7
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Post by pmithil7 »

The below are some of the definitions in the main configuration file for performance.
'''
# PROCESS PERFORMANCE DATA OPTION
# This determines whether or not Nagios will process performance
# data returned from service and host checks. If this option is
# enabled, host performance data will be processed using the
# host_perfdata_command (defined below) and service performance
# data will be processed using the service_perfdata_command (also
# defined below). Read the HTML docs for more information on
# performance data.
# Values: 1 = process performance data, 0 = do not process performance data

process_performance_data=0

# HOST AND SERVICE PERFORMANCE DATA PROCESSING COMMANDS
# These commands are run after every host and service check is
# performed. These commands are executed only if the
# enable_performance_data option (above) is set to 1. The command
# argument is the short name of a command definition that you
# define in your host configuration file. Read the HTML docs for
# more information on performance data.

#host_perfdata_command=process-host-perfdata
#service_perfdata_command=process-service-perfdata

# HOST AND SERVICE PERFORMANCE DATA FILES
# These files are used to store host and service performance data.
# Performance data is only written to these files if the
# enable_performance_data option (above) is set to 1.

host_perfdata_file=/usr/local/pnp4nagios/var/host-perfdata
service_perfdata_file=/usr/local/pnp4nagios/var/service-perfdata

# HOST AND SERVICE PERFORMANCE DATA FILE TEMPLATES
# These options determine what data is written (and how) to the
# performance data files. The templates may contain macros, special
# characters (\t for tab, \r for carriage return, \n for newline)
# and plain text. A newline is automatically added after each write
# to the performance data file. Some examples of what you can do are
# shown below.

host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$

# HOST AND SERVICE PERFORMANCE DATA FILE MODES
# This option determines whether or not the host and service
# performance data files are opened in write ("w") or append ("a")
# mode. If you want to use named pipes, you should use the special
# pipe ("p") mode which avoid blocking at startup, otherwise you will
# likely want the defult append ("a") mode.

host_perfdata_file_mode=a
service_perfdata_file_mode=a

# HOST AND SERVICE PERFORMANCE DATA FILE PROCESSING INTERVAL
# These options determine how often (in seconds) the host and service
# performance data files are processed using the commands defined
# below. A value of 0 indicates the files should not be periodically
# processed.

host_perfdata_file_processing_interval=15
service_perfdata_file_processing_interval=15

# HOST AND SERVICE PERFORMANCE DATA FILE PROCESSING COMMANDS
# These commands are used to periodically process the host and
# service performance data files. The interval at which the
# processing occurs is determined by the options above.

host_perfdata_file_processing_command=process-host-perfdata-file
service_perfdata_file_processing_command=process-service-perfdata-file

'''

Thanks.
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: host & service perfdata increasing alarmingly

Post by rkennedy »

Can you look up your command definitions for what's used to process the perf data? You may just be logging it to a file based on the name, which explains why it's filling up.

Code: Select all

process-host-perfdata-file
process-service-perfdata-file
Former Nagios Employee
Locked