host & service perfdata increasing alarmingly

An open discussion forum for obtaining help with Nagios Core. Nagios Core users of all experience levels are welcome here. Subforum have been created for the discussion of Nagios Core and Nagios Plugin development.

NOTE: The SourceForge.net mailing lists have been deprecated in favor of this forum in order to expedite support and provide additional features not available on the old mailing list.

host & service perfdata increasing alarmingly

Postby pmithil7 » Thu Feb 09, 2017 1:13 pm

Hi,

I'm facing a peculiar issue where my /var is increasing in an alrming manner. I found out that 'host-perfdata' & 'service-perdata' are increasing continuously in an alarming manner with 32G & 83G respectively and counting.

In the nagios.cfg file i even stopped processing performance data by putting the 'process_performance_data=0' and restarted nagios but still i see the two files growing. I'm not sure why is it still processing this data.

Can anyone help before it fills up my whole /var?

-Mithil
pmithil7
 
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Postby rkennedy » Thu Feb 09, 2017 1:17 pm

That's huge - it shouldn't be getting that big. Can you PM over a profile to myself, and dwhitfield for us to review? (Admin -> System profile -> Download profile)

Can you also show us the exact path / file name that seems to be filling up? What version of XI are you running?
rkennedy
 
Posts: 6545
Joined: Mon Oct 05, 2015 11:45 am

Re: host & service perfdata increasing alarmingly

Postby pmithil7 » Thu Feb 09, 2017 1:33 pm

Thank you for pitching in to help me. I have sent a PM to you as you requested. My files are in '/var/pnp4nagios/var/host-perfdata' & '/var/pnp4nagios/var/service=perfdata'.

[root@nagios-eqx-01 var]# du -sh *
3.5G host-perfdata
4.0K kohana
202M perfdata
3.7M perfdata.log
75G service-perfdata
4.0K spool
8.0K stats
pmithil7
 
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Postby dwhitfield » Thu Feb 09, 2017 1:34 pm

You posted in the XI forums, but you PMed a Core version. Could you send us a screenshot to clarify what you are running?

Thanks!
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 2109
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: host & service perfdata increasing alarmingly

Postby pmithil7 » Thu Feb 09, 2017 1:38 pm

[root@nagios-eqx-01 bin]# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 10608 services.
Checked 635 hosts.
Checked 29 host groups.
Checked 1 service groups.
Checked 33 contacts.
Checked 52 contact groups.
Checked 77 commands.
Checked 1 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 635 hosts
Checked 374 service dependencies
Checked 0 host dependencies
Checked 1 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check


Yes. It is a core 4.0.8. Can you help here or should i post it again over there?
pmithil7
 
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Postby dwhitfield » Thu Feb 09, 2017 2:23 pm

Please wrap the output of the following four commands in separate code blocks:

Code: Select all
tail -50 /usr/local/nagios/var/perfdata.log
tail -50 /usr/local/nagios/var/npcd.log
cat /etc/*release
grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg


I see you set processing to zero, but I wonder if that last one might help us understand how you got in this jam in the first place.
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 2109
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: host & service perfdata increasing alarmingly

Postby pmithil7 » Thu Feb 09, 2017 2:55 pm

1) Attached is a file containing output of perfdata.log. I can explain that to you about why i have those errors in it. Some time back my /var got full due to a similar reason and my nagios server crashed and since then perfdata has stopped growing and only the 'host-perfdata' and 'service-perfdata' files were increasing. Also nothing would be going into the 'spool' directory. Since then i haven't been able to see data in the graphs. I haven't been able to figure out that yet and hence the errors are because of that.

2) I don't have a npcd.log file in my server.

3)[root@nagios-eqx-01 ~]# cat /etc/*release
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)
CentOS release 6.6 (Final)

4) [root@nagios-eqx-01 ~]# grep perfdata_file_processing_interval /usr/local/nagios/etc/nagios.cfg
host_perfdata_file_processing_interval=15
service_perfdata_file_processing_interval=15
Attachments
perfdata log.txt
perfdata.log output
(7.32 KiB) Downloaded 8 times
pmithil7
 
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Postby rkennedy » Thu Feb 09, 2017 3:24 pm

Can you also show us all your performance data related commands definitions?
rkennedy
 
Posts: 6545
Joined: Mon Oct 05, 2015 11:45 am

Re: host & service perfdata increasing alarmingly

Postby pmithil7 » Thu Feb 09, 2017 4:30 pm

The below are some of the definitions in the main configuration file for performance.
'''
# PROCESS PERFORMANCE DATA OPTION
# This determines whether or not Nagios will process performance
# data returned from service and host checks. If this option is
# enabled, host performance data will be processed using the
# host_perfdata_command (defined below) and service performance
# data will be processed using the service_perfdata_command (also
# defined below). Read the HTML docs for more information on
# performance data.
# Values: 1 = process performance data, 0 = do not process performance data

process_performance_data=0

# HOST AND SERVICE PERFORMANCE DATA PROCESSING COMMANDS
# These commands are run after every host and service check is
# performed. These commands are executed only if the
# enable_performance_data option (above) is set to 1. The command
# argument is the short name of a command definition that you
# define in your host configuration file. Read the HTML docs for
# more information on performance data.

#host_perfdata_command=process-host-perfdata
#service_perfdata_command=process-service-perfdata

# HOST AND SERVICE PERFORMANCE DATA FILES
# These files are used to store host and service performance data.
# Performance data is only written to these files if the
# enable_performance_data option (above) is set to 1.

host_perfdata_file=/usr/local/pnp4nagios/var/host-perfdata
service_perfdata_file=/usr/local/pnp4nagios/var/service-perfdata

# HOST AND SERVICE PERFORMANCE DATA FILE TEMPLATES
# These options determine what data is written (and how) to the
# performance data files. The templates may contain macros, special
# characters (\t for tab, \r for carriage return, \n for newline)
# and plain text. A newline is automatically added after each write
# to the performance data file. Some examples of what you can do are
# shown below.

host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$

# HOST AND SERVICE PERFORMANCE DATA FILE MODES
# This option determines whether or not the host and service
# performance data files are opened in write ("w") or append ("a")
# mode. If you want to use named pipes, you should use the special
# pipe ("p") mode which avoid blocking at startup, otherwise you will
# likely want the defult append ("a") mode.

host_perfdata_file_mode=a
service_perfdata_file_mode=a

# HOST AND SERVICE PERFORMANCE DATA FILE PROCESSING INTERVAL
# These options determine how often (in seconds) the host and service
# performance data files are processed using the commands defined
# below. A value of 0 indicates the files should not be periodically
# processed.

host_perfdata_file_processing_interval=15
service_perfdata_file_processing_interval=15

# HOST AND SERVICE PERFORMANCE DATA FILE PROCESSING COMMANDS
# These commands are used to periodically process the host and
# service performance data files. The interval at which the
# processing occurs is determined by the options above.

host_perfdata_file_processing_command=process-host-perfdata-file
service_perfdata_file_processing_command=process-service-perfdata-file

'''

Thanks.
pmithil7
 
Posts: 43
Joined: Wed Dec 07, 2016 7:06 pm

Re: host & service perfdata increasing alarmingly

Postby rkennedy » Thu Feb 09, 2017 5:18 pm

Can you look up your command definitions for what's used to process the perf data? You may just be logging it to a file based on the name, which explains why it's filling up.
Code: Select all
process-host-perfdata-file
process-service-perfdata-file
rkennedy
 
Posts: 6545
Joined: Mon Oct 05, 2015 11:45 am

Next

Return to Nagios Core

Who is online

Users browsing this forum: No registered users and 6 guests