Performance Graphs Stopped Working

[nik.vu]

I have problem with server performance graphs, hosts and services collecting informations but graphs didn't show up.

npcd LOG

Code: Select all

[root@nagiosxi5 ~]# tail -25 /usr/local/nagios/var/npcd.log
[10-19-2017 14:53:59] NPCD: A thread was started on thread_counter = 1
[10-19-2017 14:53:59] NPCD: Processing file '1508417631.perfdata.host'
[10-19-2017 14:53:59] NPCD: Have to wait: Filecounter = 2 - thread_counter = 2
[10-19-2017 14:53:59] NPCD: Processing file 1508417631.perfdata.service with ID 140052249085696 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1508417631.perfdata.service
[10-19-2017 14:53:59] NPCD: Processing file '1508417631.perfdata.service'
[10-19-2017 14:54:01] NPCD: No more files to process... waiting for 15 seconds
[10-19-2017 14:54:16] NPCD: Found 4 files in /var/nagiosramdisk/spool/perfdata/
[10-19-2017 14:54:16] NPCD: DEBUG: load 6.030000/50.000000
[10-19-2017 14:54:16] NPCD: ThreadCounter 0/5 File is .
[10-19-2017 14:54:16] NPCD: DEBUG: load 6.030000/50.000000
[10-19-2017 14:54:16] NPCD: ThreadCounter 0/5 File is ..
[10-19-2017 14:54:16] NPCD: DEBUG: load 6.030000/50.000000
[10-19-2017 14:54:16] NPCD: ThreadCounter 0/5 File is 1508417645.perfdata.host
[10-19-2017 14:54:16] NPCD: Regular File: 1508417645.perfdata.host
[10-19-2017 14:54:16] NPCD: A thread was started on thread_counter = 0
[10-19-2017 14:54:16] NPCD: DEBUG: load 6.030000/50.000000
[10-19-2017 14:54:16] NPCD: Processing file 1508417645.perfdata.host with ID 140052259575552 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1508417645.perfdata.host
[10-19-2017 14:54:16] NPCD: ThreadCounter 1/5 File is 1508417645.perfdata.service
[10-19-2017 14:54:16] NPCD: Processing file '1508417645.perfdata.host'
[10-19-2017 14:54:16] NPCD: Regular File: 1508417645.perfdata.service
[10-19-2017 14:54:16] NPCD: A thread was started on thread_counter = 1
[10-19-2017 14:54:16] NPCD: Processing file 1508417645.perfdata.service with ID 140052249085696 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1508417645.perfdata.service
[10-19-2017 14:54:16] NPCD: Have to wait: Filecounter = 2 - thread_counter = 2
[10-19-2017 14:54:16] NPCD: Processing file '1508417645.perfdata.service'
[10-19-2017 14:54:18] NPCD: No more files to process... waiting for 15 seconds
[root@nagiosxi5 ~]#

Perf log

Code: Select all

[root@nagiosxi5 ~]# tail -25 /usr/local/nagios/var/npcd.log
[10-19-2017 14:53:59] NPCD: A thread was started on thread_counter = 1
[10-19-2017 14:53:59] NPCD: Processing file '1508417631.perfdata.host'
[10-19-2017 14:53:59] NPCD: Have to wait: Filecounter = 2 - thread_counter = 2
[10-19-2017 14:53:59] NPCD: Processing file 1508417631.perfdata.service with ID 140052249085696 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1508417631.perfdata.service
[10-19-2017 14:53:59] NPCD: Processing file '1508417631.perfdata.service'
[10-19-2017 14:54:01] NPCD: No more files to process... waiting for 15 seconds
[10-19-2017 14:54:16] NPCD: Found 4 files in /var/nagiosramdisk/spool/perfdata/
[10-19-2017 14:54:16] NPCD: DEBUG: load 6.030000/50.000000
[10-19-2017 14:54:16] NPCD: ThreadCounter 0/5 File is .
[10-19-2017 14:54:16] NPCD: DEBUG: load 6.030000/50.000000
[10-19-2017 14:54:16] NPCD: ThreadCounter 0/5 File is ..
[10-19-2017 14:54:16] NPCD: DEBUG: load 6.030000/50.000000
[10-19-2017 14:54:16] NPCD: ThreadCounter 0/5 File is 1508417645.perfdata.host
[10-19-2017 14:54:16] NPCD: Regular File: 1508417645.perfdata.host
[10-19-2017 14:54:16] NPCD: A thread was started on thread_counter = 0
[10-19-2017 14:54:16] NPCD: DEBUG: load 6.030000/50.000000
[10-19-2017 14:54:16] NPCD: Processing file 1508417645.perfdata.host with ID 140052259575552 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1508417645.perfdata.host
[10-19-2017 14:54:16] NPCD: ThreadCounter 1/5 File is 1508417645.perfdata.service
[10-19-2017 14:54:16] NPCD: Processing file '1508417645.perfdata.host'
[10-19-2017 14:54:16] NPCD: Regular File: 1508417645.perfdata.service
[10-19-2017 14:54:16] NPCD: A thread was started on thread_counter = 1
[10-19-2017 14:54:16] NPCD: Processing file 1508417645.perfdata.service with ID 140052249085696 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1508417645.perfdata.service
[10-19-2017 14:54:16] NPCD: Have to wait: Filecounter = 2 - thread_counter = 2
[10-19-2017 14:54:16] NPCD: Processing file '1508417645.perfdata.service'
[10-19-2017 14:54:18] NPCD: No more files to process... waiting for 15 seconds
[root@nagiosxi5 ~]# ^C
[root@nagiosxi5 ~]# ^C
[root@nagiosxi5 ~]# tail -25 /usr/local/nagios/var/perfdata.log
2017-10-19 14:54:51 [16993] [1] Found Performance Data for BG_W093-LedineCDMA / ether1-Hagleitner-Hygiene_Bandwidth (in=0Mb/s;50.00;80.00 out=0Mb/s;50.00;80.00)
2017-10-19 14:54:51 [16993] [2] No Custom Template found for check_xi_service_mrtgtraf (/usr/local/nagios/etc/pnp/check_commands/check_xi_service_mrtgtraf.cfg)
2017-10-19 14:54:51 [16993] [2] Template is check_xi_service_mrtgtraf.php
2017-10-19 14:54:51 [16993] [2] No Custom Template found for check_xi_service_mrtgtraf (/usr/local/nagios/etc/pnp/check_commands/check_xi_service_mrtgtraf.cfg)
2017-10-19 14:54:51 [16993] [2] Template is check_xi_service_mrtgtraf.php
2017-10-19 14:54:51 [16993] [2] data2rrd called
2017-10-19 14:54:51 [16993] [2] RRDs Perl Modules are not installed. Falling back to rrdtool system call.
2017-10-19 14:54:51 [16993] [2] /usr/bin/rrdtool update --daemon=unix:/var/rrdtool/rrdcached/rrdcached.sock /usr/local/nagios/share/perfdata/BG_W093-LedineCDMA/ether1-Hagleitner-Hygiene_Bandwidth.rrd 1508417675:0:0
2017-10-19 14:54:51 [16993] [1] rrdtool update returns 256
2017-10-19 14:54:51 [16993] [2] Processing Line 668
2017-10-19 14:54:51 [16993] [2] Datatype set to 'SERVICEPERFDATA'
2017-10-19 14:54:51 [16993] [1] Found Performance Data for BG_D105-JurijaGagarina36ado36v / ether3-Alied-Testing-doo_Bandwidth (in=0Mb/s;50.00;80.00 out=0Mb/s;50.00;80.00)
2017-10-19 14:54:51 [16993] [2] No Custom Template found for check_xi_service_mrtgtraf (/usr/local/nagios/etc/pnp/check_commands/check_xi_service_mrtgtraf.cfg)
2017-10-19 14:54:51 [16993] [2] Template is check_xi_service_mrtgtraf.php
2017-10-19 14:54:51 [16993] [2] No Custom Template found for check_xi_service_mrtgtraf (/usr/local/nagios/etc/pnp/check_commands/check_xi_service_mrtgtraf.cfg)
2017-10-19 14:54:51 [16993] [2] Template is check_xi_service_mrtgtraf.php
2017-10-19 14:54:51 [16993] [2] data2rrd called
2017-10-19 14:54:51 [16993] [2] RRDs Perl Modules are not installed. Falling back to rrdtool system call.
2017-10-19 14:54:51 [16993] [2] /usr/bin/rrdtool update --daemon=unix:/var/rrdtool/rrdcached/rrdcached.sock /usr/local/nagios/share/perfdata/BG_D105-JurijaGagarina36ado36v/ether3-Alied-Testing-doo_Bandwidth.rrd 1508417675:0:0
2017-10-19 14:54:51 [16993] [1] rrdtool update returns 256
2017-10-19 14:54:51 [16993] [2] Processing Line 669
2017-10-19 14:54:51 [16993] [2] No Perfdata. Skipping line 669
2017-10-19 14:54:51 [16993] [1] 669 lines processed
2017-10-19 14:54:51 [16993] [1] /var/nagiosramdisk/spool/perfdata//1508417675.perfdata.service-PID-16993 deleted
2017-10-19 14:54:51 [16993] [1] PNP exiting (runtime 1.633031s) ...
[root@nagiosxi5 ~]#

[npolovenko]

Hello, @nik.vu.
Have your graphs worked as they should before? What did you do prior to getting this problem?

1. Your log file says that RRDs Perl Modules are not installed. Please run

Code: Select all

sudo apt-get install rrdtool

or

Code: Select all

yum install rrdtool

depending on your system.
2. Please take a look at this article to troubleshoot graphing problems in XI: https://support.nagios.com/kb/article.php?id=9
3. Could you send in your Nagios XI System Profile so I can review it?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it here or send it in a PM.

[nik.vu]

Performance graphs worked normally with little problems with bandwidth graphs. To many bandwidth graphs showed 0mbps as value. On other XI servers i resolved this problem with bandwidth graphs with edit of mrtg configuration file and change number of forks.

On this server when i changed number of forks in mrtg configuration, after about one hour i issue a problem with crached DB. After i repaired DB, performance graphs didn't showed up.


1. This rrd module is allready installed

Code: Select all

Package rrdtool-1.3.8-10.el6.x86_64 already installed and latest version
Nothing to do
[root@nagiosxi5 ~]#

2.First of all i tried with this article, but result was same.

[root@nagiosxi5 ~]# ls /usr/local/nagios/var/spool/perfdata/ | wc -l
0
[root@nagiosxi5 ~]# ls /usr/local/nagios/var/spool/xidpe/ | wc -l
0

[root@nagiosxi5 ~]#

[npolovenko]

@nik.vu, Your DB might've crashed because you ran out of memory. How much ram does your machine have? It seems like you have some memory related issues:

Code: Select all

Oct 19 09:50:14 nagiosxi5 kernel: Out of memory: Kill process 29478 (httpd) score 2 or sacrifice child

[nik.vu]

Problem with RAM was in that moment when increase number of forks in mrtg conf file. After change to lower value(Forks:12), everything was ok with DB and all other Nagios XI components. The only thing that isnt't work properly is performance graphs.

[root@nagiosxi5 ~]# cat /proc/meminfo
MemTotal: 12318644 kB
MemFree: 1095520 kB
Buffers: 86176 kB
Cached: 5039256 kB
SwapCached: 7104 kB
Active: 8913888 kB
Inactive: 1663916 kB
Active(anon): 5272916 kB
Inactive(anon): 223420 kB
Active(file): 3640972 kB
Inactive(file): 1440496 kB
Unevictable: 12948 kB
Mlocked: 4408 kB
SwapTotal: 8191996 kB
SwapFree: 8166540 kB
Dirty: 18600 kB
Writeback: 0 kB
AnonPages: 5458832 kB
Mapped: 23524 kB
Shmem: 40756 kB
Slab: 430268 kB
SReclaimable: 362216 kB
SUnreclaim: 68052 kB
KernelStack: 9520 kB
PageTables: 55064 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 14351316 kB
Committed_AS: 10444696 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 307972 kB
VmallocChunk: 34359348676 kB
HardwareCorrupted: 0 kB
AnonHugePages: 704512 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7360 kB
DirectMap2M: 12566528 kB

[nik.vu]

I resolved the problem with reconfiguration of /usr/local/nagios/etc/pnp/process_perfdata.cfg.

Please just can you give me solution for bandwidth graphs. The solution from this topic is not enough good for this server where i had 10.000 bandwidth graphs. It happens again that MRTG app couldn't process all data and shows 0Mbps.

[dwasswa]

Hi @nik.vu,

We are glad that the performance graphs are now working.

Here is a possible solution to your Bandwidth graphs showing no data...https://support.nagios.com/kb/article/n ... *-cfg.html

Please try that and let me know if it helps.

[nik.vu]

This isn't works for me.

Bandwidth graphs works, but not completely. Because from time to time graps show 0mbps. This happens only with server with large number of bandwidth graphs. As you can see on this images below this is the same service on 2 different servers.

MRTG Conf

Code: Select all

HtmlDir: /var/www/mrtg
ImageDir: /var/www/mrtg
LogFormat: rrdtool
LogDir: /var/lib/mrtg
ThreshDir: /var/lib/mrtg
WorkDir: /var/lib/mrtg
Forks: 12
EnableSnmpV3: yes

Include: conf.d/*.cfg

EnableSNMPv3: yes
LibAdd: /opt/rrdtool-1.4.4/lib/perl/5.10.1
LibAdd: /opt/rrdtool-1.4.4/lib/perl/5.10.1

Also i increased number of open files in /etc/security/limits.conf

Code: Select all

# End of file
* hard nofile 10000
* soft nofile 10000

root hard nofile 10000
root soft nofile 10000

[tgriep]

Couple of things you can try. Edit the /etc/mrtg/mrtg.cfg file and increase the forks to 16 and remove these 2 lines.

Code: Select all

LibAdd: /opt/rrdtool-1.4.4/lib/perl/5.10.1
LibAdd: /opt/rrdtool-1.4.4/lib/perl/5.10.1

Save the file and restart cron by running

Code: Select all

service crond restart

I thing that it takes MRTG so long to run that is continuously skips running and that is causing the gaps.

Run this on the nagios server and post how long it takes to run

Code: Select all

time LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg

[nik.vu]

@tgriep

Thanks you for quick responce. I will try this, and let you know on Monday if still problem persist.

Have a nice weekend.