Nagios XI problem with Highcharts

[jairo.chavez]

Good Morning
A couple of days ago they installed pnp4 (0.6.26) in the Nagios XI 5.8.6, the problem we have is that Highcharts module stopped working.
The pnp graphs if they are being generated.

I add some information hoping it will be useful:

cat /usr/local/nagiosxi/var/xiversion|grep full
full=5.8.6


# perl verify_pnp_config --mode bulk+npcd --config=/usr/local/nagios/etc/nagios.cfg --pnpcfg=/usr/local/pnp4nagios/etc
[INFO] ========== Starting Environment Checks ============
[INFO] My version is: verify_pnp_config-0.6.26-R.44
[INFO] Start Options: verify_pnp_config --mode bulk+npcd --config=/usr/local/nagios/etc/nagios.cfg --pnpcfg=/usr/local/pnp4nagios/etc
[INFO] Reading /usr/local/nagios/etc/nagios.cfg
[OK ] Running product is 'nagios'
[OK ] object_cache_file is defined
[OK ] object_cache_file=/usr/local/nagios/var/objects.cache
[INFO] Reading /usr/local/nagios/var/objects.cache
[OK ] resource_file is defined
[OK ] resource_file=/usr/local/nagios/etc/resource.cfg
[INFO] Reading /usr/local/nagios/etc/resource.cfg
[INFO] Reading /usr/local/pnp4nagios/etc/process_perfdata.cfg
[INFO] Reading /usr/local/pnp4nagios/etc/pnp4nagios_release
[OK ] Found PNP4Nagios version "0.6.26"
[OK ] Configured User is 'nagios'
[OK ] Effective User is 'nagios'
[OK ] User nagios exists with ID '1001'
[OK ] Configured Group is 'nagios'
[OK ] Effective group is 'nagios'
[OK ] Group nagios exists with ID '1001'
[INFO] ========== Checking Bulk Mode + NPCD Config ============
[OK ] process_performance_data is 1 compared with '/1/'
[OK ] service_perfdata_file is defined
[OK ] service_perfdata_file=/usr/local/pnp4nagios/var/service-perfdata
[OK ] service_perfdata_file_template is defined
[OK ] service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$
[OK ] PERFDATA template looks good
[OK ] service_perfdata_file_mode is defined
[OK ] service_perfdata_file_mode=a
[OK ] service_perfdata_file_processing_interval is defined
[OK ] service_perfdata_file_processing_interval=15
[OK ] service_perfdata_file_processing_command is defined
[OK ] service_perfdata_file_processing_command=process-service-perfdata-file
[OK ] host_perfdata_file is defined
[OK ] host_perfdata_file=/usr/local/pnp4nagios/var/host-perfdata
[OK ] host_perfdata_file_template is defined
[OK ] host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
[OK ] PERFDATA template looks good
[OK ] host_perfdata_file_mode is defined
[OK ] host_perfdata_file_mode=a
[OK ] host_perfdata_file_processing_interval is defined
[OK ] host_perfdata_file_processing_interval=15
[OK ] host_perfdata_file_processing_command is defined
[OK ] host_perfdata_file_processing_command=process-host-perfdata-file
[INFO] Nagios config looks good so far
[INFO] ========== Checking config values ============
[OK ] npcd daemon is running
[OK ] /usr/local/pnp4nagios/etc/npcd.cfg is used by npcd and readable
[INFO] Reading /usr/local/pnp4nagios/etc/npcd.cfg
[OK ] perfdata_spool_dir is defined
[OK ] perfdata_spool_dir=/usr/local/pnp4nagios/var/spool
[OK ] 1 files found in /usr/local/pnp4nagios/var/spool
service_perfdata_file_processing_command at verify_pnp_config line 612.
[OK ] Command process-service-perfdata-file is defined
[OK ] '/bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$'
[OK ] Command looks good
host_perfdata_file_processing_command at verify_pnp_config line 612.
[OK ] Command process-host-perfdata-file is defined
[OK ] '/bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$'
[OK ] Command looks good
[OK ] Script /usr/local/pnp4nagios/libexec/process_perfdata.pl is executable
[INFO] ========== Starting global checks ============
[OK ] status_file is defined
[OK ] status_file=/usr/local/nagios/var/status.dat
[INFO] host_query =
[INFO] service_query =
[INFO] Reading /usr/local/nagios/var/status.dat
[INFO] ==== Starting rrdtool checks ====
[OK ] RRDTOOL is defined
[OK ] RRDTOOL=/usr/bin/rrdtool
[OK ] /usr/bin/rrdtool is executable
[OK ] RRDtool 1.4.9 Copyright 1997-2013 by Tobias Oetiker <tobi@oetiker.ch>
[OK ] USE_RRDs is defined
[OK ] USE_RRDs=1
[OK ] Perl RRDs modules are loadable
[INFO] ==== Starting directory checks ====
[OK ] RRDPATH is defined
[OK ] RRDPATH=/usr/local/pnp4nagios/var/perfdata
[OK ] Perfdata directory '/usr/local/pnp4nagios/var/perfdata' exists
[WARN] 173 hosts/services are not providing performance data
[WARN] 'process_perf_data 1' is set for 174 hosts/services which are not providing performance data!
[OK ] 'process_perf_data 1' is set for 716 of your hosts/services
[INFO] ==== System sizing ====
[OK ] 715 hosts/service objects defined
[INFO] ==== Check statistics ====
[WARN] Warning: 2, Critical: 0
[WARN] Checks finished...


# perl verify_pnp_config --mode bulk --config=/usr/local/nagios/etc/nagios.cfg --pnpcfg=/usr/local/pnp4nagios/etc
[INFO] ========== Starting Environment Checks ============
[INFO] My version is: verify_pnp_config-0.6.26-R.44
[INFO] Start Options: verify_pnp_config --mode bulk --config=/usr/local/nagios/etc/nagios.cfg --pnpcfg=/usr/local/pnp4nagios/etc
[INFO] Reading /usr/local/nagios/etc/nagios.cfg
[OK ] Running product is 'nagios'
[OK ] object_cache_file is defined
[OK ] object_cache_file=/usr/local/nagios/var/objects.cache
[INFO] Reading /usr/local/nagios/var/objects.cache
[OK ] resource_file is defined
[OK ] resource_file=/usr/local/nagios/etc/resource.cfg
[INFO] Reading /usr/local/nagios/etc/resource.cfg
[INFO] Reading /usr/local/pnp4nagios/etc/process_perfdata.cfg
[INFO] Reading /usr/local/pnp4nagios/etc/pnp4nagios_release
[OK ] Found PNP4Nagios version "0.6.26"
[OK ] Configured User is 'nagios'
[OK ] Effective User is 'nagios'
[OK ] User nagios exists with ID '1001'
[OK ] Configured Group is 'nagios'
[OK ] Effective group is 'nagios'
[OK ] Group nagios exists with ID '1001'
[INFO] ========== Checking Bulk Mode Config ============
[OK ] process_performance_data is 1 compared with '/1/'
[OK ] service_perfdata_file is defined
[OK ] service_perfdata_file=/usr/local/pnp4nagios/var/service-perfdata
[OK ] service_perfdata_file_template is defined
[OK ] service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$
[OK ] PERFDATA template looks good
[OK ] service_perfdata_file_mode is defined
[OK ] service_perfdata_file_mode=a
[OK ] service_perfdata_file_processing_interval is defined
[OK ] service_perfdata_file_processing_interval=15
[OK ] service_perfdata_file_processing_command is defined
[OK ] service_perfdata_file_processing_command=process-service-perfdata-file
[OK ] host_perfdata_file is defined
[OK ] host_perfdata_file=/usr/local/pnp4nagios/var/host-perfdata
[OK ] host_perfdata_file_template is defined
[OK ] host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
[OK ] PERFDATA template looks good
[OK ] host_perfdata_file_mode is defined
[OK ] host_perfdata_file_mode=a
[OK ] host_perfdata_file_processing_interval is defined
[OK ] host_perfdata_file_processing_interval=15
[OK ] host_perfdata_file_processing_command is defined
[OK ] host_perfdata_file_processing_command=process-host-perfdata-file
[INFO] Nagios config looks good so far
[INFO] ========== Checking config values ============
service_perfdata_file_processing_command at verify_pnp_config line 612.
[OK ] Command process-service-perfdata-file is defined
[OK ] '/bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$'
[CRIT] Command looks suspect (/bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$)



/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2020-04-28
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 673 services.
Checked 42 hosts.
Checked 11 host groups.
Checked 16 service groups.
Checked 10 contacts.
Checked 2 contact groups.
Checked 184 commands.
Checked 18 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 42 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 18 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check


tail -f /usr/local/nagios/var/npcd.log
[09-23-2021 13:26:43] NPCD: npcd Daemon (0.4.14) started with PID=28746
[09-23-2021 13:26:43] NPCD: Please have a look at 'npcd -V' to get license information
[09-23-2021 13:26:43] NPCD: HINT: load_threshold is enabled - ('10.000000')
[10-07-2021 10:37:41] NPCD: Caught Termination Signal - Hasta la vista... baby
[10-07-2021 10:37:57] NPCD: npcd Daemon (0.4.14) started with PID=1207
[10-07-2021 10:37:57] NPCD: Please have a look at 'npcd -V' to get license information
[10-07-2021 10:37:57] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-25-2021 16:10:41] NPCD: Caught Termination Signal - Hasta la vista... baby



cat /usr/local/nagios/etc/nagios.cfg
# MODIFIED
admin_email=root@localhost
admin_pager=root@localhost
translate_passive_host_checks=1
log_event_handlers=0
use_large_installation_tweaks=1
enable_environment_macros=0


# NDO 3 module
# Commented out by NDO 'make install-broker-line' on jue sep 23 13:19:51 PDT 2021
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg


# PNP settings - bulk mode with NCPD
process_performance_data=1
# service performance data
service_perfdata_file=/usr/local/nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$\tLONGSERVICEOUTPUT::$LONGSERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file-bulk
# host performance data
host_perfdata_file=/usr/local/nagios/var/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$\tLONGHOSTOUTPUT::$LONGHOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file-bulk


# service performance data
service_perfdata_file=/usr/local/pnp4nagios/var/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file


# host performance data
host_perfdata_file=/usr/local/pnp4nagios/var/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file


# OBJECTS - UNMODIFIED
#cfg_file=/usr/local/nagios/etc/objects/commands.cfg
#cfg_file=/usr/local/nagios/etc/objects/contacts.cfg
#cfg_file=/usr/local/nagios/etc/objects/localhost.cfg
#cfg_file=/usr/local/nagios/etc/objects/templates.cfg
#cfg_file=/usr/local/nagios/etc/objects/timeperiods.cfg

#Agregado:
cfg_file=/usr/local/nagios/etc/commands-perfdata-file.cfg

# STATIC OBJECT DEFINITIONS (THESE DON'T GET EXPORTED/IMPORTED BY NAGIOSQL)
cfg_dir=/usr/local/nagios/etc/static

# OBJECTS EXPORTED FROM NAGIOSQL
cfg_file=/usr/local/nagios/etc/contacttemplates.cfg
cfg_file=/usr/local/nagios/etc/contactgroups.cfg
cfg_file=/usr/local/nagios/etc/contacts.cfg
cfg_file=/usr/local/nagios/etc/timeperiods.cfg
cfg_file=/usr/local/nagios/etc/commands.cfg
cfg_file=/usr/local/nagios/etc/hostgroups.cfg
cfg_file=/usr/local/nagios/etc/servicegroups.cfg
cfg_file=/usr/local/nagios/etc/hosttemplates.cfg
cfg_file=/usr/local/nagios/etc/servicetemplates.cfg
cfg_file=/usr/local/nagios/etc/servicedependencies.cfg
cfg_file=/usr/local/nagios/etc/serviceescalations.cfg
cfg_file=/usr/local/nagios/etc/hostdependencies.cfg
cfg_file=/usr/local/nagios/etc/hostescalations.cfg
cfg_file=/usr/local/nagios/etc/hostextinfo.cfg
cfg_file=/usr/local/nagios/etc/serviceextinfo.cfg
cfg_dir=/usr/local/nagios/etc/hosts
cfg_dir=/usr/local/nagios/etc/services

# GLOBAL EVENT HANDLERS
global_host_event_handler=xi_host_event_handler
global_service_event_handler=xi_service_event_handler



# UNMODIFIED
accept_passive_host_checks=1
accept_passive_service_checks=1
additional_freshness_latency=15
auto_reschedule_checks=1
auto_rescheduling_interval=30
auto_rescheduling_window=45
bare_update_check=0
cached_host_check_horizon=15
cached_service_check_horizon=15
check_external_commands=1
check_for_orphaned_hosts=1
check_for_orphaned_services=1
check_for_updates=1
check_host_freshness=0
check_result_path=/usr/local/nagios/var/spool/checkresults
check_result_reaper_frequency=10
check_service_freshness=1
command_file=/usr/local/nagios/var/rw/nagios.cmd
daemon_dumps_core=0
date_format=us
debug_file=/usr/local/nagios/var/nagios.debug
debug_level=0
debug_verbosity=1
enable_event_handlers=1
enable_flap_detection=1
enable_notifications=1
enable_predictive_host_dependency_checks=1
enable_predictive_service_dependency_checks=1
event_broker_options=-1
event_handler_timeout=30
execute_host_checks=1
execute_service_checks=1
high_host_flap_threshold=20.0
high_service_flap_threshold=20.0
host_check_timeout=30
host_freshness_check_interval=60
host_inter_check_delay_method=s
illegal_macro_output_chars=`~$&|'"<>
illegal_object_name_chars=`~!$%^&*|'"<>?,()=
interval_length=60
lock_file=/var/run/nagios.lock
log_archive_path=/usr/local/nagios/var/archives
log_external_commands=0
log_file=/usr/local/nagios/var/nagios.log
log_host_retries=1
log_initial_states=0
log_notifications=1
log_passive_checks=0
log_rotation_method=d
log_service_retries=1
low_host_flap_threshold=5.0
low_service_flap_threshold=5.0
max_check_result_file_age=3600
max_check_result_reaper_time=30
max_concurrent_checks=0
max_debug_file_size=1000000
max_host_check_spread=30
max_service_check_spread=30
nagios_group=nagios
nagios_user=nagios
notification_timeout=30
object_cache_file=/usr/local/nagios/var/objects.cache
obsess_over_hosts=0
obsess_over_services=0
ocsp_timeout=5
passive_host_checks_are_soft=0
perfdata_timeout=5
precached_object_file=/usr/local/nagios/var/objects.precache
resource_file=/usr/local/nagios/etc/resource.cfg
retained_contact_host_attribute_mask=0
retained_contact_service_attribute_mask=0
retained_host_attribute_mask=0
retained_process_host_attribute_mask=0
retained_process_service_attribute_mask=0
retained_service_attribute_mask=0
retain_state_information=1
retention_update_interval=60
service_check_timeout=60
service_freshness_check_interval=60
service_inter_check_delay_method=s
service_interleave_factor=s
soft_state_dependencies=0
state_retention_file=/usr/local/nagios/var/retention.dat
status_file=/usr/local/nagios/var/status.dat
status_update_interval=10
temp_file=/usr/local/nagios/var/nagios.tmp
temp_path=/tmp
use_aggressive_host_checking=0
use_regexp_matching=0
use_retained_program_state=1
use_retained_scheduling_info=1
use_syslog=1
use_true_regexp_matching=0
# Added by NDO 'make install-broker-line' on jue sep 23 13:19:51 PDT 2021
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

[pbroste]

Hello @jairo.chavez

Thanks for reaching out, want to get the System Profile so we can see what is going on.

To send us your system profile.
1. Login to the Nagios XI GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Thanks,
Perry

[jairo.chavez]

Hello @jairo.chavez

Thanks for reaching out, want to get the System Profile so we can see what is going on.

To send us your system profile.
1. Login to the Nagios XI GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Thanks,
Perry

The following message appears in the private messages section:
Private messages
Compose message
We are sorry, but you are not authorised to use this feature. You may have just registered here and may need to participate more to be able to use this feature.

[jairo.chavez]

It does not appear to me enabled for private messages.
How can I enable this option?
we recently acquired nagios xi.

[jairo.chavez]

It does not allow me to send private messages, I already entered the data of Support Forum Username, Account Number and Last 6 Digits of License.

Compose message
We are sorry, but you are not authorised to use this feature. You may have just registered here and may need to participate more to be able to use this feature.

[benjaminsmith]

HI,

Sorry for the inconvenience. Please try sending this again, it should work now. There is a minimum post count to help reduce spam.

Thanks,
Benjamin

[pbroste]

Hello @jairo.chavez

Thanks for sending over the System Profile, in review we see that services and processes are getting "Caught SIGTERM", which is causing issues.

Caught SIGTERM, shutting down...
NDO-3: Ended service_status thread
Caught SIGTERM, shutting down...
Caught SIGTERM, shutting down...
Successfully shutdown... (PID=23791)
NDO-3: Started notification thread
Warning: Host performance file processing command 'process-host-perfdata-file' was not found - host performance data file will not be processed!
Warning: Service performance file processing command 'process-service-perfdata-file' was not found - service performance data file will not be processed!
Successfully launched command file worker with pid 29343
NDO-3: Ended service_status thread
Caught SIGTERM, shutting down...

Before we run the database repair let's remove the locks:

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
pkill -9 -u nagios
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
rm -rf /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagiosxi/var/event_handler.lock
rm -rf /var/run/nagios.lock
rm -rf /usr/local/nagios/var/nagios.lock
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl restart httpd

Also, I would recommend increasing the max connections for the database. We have step-by-step instructions for this process in the following knowledge-base article.
https://support.nagios.com/kb/article.php?id=513

To repair the database.

• run through the database repair.
[list]

• Code: Select all

  /usr/local/nagiosxi/scripts/repair_databases.sh

[/list]

Please let us know how things look,
Perry