Problems with XI displaying 4x duplicate services and hosts

[btmikkelsen]

ever since the upgrade to 2011R3.3, nagiosXI shows each service and host 4 times. Occasionally after an apply configuration it goes back to normal, but it has consistently done this since the upgrade. naigos core does NOT do this. viewing the operational files, the status.dat, retentionl.dat, etc are all what you would expect - no duplicates.

ideas?

For all support requests, we need to know:
CentOS release 5.8 (Final)
32 or 64bit? 64 bit
VMware Image or Manual Install of XI? Manual
no Gnome, no proxy, no ssl.

[lmiltchev]

Run the following command and post the output:

Code: Select all

tail /var/log/messages

[btmikkelsen]

is there anything you're looking at? This production monitoring system is doing a bunch of tasks. for instance...a tail of the message file will get you lots of non-applicable stuff. Regardless...here is a tail of /var/log/messages.
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [127.0.0.1]:16864
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [X]:12831
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [X]:12831
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [X]:16864
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [X]:12831
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [127.0.0.1]:16864
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [127.0.0.1]:16864
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [X]:12831
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [X]:12831
Sep 27 15:19:39 X snmpd[5862]: Connection from UDP: [X]:12831

[mguthrie]

Try running the following:

Code: Select all

service nagios stop
killall -9 nagios
service ndo2db stop
service ndo2db start
service nagios start

[btmikkelsen]

We've been living with this for quite a while now. Here is some additional information:
1) Services, not hosts duplicate 4x
2) this only happens when an "apply configuration" is performed
3) when you killall -9 nagios the service restart, the XI counters are correct
4) nagios core (/nagios) is not affected by this
5) notifications are not affected
6) retention and status files do not have 4x entries

this seems to be a XI ui issue only. Any assistance would be appreciated.
we have upgraded since this started happening:
Nagios XI Version : 2012R1.6
XX 2.6.18-348.1.1.el5 x86_64
CentOS release 5.9 (Final)
Gnome is not installed
Apache Information

PHP Version: 5.1.6
Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17
Server Name: XX
Server Address: XX
Server Port: 80
Date/Time

PHP Timezone: America/Los_Angeles
PHP Time: Tue, 26 Feb 2013 08:11:02 -0800
System Time: Tue, 26 Feb 2013 08:11:02 -0800
Nagios XI Data

nagios (pid 17667) is running...
NPCD running (pid 10989).
ndo2db (pid 27329) is running...
CPU Load 15: 3.43
Total Hosts: 2556
Total Services: 18732
Function 'get_base_uri' returns: http://XX/nagiosxi/
Function 'get_base_url' returns: http://XX/nagiosxi/
Function 'get_backend_url(internal_call=false)' returns: http://XX/nagiosxi/includes/components/ ... rofile.php
Function 'get_backend_url(internal_call=true)' returns: http://localhost/nagiosxi/backend/
Ping Test localhost

Running:
/bin/ping -c 3 localhost 2>&1
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=2 ttl=64 time=0.021 ms
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=3 ttl=64 time=0.024 ms

--- localhost.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.021/0.023/0.025/0.004 ms
Test wget To locahost

WGET From URL: http://localhost/nagiosql/index.php
Running:
/usr/bin/wget http://localhost/nagiosql/index.php
--2013-02-26 08:11:04-- http://localhost/nagiosql/index.php
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5259 (5.1K) [text/html]
Saving to: `/tmp/nagiosql_index.tmp'

0K ..... 100% 313M=0s

2013-02-26 08:11:04 (313 MB/s) - `/tmp/nagiosql_index.tmp' saved [5259/5259]

[slansing]

After you apply configuration and these duplicates appear how long does it take before they..disappear? Can you provide a screenshot of this as it happens? I'm also curious to see how many processes are running when this happens. Next time it does can you run the following command and post the output with the screenshot of your UI?:

Code: Select all

ps -ef|grep bin/nagios | grep -v grep

Also, how many checks is your system running on average per minute? And what is the load on the system.

[btmikkelsen]

nagios core information.png

after applying the configuration, duplicates happen as soon as XI starts displaying status again. these ghost services will only go away by killing the nagios processes and restarting. (killall -9 nagios; service nagios restart).
this is very easy to duplicate - it is just a matter of applying the configuration. Again - nagios core is unaffected, so these ghost host and

nagios core information.png

services must only be present in the XI ui.


Attached is a screenshot of the tactical showing both.


nagios 4589 1 16 16:05 ? 00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

FYI: http://<serveraddress>/nagiosxi/backend/?cmd=getservicestatus sometimes times out, depending on the load at the time. I don't know if this is related, or not - it is listed in the troubleshooting steps when the count is incorrect? XI Display Problems.

[mguthrie]

Do you get any errors or notices in the /usr/local/nagios/var/nagios.log file when you run:

Code: Select all

service nagios restart

Also, lets try the following:

Code: Select all

service nagios stop
rm -f /usr/local/nagios/var/retention.dat
service nagios start

This will start nagios with everything in a pending state, but should give somewhat of a clean reset.

Can you also post your /usr/local/nagios/etc/nagios.cfg file?

[btmikkelsen]

I tried these steps, without change.

nagios.cfg

Code: Select all

# MODIFIED
[email protected]
[email protected]
translate_passive_host_checks=1
log_event_handlers=0
use_large_installation_tweaks=1
enable_environment_macros=0
#debug_level=-1
# all but function enter/exit
debug_level=126


# NDOUtils module
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg


# PNP settings - bulk mode with NCPD
process_performance_data=1
# service performance data
service_perfdata_file=/var/nagiosramdisk/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
#service_perfdata_file_processing_command=process-service-perfdata-file-bulk
service_perfdata_file_processing_command=process-service-perfdata-file-pnp-bulk

# host performance data
host_perfdata_file=/var/nagiosramdisk/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
#host_perfdata_file_processing_command=process-host-perfdata-file-bulk
host_perfdata_file_processing_command=process-host-perfdata-file-pnp-bulk

# OBJECTS - UNMODIFIED
#cfg_file=/usr/local/nagios/etc/objects/commands.cfg
#cfg_file=/usr/local/nagios/etc/objects/contacts.cfg
#cfg_file=/usr/local/nagios/etc/objects/localhost.cfg
#cfg_file=/usr/local/nagios/etc/objects/templates.cfg
#cfg_file=/usr/local/nagios/etc/objects/timeperiods.cfg


# STATIC OBJECT DEFINITIONS (THESE DON'T GET EXPORTED/IMPORTED BY NAGIOSQL)
cfg_dir=/usr/local/nagios/etc/static

# OBJECTS EXPORTED FROM NAGIOSQL
cfg_file=/usr/local/nagios/etc/contacttemplates.cfg
cfg_file=/usr/local/nagios/etc/contactgroups.cfg
cfg_file=/usr/local/nagios/etc/contacts.cfg
cfg_file=/usr/local/nagios/etc/timeperiods.cfg
cfg_file=/usr/local/nagios/etc/commands.cfg
cfg_file=/usr/local/nagios/etc/hostgroups.cfg
cfg_file=/usr/local/nagios/etc/servicegroups.cfg
cfg_file=/usr/local/nagios/etc/hosttemplates.cfg
cfg_file=/usr/local/nagios/etc/servicetemplates.cfg
cfg_file=/usr/local/nagios/etc/servicedependencies.cfg
cfg_file=/usr/local/nagios/etc/serviceescalations.cfg
cfg_file=/usr/local/nagios/etc/hostdependencies.cfg
cfg_file=/usr/local/nagios/etc/hostescalations.cfg
cfg_file=/usr/local/nagios/etc/hostextinfo.cfg
cfg_file=/usr/local/nagios/etc/serviceextinfo.cfg
cfg_dir=/usr/local/nagios/etc/hosts
cfg_dir=/usr/local/nagios/etc/services

# Viad Service dependencies
cfg_file=/usr/local/nagios/etc/viad_servicedependencies.cfg

# GLOBAL EVENT HANDLERS
global_host_event_handler=xi_host_event_handler
global_service_event_handler=xi_service_event_handler



# UNMODIFIED
accept_passive_host_checks=1
accept_passive_service_checks=1
additional_freshness_latency=15
auto_reschedule_checks=0
auto_rescheduling_interval=30
auto_rescheduling_window=180
bare_update_check=0
cached_host_check_horizon=15
cached_service_check_horizon=15
check_external_commands=1
check_for_orphaned_hosts=1
check_for_orphaned_services=1
check_for_updates=1
check_host_freshness=0
check_result_path=/var/nagiosramdisk/spool/checkresults
#check_result_reaper_frequency=10
check_result_reaper_frequency=10
check_service_freshness=1
command_check_interval=-1
command_file=/usr/local/nagios/var/rw/nagios.cmd
daemon_dumps_core=0
date_format=us
debug_file=/var/nagiosramdisk/nagios.debug
debug_verbosity=1
enable_embedded_perl=0
enable_event_handlers=1
enable_flap_detection=1
enable_notifications=1
enable_predictive_host_dependency_checks=1
enable_predictive_service_dependency_checks=1
event_broker_options=-1
event_handler_timeout=30
execute_host_checks=1
execute_service_checks=1
external_command_buffer_slots=4096
high_host_flap_threshold=20.0
high_service_flap_threshold=20.0
host_check_timeout=30
host_freshness_check_interval=60
host_inter_check_delay_method=s
illegal_macro_output_chars=`~$&|'"<>
illegal_object_name_chars=`~!$%^&*|'"<>?,()=
interval_length=60
lock_file=/usr/local/nagios/var/nagios.lock
log_archive_path=/usr/local/nagios/var/archives
log_external_commands=0
log_file=/var/nagiosramdisk/nagios.log
log_host_retries=1
log_initial_states=0
log_notifications=1
log_passive_checks=0
log_rotation_method=d
log_service_retries=1
low_host_flap_threshold=5.0
low_service_flap_threshold=5.0
max_check_result_file_age=3600
#max_check_result_reaper_time=30
max_check_result_reaper_time=40
max_concurrent_checks=0
max_debug_file_size=100000000
max_host_check_spread=30
max_service_check_spread=20
nagios_group=nagios
nagios_user=nagios
notification_timeout=30
#object_cache_file=/usr/local/nagios/var/objects.cache
object_cache_file=/var/nagiosramdisk/objects.cache
obsess_over_hosts=0
obsess_over_services=0
ocsp_timeout=5
p1_file=/usr/local/nagios/bin/p1.pl
passive_host_checks_are_soft=0
perfdata_timeout=5
#precached_object_file=/usr/local/nagios/var/objects.precache
precached_object_file=/var/nagiosramdisk/objects.precache
resource_file=/usr/local/nagios/etc/resource.cfg
retained_contact_host_attribute_mask=0
retained_contact_service_attribute_mask=0
retained_host_attribute_mask=0
retained_process_host_attribute_mask=0
retained_process_service_attribute_mask=0
retained_service_attribute_mask=0
retain_state_information=1
retention_update_interval=60
service_check_timeout=90
service_freshness_check_interval=60
service_inter_check_delay_method=s
service_interleave_factor=s
sleep_time=0.25
soft_state_dependencies=0
state_retention_file=/usr/local/nagios/var/retention.dat
status_file=/usr/local/nagios/var/status.dat
#status_file=/var/nagiosramdisk/status.dat
status_update_interval=10
temp_file=/usr/local/nagios/var/nagios.tmp
temp_path=/var/nagiosramdisk/tmp
use_aggressive_host_checking=0
use_embedded_perl_implicitly=0
use_regexp_matching=0
use_retained_program_state=1
use_retained_scheduling_info=1
use_syslog=1
use_true_regexp_matching=0

[mguthrie]

Can you run the DB repair procedure? Lets make sure there's not any DB corruption so we can rule that out.
http://assets.nagios.com/downloads/nagi ... tabase.pdf