Nagios XI not displaying all the hosts

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
User avatar
chicjo01
Posts: 194
Joined: Tue Jul 28, 2015 2:52 pm

Nagios XI not displaying all the hosts

Post by chicjo01 »

I have a setup were we are using config files in the /usr/local/nagios/etc/static folder. When we check the "Service Details" it shoulds only 181 hosts, but we have over 800 hosts. The only way we found to get the number some what close is to restart ndo2db and nagios services. We have also noticed that if we view the Service Details in the Nagios Core View, it shows all the hosts.

Can you let us know what we need to do in order to have Nagios XI Service Details to display correctly when not using Nagios Core View?

Release: Nagios XI 5.2.7
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 2268 services.
Checked 892 hosts.
Checked 196 host groups.
Checked 0 service groups.
Checked 143 contacts.
Checked 193 contact groups.
Checked 136 commands.
Checked 10 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 892 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 10 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Nagios XI not displaying all the hosts

Post by rkennedy »

Could you PM over a profile for us to look at? (Admin -> System Profile -> Download Profile)

This will help us take a further look at your environment.

EDIT: Received.
Former Nagios Employee
User avatar
chicjo01
Posts: 194
Joined: Tue Jul 28, 2015 2:52 pm

Re: Nagios XI not displaying all the hosts

Post by chicjo01 »

I have PM you the results of the System Profile.
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Nagios XI not displaying all the hosts

Post by tmcdonald »

Are you actively restarting nagios when you make changes to the static folder? To be clear, simply dropping a config in that folder will not be enough - nagios will not notice the new configs until you restart the process.
Former Nagios employee
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios XI not displaying all the hosts

Post by tgriep »

I was looking through the profile and I noticed that there were errors connecting to your remote MYSQL server.
Can you login to the MYSQL server and check the logs for any error information and post it here?
Also, can you login to the XI server, run the following command and post it's output?

Code: Select all

ipcs
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
chicjo01
Posts: 194
Joined: Tue Jul 28, 2015 2:52 pm

Re: Nagios XI not displaying all the hosts

Post by chicjo01 »

I will verify that nagios is being restarted after a configuration get changed. I am reaching out to our database team about the errors in the logs. Below is the command you requested the output for.
ipcs

------ Message Queues --------
key msqid owner perms used-bytes messages
0xf4000002 6389760 nagios 600 459776 449
0x83000002 851969 nagios 600 21872640 21360
0xa1000002 458754 nagios 600 131047424 127976
0xc1000002 655363 nagios 600 4393984 4291
0xb3000002 557060 nagios 600 131046400 127975
0x51000002 1015813 nagios 600 16334848 15952
0xa6000002 5242886 nagios 600 309248 302
0x3d000002 1310727 nagios 600 4065280 3970
0x52000002 7012360 nagios 600 4605952 4498
0xed000002 1605641 nagios 600 4391936 4289
0x42000002 1671178 nagios 600 13146112 12838
0x1f000002 1736715 nagios 600 12904448 12602
0x09000002 1802252 nagios 600 0 0
0x10000002 2621453 nagios 600 419840 410
0x6b000002 1933326 nagios 600 0 0
0x4b000002 1998863 nagios 600 13123584 12816
0xd4000002 2031632 nagios 600 240640 235
0xc3000002 2064401 nagios 600 2848768 2782
0x3f000002 2097170 nagios 600 2048 2
0xb1000002 5701651 nagios 600 481280 470
0xcc000002 4980756 nagios 600 521216 509
0x8e000002 4849685 nagios 600 238592 233
0x79000002 2359318 nagios 600 3706880 3620
0x66000002 2392087 nagios 600 104448 102
0x8a000002 6783000 nagios 600 487424 476
0x89000002 4227097 nagios 600 207872 203
0x87000002 6848538 nagios 600 378880 370
0xdf000002 2588699 nagios 600 0 0
0x44000002 2719772 nagios 600 528384 516
0xf8000002 2818077 nagios 600 175104 171
0x07000002 2883614 nagios 600 102400 100
0x54000002 2949151 nagios 600 402432 393
0x4f000002 3014688 nagios 600 287744 281
0x28000002 3637281 nagios 600 257024 251
0x78000002 3112994 nagios 600 352256 344
0xfd000002 3145763 nagios 600 4494336 4389
0x7b000002 3211300 nagios 600 460800 450
0xc4000002 3276837 nagios 600 202752 198
0x5c000002 6455334 nagios 600 1098752 1073
0xea000002 3407911 nagios 600 935936 914
0x05000002 6062120 nagios 600 389120 380
0xb5000002 3538985 nagios 600 498688 487
0x29000002 3604522 nagios 600 776192 758
0x77000002 3702827 nagios 600 209920 205
0xcb000002 3768364 nagios 600 817152 798
0xc8000002 3801133 nagios 600 4576256 4469
0x8d000002 3866670 nagios 600 417792 408
0x2f000002 3932207 nagios 600 271360 265
0xff000002 5996592 nagios 600 214016 209
0x22000002 4030513 nagios 600 744448 727
0x2e000002 4096050 nagios 600 224256 219
0xae000002 4194355 nagios 600 181248 177
0xe5000002 4292660 nagios 600 461824 451
0xb4000002 4358197 nagios 600 591872 578
0x96000002 4423734 nagios 600 884736 864
0x84000002 4489271 nagios 600 732160 715
0xe1000002 4554808 nagios 600 4883456 4769
0x88000002 4620345 nagios 600 614400 600
0xe7000002 4685882 nagios 600 455680 445
0xad000002 5767227 nagios 600 91136 89
0x40000002 4816956 nagios 600 1676288 1637
0xcd000002 7143485 nagios 600 915456 894
0xc0000002 7405630 nagios 600 1074176 1049
0x37000002 5079103 nagios 600 979968 957
0x4a000002 5144640 nagios 600 163840 160
0xba000002 5210177 nagios 600 5142528 5022
0x13000002 7110722 nagios 600 539648 527
0x55000002 5374019 nagios 600 435200 425
0xc2000002 5439556 nagios 600 694272 678
0xa5000002 5505093 nagios 600 93184 91
0xb2000002 5537862 nagios 600 4496384 4391
0xf9000002 5570631 nagios 600 4403200 4300
0x3c000002 5865544 nagios 600 1225728 1197
0xe9000002 7209033 nagios 600 321536 314
0x80000002 6029387 nagios 600 4460544 4356
0x74000002 6127692 nagios 600 1076224 1051
0x2a000002 6193229 nagios 600 331776 324
0x19000002 6258766 nagios 600 120832 118
0x58000002 6324303 nagios 600 87040 85
0xe2000002 6520912 nagios 600 171008 167
0xe0000002 6586449 nagios 600 328704 321
0x68000002 6651986 nagios 600 503808 492
0x60000002 6684755 nagios 600 6244352 6098
0xc9000002 6750292 nagios 600 1317888 1287
0x5d000002 6946901 nagios 600 2689024 2626
0xef000002 7077974 nagios 600 1167360 1140
0x1d000002 7274583 nagios 600 576512 563
0x8f000002 7340120 nagios 600 195584 191
0x59000002 7471193 nagios 600 450560 440
0x85000002 7536730 nagios 600 428032 418

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x01131425 884737 root 600 1000 12

------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 6684672 apache 600 1
0x00000000 6717441 apache 600 1
0x00000000 6488066 apache 600 1
0x00000000 6750211 apache 600 1
0x00000000 6782980 apache 600 1
0x00000000 6815749 apache 600 1
User avatar
chicjo01
Posts: 194
Joined: Tue Jul 28, 2015 2:52 pm

Re: Nagios XI not displaying all the hosts

Post by chicjo01 »

Below is the error and warning we are getting in the Remote MySQL server.
2016-05-03 00:45:16 4923 [Warning] Aborted connection 6089678 to db: '<DB Schema>' user: '<DB User>' host: '<DB Server>' (Got an error reading communication packets)
2016-05-03 00:50:13 4923 [Warning] Aborted connection 6089679 to db: '<DB Schema>' user: '<DB User>' host: '<DB Server>' (Got an error reading communication packets)
2016-05-03 00:55:16 4923 [Warning] Aborted connection 6090566 to db: '<DB Schema>' user: '<DB User>' host: '<DB Server>' (Got an error reading communication packets)
2016-05-03 01:00:14 4923 [Warning] Aborted connection 6090565 to db: '<DB Schema>' user: '<DB User>' host: '<DB Server>' (Got an error reading communication packets)
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'last_modified'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'last_modified'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'entry_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'entry_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'entry_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'entry_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'entry_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'entry_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'connect_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'connect_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'start_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'start_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'start_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'start_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'status_update_time'
2016-05-03 01:10:02 4875 [ERROR] Invalid default value for 'status_update_time'
2016-05-03 01:10:03 4875 [ERROR] Invalid default value for 'status_update_time'
2016-05-03 01:10:03 4875 [ERROR] Invalid default value for 'entry_time'
2016-05-03 01:10:03 4875 [ERROR] Invalid default value for 'status_update_time'
2016-05-03 01:10:03 4875 [ERROR] Invalid default value for 'start_time'
2016-05-03 01:10:03 4875 [ERROR] Invalid default value for 'entry_time'
2016-05-03 01:10:03 4875 [ERROR] Invalid default value for 'event_time'
2016-05-03 01:10:03 4875 [ERROR] Invalid default value for 'entry_time'
Lots of these in the mysql instance log…..
User avatar
chicjo01
Posts: 194
Joined: Tue Jul 28, 2015 2:52 pm

Re: Nagios XI not displaying all the hosts

Post by chicjo01 »

Confirmed with the Team supporting the instance that they are indeed restarting nagios after a change to the configuration files.
systemctl restart nagios; systemctl restart nagiosxi; systemctl restart ndo2db
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios XI not displaying all the hosts

Post by tgriep »

Thanks for the information, it looks like the Nagios server is having problems connecting and staying connected to the MYSQL server.
Can you provide any information about the MYSQL server and also post the configuration files for it?

Can you run a repair of the mysql database?

Also, there are a lot of stuck messages in the queue in the XI server, login as root to the server, run the following commands to clear them out.

Code: Select all

service nagios stop
service ndo2db stop
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
service ndo2db start
service nagios start
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
chicjo01
Posts: 194
Joined: Tue Jul 28, 2015 2:52 pm

Re: Nagios XI not displaying all the hosts

Post by chicjo01 »

I did a check on the repair_database.sh script in the /usr/local/nagiosxi/scripts, it is designed specifically to handle the database on the localhost and not an offloaded server or am I incorrect and can you provide the specific adjustments or options I need to use?

After running the requested commands as root:
ipcs

------ Message Queues --------
key msqid owner perms used-bytes messages
0x50000002 7569408 nagios 600 12991488 12687
0x9d000002 7602177 nagios 600 12042240 11760

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x01131425 884737 root 600 1000 11

------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 6684672 apache 600 1
0x00000000 6717441 apache 600 1
0x00000000 6488066 apache 600 1
0x00000000 6750211 apache 600 1
0x00000000 6782980 apache 600 1
0x00000000 6815749 apache 600 1
Locked