Nagios XI stuck applying configuration

[tgriep]

When you apply the configuration, can you click on the show errors and post that back to the forum?

Also, it looks like cron isn't running. Can you start the cron process by running the following?

Code: Select all

service crond start

[JakeHatMacys]

There wasn't any error it was just stuck at applying config................................................................................

But I did figure it out. Done goofed and had /usr/local at 100% which locked us up.

We had the space increased from 1 gig on that file system to 6... and now it's filling up again at 72%

There a good way to manage that or minimize the space used by that file system???

[tgriep]

Can you run this command and post the results so we can see which folder is filling up?

Code: Select all

du --max-depth=3 /usr/local

[JakeHatMacys]

Code: Select all

$ du --max-depth=3 /usr/local
2836    /usr/local/nagvis/share/frontend
1500    /usr/local/nagvis/share/netmap
4       /usr/local/nagvis/share/var
780     /usr/local/nagvis/share/docs
1152    /usr/local/nagvis/share/server
736     /usr/local/nagvis/share/userfiles
7020    /usr/local/nagvis/share
8       /usr/local/nagvis/etc/automaps
20      /usr/local/nagvis/etc/geomap
20      /usr/local/nagvis/etc/maps
92      /usr/local/nagvis/etc
12      /usr/local/nagvis/var/tmpl
16      /usr/local/nagvis/var
7156    /usr/local/nagvis
136     /usr/local/lib/python/pysamba
28356   /usr/local/lib/python
28360   /usr/local/lib
4       /usr/local/include
23232   /usr/local/bin
12528   /usr/local/nrdp/plugins/SunOS
4260    /usr/local/nrdp/plugins/Windows
23632   /usr/local/nrdp/plugins/AIX
3648    /usr/local/nrdp/plugins/Darwin
8344    /usr/local/nrdp/plugins/Linux
952     /usr/local/nrdp/plugins/Generic
53368   /usr/local/nrdp/plugins
28      /usr/local/nrdp/clients/nrds
88      /usr/local/nrdp/clients
104     /usr/local/nrdp/win_clients/NRDS_win_source
4       /usr/local/nrdp/win_clients/tmp
372     /usr/local/nrdp/win_clients/build_files
488     /usr/local/nrdp/win_clients
12      /usr/local/nrdp/configs
36      /usr/local/nrdp/server/plugins
16      /usr/local/nrdp/server/includes
68      /usr/local/nrdp/server
54048   /usr/local/nrdp
3372    /usr/local/nagiosxi/nom/checkpoints
3376    /usr/local/nagiosxi/nom
4       /usr/local/nagiosxi/tmp
504     /usr/local/nagiosxi/scripts
36      /usr/local/nagiosxi/html/views
476     /usr/local/nagiosxi/html/config
108     /usr/local/nagiosxi/html/account
768     /usr/local/nagiosxi/html/reports
1360    /usr/local/nagiosxi/html/images
12      /usr/local/nagiosxi/html/help
596     /usr/local/nagiosxi/html/admin
2796    /usr/local/nagiosxi/html/db
40      /usr/local/nagiosxi/html/perfgraphs
60      /usr/local/nagiosxi/html/dashboards
16      /usr/local/nagiosxi/html/about
95600   /usr/local/nagiosxi/html/includes
12      /usr/local/nagiosxi/html/addons
132     /usr/local/nagiosxi/html/backend
4264    /usr/local/nagiosxi/html/downloads
32      /usr/local/nagiosxi/html/tools
106556  /usr/local/nagiosxi/html
136     /usr/local/nagiosxi/cron
80      /usr/local/nagiosxi/etc/components
88      /usr/local/nagiosxi/etc
4       /usr/local/nagiosxi/var/components
8       /usr/local/nagiosxi/var/subsys
164     /usr/local/nagiosxi/var
64      /usr/local/nagiosxi/tools
110896  /usr/local/nagiosxi
4       /usr/local/share/info
4       /usr/local/share/man/man4x
4       /usr/local/share/man/man2x
4       /usr/local/share/man/man1x
4       /usr/local/share/man/man7x
4       /usr/local/share/man/man3x
4       /usr/local/share/man/man5
4       /usr/local/share/man/man7
4       /usr/local/share/man/man6
4       /usr/local/share/man/man8x
4       /usr/local/share/man/man5x
4       /usr/local/share/man/man9x
4       /usr/local/share/man/man6x
4       /usr/local/share/man/man4
4       /usr/local/share/man/man3
12      /usr/local/share/man/man1
4       /usr/local/share/man/man2
4       /usr/local/share/man/man8
4       /usr/local/share/man/man9
4       /usr/local/share/man/mann
88      /usr/local/share/man
4       /usr/local/share/applications
100     /usr/local/share
4       /usr/local/nagios/include
1608    /usr/local/nagios/bin
1256    /usr/local/nagios/share/pnp
300     /usr/local/nagios/share/ajaxterm
4       /usr/local/nagios/share/media
256     /usr/local/nagios/share/contexthelp
124     /usr/local/nagios/share/locale
7444    /usr/local/nagios/share/images
3908344 /usr/local/nagios/share/perfdata
196     /usr/local/nagios/share/includes
8       /usr/local/nagios/share/docs
4       /usr/local/nagios/share/ssi
100     /usr/local/nagios/share/stylesheets
116     /usr/local/nagios/share/js
3918212 /usr/local/nagios/share
10692   /usr/local/nagios/libexec
5728    /usr/local/nagios/sbin
5208    /usr/local/nagios/etc/services
64      /usr/local/nagios/etc/pnp
1040    /usr/local/nagios/etc/import
84      /usr/local/nagios/etc/etc_back2
12      /usr/local/nagios/etc/static
5208    /usr/local/nagios/etc/hosts
11796   /usr/local/nagios/etc
8       /usr/local/nagios/var/stats
4       /usr/local/nagios/var/rw
184     /usr/local/nagios/var/spool
436     /usr/local/nagios/var/archives
11272   /usr/local/nagios/var
3959316 /usr/local/nagios
28      /usr/local/nagiosmobile/class
28      /usr/local/nagiosmobile/jquery.mobile-1.0/images
280     /usr/local/nagiosmobile/jquery.mobile-1.0
80      /usr/local/nagiosmobile/includes
104     /usr/local/nagiosmobile/js
536     /usr/local/nagiosmobile
4       /usr/local/libexec
4       /usr/local/sbin
du: cannot read directory `/usr/local/lost+found': Permission denied
20      /usr/local/lost+found
4       /usr/local/lib64
4       /usr/local/etc
4       /usr/local/src
108     /usr/local/import_nag_conig/services
8       /usr/local/import_nag_conig/pnp/pages
8       /usr/local/import_nag_conig/pnp/check_commands
64      /usr/local/import_nag_conig/pnp
4       /usr/local/import_nag_conig/import
12      /usr/local/import_nag_conig/static
2020    /usr/local/import_nag_conig/hosts
2400    /usr/local/import_nag_conig
4       /usr/local/games
448     /usr/local/nsis/bin
6044    /usr/local/nsis/share/nsis
6048    /usr/local/nsis/share
6500    /usr/local/nsis
4192600 /usr/local
$

[tgriep]

Looks like the perfdata folder has taken up the most space, run this and post the results.

Code: Select all

du --max-depth=1 /usr/local/nagios/share/perfdata

That folder is where all of the hosts and service's data is stored.

How many hosts and service's are you checking now?
Earlier in your post, you said that you were trying to import 1200 servers, are you still doing that many?

[JakeHatMacys]

Looks like the perfdata folder has taken up the most space, run this and post the results.

Code: Select all

du --max-depth=1 /usr/local/nagios/share/perfdata

That folder is where all of the hosts and service's data is stored.

How many hosts and service's are you checking now?
Earlier in your post, you said that you were trying to import 1200 servers, are you still doing that many?

Yep, we're actually up to 1700 atm which is just Linux & Solaris host that we plan on running file system monitoring for. Really we're stress testing to an extent. But we'll need to use thousands of hosts as we plan on using Nagios XI for in store devices + File system monitoring for the enterprise.

The store devices you're looking at 30 CVT's per store and we have over 900 stores. Each CVT will be a host technically to boot. Is there a recommended threshold where to stop???

And yeah my screen maxed out:

Code: Select all

3020    /usr/local/nagios/share/perfdata/esu2v547
3020    /usr/local/nagios/share/perfdata/s66d0z10

5339840 /usr/local/nagios/share/perfdata
$

I deleted all but 2 of them since I'm sure my boss doesn't want me publishing thousands of server names on a community forum. If this is just performance data is there a setting we can tweak to minimize the data stored? I can talk with my bosses on what they want to see, we did increase the file system to 60 gigs so that should give us plenty of breathing room for now.

[tgriep]

There isn't a way to shrink that folder with out changing the way the plugin's store their information but you could do some performance and database optimizations that will help a bit.
Here are a couple of links to documents that show you how to do that

http://assets.nagios.com/downloads/nagi ... ios-XI.pdf

and

http://assets.nagios.com/downloads/nagi ... zation.pdf

[JakeHatMacys]

Thanks, I'll take a look at these.

Also back to applying configurations. When we apply a config does it only apply what was changed or the entire config every time?

Just curious as we grow the server if we'll have increased wait times on applying the configurations. Right now we have about 2000 hosts and now adding services into the mix. Applying configs takes around 10 to 15 seconds.

Just curious.

[tmcdonald]

Also back to applying configurations. When we apply a config does it only apply what was changed or the entire config every time?

Depends on your version. From our changelog:

- Changed the Apply Configuration process to only write out changed configuration files -SW, AB

This was implemented in 2014R1.1 primarily by our support manager, Andy Brist (abrist on the forums).

[lmiltchev]

Also back to applying configurations. When we apply a config does it only apply what was changed or the entire config every time?

The apply configuration will only touch the files that were modified (after 2014R1.1).

You can easily test this - check the timestamps on the hosts/services:

Code: Select all

ll /usr/local/nagios/etc/hosts
ll /usr/local/nagios/etc/services

Modify one host/service in the CCM and apply configuration, then check the timestamps again. Only the config that was modified would have an "updated" timestamp.