Page 1 of 1
Long Apply Config Time with Large Imports
Posted: Thu Aug 04, 2016 2:18 am
by Deantwo
In an attempt to make the system more automated I have created a little bash script that I can call externally to create an new host using specific templates.
The script works nicely and is not really my issue, the problem is that with the amount of hosts I am importing at a time is making "Apply Configuration" run for a rather long time.
I add about 850 files to the
/usr/local/nagios/etc/import folder at a time, this is in order to override existing host configurations with possible changes. Then I apply the configuration using NagiosXI's web interface.
I would use
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh, but since it takes so long to complete I decided against it for now because it would freeze my application while it waits for it to finish.
So my question is, is there anyway I can speed this up?
Of course now I am thinking up ways of how I can improve this on my end, but still curious about if there is anything that can be done on the Nagios side.
It is kinda hilarious how many dots the "Apply Configuration" screen will append. Does it make one for each imported file?
Apply Slowly.png
Re: Long Apply Config Time with Large Imports
Posted: Thu Aug 04, 2016 9:43 am
by lmiltchev
When you are importing so many configs at a time, apply configuration will take a long time. I believe each dot corresponds to each second.
Most probably, apply configuration is timing out.
Can you show us the resource limits in the "/etc/php.ini" file? Run the following command and show the output:
Code: Select all
grep 'max_execution_time =\|max_input_time =\|memory_limit\|max_input_vars =' /etc/php.ini
Re: Long Apply Config Time with Large Imports
Posted: Fri Aug 05, 2016 1:06 am
by Deantwo
lmiltchev wrote:Can you show us the resource limits in the "/etc/php.ini" file? Run the following command and show the output:
Code: Select all
grep 'max_execution_time =\|max_input_time =\|memory_limit\|max_input_vars =' /etc/php.ini
Code: Select all
[root@localhost ~]# grep 'max_execution_time =\|max_input_time =\|memory_limit\|max_input_vars =' /etc/php.ini
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
[root@localhost ~]#
Would increasing this maybe make it faster?
I didn't really say how long it actually takes to apply the configuration with all the import files, I am guessing it is like 5 minutes or so, but I haven't really timed it.
Re: Long Apply Config Time with Large Imports
Posted: Fri Aug 05, 2016 10:02 am
by ssax
As long as it completes the only thing you can really do is add faster disks, more RAM, more CPU, etc. Actually, you might be able to setup a RAMDisk and mount it to the
/usr/local/nagios/etc/import directory but I haven't tested it yet, but that will only speed up the import. The apply configuration imports the data into the DB, exports the data to files, and then restarts the nagios service. You could see where it's taking the most time by running this tail command:
Code: Select all
tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Then apply configuration again and watch the tail output and it will show you what it's doing.
Thank you
Re: Long Apply Config Time with Large Imports
Posted: Tue Aug 09, 2016 1:25 am
by Deantwo
ssax wrote:You could see where it's taking the most time by running this tail command:
Code: Select all
tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Then apply configuration again and watch the tail output and it will show you what it's doing.
Most I see stuff like this repeated over and over:
Code: Select all
IMPORTING /usr/local/nagios/etc/import/10.x.x.1.cfg
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=admin&type=import&importsubmitted=true&chbOverwrite=1&subForm=Import&selImportFile[]=/usr/local/nagios/etc/import/10.x.x.1.cfg' -O nagiosql.import.monitoring
--2016-08-09 08:21:18-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.import.monitoring”
0K .......... .......... .......... .......... .......... 88.0K
50K .......... .......... .......... .......... .......... 43.7M
100K .......... .......... .......... .......... .......... 47.3M
150K .......... .......... .. 1016K=0.6s
2016-08-09 08:21:19 (291 KB/s) - “nagiosql.import.monitoring” saved [176237]
Of course it is not for the same file more than once.
So this is where it is loading the files into the database I guess?
Re: Long Apply Config Time with Large Imports
Posted: Tue Aug 09, 2016 12:30 pm
by tgriep
One thing you can try is to do some changes in the php.ini file and see if that helps.
Can you edit the /etc/php.ini file and change the following from
Code: Select all
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
to
Code: Select all
max_execution_time = 60
max_input_time = 120
memory_limit = 512M
Save the file and restart apache by running
Try this to see if it helps out.