Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

NagiosXI Bulk modification tool not working properly

https://support.nagios.com/forum/viewtopic.php?t=53307

Page 1 of 2

NagiosXI Bulk modification tool not working properly

Posted: Wed Apr 03, 2019 2:48 pm
by thanigaivel.a
I've been using nagiosxi for a while and observed that NagiosXI Bulk modification tool was not working properly.
After any bulk update it's not reflecting across the services. kindly provide a solution.

Eg. Tried to switch on the "Notification enabled" to "ON", but bulk modification tool not enabled to all the services which I mentioned.

Re: NagiosXI Bulk modification tool not working properly

Posted: Wed Apr 03, 2019 4:20 pm
by ssax
You have an issue with your commands for your event handlers:

Code: Select all

Feb 26 13:38:57 usa0300lv6348 nagios: wproc: SERVICE EVENTHANDLER job 3536 from worker Core Worker 28357 is a non-check helper but exited with return code 1
Feb 26 13:38:57 usa0300lv6348 nagios: wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
Feb 26 13:38:57 usa0300lv6348 nagios: wproc:   stderr line 01: /bin/sh: -c: line 0: syntax error near unexpected token `('
Feb 26 13:38:57 usa0300lv6348 nagios: wproc:   stderr line 02: /bin/sh: -c: line 0: `/usr/local/nagios/libexec/event_handler_service discoverylab.sdsp.mc.xerox.com discoverylab 10.15.24.48 Memory_Usage Memory_Usage WARNING SOFT 162450 162315 336481  WARNING - Memory usage: total:1905140 MB - used:1832080 MB(96%) - free:73060 MB(4%):Top 5 Memory Processes(memKB,pname,pid):  175441 automount 2069, 153597 ManagementAgent 3868, 147252 EracentEPAServi 2322, 147252 EracentEPAServi 1520, 143992 EracentEDAServi 2302  '
Go to Configure > Core Config Manager > Commands:
- Search for event_handler_service
- Change all of the ones you added from this:

Code: Select all

$USER1$/event_handler_service $HOSTNAME$ $HOSTALIAS$ $HOSTADDRESS$ $SERVICEDISPLAYNAME$ $SERVICEDESC$ $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEPROBLEMID$ $LASTSERVICEPROBLEMID$ $SERVICEEVENTID$  $SERVICEOUTPUT$  $LONGSERVICEOUTPUT$
To surrounding the ones that can contain special characters in single quotes like this:

Code: Select all

$USER1$/event_handler_service '$HOSTNAME$' '$HOSTALIAS$' $HOSTADDRESS$ '$SERVICEDISPLAYNAME$' '$SERVICEDESC$' $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEPROBLEMID$ $LASTSERVICEPROBLEMID$ $SERVICEEVENTID$  '$SERVICEOUTPUT$'  '$LONGSERVICEOUTPUT$'
Save and Apply Configuration.

Then run these commands:

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl start httpd
systemctl restart snmptt
Then once it's back up and running, go to Configure > Core Config Manager > Tools > Config File Management:
- Click the Delete Files button (don't worry, it's safe, they will be rewritten)
- Then click the Write Configs button
- Once the files are done being written:

Run this tail command (and leave it running):

Code: Select all

tail -F /usr/local/nagiosxi/var/cmdsubsys.log /var/log/httpd/error_log /var/log/httpd/ssl_error_log
Then re-run through the Bulk Mods Tool changes again and see if it works properly now.

If it did NOT work properly, please send me a fresh copy of your profile and the full output from the command above.

Additionally, please let me know the exact name of a host/service that it DID work for and the host/service name of one that DID NOT get changed.

Thank you.

Re: NagiosXI Bulk modification tool not working properly

Posted: Thu Apr 04, 2019 8:36 am
by gnikolov
Hi, I have noticed a similar issue(though it is on 5.4.13), I found out that the bulk modification tool will update about ~300 services or hosts and then the next don't get edited.
When deleting I cannot delete more than the same amount(I have tried with 600, 1000 and 2900 and it spits error) - do you experience this as well?
I will be following this topic to see if there is resolution.

Re: NagiosXI Bulk modification tool not working properly

Posted: Thu Apr 04, 2019 9:37 am
by lmiltchev
In order to eliminate timeouts, and issues with not being able to modify a large number of objects, try increasing the value of the max_input_vars in the /etc/php.ini file, for example:

Code: Select all

max_input_vars = 100000

Re: NagiosXI Bulk modification tool not working properly

Posted: Fri Apr 05, 2019 7:22 am
by thanigaivel.a
After the work around you have given, now its working fine.

However, I'm not sure same issues will reoccur or not if I update more than 500 or so... services in a single go.

Now, I've changed the event_handlers syntax with single quotes ''.

Attached the updated profile. Could you please confirm now are you seeing any issues from the profile ?

Re: NagiosXI Bulk modification tool not working properly

Posted: Fri Apr 05, 2019 11:02 am
by ssax
Please attach these files as well:

Code: Select all

/etc/php.ini
/etc/my.cnf

Thank you!

Re: NagiosXI Bulk modification tool not working properly

Posted: Fri Apr 05, 2019 11:07 am
by ssax
It seems okay based on your profile but please send a fresh copy of your profile so that I can review the latest logs.

Looks like you're close to running out of space, make sure to keep an eye on it"

Code: Select all

Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/vg00-lv_root       45G   37G  8.3G  82% /
devtmpfs                      5.5G     0  5.5G   0% /dev
tmpfs                         5.5G     0  5.5G   0% /dev/shm
tmpfs                         5.5G     0  5.5G   0% /sys/fs/cgroup
tmpfs                         5.5G  570M  5.0G  11% /run
/dev/sda1                     976M  190M  719M  21% /boot
/dev/mapper/vg00-lv_export    4.0G  2.8G  1.3G  70% /export
/dev/mapper/vg00-lv_openv      10G  1.3G  8.8G  13% /usr/openv
/dev/mapper/vg00-lv_opt        50G  248M   50G   1% /opt
/dev/mapper/vg00-lv_var        90G   30G   61G  33% /var
/dev/mapper/vg00-lv_tmp        10G   36M   10G   1% /tmp
/dev/mapper/vg00-lv_audit    1014M  961M   54M  95% /var/log/audit
tmpfs                         800M   30M  771M   4% /var/nagiosramdisk
/dev/mapper/vg00-lv_nagiosxi   50G   36G   14G  73% /usr/local
tmpfs                         1.1G     0  1.1G   0% /run/user/DDD
vnx-01:/fs101/home/XXX  3.6T  3.0T  650G  83% /home/XXX
tmpfs                         1.1G     0  1.1G   0% /run/user/DDD
ns80:/fs102/home/XXX     3.1T  2.9T  225G  93% /home/XXX
tmpfs                         1.1G     0  1.1G   0% /run/user/DDD
tmpfs                         1.1G     0  1.1G   0% /run/user/DDD
ns80:/fs001/home/XXX629G  625G  4.4G 100% /home/XXX

Re: NagiosXI Bulk modification tool not working properly

Posted: Mon Apr 08, 2019 1:15 pm
by thanigaivel.a

Code: Select all

[root@usa0300lv6348 libexec]# cat /etc/my.cnf
[mysqld]
innodb_file_per_table=1

innodb_file_per_table=1

innodb_file_per_table=1

query_cache_size=16M
query_cache_limit=4M
tmp_table_size=64M
max_heap_table_size=64M
key_buffer_size=32M
table_open_cache=32

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
Attached the profile and php.ini. copied my.cnf in code section

Re: NagiosXI Bulk modification tool not working properly

Posted: Mon Apr 08, 2019 1:24 pm
by ssax
You have a ton of these:
- The commands I sent below will clean them up but you should investigate why the ones listed are not finishing properly.

Code: Select all

ps aux | grep '/usr/bin/ssh'
You're also swapping, add some more memory:

total used free shared buff/cache available
Mem: 11227 2192 839 577 8195 7840
Swap: 8191 101 8090

Please edit your /etc/php.ini and change these:

Code: Select all

max_execution_time = 90
max_input_time = 90
memory_limit = 512M
To these:

Code: Select all

max_execution_time = 300
max_input_time = 300
memory_limit = 1G
Then run these commands and see if the issue is now resolved when you try it again:

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mysqld || systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl start httpd
systemctl restart snmptt

Re: NagiosXI Bulk modification tool not working properly

Posted: Wed Apr 10, 2019 8:47 am
by thanigaivel.a
Regarding "ps -ef |grep /usr/bin/ssh"

we have configured all servers in nagios to monitor server accessibility through check_by_ssh plugin to track server is accessible via ssh or not, sometime due to hung state server will goes in to hung state.

As per your suggestion, Increased the ram size and update the php.ini values.
Attached the updated profile .
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited