Page 1 of 2
Contact Recieving Notification which are removed
Posted: Fri Dec 28, 2018 2:31 am
by biswajit.banerjee
Hi
We have been struggling to find out how the notification are going to removed imported contacts . We have checked in system profile but we only have traces of notification sent in logs but object.cache or any other file does not have any traces of these contacts . The sample contacts are rsharma , AmalGhosh , RaviKuhar . Sample Below from Notification screen of XI
Page 1 of 2
Date / Time Host Service Reason Escalated State Contact Dispatcher Information
2018-12-28 12:55:47 UCORPVGRAC03 Total Processes Service Problem No CRITICAL AmalGhosh Custom: notify-service-by-sms PROCS CRITICAL: 393 processes
2018-12-28 12:55:47 UCORPVGRAC03 Total Processes Service Problem No CRITICAL AmalGhosh Custom: notify-service-by-email PROCS CRITICAL: 393 processes
2018-12-28 12:55:42 AM-UAT-ECARE Memory Usage Service Problem Yes CRITICAL AmalGhosh Custom: notify-service-by-sms NRPE: Command 'check_mem' not
I have Attached the System profile for your reference , Please help us to diagnose .
Thanks
Regards
Biswajit Banerjee
Re: Contact Recieving Notification which are removed
Posted: Fri Dec 28, 2018 11:13 am
by ssax
You are likely hitting a bug, please run these commands and see if it resolves your issue:
Code: Select all
systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mysqld || systemctl restart mariadb
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
Re: Contact Recieving Notification which are removed
Posted: Fri Dec 28, 2018 11:53 pm
by biswajit.banerjee
Thanks for the reply .
We have executed the suggested commands , But it did not sort out the issue .
Do you thing updating to 5.5.8 from 5.5.5 will help.
Or suggest what can be done ?
Thanks
Biswajit
Re: Contact Recieving Notification which are removed
Posted: Wed Jan 02, 2019 11:46 am
by ssax
Upgrading likely won't fix the issue but there were many other bugs fixed since 5.5.5, I would upgrade anyways and test, it's not a requirement though.
Please send me a fresh copy of your profile, you can download it from Admin > System Profile > Download Profile.
What is the output of this command on your XI server as well:
Code: Select all
grep -R "rsharma\|AmalGhosh\|RaviKuhar" /usr/local/nagios/etc
Re: Contact Recieving Notification which are removed
Posted: Wed Jan 02, 2019 8:19 pm
by biswajit.banerjee
Hi ,
Please find the profile as attached
Output of
grep -R "rsharma\|AmalGhosh\|RaviKuhar" /usr/local/nagios/etc
/usr/local/nagios/etc/contacts.cfg_bkp20_2018: contact_name rsharma
/usr/local/nagios/etc/contacts.cfg_bkp20_2018: host_notification_period rsharma_notification_times
/usr/local/nagios/etc/contacts.cfg_bkp20_2018: service_notification_period rsharma_notification_times
Some one from my team must have created the backup .
Any way , we are still receiving alert message for mentioned users .
Regards
Biswajit
Re: Contact Recieving Notification which are removed
Posted: Thu Jan 03, 2019 10:09 am
by ssax
Everything looks correct based on your profile.
Did this command run okay when you ran it?
Code: Select all
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
What is the output of this command?
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table
Re: Contact Recieving Notification which are removed
Posted: Thu Jan 03, 2019 11:32 pm
by biswajit.banerjee
Yes, the command
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
Run Okay .
The output of
# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table
+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.00 |
| nagios_commands | 0.02 |
| nagios_commenthistory | 10.32 |
| nagios_comments | 0.01 |
| nagios_configfiles | 0.00 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.39 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.04 |
| nagios_contactgroup_members | 0.00 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 232.95 |
| nagios_contactnotifications | 138.83 |
| nagios_contacts | 0.01 |
| nagios_contactstatus | 0.01 |
| nagios_customvariables | 0.23 |
| nagios_customvariablestatus | 0.25 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 0.00 |
| nagios_eventhandlers | 0.10 |
| nagios_externalcommands | 0.02 |
| nagios_flappinghistory | 4.21 |
| nagios_host_contactgroups | 0.02 |
| nagios_host_contacts | 0.00 |
| nagios_host_parenthosts | 0.00 |
| nagios_hostchecks | 0.00 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.02 |
| nagios_hostescalation_contacts | 0.09 |
| nagios_hostescalations | 0.06 |
| nagios_hostgroup_members | 0.02 |
| nagios_hostgroups | 0.00 |
| nagios_hosts | 0.08 |
| nagios_hoststatus | 0.21 |
| nagios_instances | 0.00 |
| nagios_logentries | 819.52 |
| nagios_notifications | 76.87 |
| nagios_objects | 4.38 |
| nagios_processevents | 0.96 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.00 |
| nagios_service_contactgroups | 0.14 |
| nagios_service_contacts | 0.02 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.00 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.02 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.02 |
| nagios_servicegroup_members | 0.02 |
| nagios_servicegroups | 0.00 |
| nagios_services | 0.90 |
| nagios_servicestatus | 2.22 |
| nagios_statehistory | 200.87 |
| nagios_systemcommands | 0.04 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.03 |
| nagios_timeperiods | 0.01 |
| tbl_command | 0.08 |
| tbl_contact | 0.03 |
| tbl_contactgroup | 0.03 |
| tbl_contacttemplate | 0.03 |
| tbl_domain | 0.03 |
| tbl_host | 0.11 |
| tbl_hostdependency | 0.03 |
| tbl_hostescalation | 0.03 |
| tbl_hostextinfo | 0.03 |
| tbl_hostgroup | 0.03 |
| tbl_hosttemplate | 0.03 |
| tbl_info | 0.17 |
| tbl_lnkContactToCommandHost | 0.02 |
| tbl_lnkContactToCommandService | 0.02 |
| tbl_lnkContactToContactgroup | 0.02 |
| tbl_lnkContactToContacttemplate | 0.02 |
| tbl_lnkContactToVariabledefinition | 0.02 |
| tbl_lnkContactgroupToContact | 0.02 |
| tbl_lnkContactgroupToContactgroup | 0.02 |
| tbl_lnkContacttemplateToCommandHost | 0.02 |
| tbl_lnkContacttemplateToCommandService | 0.02 |
| tbl_lnkContacttemplateToContactgroup | 0.02 |
| tbl_lnkContacttemplateToContacttemplate | 0.02 |
| tbl_lnkContacttemplateToVariabledefinition | 0.02 |
| tbl_lnkHostToContact | 0.02 |
| tbl_lnkHostToContactgroup | 0.02 |
| tbl_lnkHostToHost | 0.02 |
| tbl_lnkHostToHostgroup | 0.02 |
| tbl_lnkHostToHosttemplate | 0.02 |
| tbl_lnkHostToVariabledefinition | 0.02 |
| tbl_lnkHostdependencyToHost_DH | 0.02 |
| tbl_lnkHostdependencyToHost_H | 0.02 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.02 |
| tbl_lnkHostdependencyToHostgroup_H | 0.02 |
| tbl_lnkHostescalationToContact | 0.02 |
| tbl_lnkHostescalationToContactgroup | 0.02 |
| tbl_lnkHostescalationToHost | 0.02 |
| tbl_lnkHostescalationToHostgroup | 0.02 |
| tbl_lnkHostgroupToHost | 0.02 |
| tbl_lnkHostgroupToHostgroup | 0.02 |
| tbl_lnkHosttemplateToContact | 0.02 |
| tbl_lnkHosttemplateToContactgroup | 0.02 |
| tbl_lnkHosttemplateToHost | 0.02 |
| tbl_lnkHosttemplateToHostgroup | 0.02 |
| tbl_lnkHosttemplateToHosttemplate | 0.02 |
| tbl_lnkHosttemplateToVariabledefinition | 0.02 |
| tbl_lnkServiceToContact | 0.02 |
| tbl_lnkServiceToContactgroup | 0.02 |
| tbl_lnkServiceToHost | 0.16 |
| tbl_lnkServiceToHostgroup | 0.02 |
| tbl_lnkServiceToServicegroup | 0.02 |
| tbl_lnkServiceToServicetemplate | 0.16 |
| tbl_lnkServiceToVariabledefinition | 0.13 |
| tbl_lnkServicedependencyToHost_DH | 0.02 |
| tbl_lnkServicedependencyToHost_H | 0.02 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.02 |
| tbl_lnkServicedependencyToHostgroup_H | 0.02 |
| tbl_lnkServicedependencyToService_DS | 0.02 |
| tbl_lnkServicedependencyToService_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.02 |
| tbl_lnkServiceescalationToContactgroup | 0.02 |
| tbl_lnkServiceescalationToHost | 0.02 |
| tbl_lnkServiceescalationToHostgroup | 0.02 |
| tbl_lnkServiceescalationToService | 0.02 |
| tbl_lnkServicegroupToService | 0.02 |
| tbl_lnkServicegroupToServicegroup | 0.02 |
| tbl_lnkServicetemplateToContact | 0.02 |
| tbl_lnkServicetemplateToContactgroup | 0.02 |
| tbl_lnkServicetemplateToHost | 0.02 |
| tbl_lnkServicetemplateToHostgroup | 0.02 |
| tbl_lnkServicetemplateToServicegroup | 0.02 |
| tbl_lnkServicetemplateToServicetemplate | 0.02 |
| tbl_lnkServicetemplateToVariabledefinition | 0.02 |
| tbl_lnkTimeperiodToTimeperiod | 0.02 |
| tbl_logbook | 0.02 |
| tbl_mainmenu | 0.02 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 1.52 |
| tbl_servicedependency | 0.03 |
| tbl_serviceescalation | 0.03 |
| tbl_serviceextinfo | 0.03 |
| tbl_servicegroup | 0.03 |
| tbl_servicetemplate | 0.03 |
| tbl_session | 0.02 |
| tbl_session_locks | 0.02 |
| tbl_settings | 0.03 |
| tbl_submenu | 0.02 |
| tbl_timedefinition | 0.06 |
| tbl_timeperiod | 0.03 |
| tbl_user | 0.03 |
| tbl_variabledefinition | 0.22 |
| xi_auditlog | 11.58 |
| xi_auth_tokens | 4.59 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.02 |
| xi_eventqueue | 0.03 |
| xi_events | 0.63 |
| xi_incidents | 0.02 |
| xi_meta | 8.05 |
| xi_options | 0.06 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.03 |
| xi_usermeta | 1.73 |
| xi_users | 0.06 |
+--------------------------------------------+------------+
Re: Contact Recieving Notification which are removed
Posted: Fri Jan 04, 2019 10:08 am
by ssax
That all looks proper as well, let's check the DB values:
Please run these commands and PM me the resulting
/tmp/notifications_dump.zip file:
Code: Select all
mysqldump -uroot -pnagiosxi --add-drop-database nagiosxi xi_users xi_usermeta > /tmp/nagiosxi_users.sql
mysqldump -uroot -pnagiosxi --add-drop-database nagios nagios_notifications nagios_objects nagios_contactnotifications nagios_contactnotificationmethods > /tmp/nagios_objects.sql
zip /tmp/notifications_dump.zip /tmp/nagiosxi_users.sql /tmp/nagios_objects.sql
Thank you
Re: Contact Recieving Notification which are removed
Posted: Fri Jan 04, 2019 11:59 am
by biswajit.banerjee
As suggested we have generated /tmp/notifications_dump.zip , but we are not able to upload on this ticket as the size is 71 MB .
We have shared this at (LINK REMOVED, RECEIVED) . Please get from there .
I hope it works
Re: Contact Recieving Notification which are removed
Posted: Fri Jan 04, 2019 4:26 pm
by ssax
It looks like you have corrupt data in your nagios_objects table:
Code: Select all
| 43183 | 1 | 2 | Ðgœ | lœ | 0 |
| 43184 | 1 | 2 | Phœ | Ðgœ | 0 |
| 43185 | 1 | 2 | pnœ | lœ | 0 |
| 43186 | 1 | 2 | Àhœ | Phœ | 0 |
| 43187 | 1 | 2 | `iœ | Ì¥ | 0 |
| 43188 | 1 | 2 | rœ | {œ | 0 |
| 43189 | 1 | 2 | pín | Ì¥ | 0 |
Then this command to create a backup of your nagios_objects table:
Code: Select all
mysqldump -uroot -pnagiosxi --add-drop-database nagios nagios_objects > /tmp/NAGIOS_OBJECTS_ORIG.sql
Then run this command to truncate the nagios_objects table:
Code: Select all
mysql -uroot -pnagiosxi nagios -e "truncate table nagios_objects;"
Please go to Configure > Core Config Manager > Tools > Config File Management:
- Then click the Write Configs button
- Now apply configuration
Then once that's done, test to see if you're still seeing the issue.
Then run this command and send me the resulting
/tmp/FRESHNAGIOSOBJECTS.sql file so that I can review it:
Code: Select all
mysqldump -uroot -pnagiosxi --add-drop-database nagios nagios_objects > /tmp/FRESHNAGIOSOBJECTS.sql
Code: Select all
mysql -uroot -pnagiosxi nagios < /tmp/NAGIOS_OBJECTS_ORIG.sql
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
Thank you