Contact Recieving Notification which are removed

[biswajit.banerjee]

Hi
We have been struggling to find out how the notification are going to removed imported contacts . We have checked in system profile but we only have traces of notification sent in logs but object.cache or any other file does not have any traces of these contacts . The sample contacts are rsharma , AmalGhosh , RaviKuhar . Sample Below from Notification screen of XI

Page 1 of 2
Date / Time Host Service Reason Escalated State Contact Dispatcher Information
2018-12-28 12:55:47 UCORPVGRAC03 Total Processes Service Problem No CRITICAL AmalGhosh Custom: notify-service-by-sms PROCS CRITICAL: 393 processes
2018-12-28 12:55:47 UCORPVGRAC03 Total Processes Service Problem No CRITICAL AmalGhosh Custom: notify-service-by-email PROCS CRITICAL: 393 processes
2018-12-28 12:55:42 AM-UAT-ECARE Memory Usage Service Problem Yes CRITICAL AmalGhosh Custom: notify-service-by-sms NRPE: Command 'check_mem' not

I have Attached the System profile for your reference , Please help us to diagnose .

Thanks
Regards
Biswajit Banerjee

[ssax]

You are likely hitting a bug, please run these commands and see if it resolves your issue:

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mysqld || systemctl restart mariadb
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

[biswajit.banerjee]

Thanks for the reply .
We have executed the suggested commands , But it did not sort out the issue .

Do you thing updating to 5.5.8 from 5.5.5 will help.
Or suggest what can be done ?

Thanks
Biswajit

[ssax]

Upgrading likely won't fix the issue but there were many other bugs fixed since 5.5.5, I would upgrade anyways and test, it's not a requirement though.

Please send me a fresh copy of your profile, you can download it from Admin > System Profile > Download Profile.

What is the output of this command on your XI server as well:

Code: Select all

grep -R "rsharma\|AmalGhosh\|RaviKuhar" /usr/local/nagios/etc

[biswajit.banerjee]

Hi ,
Please find the profile as attached
Output of
grep -R "rsharma\|AmalGhosh\|RaviKuhar" /usr/local/nagios/etc
/usr/local/nagios/etc/contacts.cfg_bkp20_2018: contact_name rsharma
/usr/local/nagios/etc/contacts.cfg_bkp20_2018: host_notification_period rsharma_notification_times
/usr/local/nagios/etc/contacts.cfg_bkp20_2018: service_notification_period rsharma_notification_times

Some one from my team must have created the backup .
Any way , we are still receiving alert message for mentioned users .

Regards
Biswajit

[ssax]

Everything looks correct based on your profile.

Did this command run okay when you ran it?

Code: Select all

echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi

What is the output of this command?

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table

[biswajit.banerjee]

Yes, the command
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
Run Okay .

The output of
# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table
+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.00 |
| nagios_commands | 0.02 |
| nagios_commenthistory | 10.32 |
| nagios_comments | 0.01 |
| nagios_configfiles | 0.00 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.39 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.04 |
| nagios_contactgroup_members | 0.00 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 232.95 |
| nagios_contactnotifications | 138.83 |
| nagios_contacts | 0.01 |
| nagios_contactstatus | 0.01 |
| nagios_customvariables | 0.23 |
| nagios_customvariablestatus | 0.25 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 0.00 |
| nagios_eventhandlers | 0.10 |
| nagios_externalcommands | 0.02 |
| nagios_flappinghistory | 4.21 |
| nagios_host_contactgroups | 0.02 |
| nagios_host_contacts | 0.00 |
| nagios_host_parenthosts | 0.00 |
| nagios_hostchecks | 0.00 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.02 |
| nagios_hostescalation_contacts | 0.09 |
| nagios_hostescalations | 0.06 |
| nagios_hostgroup_members | 0.02 |
| nagios_hostgroups | 0.00 |
| nagios_hosts | 0.08 |
| nagios_hoststatus | 0.21 |
| nagios_instances | 0.00 |
| nagios_logentries | 819.52 |
| nagios_notifications | 76.87 |
| nagios_objects | 4.38 |
| nagios_processevents | 0.96 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.00 |
| nagios_service_contactgroups | 0.14 |
| nagios_service_contacts | 0.02 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.00 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.02 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.02 |
| nagios_servicegroup_members | 0.02 |
| nagios_servicegroups | 0.00 |
| nagios_services | 0.90 |
| nagios_servicestatus | 2.22 |
| nagios_statehistory | 200.87 |
| nagios_systemcommands | 0.04 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.03 |
| nagios_timeperiods | 0.01 |
| tbl_command | 0.08 |
| tbl_contact | 0.03 |
| tbl_contactgroup | 0.03 |
| tbl_contacttemplate | 0.03 |
| tbl_domain | 0.03 |
| tbl_host | 0.11 |
| tbl_hostdependency | 0.03 |
| tbl_hostescalation | 0.03 |
| tbl_hostextinfo | 0.03 |
| tbl_hostgroup | 0.03 |
| tbl_hosttemplate | 0.03 |
| tbl_info | 0.17 |
| tbl_lnkContactToCommandHost | 0.02 |
| tbl_lnkContactToCommandService | 0.02 |
| tbl_lnkContactToContactgroup | 0.02 |
| tbl_lnkContactToContacttemplate | 0.02 |
| tbl_lnkContactToVariabledefinition | 0.02 |
| tbl_lnkContactgroupToContact | 0.02 |
| tbl_lnkContactgroupToContactgroup | 0.02 |
| tbl_lnkContacttemplateToCommandHost | 0.02 |
| tbl_lnkContacttemplateToCommandService | 0.02 |
| tbl_lnkContacttemplateToContactgroup | 0.02 |
| tbl_lnkContacttemplateToContacttemplate | 0.02 |
| tbl_lnkContacttemplateToVariabledefinition | 0.02 |
| tbl_lnkHostToContact | 0.02 |
| tbl_lnkHostToContactgroup | 0.02 |
| tbl_lnkHostToHost | 0.02 |
| tbl_lnkHostToHostgroup | 0.02 |
| tbl_lnkHostToHosttemplate | 0.02 |
| tbl_lnkHostToVariabledefinition | 0.02 |
| tbl_lnkHostdependencyToHost_DH | 0.02 |
| tbl_lnkHostdependencyToHost_H | 0.02 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.02 |
| tbl_lnkHostdependencyToHostgroup_H | 0.02 |
| tbl_lnkHostescalationToContact | 0.02 |
| tbl_lnkHostescalationToContactgroup | 0.02 |
| tbl_lnkHostescalationToHost | 0.02 |
| tbl_lnkHostescalationToHostgroup | 0.02 |
| tbl_lnkHostgroupToHost | 0.02 |
| tbl_lnkHostgroupToHostgroup | 0.02 |
| tbl_lnkHosttemplateToContact | 0.02 |
| tbl_lnkHosttemplateToContactgroup | 0.02 |
| tbl_lnkHosttemplateToHost | 0.02 |
| tbl_lnkHosttemplateToHostgroup | 0.02 |
| tbl_lnkHosttemplateToHosttemplate | 0.02 |
| tbl_lnkHosttemplateToVariabledefinition | 0.02 |
| tbl_lnkServiceToContact | 0.02 |
| tbl_lnkServiceToContactgroup | 0.02 |
| tbl_lnkServiceToHost | 0.16 |
| tbl_lnkServiceToHostgroup | 0.02 |
| tbl_lnkServiceToServicegroup | 0.02 |
| tbl_lnkServiceToServicetemplate | 0.16 |
| tbl_lnkServiceToVariabledefinition | 0.13 |
| tbl_lnkServicedependencyToHost_DH | 0.02 |
| tbl_lnkServicedependencyToHost_H | 0.02 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.02 |
| tbl_lnkServicedependencyToHostgroup_H | 0.02 |
| tbl_lnkServicedependencyToService_DS | 0.02 |
| tbl_lnkServicedependencyToService_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.02 |
| tbl_lnkServiceescalationToContactgroup | 0.02 |
| tbl_lnkServiceescalationToHost | 0.02 |
| tbl_lnkServiceescalationToHostgroup | 0.02 |
| tbl_lnkServiceescalationToService | 0.02 |
| tbl_lnkServicegroupToService | 0.02 |
| tbl_lnkServicegroupToServicegroup | 0.02 |
| tbl_lnkServicetemplateToContact | 0.02 |
| tbl_lnkServicetemplateToContactgroup | 0.02 |
| tbl_lnkServicetemplateToHost | 0.02 |
| tbl_lnkServicetemplateToHostgroup | 0.02 |
| tbl_lnkServicetemplateToServicegroup | 0.02 |
| tbl_lnkServicetemplateToServicetemplate | 0.02 |
| tbl_lnkServicetemplateToVariabledefinition | 0.02 |
| tbl_lnkTimeperiodToTimeperiod | 0.02 |
| tbl_logbook | 0.02 |
| tbl_mainmenu | 0.02 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 1.52 |
| tbl_servicedependency | 0.03 |
| tbl_serviceescalation | 0.03 |
| tbl_serviceextinfo | 0.03 |
| tbl_servicegroup | 0.03 |
| tbl_servicetemplate | 0.03 |
| tbl_session | 0.02 |
| tbl_session_locks | 0.02 |
| tbl_settings | 0.03 |
| tbl_submenu | 0.02 |
| tbl_timedefinition | 0.06 |
| tbl_timeperiod | 0.03 |
| tbl_user | 0.03 |
| tbl_variabledefinition | 0.22 |
| xi_auditlog | 11.58 |
| xi_auth_tokens | 4.59 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.02 |
| xi_eventqueue | 0.03 |
| xi_events | 0.63 |
| xi_incidents | 0.02 |
| xi_meta | 8.05 |
| xi_options | 0.06 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.03 |
| xi_usermeta | 1.73 |
| xi_users | 0.06 |
+--------------------------------------------+------------+

[ssax]

That all looks proper as well, let's check the DB values:

Please run these commands and PM me the resulting /tmp/notifications_dump.zip file:

Code: Select all

mysqldump -uroot -pnagiosxi --add-drop-database nagiosxi xi_users xi_usermeta > /tmp/nagiosxi_users.sql
mysqldump -uroot -pnagiosxi --add-drop-database nagios nagios_notifications nagios_objects nagios_contactnotifications nagios_contactnotificationmethods  > /tmp/nagios_objects.sql
zip /tmp/notifications_dump.zip /tmp/nagiosxi_users.sql /tmp/nagios_objects.sql

Thank you

[biswajit.banerjee]

As suggested we have generated /tmp/notifications_dump.zip , but we are not able to upload on this ticket as the size is 71 MB .
We have shared this at (LINK REMOVED, RECEIVED) . Please get from there .
I hope it works

[ssax]

It looks like you have corrupt data in your nagios_objects table:

Code: Select all

|     43183 |           1 |             2 | Ðgœ                                                 |  lœ                                                                                        |         0 |
|     43184 |           1 |             2 | Phœ                                                 | Ðgœ                                                                                        |         0 |
|     43185 |           1 |             2 | pnœ                                                 |  lœ                                                                                        |         0 |
|     43186 |           1 |             2 | Àhœ                                                 | Phœ                                                                                        |         0 |
|     43187 |           1 |             2 | `iœ                                                 |  Ì¥                                                                                        |         0 |
|     43188 |           1 |             2 |  rœ                                                 |  {œ                                                                                        |         0 |
|     43189 |           1 |             2 | pín                                                 |  Ì¥                                                                                        |         0 |

First, take a VM snapshot (if it's a VM) and an XI backup just in case:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf


Then this command to create a backup of your nagios_objects table:

Code: Select all

mysqldump -uroot -pnagiosxi --add-drop-database nagios nagios_objects > /tmp/NAGIOS_OBJECTS_ORIG.sql

Then run this command to truncate the nagios_objects table:

Code: Select all

mysql -uroot -pnagiosxi nagios -e "truncate table nagios_objects;"

Please go to Configure > Core Config Manager > Tools > Config File Management:
- Then click the Write Configs button
- Now apply configuration

Then once that's done, test to see if you're still seeing the issue.

Then run this command and send me the resulting /tmp/FRESHNAGIOSOBJECTS.sql file so that I can review it:

Code: Select all

mysqldump -uroot -pnagiosxi --add-drop-database nagios nagios_objects > /tmp/FRESHNAGIOSOBJECTS.sql

In the event you have issues and need to revert the nagios_objects table you can run these commands to do it:

Code: Select all

mysql -uroot -pnagiosxi nagios < /tmp/NAGIOS_OBJECTS_ORIG.sql
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

Thank you