missing service alert notification

[scharft]

Hi,

we have configured service alerts to send notification to our ticketsystem but for a period of time, the notification were not send for a specific service "DISK-F" on 2 hosts and we are wondering why.
Is it possible to find out if a notification was disabled or the host was in a downtime?
Can you assist please during a remote session or do you neet specific log files?


Best Regards
Thomas

[scottwilkerson]

The easiest thing to do would be to go to
Reports -> Event Log
search for hostname top right
change time period to cover what you are looking for and verify if it was changed, acknowledged, went in downtime, etc.

[scharft]

Thats weird, i don't find anything.. please see the attachment.

One Step back, we use the Service Alert notification and it stops, why?

Can you assist please?

[scottwilkerson]

without the search do you see anything?
If you go to state history report do you see this server?

[scharft]

withour the search, no i don't see anything.. hm..
Yes, on the state history report i see the host and his events



between 3AM & 8PM on April 1st
Why Service alert notification to ServiceNow stopped?

[scottwilkerson]

withour the search, no i don't see anything.. hm..

this leads me to believe you may have db corruption
do you see errors when you run this

Code: Select all

tail -50 /var/log/mysqld.log

If so you may need to repair the databases
https://assets.nagios.com/downloads/nag ... tabase.pdf

[scharft]

in the repository /var/log/ is no mysqld.log file, i checked this also on another instance, It's the same.
so you mean we have a db error on 2 nagios instances?

btw: for us it's neccassary to clear the situation of:
between 3AM & 8PM on April 1st
Why Service alert notification to ServiceNow stopped?

Reason: we had an outage and need this for an rca, please assist asap!

[scottwilkerson]

Please send your system profile.zip from Admin -> System Profile

so we can analyze the logs

[scharft]

thank you for analyzing the log.
The zip file is attached.

[scottwilkerson]

So you have a newer system and the log name is /var/log/mariadb/mariadb.log and it does show database corruption which would cause all these issues.

You need to repair the databases
https://assets.nagios.com/downloads/nag ... tabase.pdf