old notification data

[teh0015]

We had an issue on our XI box and I had to roll back to a previous snapshot (it's in vsphere).
Logging in a bunch of my dashboard widgets were returning " './nagios/nagios_servicestatus' is marked as crashed" so I ran
"/usr/local/nagiosxi/scripts/repairmysql.sh nagios"

Everything seems back but if I go to 'Notifications' and try to select 'last year' it returns nothing.
The earliest notification I can pull up is from 1/3/18 - nothing from 17.
Performance data on certain monitors does go back into 17 but no notification data.

[npolovenko]

Hello, @teh0015.
How are you generating the notifications report? Are you using the Notifications report on the Home page? If so, try to go to the Reports menu, and run the Notifications report from there. Another option is to run the Notification Report from the legacy reports section:

Untitled.png

Reports->Legacy Reports->Notifications

[teh0015]

Going through Home->Notifications I can't see anything earlier than 1/3/18.
If I try to look at 'last year' I get nothing.
If I put in a 'custom' date with start sometime in 17 the earliest result I get is 1/3/18

Going through Reports->Available Reports->Notifications is the same.

Going through Reports->Legacy Reports->Notifications I * see notifications before 1/3/18.
(I only went back as early as 11/17 so I don't know how far it goes)
I've got eventman.log-* files going back to may of 17 in /usr/local/nagiosxi/var/

[npolovenko]

Going through Reports->Legacy Reports->Notifications I * see notifications before 1/3/18.

@teh0015, The first two reports pull the data from the MYSQL database, and the third one uses the actual log files. Looks like you're still able to use the third report option. Assuming that the database tables that used to store old notifications got corrupted there's not much we can do, except if you had made a backup before the incident happened?

[teh0015]

I've crawled our backups and this issue exists as far back as we have them.

Is there a way to import them ?

[npolovenko]

@teh0015, You could dump everything from the notifications table to see if the old entries are still there:

Code: Select all

 echo "select * from nagios_contactnotifications;" | mysql -uroot -pnagiosxi nagiosxi

However, I checked my DB settings, and the Notification period is set to 90 days. You can check the same on your system:

Untitled.png

So the reason why last years entries are gone is probably because of the retention period rather than because of corrupted tables.

[teh0015]

interesting ... so the default is to only hold notification data for 90 days?

How much of a hit would it be for me to set that option to 2 years - and how would I go about getting all of the previous data into the db ?
/usr/local/nagios/var/archives/ *has all of the data.
I checked the db and it wasn't in there though.

Code: Select all

mysql> select start_time, output from nagios_notifications where start_time like '%2017%';
Empty set, 1 warning (0.08 sec)

As per the config option you pointed out it only goes back 90 days.

It's obviously not something that comes up much as it must have always been like this and we've only just noticed it.
The example is wanting to pull a list of notifications on a specific day in 12/17 to verify what notifications got sent to who.
I can grep through files find what I want but my boss's boss is probably going to be less than happy with having to do that.
... which is what led to this thread.

[npolovenko]

@teh0015, I haven't done it personally, but you could set it to two years. As long as you keep an eye on size of the "nagios_notifications" table:

Code: Select all

mysql -uroot -pnagiosxi nagios

Code: Select all

SELECT table_name AS "Table", ROUND(((data_length + index_length) / 1024 / 1024), 2) AS "Size (MB)" FROM information_schema.TABLES WHERE table_schema = "nagios" ORDER BY (data_length + index_length) DESC;

You don't want your system to run out of space. It's hard to project how big your table is going to be in a year because that depends on: number of services and hosts and number of state changes and notifications during the period.

We don't have a script to reimport the entries back into the database and we haven't really tried doing that. You may try coming up with your own script or contact the sales for a custom feature development.

In your case, I would suggest using a third party log-reading solution, like Nagios Log Server or similar, that will import all the log files from /usr/local/nagios/var/archives/ folder and allow you to search for what you need using custom queries.