Old indexes open and backups in progress after red

[CFT6Server]

Our cluster ran into red yesterday during an upgrade, and looks like it opened some really old indexes. I figure the backup/maintenance might just close these and remove them, but looks like it is either stuck, or just not completed.

1. How can I check what the cluster is doing when backup is in progress?
2. Wonder why these old closed/delete indexes are reopened and not closed.
3. Will this just sort itself out, or would it require intervention?

[jolson]

1. How can I check what the cluster is doing when backup is in progress?

The standard procedure is to open up a shell to all of your NLS boxes and run the following command on each of them:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log

After starting the tail, run the backup command from 'Administration -> Command Subsystem'.

2. Wonder why these old closed/delete indexes are reopened and not closed.

I'm not sure why that may have happened. Were there any unusual troubles experienced throughout the course of your upgrade? Running out of memory/disk space or similar?

3. Will this just sort itself out, or would it require intervention?

At worst you may have to close/delete your old indices. From the looks of it, you won't have to do anything beyond that.

[CFT6Server]

So since the backup is already showing "in progress", I went and looked at the jobs log which are empty. Any difference if I just initiate backup again?

[CFT6Server]

Looks like the jobs were still running, but it finally finished and the indexes are closed again and backup is completed.

[jolson]

So since the backup is already showing "in progress", I went and looked at the jobs log which are empty. Any difference if I just initiate backup again?

jobs.log is a unique log file in that it is truncated every minute or so - which is why a 'tail -f' is necessary to view the output of the log. It would grow excessively if it were not truncated often, which is the rationale.

it finally finished and the indexes are closed again and backup is completed.

That's great to hear. Are we good to close this thread?

[CFT6Server]

Thanks. Please close.