Log Server does not create backups

[mcapra]

You don't even need to touch the old repository at this point. Just create a new one via the NLS GUI located in some other path:

2016_10_14_11_15_52_Backup_Maintenance_Nagios_Log_Server.png

Then set that as the repository NLS will use to store backups:

2016_10_14_11_16_28_Backup_Maintenance_Nagios_Log_Server.png

[vmesquita]

We found that that actually the backup area is running out of space, so that's the likelly cause. However it got full very fast (less than 1 month I think). It would be ideal if we could use a backup solution we have to store the files on the Log Server Backup area and then delete the files (otherwise we'll need a huge ammount of space to keep backups). But as I understand, there's no way to remove part of the content of the backup folder and then put it back for a restore if we need it, right?

[mcapra]

there's no way to remove part of the content of the backup folder and then put it back for a restore if we need it, right?

That is correct. The repositories are pretty all-or-nothing most of the time. If you know which snapshots reference which indices, you could remove specific items in the repository. This can be awfully tricky, though.

If you're worried about the size of your backup repository, why not adjust the "Delete backups older than" value?

[vmesquita]

Ok, a few more questions then:

1) Do every backup contains all closed indexes?

2) Is it possible to adjust the backup frequency to something other than daily?

Unfortunatelly adjusting the "Delete backups older than" only wouldn't solve the problem. We would be forced to keep backups that go back only a few days (unless question #2 is possible and question #1 is true)

[mcapra]

1) Do every backup contains all closed indexes?

The snapshots contain every index that was open when the snapshot was taken. Which is to say each snapshot is quite literally a "snapshot" of what indices are currently open in NLS, not terribly different from how a VM snapshot works. Closed indices are not generally included in snapshots, but if an index was open when a previous snapshot was created then it is effectively backed-up.

2) Is it possible to adjust the backup frequency to something other than daily?

Yes, you can adjust the frequency of these jobs in the "Command Subsystem" section of the Administration page. For snapshots, adjust the "backup_maintenance" job to the desired frequency.

Here's a high-level overview of how elasticsearch snapshots work:
https://www.elastic.co/guide/en/elastic ... shots.html

[vmesquita]

Thanks! A few more questions:
What's the difference between backups and backup_maintenance in the Command Subsystem area?

If I set "Close indexes older than" to 7 days, and set backups to every 7 days, I will have snapshots that contain the last 7 days (i.e., I won't loose any index). Or am I mistaken?

[mcapra]

What's the difference between backups and backup_maintenance in the Command Subsystem area?

backup_maintenance is what handles optimizing/closing/deleting of snapshots and indices. So maybe not totally ideal if all you care about is rate-limiting the creation of snapshots. You would have to adjust other values such as index close/delete periods if your system is reaching capacity. Lots of fudging would go into this. The snapshots process should be separated into it's own "job" to address other issues, but I digress.

backups runs the create_backup.sh script which does the following:

Code: Select all

# Create mapping files with the index mapping data
# Create a backup of each of the indexes and store them in our temp directory
# Compress everything into a single file

If I set "Close indexes older than" to 7 days, and set backups to every 7 days, I will have snapshots that contain the last 7 days (i.e., I won't loose any index). Or am I mistaken?

How many days worth of data you retain via snapshots, in this case, would depend on your "Delete backups older than" setting in addition to "Close indexes older than". However, in saying that if "Close indexes older than" is set to 7 days, then you will actually have 6 days worth of indices per snapshot since the current day's index is not included in the snapshot. The current day's index is not included since it is constantly having data written to it in most environments.

The wording and how things are structured is admittedly confusing. You are not the first person to have brought this up and I suspect you will not be the last

[vmesquita]

Sorry for the late reply. We took a while deciding what to do, and we decided to increase storage space. It also took a while to provide this space.
So I deleted all the contents of the Shared folder and I was able to manually run the backup. So I left as it was and waited for subsequent backups to come up. Next day I tried to access the Backup area on web interface and the server becomes extremally slow and unresponsive. I even tried to open a new tab with the main page, and it doesn't open. Finally I went to the command line and restarted both elasticsearch and logstash services. Then the tab loading the Backup area finally loads, but no backups are shown there (not even the one done the day before, manually).
I did this on both nodes.

Also I took a look at the backup area and it has 50 Gb ocuppied, while the manual Backup I did yesterday used about 1.8Gb.

Any ideas on what happened now?

[rkennedy]

Usually this happens if you're running low on resources, specifically memory. Could you PM over a profile for us to review? (Admin -> System Status -> Download System Profile)

With this we should be able to see where the congestion is on your system.

[vmesquita]

Ok, I just sent you a PM with the system profile.