what to be done if nagios log server is almost full

[hlyeung]

Hi,
we have deployed three nagios log servers and now the disk will be full soon. what is the best solution to add space to it?
can we add volume and then copy old data volume for each servers?
thanks.

Regards,
Lam

[tgriep]

Here is a manual on changing data paths for log server but it can be used to add space to the server.
The section "Adding Multiple Paths" describes that.

http://assets.nagios.com/downloads/nagi ... Server.pdf

[eloyd]

So more generally, rather than adding disk space, is there a way to delete old logs? Say, nuke or archive stuff 90 days or older?

[lgroschen]

@eloyd in the Backup & Maintenance section you can set the amount of days to keep the backups.

@hlyeung:
There are a few options- I'll give you the one I think is the best:

You can always add another instance (server) into your cluster and then all the shards that are allocated amongst the cluster (1 primary shard and 1 backup shard per instance) will include the new server and the disk usage will be spread evenly.

I recommend this option since it will increase the clusters overall performance. Searching, displaying, querying, and filtering of logs will be faster, more space on drives for the cluster as a whole. Better yet you can add 2 more servers if you anticipate having a large amount of data in the future. Then when you come to a point in the clusters life where you are closing in on 50% drive capacity on all of the servers, that might be a good time to bring one server out of the 5 in the cluster down to upgrade the hard drive space. Since you have 4 servers to carry the load that the 5th had when you bring it down you will not lose any data, then add the upgraded server back into the cluster. Then upgrade each instance in turn until you are back to an acceptable disk usage %.

Keep in mind the license you have or plan to buy so you are able to increase the number of instances to a comfortable number without reaching a cap.

[eloyd]

But that doesn't really do what I asked.

is there a way to delete old logs? Say, nuke or archive stuff 90 days or older?

I mean, if I only need 90 days worth of material for auditing and security purposes (let's just pretend) then I save them for an extra 90 days just to be sure, and I don't care about stuff that's older than 180 days, is there a way to delete that 180+ day information?

Similar to log file rotation, I don't need kernel logs from six months (or six years) ago, just the past six weeks, most likely.

[lgroschen]

eloyd,

I was answering the OPs post in that reponse. So I was answering what he asked. I also edited that post after and answered your question.

[eloyd]

Sorry, didn't mean to sound upset. It's hard to convey proper connotation in a forum post.

Thanks for the edit. I'll dig deeper into the admin screen. Is there a way to be more granular? Say, delete host X data after 30 days but host Y data after 90 and everything after 180?

[lgroschen]

The part that I'm certain about is when you set the amount of days it will control the indices being active or not, but the trick and the part i'm not as experienced with is setting the backups and removing, relocating or deleting them. If you offload them it makes the backups a lot easier to manage since they won't take any room up on your local machine.

I think there isn't actually a way to set X and Y backups, but I think that would be something awesome to be added (if it isn't already - i'll ask around) so I'll figure that out today.

[eloyd]

Okay, but to be clear, I'm not asking about backups. I'm asking about retention. Keep host X for A days, keep host Y for B days. No need to back 'em up, just delete 'em after the appropriate number of days.

[lgroschen]

Yes, there is a field in backups that says "Delete indexes older than:" X (days) which can be disabled by setting to 0.