NLS runs slowly out of disk space - how to replace?

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
mafamgm
Posts: 25
Joined: Tue May 25, 2021 10:07 am

NLS runs slowly out of disk space - how to replace?

Post by mafamgm »

Hello,

I have an NLS that is slowly running out of space. Currently there are about 300 GB free.

I have already added a 2nd server, which has significantly more disk space available, as a 2nd cluster node.

I have noticed that no virtual IP is created.

Now I have 2 questions.

What is the cluster functionality for if no virtual IP is created for the cluster?
All devices that I have configured, send their log data now continue only to the first node.

How can I replace the first server that runs out of disk space with a newer one with more disk space without loosing any data?

Thanks a lot
gsmith
Posts: 1253
Joined: Tue Mar 02, 2021 11:15 am

Re: NLS runs slowly out of disk space - how to replace?

Post by gsmith »

Hi

It sounds like there is a configuration problem. The two nodes should be using approximately the same
storage space. The functionality is such that all the data is captured in 5 pieces, aka shards. And in a
cluster shards 1,3,5 live on the master, and shards 2 and 4 live on the secondary. In addition to that,
shards 1,3,5 should be replicated to the secondary while shards 2 and 4 should be replicated to the master.

On the master node please run:

Code: Select all

tar cvzf /tmp/logstshmstr.tar.gz /usr/local/nagioslogserver/var/*
and on the secondary node please run:

Code: Select all

tar cvzf /tmp/logstshsec.tar.gz /usr/local/nagioslogserver/var/*
then please send the resulting files to me:
/tmp/logstshmstr.tar.gz
/tmp/logstshsec.tar.gz

Thanks
mafamgm
Posts: 25
Joined: Tue May 25, 2021 10:07 am

Re: NLS runs slowly out of disk space - how to replace?

Post by mafamgm »

Hi,
According to the "Total Elasticsearch Disk Usage" display, the master consumes 2.397 GiB of disk space and the slave consumes 585 GiB

The master generally has less disk space physically available than the new slave node

I send you the data.

Thanks
gsmith
Posts: 1253
Joined: Tue Mar 02, 2021 11:15 am

Re: NLS runs slowly out of disk space - how to replace?

Post by gsmith »

Hi,

1. Do you have a repository setup?
2. If so do both nodes have access to it?
3. On each node go to Admin, Snapshots & Maintenance, and please take a screenshot of the
Maintenance and Repository Settings tab.

Thank you
mafamgm
Posts: 25
Joined: Tue May 25, 2021 10:07 am

Re: NLS runs slowly out of disk space - how to replace?

Post by mafamgm »

Hi,

i've added the screenshots.

They have a snapshot repos, but it's not a shared one.
Every node has it's own local repos configured.

Thanks
You do not have the required permissions to view the files attached to this post.
gsmith
Posts: 1253
Joined: Tue Mar 02, 2021 11:15 am

Re: NLS runs slowly out of disk space - how to replace?

Post by gsmith »

Hi,

It seems like your two nodes aren't really clustered. There is not supposed to be a virtual IP created, the master node
will delegate compute and storage to a slave node when things are configured properly.

Please open a ticket at:
https://info.nagios.com/answer-hub-access-new-users

This is the new location of our ticketing system. If you need help logging in/registering please let me know.

Thank you!
mafamgm
Posts: 25
Joined: Tue May 25, 2021 10:07 am

Re: NLS runs slowly out of disk space - how to replace?

Post by mafamgm »

Ok, will create a Ticket.

Thanks.
Locked