Nagios LS not creating new indices

[TimePlan]

Hey

To help you help me with my issue, here's some background to the problem:

• I have configured our Nagios Log Server to be Single Cluster.
• I have moved the Nagios Log data location to a secondary drive.
  During this process I altered the owner of the new location to be the Nagios Log Server user (nagios:nagios)
  And I changed the permissions to 755 (with -R parameter).
  Lastly I moved the existing data files.
• I have moved the Nagios Log Server log location to a secondary drive aswell.
  The process was the same as with the Nagios Log Server data location.
• I have moved the Logstash log location (at least I think I did so correctly)
  During this process, I deleted some .tar.gz files from /var/log/logstash/.
  I also altered the owner of the new location to be the Nagios Log Server user (nagios:nagios)
  And I changed the permissions to 755 (with -R parameter).
  Lastly I moved the existing logstash.log file to the new location

Since I made the changes listed above, the log server has not received any logs.
I have attached an excerpt of logstash.log.

Can you please help me fix this problem?

On behalf of TimePlan Software A/S
Lasse Greve Pedersen
IT-supporter

[npolovenko]

Hello, @TimePlan. Can you list the exact commands you ran to configure the changes you listed?

Just to confirm, did you use this tutorial to change the data store path for the log server?
https://assets.nagios.com/downloads/nag ... Server.pdf

[TimePlan]

Hey @npolovenko

I'll try to dig out the exact commands from our documentation.

Yes. I followed the guide.
I had actually done the changes to data dir path and log dir path a long time ago.
Our Nagios Log Server installation ha not worked for a long time, because the secondary drive, on which the data and log files were stored, ran out of space.
Before I moved the logstash.log file, I had installed a new drive in the same mount folder as the old drive, installed the old drive in a new mount location and moved the log and data files to the new drive.
As part of troubleshooting, I've tried to set the data_dir path back to the original placement and back to the altered location. It didn't have an effect.
It seems to me, elasticsearch no longer creates new indices for some reason.

EDIT:
I forgot to mention that I had to restore the nagioslogserver shard from a backup, as it had been corrupted and I could not log in to the Web Interface.
During the process, I was forced to delete 3 other shards with data from certain dates, as they were stuck in the INITIALIZING phase even after restoration of the nagioslogserver shard.
I doubt it's the cause of the problem, but I thought you might like to know anyway.

[TimePlan]

I will slowly fill this post with commands, I've issued to make the changes made in original post.
Therefore, there will be a lot of edits of this.

Note:
The secondary drive for our CentOS 7 installation is mounted in this folder: /mnt/nagiosdata

Change of logstash location:

• mkdir /mnt/nagiosdata/logstash
  chown -R nagios:nagios /mnt/nagiosdata/logstash/
  chmod 777 /mnt/nagiosdata/logstash
  vi /etc/sysconfig/logstash
  Changed #LS_LOG_FILE=/var/log/logstash/logstash.log to LS_LOG_FILE=/mnt/nagiosdata/logstash/logstash.log
  mv /var/log/logstash/logstash.log /mnt/nagiosdata/logstash/

Change to Single Cluster mode:

• vi /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml
  Changed # index.number_of_replicas: 0 to index.number_of_replicas: 0
  systemctl restart elasticsearch

Change of Nagios Log Server Data and Log directory:

• systemctl stop elasticsearch
  mkdir /mnt/nagiosdata/nagiosdata
  mkdir /mnt/nagiosdata/logs
  chown -R nagios:nagios /mnt/nagiosdata/
  chmod -R 775 /mnt/nagiosdata/
  vi /etc/sysconfig/elasticsearch
  Changed DATA_DIR="/usr/local/nagioslogserver/elasticsearch/data" to DATA_DIR="/mnt/nagiosdata/nagiosdata
  mv /usr/local/nagioslogserver/elasticsearch/data/* /mnt/nagiosdata/nagiosdata
  mv /var/log/elasticsearch /mnt/nagiosdata/logs

[npolovenko]

@TimePlan, Thanks for providing the details on the steps you took. I don't notice anything out of the ordinary so far.
Please use the script I attached to generate a system profile on the log server and upload it in this thread.
Place the script in the /tmp/ folder and execute it. It should generate the system-profile.tar.gz archive in the /tmp/ folder. Use FileZilla or WinSCP to download it from the server.
Also, where did you take a screenshot attached to your first post?

[TimePlan]

@npolovenko, here's the profile, you requested.

The screenshot, I provided in the original post, is an excerpt from logstash.log.
Let me know if you need more information from it.

This might also have something to do with the shards not being made:
I upgraded Nagios Log Server as described in this: https://assets.nagios.com/downloads/nag ... Server.pdf.
I made sure to execute the command to reenable shard allocation, and it gave me the correct output as far as I could tell.
I've attached a screenshot of the output, I recieved.

[npolovenko]

@TimePlan, Thank you. I'd like to verify the cluster settings. Please run the following command and update us with the output:

curl -XGET localhost:9200/_cluster/settings

Also, please upload all logs from the following directory:

/mnt/nagiosdata/logs/elasticsearch

[TimePlan]

@npolovenko

I have attached the cluster settings piped to a .txt-file.
I've also attached a .zip-file containing all 3 elasticsearch logs.

[npolovenko]

@TimePlan, I'm seeing the following error in the elasticsearch log:

Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK (ulimit).

Please follow the following article to set the memlock limit to unlimited.

Can you also show me the output of:

ulimit -l

https://timonweb.com/posts/elasticsearc ... hat-to-do/

[TimePlan]

Hey @npolovenko

I have attached the output of

ulimit -l

Both from before I followed the instructions in the link, you provided, as well as after.

However, it didn't fix the problem of elasticsearch not creating indices.

Is it possible to export the elasticsearch indices to be used in another Nagios Log Server Cluster?
Because a solution might be to export the existing indices to a new functioning Nagios Log Server Cluster.