Daily indexes are not created

[tomasvalenta]

Hello,
we built new cluster with 2 Nagios Log server, restored indexes from pilot installation and changed IP addresses of the new servers.
I checked cluster_hosts file and it contain new addresses. In the UI is cluster in green, instances also OK. But in Admin menu/System status
I see in the instance dropdown list old IP addresses. Daily indexes are not creating and in the logstash.log I found error (Connection refused)", :class=>"Manticore::SocketException", :level=>:error. In the attachment is system profile of the one instance.
Thanks for any help.
Tomas

[cdienger]

Step me through the process again with a few more details. What where the IP addresses of the pilot system? What were the IP addresses of the machines when they were initially clustered? What were they changed to? What indexes did you import and what did you do to import them? Are the IPs displayed in the drop down the IPs from the pilot system?

[tomasvalenta]

We tested Nagios LogServer in virtual environment. 2 instances in the cluster. We bought new physical HW (2 servers) for live environment and based on recommendation from Nagios support we installed new separate cluster, restored configuration from backup of old cluster. We tested functionality of this new cluster by logging of events from clients and it works. Then we connect these servers to the same snapshot repository as old cluster and we did restore of indexes from UI. I do not know exactly when the new cluster stopped creating daily logs but it could be after restore of backup or restore of logs. The new cluster had IP 10.209.0.33 and 10.209.0.34. We moved this cluster to different network, changed IP addresses to new ones (209.254.210.7 and 8) by nmtui and edited the file cluster_hosts on both instances. I attached the snapshots from UI where you can see everything is OK and in System status are old IP addresses.

[tomasvalenta]

I tested the daily logfile creation and it looks like the file is created when the system is received first event of the day. As I checked no events are in the new system but also in the old one ! We changed DNS records for the name mentioned in client configuration. On windows computers we are using nxlog and when I restarted nxlog service it starts to send events to the new system successfully. The same is for rsyslogd. I understand this situation for about 10 minutes because the client have local DNS cache with expiration 10 minutes. But our situation is different - more than 1 day. So now we are restarting agents on all clients and I hope one part of our problem is fixed. The second - old IP address - still needs attention and I will be happy for any hint.
Thanks

[cdienger]

The new addresses should be picked and displayed after a restart of the elasticsearch database. Run the following:

service elasticsearch restart

[tomasvalenta]

This does not help but as I checked logfile for CRON I found errors the /home/nagios directory not found. So I created
this directory, assigned the chown nagios, chgrp nagios and after 1 hour everything started to be OK.
Anyway thanks for your help and time.
Tomas

[cdienger]

Thanks for following up with your findings!