Defaults missing

[Fred Kroeger]

Just set up a brand new 2 node cluster but think it is broken.
All seemed fine until I connected the second node to the cluster.
My nagiosadmin account and my personal account disappeared so I couldn't login. Trawling through the forum I found a reference to this and recreated the nagiosadmin account am able to login again.
My personal account had been deleted too.
Tried to setup a Linux server to send logs - all appears to work OK but can't see the logs appear in NLS.
Noticed that the Input Defaults were missing, so I created a Syslog & Windows event Log Input using the custom add input, They are both showing active.
Also noticed that there are no Global Queries or Global dashboards listed on the home page.

It seems something happened when I added the second instance. Is there anyway to get back to a default installation ?
The only extra thing I have done is added a couple of disks and allocated it for the Data Store - configured as per your doco.

running a tail on logstash.log I am getting constant error messages on the second node

Code: Select all

log4j, [2015-04-08T05:36:22.584]  WARN: org.elasticsearch.client.transport: [b1203134-8b58-46d0-acaa-a80bd658fba2] node [#transport#-1][node2.xxx.com.au][inet[localhost/127.0.0.1:9300]] not part of the cluster Cluster [19e0d039-931e-4d73-b09e-cc430bdd77d0], ignoring...
log4j, [2015-04-08T05:36:27.586]  WARN: org.elasticsearch.client.transport: [b1203134-8b58-46d0-acaa-a80bd658fba2] node [#transport#-1][node2.xxx.com.au][inet[localhost/127.0.0.1:9300]] not part of the cluster Cluster [19e0d039-931e-4d73-b09e-cc430bdd77d0], ignoring...
log4j, [2015-04-08T05:36:32.591]  WARN: org.elasticsearch.client.transport: [b1203134-8b58-46d0-acaa-a80bd658fba2] node [#transport#-1][node2.xxx.com.au][inet[localhost/127.0.0.1:9300]] not part of the cluster Cluster [19e0d039-931e-4d73-b09e-cc430bdd77d0], ignoring...
log4j, [2015-04-08T05:36:37.595]  WARN: org.elasticsearch.client.transport: [b1203134-8b58-46d0-acaa-a80bd658fba2] node [#transport#-1][node2.xxx.com.au][inet[localhost/127.0.0.1:9300]] not part of the cluster Cluster [19e0d039-931e-4d73-b09e-cc430bdd77d0], ignoring...

Listing /usr/local/nagioslogserver/elasticsearch/data shows both cluster numbers however 6deefb92-cbc9-4461-aa52-4b283a133a60 is the real cluster number from Node1
[root@node2 data]# ls -l
total 8
drwxr-xr-x 3 nagios users 4096 Apr 8 12:08 19e0d039-931e-4d73-b09e-cc430bdd77d0
drwxr-xr-x 3 nagios users 4096 Apr 8 12:03 6deefb92-cbc9-4461-aa52-4b283a133a60


Looking at the Cluster status on both nodes for 6deefb92-cbc9-4461-aa52-4b283a133a60 shows 2 instances and Status is Green

Interestingly when I select System Status I can select either instance but the icons under the Core Service heading appear to be greyed out. When I hover the mouse over the icons the following is displayed
"Could not authenticate. Invalid token given"

Any ideas on what went wrong ?

Regards... Fred

[jolson]

Interesting - I haven't come across this particular error before. Did you set this cluster up using Virtual Machines, or did you build the machines manually?

I assume you did not perform any setup or do anything unusual before joining the clusters together?

Could you please show us your bash history and upload it here?

history > history.txt

I would like to see if I can reproduce this / fix it. The fastest way to resolve this would likely be to set up new machines using our virtual template - please be sure that selinux is turned off before attempting to join the clusters:

Code: Select all

sestatus

Also, what happens if you change the data store back to the default value? I feel like changing the data store may be what caused this to happen.

[Fred Kroeger]

These are both OVF templates freshly downloaded.
Having a quick look at the history myself, it appears that you may be right about changing the datastore path.
It was right after I changed it that I had to recreate the nagiosadmin account.
I will delete the VMs and recreate them - this time without changing the datastore paths.

regards.. Fred

[Fred Kroeger]

I recreated both VMs - this time without changing the datastore path and all worked perfectly.
So it would appear that something happened after I made that change? I will leave as is until next and then try to change the datastore path again.

thanks for your help.... Fred