Cannot login to web int. - Could not get user....

[tomasvalenta]

After more than 6 months of running Log server in free edition now I am not able to logon
to the web interface - after filling logon details for long time nothing happened and then
error message:
Could not get user. Specified ID may be wrong.
The same message for my account and for nagiosadmin also
I am able to logon by ssh, I tried to run reset_nagiosadmin_password.sh script but this script
has no affect and does not end. Service elasticsearch is running.
Can you help me somebody, please ?
Thanks
Tomas

[rkennedy]

You can run the following to create an account through the CLI -

Code: Select all

curl -XPUT 'http://localhost:9200/nagioslogserver/user/2' -d '{"username":"someuser","password":"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19","auth_type":"admin","email":"[email protected]","language":"default","apiaccess":"1","apikey":"1396e08757545557073844695e5b64caa0bd3ad3","created":"2015-01-23 10:00:00","created_by":0,"default_dashboard":"/dashboard/elasticsearch/default"}'

This will create a user named "someuser" with the password "nagiosls". You can use this account to log in and optionally create the accounts that you need, then delete the "someuser" account as one of the other users when done.

[tomasvalenta]

I run the command with this result:
{"error":"UnavailableShardsException[[nagioslogserver][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: index {[nagiosl ogserver][user][2], source[{\"username\":\"someuser\",\"password\":\"c678bcf3b51 38b9263a95c44d28097f22c2e02877193d2c25313478821d45c19\",\"auth_type\":\"admin\", \"email\":\"[email protected]\",\"language\":\"default\",\"apiaccess\":\"1\",\"ap ikey\":\"1396e08757545557073844695e5b64caa0bd3ad3\",\"created\":\"2015-01-23 10: 00:00\",\"created_by\":0,\"default_dashboard\":\"/dashboard/elasticsearch/defaul

[tomasvalenta]

result of your command is the following:
{"error":"UnavailableShardsException[[nagioslogserver][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m]
Based on this result and my searches it is important to tell you the server filled disk space before. I increased size of the volume by nagios how-to and now
the server has 47GB of free space.
Here is info about health of our log server...
[root@logserver ~]# curl -XGET http://localhost:9200/_cluster/health?
{"cluster_name":"77ebcce0-c6c7-4a1c-b7de-19ece082d9a2","status":"red","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":1153,"active_shards":1153,"relocating_shards":0,"initializing_shards":3,"unassigned_shards":1176,"number_of_pending_tasks":1,"number_of_in_flight_fetch":0}

[mcapra]

Can you try re-running that command:

Code: Select all

curl -XPUT 'http://localhost:9200/nagioslogserver/user/99' -d '{"username":"someuser","password":"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19","auth_type":"admin","email":"[email protected]","language":"default","apiaccess":"1","apikey":"1396e08757545557073844695e5b64caa0bd3ad3","created":"2015-01-23 10:00:00","created_by":0,"default_dashboard":"/dashboard/elasticsearch/default"}'

If it still doesn't work, I would suggest restarting the entire machine. If that doesn't allow the command to work, please share the output of:

Code: Select all

curl -XGET http://localhost:9200/_cat/shards | grep INITIALIZING

[tomasvalenta]

the command has the same error - Primary shard is not active or isn't assigned to a known node
and
nagioslogserver shard is in INITIALIZING state forewer....
there are also several unassigned shards
nagioslogserver_log 2 r UNASSIGNED
nagioslogserver_log 0 r UNASSIGNED
nagioslogserver_log 3 r UNASSIGNED
nagioslogserver_log 1 r UNASSIGNED
nagioslogserver_log 4 r UNASSIGNED
nagioslogserver 0 r UNASSIGNED
logstash-2017.02.02 4 r UNASSIGNED
logstash-2017.02.02 0 r UNASSIGNED
logstash-2017.02.02 3 r UNASSIGNED
logstash-2017.02.02 1 r UNASSIGNED
logstash-2017.02.02 2 r UNASSIGNED
logstash-2017.02.03 2 r UNASSIGNED
logstash-2017.02.03 0 r UNASSIGNED
logstash-2017.02.03 3 r UNASSIGNED
logstash-2017.02.03 1 r UNASSIGNED
logstash-2017.02.03 4 r UNASSIGNED
logstash-2017.02.01 4 r UNASSIGNED
logstash-2017.02.01 0 r UNASSIGNED
logstash-2017.02.01 3 r UNASSIGNED
logstash-2017.02.01 1 r UNASSIGNED
logstash-2017.02.01 2 r UNASSIGNED

[rkennedy]

Can you show us the full output of curl -XGET http://localhost:9200/_cat/shards?

Also, did you reboot the machine as @mcapra mentioned? This should push elasticsearch into re-assigning / allocating.

[tomasvalenta]

Code: Select all

[root@logserver ~]# curl -XGET http://localhost:9200/_cat/shards
nagioslogserver_log 2 p STARTED        1044 120.6kb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
nagioslogserver_log 2 r UNASSIGNED
nagioslogserver_log 0 p STARTED        1178 129.9kb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
nagioslogserver_log 0 r UNASSIGNED
nagioslogserver_log 3 p STARTED        1108 101.1kb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
nagioslogserver_log 3 r UNASSIGNED
nagioslogserver_log 1 p STARTED        1129 115.3kb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
nagioslogserver_log 1 r UNASSIGNED
nagioslogserver_log 4 p STARTED        1081  99.5kb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
nagioslogserver_log 4 r UNASSIGNED
logstash-2017.02.06 2 p STARTED      103083 102.6mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.06 2 r UNASSIGNED
logstash-2017.02.06 0 p STARTED      102956 102.4mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.06 0 r UNASSIGNED
logstash-2017.02.06 3 p STARTED      103022 104.3mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.06 3 r UNASSIGNED
logstash-2017.02.06 1 p STARTED      103106 101.7mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.06 1 r UNASSIGNED
logstash-2017.02.06 4 p STARTED      103061 102.2mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.06 4 r UNASSIGNED
logstash-2017.02.04 2 p STARTED      214686 200.1mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.04 2 r UNASSIGNED
logstash-2017.02.04 0 p STARTED      214715 200.3mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.04 0 r UNASSIGNED
logstash-2017.02.04 3 p STARTED      214813 200.2mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.04 3 r UNASSIGNED
logstash-2017.02.04 1 p STARTED      214913 200.4mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.04 1 r UNASSIGNED
logstash-2017.02.04 4 p STARTED      214490 199.7mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.04 4 r UNASSIGNED
nagioslogserver     0 p INITIALIZING                209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
nagioslogserver     0 r UNASSIGNED
logstash-2017.02.05 2 p STARTED      216661 202.9mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.05 2 r UNASSIGNED
logstash-2017.02.05 0 p STARTED      216463 202.8mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.05 0 r UNASSIGNED
logstash-2017.02.05 3 p STARTED      216867 203.3mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.05 3 r UNASSIGNED
logstash-2017.02.05 1 p STARTED      216835 202.9mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.05 1 r UNASSIGNED
logstash-2017.02.05 4 p STARTED      216638 202.8mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.05 4 r UNASSIGNED
logstash-2017.02.02 4 p STARTED      365765 361.9mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.02 4 r UNASSIGNED
logstash-2017.02.02 0 p STARTED      365549 361.7mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.02 0 r UNASSIGNED
logstash-2017.02.02 3 p STARTED      365415 361.6mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.02 3 r UNASSIGNED
logstash-2017.02.02 1 p STARTED      365930 361.2mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.02 1 r UNASSIGNED
logstash-2017.02.02 2 p STARTED      365536 362.3mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.02 2 r UNASSIGNED
logstash-2017.02.03 2 p STARTED      263577 257.4mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.03 2 r UNASSIGNED
logstash-2017.02.03 0 p STARTED      263588   257mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.03 0 r UNASSIGNED
logstash-2017.02.03 3 p STARTED      263540 256.9mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.03 3 r UNASSIGNED
logstash-2017.02.03 1 p STARTED      263613 257.4mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.03 1 r UNASSIGNED
logstash-2017.02.03 4 p STARTED      263699 256.6mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.03 4 r UNASSIGNED
logstash-2017.02.01 4 p STARTED      244363 178.2mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.01 4 r UNASSIGNED
logstash-2017.02.01 0 p STARTED      244476   178mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.01 0 r UNASSIGNED
logstash-2017.02.01 3 p STARTED      244555 178.4mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.01 3 r UNASSIGNED
logstash-2017.02.01 1 p STARTED      244311 178.5mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.01 1 r UNASSIGNED
logstash-2017.02.01 2 p STARTED      244523 177.9mb 209.52.249.9 73d2efd4-e75b-4230-a0cb-1929403d6874
logstash-2017.02.01 2 r UNASSIGNED
We did server restart several times but does not help. THe nagioslogserver shard still remain in INITIALIZING status.
Can I safely delete it and restart elasticsearch or the whole server ?

[mcapra]

Can you share the contents of your Elasticsearch logs? They should be located at /var/log/elasticsearch. Please zip up that path and either PM it or post as an attachment.

Can you also share the outputs of the following commands executed from the CLI of one of your Nagios Log Server instances:

Code: Select all

df -h
free -m

[tomasvalenta]

just to be sure - we have only one server.
[root@logserver ~]# df -h && free -m
Filesystem Size Used Avail Use% Mounted on
rootfs 148G 12G 135G 8% /
devtmpfs 3.9G 160K 3.9G 1% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/xvda1 148G 12G 135G 8% /
total used free shared buffers cached
Mem: 7994 7339 655 0 102 1877
-/+ buffers/cache: 5359 2635
Swap: 255 0 255

I searched Internet little bit and found one hint and it works.
I deleted all files with *.recovery in elasticsearch data/.... directory and restarted elasticsearch.
Now I am able to logon back. It looks like all my custom reports and lookups are lost but I will create
them again. Thank you very much for your help