Page 2 of 2
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 10:57 am
by jolson
What does your cron log look like?
Since we've verified permissions on the directory in question, let's take a look at what indices are included in the data you moved over:
This is where your indices should be listed:
Code: Select all
ls /data01/elasticsearch/data/7f4cab6f-741e-454a-a588-027dce70bfcb/nodes/0/indices/
I'm not sure what could be in this directory:
Code: Select all
ls -lR /data01/elasticsearch/data/nodes/
Let me know, thank you!
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 11:23 am
by vAJ
cron:
Code: Select all
Aug 5 11:19:01 aus02gdnls01 CROND[5716]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 5 11:19:01 aus02gdnls01 CROND[5717]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
indices:
Code: Select all
ls /data01/elasticsearch/data/7f4cab6f-741e-454a-a588-027dce70bfcb/nodes/0/indices/
nagioslogserver nagioslogserver_log
nodes:
Code: Select all
/data01/elasticsearch/data/nodes/:
total 4
drwxr-xr-x 4 nagios nagios 4096 Aug 4 10:32 0
/data01/elasticsearch/data/nodes/0:
total 8
drwxr-xr-x 6 nagios nagios 4096 Aug 4 10:33 indices
-rw-r--r-- 1 nagios nagios 0 Aug 4 10:19 node.lock
drwxr-xr-x 2 nagios nagios 4096 Aug 4 10:32 _state
/data01/elasticsearch/data/nodes/0/indices:
total 16
drwxr-xr-x 8 nagios nagios 4096 Aug 4 10:32 kibana-int
drwxr-xr-x 8 nagios nagios 4096 Aug 4 10:33 logstash-2015.08.04
drwxr-xr-x 4 nagios nagios 4096 Aug 4 10:32 nagioslogserver
drwxr-xr-x 8 nagios nagios 4096 Aug 4 10:32 nagioslogserver_log
/data01/elasticsearch/data/nodes/0/indices/kibana-int:
total 24
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:32 0
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:32 1
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:32 2
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:32 3
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:32 4
drwxr-xr-x 2 nagios nagios 4096 Aug 4 10:32 _state
/data01/elasticsearch/data/nodes/0/indices/kibana-int/0:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug 4 12:03 index
drwxr-xr-x 2 nagios nagios 4096 Aug 4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug 4 11:56 translog
/data01/elasticsearch/data/nodes/0/indices/kibana-int/0/index:
total 48
-rw-r--r-- 1 nagios nagios 350 Aug 4 11:56 _0.cfe
-rw-r--r-- 1 nagios nagios 9937 Aug 4 11:56 _0.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 11:56 _0.si
-rw-r--r-- 1 nagios nagios 350 Aug 4 11:56 _1.cfe
-rw-r--r-- 1 nagios nagios 9942 Aug 4 11:56 _1.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 11:56 _1.si
-rw-r--r-- 1 nagios nagios 206 Aug 4 12:03 segments_4
-rw-r--r-- 1 nagios nagios 36 Aug 4 12:03 segments.gen
-rw-r--r-- 1 nagios nagios 0 Aug 4 10:32 write.lock
/data01/elasticsearch/data/nodes/0/indices/kibana-int/0/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug 4 10:32 state-0.st
/data01/elasticsearch/data/nodes/0/indices/kibana-int/0/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug 4 11:56 translog-1438702359238
/data01/elasticsearch/data/nodes/0/indices/kibana-int/1:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug 4 12:33 index
drwxr-xr-x 2 nagios nagios 4096 Aug 4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug 4 12:27 translog
/data01/elasticsearch/data/nodes/0/indices/kibana-int/1/index:
total 96
-rw-r--r-- 1 nagios nagios 350 Aug 4 11:59 _a.cfe
-rw-r--r-- 1 nagios nagios 9948 Aug 4 11:59 _a.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 11:59 _a.si
-rw-r--r-- 1 nagios nagios 5800 Aug 4 11:59 _b.fdt
-rw-r--r-- 1 nagios nagios 64 Aug 4 11:59 _b.fdx
-rw-r--r-- 1 nagios nagios 838 Aug 4 11:59 _b.fnm
-rw-r--r-- 1 nagios nagios 52 Aug 4 11:59 _b_Lucene410_0.dvd
-rw-r--r-- 1 nagios nagios 90 Aug 4 11:59 _b_Lucene410_0.dvm
-rw-r--r-- 1 nagios nagios 5080 Aug 4 11:59 _b_Lucene41_0.doc
-rw-r--r-- 1 nagios nagios 10024 Aug 4 11:59 _b_Lucene41_0.pos
-rw-r--r-- 1 nagios nagios 6767 Aug 4 11:59 _b_Lucene41_0.tim
-rw-r--r-- 1 nagios nagios 287 Aug 4 11:59 _b_Lucene41_0.tip
-rw-r--r-- 1 nagios nagios 126 Aug 4 11:59 _b.nvd
-rw-r--r-- 1 nagios nagios 101 Aug 4 11:59 _b.nvm
-rw-r--r-- 1 nagios nagios 426 Aug 4 11:59 _b.si
-rw-r--r-- 1 nagios nagios 206 Aug 4 12:33 segments_6
-rw-r--r-- 1 nagios nagios 36 Aug 4 12:33 segments.gen
-rw-r--r-- 1 nagios nagios 0 Aug 4 10:32 write.lock
/data01/elasticsearch/data/nodes/0/indices/kibana-int/1/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug 4 10:32 state-0.st
/data01/elasticsearch/data/nodes/0/indices/kibana-int/1/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug 4 12:27 translog-1438702359275
/data01/elasticsearch/data/nodes/0/indices/kibana-int/2:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug 4 12:33 index
drwxr-xr-x 2 nagios nagios 4096 Aug 4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug 4 12:27 translog
/data01/elasticsearch/data/nodes/0/indices/kibana-int/2/index:
total 76
-rw-r--r-- 1 nagios nagios 4992 Aug 4 11:59 _a.fdt
-rw-r--r-- 1 nagios nagios 64 Aug 4 11:59 _a.fdx
-rw-r--r-- 1 nagios nagios 838 Aug 4 11:59 _a.fnm
-rw-r--r-- 1 nagios nagios 52 Aug 4 11:59 _a_Lucene410_0.dvd
-rw-r--r-- 1 nagios nagios 90 Aug 4 11:59 _a_Lucene410_0.dvm
-rw-r--r-- 1 nagios nagios 5012 Aug 4 11:59 _a_Lucene41_0.doc
-rw-r--r-- 1 nagios nagios 9664 Aug 4 11:59 _a_Lucene41_0.pos
-rw-r--r-- 1 nagios nagios 6471 Aug 4 11:59 _a_Lucene41_0.tim
-rw-r--r-- 1 nagios nagios 287 Aug 4 11:59 _a_Lucene41_0.tip
-rw-r--r-- 1 nagios nagios 98 Aug 4 11:59 _a.nvd
-rw-r--r-- 1 nagios nagios 101 Aug 4 11:59 _a.nvm
-rw-r--r-- 1 nagios nagios 426 Aug 4 11:59 _a.si
-rw-r--r-- 1 nagios nagios 157 Aug 4 12:33 segments_6
-rw-r--r-- 1 nagios nagios 36 Aug 4 12:33 segments.gen
-rw-r--r-- 1 nagios nagios 0 Aug 4 10:32 write.lock
/data01/elasticsearch/data/nodes/0/indices/kibana-int/2/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug 4 10:32 state-0.st
/data01/elasticsearch/data/nodes/0/indices/kibana-int/2/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug 4 12:27 translog-1438702359304
/data01/elasticsearch/data/nodes/0/indices/kibana-int/3:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug 4 12:33 index
drwxr-xr-x 2 nagios nagios 4096 Aug 4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug 4 12:27 translog
/data01/elasticsearch/data/nodes/0/indices/kibana-int/3/index:
total 108
-rw-r--r-- 1 nagios nagios 350 Aug 4 10:32 _0.cfe
-rw-r--r-- 1 nagios nagios 10496 Aug 4 10:32 _0.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 10:32 _0.si
-rw-r--r-- 1 nagios nagios 350 Aug 4 11:56 _1.cfe
-rw-r--r-- 1 nagios nagios 9932 Aug 4 11:56 _1.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 11:56 _1.si
-rw-r--r-- 1 nagios nagios 350 Aug 4 11:56 _2.cfe
-rw-r--r-- 1 nagios nagios 9958 Aug 4 11:56 _2.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 11:56 _2.si
-rw-r--r-- 1 nagios nagios 350 Aug 4 11:59 _3.cfe
-rw-r--r-- 1 nagios nagios 9912 Aug 4 11:59 _3.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 11:59 _3.si
-rw-r--r-- 1 nagios nagios 350 Aug 4 11:59 _4.cfe
-rw-r--r-- 1 nagios nagios 9937 Aug 4 11:59 _4.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 11:59 _4.si
-rw-r--r-- 1 nagios nagios 353 Aug 4 12:33 segments_6
-rw-r--r-- 1 nagios nagios 36 Aug 4 12:33 segments.gen
-rw-r--r-- 1 nagios nagios 0 Aug 4 10:32 write.lock
/data01/elasticsearch/data/nodes/0/indices/kibana-int/3/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug 4 10:32 state-0.st
/data01/elasticsearch/data/nodes/0/indices/kibana-int/3/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug 4 12:27 translog-1438702359257
/data01/elasticsearch/data/nodes/0/indices/kibana-int/4:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug 4 12:05 index
drwxr-xr-x 2 nagios nagios 4096 Aug 4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug 4 11:59 translog
/data01/elasticsearch/data/nodes/0/indices/kibana-int/4/index:
total 28
-rw-r--r-- 1 nagios nagios 350 Aug 4 11:59 _0.cfe
-rw-r--r-- 1 nagios nagios 9922 Aug 4 11:59 _0.cfs
-rw-r--r-- 1 nagios nagios 257 Aug 4 11:59 _0.si
-rw-r--r-- 1 nagios nagios 157 Aug 4 12:05 segments_4
-rw-r--r-- 1 nagios nagios 36 Aug 4 12:05 segments.gen
-rw-r--r-- 1 nagios nagios 0 Aug 4 10:32 write.lock
/data01/elasticsearch/data/nodes/0/indices/kibana-int/4/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug 4 10:32 state-0.st
/data01/elasticsearch/data/nodes/0/indices/kibana-int/4/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug 4 11:59 translog-1438702359328
/data01/elasticsearch/data/nodes/0/indices/kibana-int/_state:
total 4
-rw-r--r-- 1 nagios nagios 354 Aug 4 10:32 state-1.st
/data01/elasticsearch/data/nodes/0/indices/logstash-2015.08.04:
total 24
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:33 0
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:33 1
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:33 2
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:33 3
drwxr-xr-x 5 nagios nagios 4096 Aug 4 10:33 4
drwxr-xr-x 2 nagios nagios 4096 Aug 4 11:01 _state
/data01/elasticsearch/data/nodes/0/indices/logstash-2015.08.04/0:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug 4 13:51 index
drwxr-xr-x 2 nagios nagios 4096 Aug 4 10:33 _state
drwxr-xr-x 2 nagios nagios 4096 Aug 4 13:51 translog
/data01/elasticsearch/data/nodes/0/indices/logstash-2015.08.04/0/index:
total 312
-rw-r--r-- 1 nagios nagios 62827 Aug 4 13:51 _n2.fdt
-rw-r--r-- 1 nagios nagios 127 Aug 4 13:51 _n2.fdx
-rw-r--r-- 1 nagios nagios 2661 Aug 4 13:51 _n2.fnm
-rw-r--r-- 1 nagios nagios 216 Aug 4 13:51 _n2_Lucene410_0.dvd
-rw-r--r-- 1 nagios nagios 91 Aug 4 13:51 _n2_Lucene410_0.dvm
-rw-r--r-- 1 nagios nagios 57408 Aug 4 13:51 _n2_Lucene41_0.doc
-rw-r--r-- 1 nagios nagios 38743 Aug 4 13:51 _n2_Lucene41_0.pos
-rw-r--r-- 1 nagios nagios 52612 Aug 4 13:51 _n2_Lucene41_0.tim
-rw-r--r-- 1 nagios nagios 1529 Aug 4 13:51 _n2_Lucene41_0.tip
-rw-r--r-- 1 nagios nagios 422 Aug 4 13:51 _n2.si
-rw-r--r-- 1 nagios nagios 308 Aug 4 13:51 _n3.cfe
-rw-r--r-- 1 nagios nagios 6506 Aug 4 13:51 _n3.cfs
-rw-r--r-- 1 nagios nagios 260 Aug 4 13:51 _n3.si
-rw-r--r-- 1 nagios nagios 308 Aug 4 13:51 _n4.cfe
-rw-r--r-- 1 nagios nagios 6286 Aug 4 13:51 _n4.cfs
-rw-r--r-- 1 nagios nagios 260 Aug 4 13:51 _n4.si
-rw-r--r-- 1 nagios nagios 308 Aug 4 13:51 _n5.cfe
-rw-r--r-- 1 nagios nagios 6286 Aug 4 13:51 _n5.cfs
-rw-r--r-- 1 nagios nagios 260 Aug 4 13:51 _n5.si
-rw-r--r-- 1 nagios nagios 308 Aug 4 13:51 _n6.cfe
-rw-r--r-- 1 nagios nagios 7065 Aug 4 13:51 _n6.cfs
-rw-r--r-- 1 nagios nagios 260 Aug 4 13:51 _n6.si
......
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 11:33 am
by jolson
Try this.
Code: Select all
service elasticsearch stop
mv /data01/elasticsearch/data/7f4cab6f-741e-454a-a588-027dce70bfcb/nodes ~/
mv /data01/elasticsearch/data/nodes/ /data01/elasticsearch/data/7f4cab6f-741e-454a-a588-027dce70bfcb/nodes
service elasticsearch start
Let me know if the above procedure helps. Are you able to log in to your dashboard? Do any more indices appear when you run the index health command?
curl 'localhost:9200/_cluster/health?level=indices&pretty'
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 11:39 am
by vAJ
Progress!
That brought back all of the users I had setup and System Status no longer shows the Invalid Token error. Logstash will not stay started though. Log shows same "no configs" error.
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 12:06 pm
by jolson
Try running an 'Apply Configuration' from the Web GUI. You can find this at 'Administration -> Apply Configuration -> Apply'. The configs should populate in the following location:
Code: Select all
ls /usr/local/nagioslogserver/logstash/etc/conf.d/
After you get them populated, try restarting logstash:
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 12:11 pm
by vAJ
dev_nls_apply.JPG
Seeing the IP in there twice. My production instance only shows this once.
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 12:16 pm
by jolson
That problem can be caused by a strange cluster_hosts file. Check on the following file:
Code: Select all
cat /usr/local/nagioslogserver/var/cluster_hosts
If you see any duplicate or invalid IP addresses/hostnames, feel free to remove them. The file should look similar to this:
localhost
192.168.1.1
192.168.1.2
It's common to see 'localhost' in addition to the external IP address (in this case, 192.168.1.1).
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 12:17 pm
by vAJ
FYI, starting from shell nets the following response:
Code: Select all
[root@aus02gdnls01 logstash]# service logstash start
Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
[ OK ]
[root@aus02gdnls01 logstash]# Aug 05, 2015 12:16:40 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Aug 05, 2015 12:16:41 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Aug 05, 2015 12:16:41 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Aug 05, 2015 12:16:41 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Aug 05, 2015 12:16:41 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 12:18 pm
by vAJ
localhost
10.87.200.66
That's all I've got.
In instance status, it showed both, one with just the IP and no stats listed. I trashed that and everything looks clean now. Apply config only shows the one.
I think we can close this now. But might review the fullinstall script packages with 2.1 and make sure user creation, et al look right. I followed that and the data store documentation to the T.
-Andrew
Re: Core services - Could not authenticate, Invalid Token Gi
Posted: Wed Aug 05, 2015 12:44 pm
by jolson
Try taking a peek at your 'per instance' tab - is there more than one server listed?
2015-08-05 12_43_47-Admin Dashboard • Nagios Log Server - Firefox Developer Edition.png
How is your Logstash config doing?