Core services - Could not authenticate, Invalid Token Given

[jolson]

What does your cron log look like?

Code: Select all

cat /var/log/cron

Since we've verified permissions on the directory in question, let's take a look at what indices are included in the data you moved over:
This is where your indices should be listed:

Code: Select all

ls /data01/elasticsearch/data/7f4cab6f-741e-454a-a588-027dce70bfcb/nodes/0/indices/

I'm not sure what could be in this directory:

Code: Select all

ls -lR /data01/elasticsearch/data/nodes/

Let me know, thank you!

[vAJ]

cron:

Code: Select all

Aug  5 11:19:01 aus02gdnls01 CROND[5716]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug  5 11:19:01 aus02gdnls01 CROND[5717]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)

indices:

Code: Select all

ls /data01/elasticsearch/data/7f4cab6f-741e-454a-a588-027dce70bfcb/nodes/0/indices/
nagioslogserver  nagioslogserver_log

nodes:

Code: Select all

/data01/elasticsearch/data/nodes/:
total 4
drwxr-xr-x 4 nagios nagios 4096 Aug  4 10:32 0

/data01/elasticsearch/data/nodes/0:
total 8
drwxr-xr-x 6 nagios nagios 4096 Aug  4 10:33 indices
-rw-r--r-- 1 nagios nagios    0 Aug  4 10:19 node.lock
drwxr-xr-x 2 nagios nagios 4096 Aug  4 10:32 _state

/data01/elasticsearch/data/nodes/0/indices:
total 16
drwxr-xr-x 8 nagios nagios 4096 Aug  4 10:32 kibana-int
drwxr-xr-x 8 nagios nagios 4096 Aug  4 10:33 logstash-2015.08.04
drwxr-xr-x 4 nagios nagios 4096 Aug  4 10:32 nagioslogserver
drwxr-xr-x 8 nagios nagios 4096 Aug  4 10:32 nagioslogserver_log

/data01/elasticsearch/data/nodes/0/indices/kibana-int:
total 24
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:32 0
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:32 1
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:32 2
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:32 3
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:32 4
drwxr-xr-x 2 nagios nagios 4096 Aug  4 10:32 _state

/data01/elasticsearch/data/nodes/0/indices/kibana-int/0:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug  4 12:03 index
drwxr-xr-x 2 nagios nagios 4096 Aug  4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug  4 11:56 translog

/data01/elasticsearch/data/nodes/0/indices/kibana-int/0/index:
total 48
-rw-r--r-- 1 nagios nagios  350 Aug  4 11:56 _0.cfe
-rw-r--r-- 1 nagios nagios 9937 Aug  4 11:56 _0.cfs
-rw-r--r-- 1 nagios nagios  257 Aug  4 11:56 _0.si
-rw-r--r-- 1 nagios nagios  350 Aug  4 11:56 _1.cfe
-rw-r--r-- 1 nagios nagios 9942 Aug  4 11:56 _1.cfs
-rw-r--r-- 1 nagios nagios  257 Aug  4 11:56 _1.si
-rw-r--r-- 1 nagios nagios  206 Aug  4 12:03 segments_4
-rw-r--r-- 1 nagios nagios   36 Aug  4 12:03 segments.gen
-rw-r--r-- 1 nagios nagios    0 Aug  4 10:32 write.lock

/data01/elasticsearch/data/nodes/0/indices/kibana-int/0/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug  4 10:32 state-0.st

/data01/elasticsearch/data/nodes/0/indices/kibana-int/0/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug  4 11:56 translog-1438702359238

/data01/elasticsearch/data/nodes/0/indices/kibana-int/1:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug  4 12:33 index
drwxr-xr-x 2 nagios nagios 4096 Aug  4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug  4 12:27 translog

/data01/elasticsearch/data/nodes/0/indices/kibana-int/1/index:
total 96
-rw-r--r-- 1 nagios nagios   350 Aug  4 11:59 _a.cfe
-rw-r--r-- 1 nagios nagios  9948 Aug  4 11:59 _a.cfs
-rw-r--r-- 1 nagios nagios   257 Aug  4 11:59 _a.si
-rw-r--r-- 1 nagios nagios  5800 Aug  4 11:59 _b.fdt
-rw-r--r-- 1 nagios nagios    64 Aug  4 11:59 _b.fdx
-rw-r--r-- 1 nagios nagios   838 Aug  4 11:59 _b.fnm
-rw-r--r-- 1 nagios nagios    52 Aug  4 11:59 _b_Lucene410_0.dvd
-rw-r--r-- 1 nagios nagios    90 Aug  4 11:59 _b_Lucene410_0.dvm
-rw-r--r-- 1 nagios nagios  5080 Aug  4 11:59 _b_Lucene41_0.doc
-rw-r--r-- 1 nagios nagios 10024 Aug  4 11:59 _b_Lucene41_0.pos
-rw-r--r-- 1 nagios nagios  6767 Aug  4 11:59 _b_Lucene41_0.tim
-rw-r--r-- 1 nagios nagios   287 Aug  4 11:59 _b_Lucene41_0.tip
-rw-r--r-- 1 nagios nagios   126 Aug  4 11:59 _b.nvd
-rw-r--r-- 1 nagios nagios   101 Aug  4 11:59 _b.nvm
-rw-r--r-- 1 nagios nagios   426 Aug  4 11:59 _b.si
-rw-r--r-- 1 nagios nagios   206 Aug  4 12:33 segments_6
-rw-r--r-- 1 nagios nagios    36 Aug  4 12:33 segments.gen
-rw-r--r-- 1 nagios nagios     0 Aug  4 10:32 write.lock

/data01/elasticsearch/data/nodes/0/indices/kibana-int/1/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug  4 10:32 state-0.st

/data01/elasticsearch/data/nodes/0/indices/kibana-int/1/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug  4 12:27 translog-1438702359275

/data01/elasticsearch/data/nodes/0/indices/kibana-int/2:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug  4 12:33 index
drwxr-xr-x 2 nagios nagios 4096 Aug  4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug  4 12:27 translog

/data01/elasticsearch/data/nodes/0/indices/kibana-int/2/index:
total 76
-rw-r--r-- 1 nagios nagios 4992 Aug  4 11:59 _a.fdt
-rw-r--r-- 1 nagios nagios   64 Aug  4 11:59 _a.fdx
-rw-r--r-- 1 nagios nagios  838 Aug  4 11:59 _a.fnm
-rw-r--r-- 1 nagios nagios   52 Aug  4 11:59 _a_Lucene410_0.dvd
-rw-r--r-- 1 nagios nagios   90 Aug  4 11:59 _a_Lucene410_0.dvm
-rw-r--r-- 1 nagios nagios 5012 Aug  4 11:59 _a_Lucene41_0.doc
-rw-r--r-- 1 nagios nagios 9664 Aug  4 11:59 _a_Lucene41_0.pos
-rw-r--r-- 1 nagios nagios 6471 Aug  4 11:59 _a_Lucene41_0.tim
-rw-r--r-- 1 nagios nagios  287 Aug  4 11:59 _a_Lucene41_0.tip
-rw-r--r-- 1 nagios nagios   98 Aug  4 11:59 _a.nvd
-rw-r--r-- 1 nagios nagios  101 Aug  4 11:59 _a.nvm
-rw-r--r-- 1 nagios nagios  426 Aug  4 11:59 _a.si
-rw-r--r-- 1 nagios nagios  157 Aug  4 12:33 segments_6
-rw-r--r-- 1 nagios nagios   36 Aug  4 12:33 segments.gen
-rw-r--r-- 1 nagios nagios    0 Aug  4 10:32 write.lock

/data01/elasticsearch/data/nodes/0/indices/kibana-int/2/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug  4 10:32 state-0.st

/data01/elasticsearch/data/nodes/0/indices/kibana-int/2/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug  4 12:27 translog-1438702359304

/data01/elasticsearch/data/nodes/0/indices/kibana-int/3:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug  4 12:33 index
drwxr-xr-x 2 nagios nagios 4096 Aug  4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug  4 12:27 translog

/data01/elasticsearch/data/nodes/0/indices/kibana-int/3/index:
total 108
-rw-r--r-- 1 nagios nagios   350 Aug  4 10:32 _0.cfe
-rw-r--r-- 1 nagios nagios 10496 Aug  4 10:32 _0.cfs
-rw-r--r-- 1 nagios nagios   257 Aug  4 10:32 _0.si
-rw-r--r-- 1 nagios nagios   350 Aug  4 11:56 _1.cfe
-rw-r--r-- 1 nagios nagios  9932 Aug  4 11:56 _1.cfs
-rw-r--r-- 1 nagios nagios   257 Aug  4 11:56 _1.si
-rw-r--r-- 1 nagios nagios   350 Aug  4 11:56 _2.cfe
-rw-r--r-- 1 nagios nagios  9958 Aug  4 11:56 _2.cfs
-rw-r--r-- 1 nagios nagios   257 Aug  4 11:56 _2.si
-rw-r--r-- 1 nagios nagios   350 Aug  4 11:59 _3.cfe
-rw-r--r-- 1 nagios nagios  9912 Aug  4 11:59 _3.cfs
-rw-r--r-- 1 nagios nagios   257 Aug  4 11:59 _3.si
-rw-r--r-- 1 nagios nagios   350 Aug  4 11:59 _4.cfe
-rw-r--r-- 1 nagios nagios  9937 Aug  4 11:59 _4.cfs
-rw-r--r-- 1 nagios nagios   257 Aug  4 11:59 _4.si
-rw-r--r-- 1 nagios nagios   353 Aug  4 12:33 segments_6
-rw-r--r-- 1 nagios nagios    36 Aug  4 12:33 segments.gen
-rw-r--r-- 1 nagios nagios     0 Aug  4 10:32 write.lock

/data01/elasticsearch/data/nodes/0/indices/kibana-int/3/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug  4 10:32 state-0.st

/data01/elasticsearch/data/nodes/0/indices/kibana-int/3/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug  4 12:27 translog-1438702359257

/data01/elasticsearch/data/nodes/0/indices/kibana-int/4:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug  4 12:05 index
drwxr-xr-x 2 nagios nagios 4096 Aug  4 10:32 _state
drwxr-xr-x 2 nagios nagios 4096 Aug  4 11:59 translog

/data01/elasticsearch/data/nodes/0/indices/kibana-int/4/index:
total 28
-rw-r--r-- 1 nagios nagios  350 Aug  4 11:59 _0.cfe
-rw-r--r-- 1 nagios nagios 9922 Aug  4 11:59 _0.cfs
-rw-r--r-- 1 nagios nagios  257 Aug  4 11:59 _0.si
-rw-r--r-- 1 nagios nagios  157 Aug  4 12:05 segments_4
-rw-r--r-- 1 nagios nagios   36 Aug  4 12:05 segments.gen
-rw-r--r-- 1 nagios nagios    0 Aug  4 10:32 write.lock

/data01/elasticsearch/data/nodes/0/indices/kibana-int/4/_state:
total 4
-rw-r--r-- 1 nagios nagios 81 Aug  4 10:32 state-0.st

/data01/elasticsearch/data/nodes/0/indices/kibana-int/4/translog:
total 4
-rw-r--r-- 1 nagios nagios 17 Aug  4 11:59 translog-1438702359328

/data01/elasticsearch/data/nodes/0/indices/kibana-int/_state:
total 4
-rw-r--r-- 1 nagios nagios 354 Aug  4 10:32 state-1.st

/data01/elasticsearch/data/nodes/0/indices/logstash-2015.08.04:
total 24
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:33 0
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:33 1
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:33 2
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:33 3
drwxr-xr-x 5 nagios nagios 4096 Aug  4 10:33 4
drwxr-xr-x 2 nagios nagios 4096 Aug  4 11:01 _state

/data01/elasticsearch/data/nodes/0/indices/logstash-2015.08.04/0:
total 12
drwxr-xr-x 2 nagios nagios 4096 Aug  4 13:51 index
drwxr-xr-x 2 nagios nagios 4096 Aug  4 10:33 _state
drwxr-xr-x 2 nagios nagios 4096 Aug  4 13:51 translog

/data01/elasticsearch/data/nodes/0/indices/logstash-2015.08.04/0/index:
total 312
-rw-r--r-- 1 nagios nagios 62827 Aug  4 13:51 _n2.fdt
-rw-r--r-- 1 nagios nagios   127 Aug  4 13:51 _n2.fdx
-rw-r--r-- 1 nagios nagios  2661 Aug  4 13:51 _n2.fnm
-rw-r--r-- 1 nagios nagios   216 Aug  4 13:51 _n2_Lucene410_0.dvd
-rw-r--r-- 1 nagios nagios    91 Aug  4 13:51 _n2_Lucene410_0.dvm
-rw-r--r-- 1 nagios nagios 57408 Aug  4 13:51 _n2_Lucene41_0.doc
-rw-r--r-- 1 nagios nagios 38743 Aug  4 13:51 _n2_Lucene41_0.pos
-rw-r--r-- 1 nagios nagios 52612 Aug  4 13:51 _n2_Lucene41_0.tim
-rw-r--r-- 1 nagios nagios  1529 Aug  4 13:51 _n2_Lucene41_0.tip
-rw-r--r-- 1 nagios nagios   422 Aug  4 13:51 _n2.si
-rw-r--r-- 1 nagios nagios   308 Aug  4 13:51 _n3.cfe
-rw-r--r-- 1 nagios nagios  6506 Aug  4 13:51 _n3.cfs
-rw-r--r-- 1 nagios nagios   260 Aug  4 13:51 _n3.si
-rw-r--r-- 1 nagios nagios   308 Aug  4 13:51 _n4.cfe
-rw-r--r-- 1 nagios nagios  6286 Aug  4 13:51 _n4.cfs
-rw-r--r-- 1 nagios nagios   260 Aug  4 13:51 _n4.si
-rw-r--r-- 1 nagios nagios   308 Aug  4 13:51 _n5.cfe
-rw-r--r-- 1 nagios nagios  6286 Aug  4 13:51 _n5.cfs
-rw-r--r-- 1 nagios nagios   260 Aug  4 13:51 _n5.si
-rw-r--r-- 1 nagios nagios   308 Aug  4 13:51 _n6.cfe
-rw-r--r-- 1 nagios nagios  7065 Aug  4 13:51 _n6.cfs
-rw-r--r-- 1 nagios nagios   260 Aug  4 13:51 _n6.si


......

[jolson]

Try this.

Code: Select all

service elasticsearch stop
mv /data01/elasticsearch/data/7f4cab6f-741e-454a-a588-027dce70bfcb/nodes ~/
mv /data01/elasticsearch/data/nodes/ /data01/elasticsearch/data/7f4cab6f-741e-454a-a588-027dce70bfcb/nodes
service elasticsearch start

Let me know if the above procedure helps. Are you able to log in to your dashboard? Do any more indices appear when you run the index health command?
curl 'localhost:9200/_cluster/health?level=indices&pretty'

[vAJ]

Progress!

That brought back all of the users I had setup and System Status no longer shows the Invalid Token error. Logstash will not stay started though. Log shows same "no configs" error.

[jolson]

Try running an 'Apply Configuration' from the Web GUI. You can find this at 'Administration -> Apply Configuration -> Apply'. The configs should populate in the following location:

Code: Select all

ls /usr/local/nagioslogserver/logstash/etc/conf.d/

After you get them populated, try restarting logstash:

Code: Select all

service logstash restart

[vAJ]

dev_nls_apply.JPG

Seeing the IP in there twice. My production instance only shows this once.

[jolson]

That problem can be caused by a strange cluster_hosts file. Check on the following file:

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_hosts

If you see any duplicate or invalid IP addresses/hostnames, feel free to remove them. The file should look similar to this:

localhost
192.168.1.1
192.168.1.2

It's common to see 'localhost' in addition to the external IP address (in this case, 192.168.1.1).

[vAJ]

FYI, starting from shell nets the following response:

Code: Select all

[root@aus02gdnls01 logstash]# service logstash start
Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
                                                           [  OK  ]
[root@aus02gdnls01 logstash]# Aug 05, 2015 12:16:40 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Aug 05, 2015 12:16:41 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Aug 05, 2015 12:16:41 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Aug 05, 2015 12:16:41 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []
Aug 05, 2015 12:16:41 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [bb316fa2-7742-416c-a589-4459874ebb4f] loaded [], sites []

[vAJ]

localhost
10.87.200.66

That's all I've got.

In instance status, it showed both, one with just the IP and no stats listed. I trashed that and everything looks clean now. Apply config only shows the one.

I think we can close this now. But might review the fullinstall script packages with 2.1 and make sure user creation, et al look right. I followed that and the data store documentation to the T.

-Andrew

[jolson]

Try taking a peek at your 'per instance' tab - is there more than one server listed?

2015-08-05 12_43_47-Admin Dashboard • Nagios Log Server - Firefox Developer Edition.png

How is your Logstash config doing?