nagios ls - status = red

[newmember]

I have a red status

Code: Select all

[nagios@hnapxlamslog01 ~]$ curl -XGET 'http://localhost:9200/_cat/shards?v'

I have hundreds of these "UNASSIGNED"

Code: Select all

logstash-2018.09.09 0     p      STARTED         19   84.1kb 170.108.74.80 1ccd6723-32a1-4fbe-8de4-2fa95babec98 
logstash-2018.09.09 0     r      UNASSIGNED                                                                     
logstash-2018.09.09 3     p      STARTED         21   22.2kb 170.108.74.80 1ccd6723-32a1-4fbe-8de4-2fa95babec98 
logstash-2018.09.09 3     r      UNASSIGNED                                                                     
logstash-2018.09.09 1     p      STARTED         18   60.4kb 170.108.74.80 1ccd6723-32a1-4fbe-8de4-2fa95babec98 
logstash-2018.09.09 1     r      UNASSIGNED                                                                     
logstash-2018.09.09 2     p      STARTED         21   22.8kb 170.108.74.80 1ccd6723-32a1-4fbe-8de4-2fa95babec98 
logstash-2018.09.09 2     r      UNASSIGNED                                                                     
[nagios@hnapxlamslog01 ~]$ 
[nagios@hnapxlamslog01 ~]$ 

Code: Select all

[nagios@hnapxlamslog01 ~]$ curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ { "allocate" : { "index" : "logstash-2018.09.09",  "shard" : 2,  "node" : "
1ccd6723-32a1-4fbe-8de4-2fa95babec98",  "allow_primary" : true } } ] }'                                                                                                  
{"error":"ElasticsearchIllegalArgumentException[[allocate] allocation of [logstash-2018.09.09][2] on node [1ccd6723-32a1-4fbe-8de4-2fa95babec98][FdcoENIXSQWkFbmLdIwkdQ][hnapxlamslog01][inet[/170.108.74.80:9300]]{max_local_storage_nodes=1} is not allowed, reason: [NO(shard cannot be allocated on same node [FdcoENIXSQWkFbmLdIwkdQ] it already exists on)][YES(node passes include/exclude/require filters)][YES(primary is already active)][YES(below shard recovery limit of [2])][YES(allocation disabling is ignored)][YES(allocation disabling is ignored)][YES(no allocation awareness enabled)][YES(total shard limit disabled: [-1] <= 0)][YES(target node version [1.6.0] is same or newer than source node version [1.6.0])][YES(only a single node is present)][YES(shard not primary or relocation disabled)]]","status":400}
[nagios@hnapxlamslog01 ~]$ 

How do I overcome this issue?

Thanks

[swolf]

Hi @newmember,

I don't think your issue and the output you posted are related. If you have nagios log server running on just one machine, it's normal to see half of your shards (all of the replica shards) as 'UNASSIGNED'. They only exist to provide failover, so there's no point in starting that second shard on your main machine.

However, if everything is working properly on a one-instance log server, you should see yellow status, not red. What are the statuses of your main services?

Code: Select all

systemctl status httpd
systemctl status elasticsearch
systemctl status logstash

[newmember]

[nagios@hnapxlamslog01 ~]$ systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-10-24 07:49:13 PDT; 1 years 7 months ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 112408 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 68546 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─ 68546 /usr/sbin/httpd -DFOREGROUND
├─ 72048 /usr/sbin/httpd -DFOREGROUND
├─ 72131 /usr/sbin/httpd -DFOREGROUND
├─ 94834 /usr/sbin/httpd -DFOREGROUND
├─112438 /usr/sbin/httpd -DFOREGROUND
├─112439 /usr/sbin/httpd -DFOREGROUND
├─112440 /usr/sbin/httpd -DFOREGROUND
├─112441 /usr/sbin/httpd -DFOREGROUND
├─112442 /usr/sbin/httpd -DFOREGROUND
├─112513 /usr/sbin/httpd -DFOREGROUND
└─112515 /usr/sbin/httpd -DFOREGROUND
[nagios@hnapxlamslog01 ~]$
[nagios@hnapxlamslog01 ~]$ systemctl status elasticsearch
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: active (running) since Mon 2019-06-03 23:06:20 PDT; 1 weeks 0 days ago
Docs: man:systemd-sysv-generator(8)
Process: 94157 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 94324 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─94344 /bin/java -Xms7943m -Xmx7943m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFrac...
[nagios@hnapxlamslog01 ~]$
[nagios@hnapxlamslog01 ~]$ systemctl status logstash
● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Fri 2019-06-07 18:05:27 PDT; 3 days ago
Docs: man:systemd-sysv-generator(8)
Process: 79112 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 79126 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
├─79132 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/c...
└─79134 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSI...
[nagios@hnapxlamslog01 ~]$

[cdienger]

What is the full output of "curl -XGET 'http://localhost:9200/_cat/shards?v'" ? Red usually indicates a primary shard is unassigned but we don't see that in the snippet that was provided.

[newmember]

I see the output to a txt file

please see attached

Thanks

[tgriep]

Is the Logserver part of a cluster or is it a standalone system?

Take a look at this KB article for troubleshooting tips and possible solutions on fixing the Red Status on the server.
https://support.nagios.com/kb/article/n ... th-90.html