Page 1 of 1

nagios ls - status = red

Posted: Sun Jun 09, 2019 2:40 am
by newmember
I have a red status

Code: Select all

[nagios@hnapxlamslog01 ~]$ curl -XGET 'http://localhost:9200/_cat/shards?v'
I have hundreds of these "UNASSIGNED"

Code: Select all

logstash-2018.09.09 0     p      STARTED         19   84.1kb 170.108.74.80 1ccd6723-32a1-4fbe-8de4-2fa95babec98 
logstash-2018.09.09 0     r      UNASSIGNED                                                                     
logstash-2018.09.09 3     p      STARTED         21   22.2kb 170.108.74.80 1ccd6723-32a1-4fbe-8de4-2fa95babec98 
logstash-2018.09.09 3     r      UNASSIGNED                                                                     
logstash-2018.09.09 1     p      STARTED         18   60.4kb 170.108.74.80 1ccd6723-32a1-4fbe-8de4-2fa95babec98 
logstash-2018.09.09 1     r      UNASSIGNED                                                                     
logstash-2018.09.09 2     p      STARTED         21   22.8kb 170.108.74.80 1ccd6723-32a1-4fbe-8de4-2fa95babec98 
logstash-2018.09.09 2     r      UNASSIGNED                                                                     
[nagios@hnapxlamslog01 ~]$ 
[nagios@hnapxlamslog01 ~]$ 

Code: Select all

[nagios@hnapxlamslog01 ~]$ curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ { "allocate" : { "index" : "logstash-2018.09.09",  "shard" : 2,  "node" : "
1ccd6723-32a1-4fbe-8de4-2fa95babec98",  "allow_primary" : true } } ] }'                                                                                                  
{"error":"ElasticsearchIllegalArgumentException[[allocate] allocation of [logstash-2018.09.09][2] on node [1ccd6723-32a1-4fbe-8de4-2fa95babec98][FdcoENIXSQWkFbmLdIwkdQ][hnapxlamslog01][inet[/170.108.74.80:9300]]{max_local_storage_nodes=1} is not allowed, reason: [NO(shard cannot be allocated on same node [FdcoENIXSQWkFbmLdIwkdQ] it already exists on)][YES(node passes include/exclude/require filters)][YES(primary is already active)][YES(below shard recovery limit of [2])][YES(allocation disabling is ignored)][YES(allocation disabling is ignored)][YES(no allocation awareness enabled)][YES(total shard limit disabled: [-1] <= 0)][YES(target node version [1.6.0] is same or newer than source node version [1.6.0])][YES(only a single node is present)][YES(shard not primary or relocation disabled)]]","status":400}
[nagios@hnapxlamslog01 ~]$ 


How do I overcome this issue?

Thanks

Re: nagios ls - status = red

Posted: Mon Jun 10, 2019 9:41 am
by swolf
Hi @newmember,

I don't think your issue and the output you posted are related. If you have nagios log server running on just one machine, it's normal to see half of your shards (all of the replica shards) as 'UNASSIGNED'. They only exist to provide failover, so there's no point in starting that second shard on your main machine.

However, if everything is working properly on a one-instance log server, you should see yellow status, not red. What are the statuses of your main services?

Code: Select all

systemctl status httpd
systemctl status elasticsearch
systemctl status logstash

Re: nagios ls - status = red

Posted: Tue Jun 11, 2019 1:37 am
by newmember
[nagios@hnapxlamslog01 ~]$ systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-10-24 07:49:13 PDT; 1 years 7 months ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 112408 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 68546 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─ 68546 /usr/sbin/httpd -DFOREGROUND
├─ 72048 /usr/sbin/httpd -DFOREGROUND
├─ 72131 /usr/sbin/httpd -DFOREGROUND
├─ 94834 /usr/sbin/httpd -DFOREGROUND
├─112438 /usr/sbin/httpd -DFOREGROUND
├─112439 /usr/sbin/httpd -DFOREGROUND
├─112440 /usr/sbin/httpd -DFOREGROUND
├─112441 /usr/sbin/httpd -DFOREGROUND
├─112442 /usr/sbin/httpd -DFOREGROUND
├─112513 /usr/sbin/httpd -DFOREGROUND
└─112515 /usr/sbin/httpd -DFOREGROUND
[nagios@hnapxlamslog01 ~]$
[nagios@hnapxlamslog01 ~]$ systemctl status elasticsearch
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: active (running) since Mon 2019-06-03 23:06:20 PDT; 1 weeks 0 days ago
Docs: man:systemd-sysv-generator(8)
Process: 94157 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 94324 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─94344 /bin/java -Xms7943m -Xmx7943m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFrac...
[nagios@hnapxlamslog01 ~]$
[nagios@hnapxlamslog01 ~]$ systemctl status logstash
● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Fri 2019-06-07 18:05:27 PDT; 3 days ago
Docs: man:systemd-sysv-generator(8)
Process: 79112 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 79126 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
├─79132 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/c...
└─79134 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSI...
[nagios@hnapxlamslog01 ~]$

Re: nagios ls - status = red

Posted: Tue Jun 11, 2019 4:51 pm
by cdienger
What is the full output of "curl -XGET 'http://localhost:9200/_cat/shards?v'" ? Red usually indicates a primary shard is unassigned but we don't see that in the snippet that was provided.

Re: nagios ls - status = red

Posted: Tue Jun 11, 2019 10:00 pm
by newmember
I see the output to a txt file

please see attached

Thanks

Re: nagios ls - status = red

Posted: Wed Jun 12, 2019 2:04 pm
by tgriep
Is the Logserver part of a cluster or is it a standalone system?

Take a look at this KB article for troubleshooting tips and possible solutions on fixing the Red Status on the server.
https://support.nagios.com/kb/article/n ... th-90.html