Nagios Support Forum

Posted: **Tue May 11, 2021 2:14 am**

hi,
I have some logs from many system push log to NLS.
i am facing to this issue :
Log not push real time, sometime it push many logs after that stuck and work fine and stuck again, like time out error.

I have log like this from /var/log:

{:timestamp=>"2021-05-11T11:43:47.391000+0700", :message=>"retrying failed action with response code: 429 (RemoteTransportException[[df0079ec-6938-4c2e-8426-f836858b16d1][inet[/192.168.165.6:9300]][indices:data/write/bulk]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1@3cb79892]; )", :level=>:info}

After restart, it getting log :

{:timestamp=>"2021-05-11T08:41:25.030000+0700", :message=>"retrying failed action with response code: 503 (UnavailableShardsException[[logstash-2021.05.11][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@43606f07])", :level=>:info}

Log from client :

2021-05-11T14:34:16.606+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: read tcp 192.168.xxx.xxx:57540->192.168.xxx.x:5010: i/o timeout
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:16.606+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: read tcp 192.168.xxx.xxx:57540->192.168.xxx.x:5010: i/o timeout
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:16.685+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: client is not connected
2021-05-11T14:34:16.685+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.685+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:17.739+0700 ERROR [publisher_pipeline_output] pipeline/output.go:181 failed to publish events: client is not connected
2021-05-11T14:34:17.739+0700 INFO [publisher_pipeline_output] pipeline/output.go:144 Connecting to backoff(async(tcp://192.168.xxx.x:5010))
2021-05-11T14:34:17.740+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:17.740+0700 INFO [publisher] pipeline/retry.go:225 done

Posted: **Tue May 11, 2021 2:11 pm**

Please PM me a copy of your profile so I can review your logs/shard status, you can download it from Admin > System Status by clicking the Download System Profile button.

Posted: **Thu May 13, 2021 11:13 am**

I see an ton of old jobs that a must be stuck in your ps aux output.

What is the output of these commands?

Code: Select all

chage -l nagios
tail -n30 /var/log/cron

Try running these commands and see if it helps (I would do this on all nodes):

Code: Select all

systemctl stop logstash elasticsearch crond
pkill -9 -u nagios
pkill -9 CROND
systemctl start logstash elasticsearch crond

Then go to Admin > Command Subsystem and click the Reset All Jobs button.

If it occurs again after that, please send a FRESH copy of your profile so I can review the latest logs.

Posted: **Sun May 16, 2021 10:04 pm**

our system work fine now, after we do clearing cache.
pls close the ticket
thanks

Posted: **Mon May 17, 2021 6:30 am**

sacom01 wrote:our system work fine now, after we do clearing cache.
pls close the ticket
thanks

Locking thread

Nagios Support Forum

logs do not push realtime

logs do not push realtime

Re: logs do not push realtime

Re: logs do not push realtime

Re: logs do not push realtime

Re: logs do not push realtime