logs do not push realtime

[sacom01]

hi,
I have some logs from many system push log to NLS.
i am facing to this issue :
Log not push real time, sometime it push many logs after that stuck and work fine and stuck again, like time out error.

I have log like this from /var/log:

{:timestamp=>"2021-05-11T11:43:47.391000+0700", :message=>"retrying failed action with response code: 429 (RemoteTransportException[[df0079ec-6938-4c2e-8426-f836858b16d1][inet[/192.168.165.6:9300]][indices:data/write/bulk]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1@3cb79892]; )", :level=>:info}

After restart, it getting log :

{​:timestamp=>"2021-05-11T08:41:25.030000+0700", :message=>"retrying failed action with response code: 503 (UnavailableShardsException[[logstash-2021.05.11][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@43606f07])", :level=>:info}​

Log from client :

2021-05-11T14:34:16.606+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: read tcp 192.168.xxx.xxx:57540->192.168.xxx.x:5010: i/o timeout
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:16.606+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: read tcp 192.168.xxx.xxx:57540->192.168.xxx.x:5010: i/o timeout
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:16.685+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: client is not connected
2021-05-11T14:34:16.685+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.685+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:17.739+0700 ERROR [publisher_pipeline_output] pipeline/output.go:181 failed to publish events: client is not connected
2021-05-11T14:34:17.739+0700 INFO [publisher_pipeline_output] pipeline/output.go:144 Connecting to backoff(async(tcp://192.168.xxx.x:5010))
2021-05-11T14:34:17.740+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:17.740+0700 INFO [publisher] pipeline/retry.go:225 done

[ssax]

Please PM me a copy of your profile so I can review your logs/shard status, you can download it from Admin > System Status by clicking the Download System Profile button​.

[ssax]

I see an ton of old jobs that a must be stuck in your ps aux output.

What is the output of these commands?

Code: Select all

chage -l nagios
tail -n30 /var/log/cron

Try running these commands and see if it helps (I would do this on all nodes):

Code: Select all

systemctl stop logstash elasticsearch crond
pkill -9 -u nagios
pkill -9 CROND
systemctl start logstash elasticsearch crond

Then go to Admin > Command Subsystem and click the Reset All Jobs button.

If it occurs again after that, please send a FRESH copy of your profile so I can review the latest logs.

[sacom01]

our system work fine now, after we do clearing cache.
pls close the ticket
thanks

[scottwilkerson]

our system work fine now, after we do clearing cache.
pls close the ticket
thanks

Locking thread