hi,
I have some logs from many system push log to NLS.
i am facing to this issue :
Log not push real time, sometime it push many logs after that stuck and work fine and stuck again, like time out error.
I have log like this from /var/log:
{:timestamp=>"2021-05-11T11:43:47.391000+0700", :message=>"retrying failed action with response code: 429 (RemoteTransportException[[df0079ec-6938-4c2e-8426-f836858b16d1][inet[/192.168.165.6:9300]][indices:data/write/bulk]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1@3cb79892]; )", :level=>:info}
After restart, it getting log :
{:timestamp=>"2021-05-11T08:41:25.030000+0700", :message=>"retrying failed action with response code: 503 (UnavailableShardsException[[logstash-2021.05.11][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@43606f07])", :level=>:info}
Log from client :
2021-05-11T14:34:16.606+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: read tcp 192.168.xxx.xxx:57540->192.168.xxx.x:5010: i/o timeout
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:16.606+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: read tcp 192.168.xxx.xxx:57540->192.168.xxx.x:5010: i/o timeout
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.606+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:16.685+0700 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: client is not connected
2021-05-11T14:34:16.685+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:16.685+0700 INFO [publisher] pipeline/retry.go:225 done
2021-05-11T14:34:17.739+0700 ERROR [publisher_pipeline_output] pipeline/output.go:181 failed to publish events: client is not connected
2021-05-11T14:34:17.739+0700 INFO [publisher_pipeline_output] pipeline/output.go:144 Connecting to backoff(async(tcp://192.168.xxx.x:5010))
2021-05-11T14:34:17.740+0700 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2021-05-11T14:34:17.740+0700 INFO [publisher] pipeline/retry.go:225 done
logs do not push realtime
logs do not push realtime
You do not have the required permissions to view the files attached to this post.
Re: logs do not push realtime
Please PM me a copy of your profile so I can review your logs/shard status, you can download it from Admin > System Status by clicking the Download System Profile button.
Re: logs do not push realtime
I see an ton of old jobs that a must be stuck in your ps aux output.
What is the output of these commands?
Try running these commands and see if it helps (I would do this on all nodes):
Then go to Admin > Command Subsystem and click the Reset All Jobs button.
If it occurs again after that, please send a FRESH copy of your profile so I can review the latest logs.
What is the output of these commands?
Code: Select all
chage -l nagios
tail -n30 /var/log/cron
Code: Select all
systemctl stop logstash elasticsearch crond
pkill -9 -u nagios
pkill -9 CROND
systemctl start logstash elasticsearch crond
If it occurs again after that, please send a FRESH copy of your profile so I can review the latest logs.
Re: logs do not push realtime
our system work fine now, after we do clearing cache.
pls close the ticket
thanks
pls close the ticket
thanks
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: logs do not push realtime
Locking threadsacom01 wrote:our system work fine now, after we do clearing cache.
pls close the ticket
thanks