Search found 19 matches

by li_alm
Fri Mar 22, 2019 4:03 am
Forum: Nagios Log Server
Topic: Error: All shards failed for phase
Replies: 5
Views: 1612

Re: Error: All shards failed for phase

OK, @cdienger, thanks, we will try to increase the RAM for the machine using only 2GB.

My main concern was about MSG1 and MSG2 (see my initial post), because I had the impression Nagios Log Servers would not start.

Regards,
Liviu
by li_alm
Wed Mar 20, 2019 3:32 am
Forum: Nagios Log Server
Topic: Error: All shards failed for phase
Replies: 5
Views: 1612

Re: Error: All shards failed for phase

Hello, @npolovenko, Thank you for your reply. I have 2 nagios deployments (completely independent, separate), both behave the same (a lot of "rejected" messages in the logs). Deployment1: 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz) 2 GB RAM Deployment2: 1 CPU core (Intel(R) Xeon...
by li_alm
Tue Mar 19, 2019 3:01 am
Forum: Nagios Log Server
Topic: Error: All shards failed for phase
Replies: 5
Views: 1612

Error: All shards failed for phase

Hello, Our Nagios Log Server has stopped working (the elasticsearch service seemed to have stopped). We restarted the elasticsearch service and a lot of messages of the following type appeared in the log: MSG1: All shards failed for phase: [query] org.elasticsearch.action.NoShardAvailableActionExcep...
by li_alm
Wed Aug 08, 2018 3:32 am
Forum: Nagios Log Server
Topic: elasticsearch "rejected execution (queue capacity 1000)" mes
Replies: 3
Views: 1891

Re: elasticsearch "rejected execution (queue capacity 1000)"

scottwilkerson Hi, Scott, Thank you for your reply. Yes, I also found this link, but I really can't tell which solution applies to us. I have 2 nagios deployments (completely independent, separate), both behave the same (a lot of "rejected" messages in the logs). Deployment1: 1 CPU core (...
by li_alm
Fri Jun 29, 2018 3:29 am
Forum: Nagios Log Server
Topic: elasticsearch "rejected execution (queue capacity 1000)" mes
Replies: 3
Views: 1891

elasticsearch "rejected execution (queue capacity 1000)" mes

Hello, We receive a lot of elasticsearch "query rejected" messages in nagios. E.g.: 2018-06-21 03:50:11,996][DEBUG][action.count ] [04c4efb4-9365-45d3-9c7b-162e3cbcc051] [logstash-2017.07.16][2], node[ac07Vu3JSl6vogmMAT4AYA], [P], s[STARTED]: failed to execute [[[]][], source[{"query&...
by li_alm
Fri Oct 06, 2017 3:32 pm
Forum: Nagios Log Server
Topic: When does Elasticsearch stop writing to an index?
Replies: 7
Views: 2537

Re: When does Elasticsearch stop writing to an index?

Yes, the timestamp in the message received by logstash is used as the timestamp in the nagios system.

match => { "message" => ...(?:%{TIMESTAMP_ISO8601:syslog_ts}|-)...
date {
match => [ "syslog_ts", "ISO8601" ]
}
by li_alm
Fri Oct 06, 2017 11:41 am
Forum: Nagios Log Server
Topic: When does Elasticsearch stop writing to an index?
Replies: 7
Views: 2537

Re: When does Elasticsearch stop writing to an index?

Wow, thanks.
So you're saying if today=06.10.2017 and ES receives from logstash a message timestamped=04.10.2017, this message will be written into logstash-2017.10.04?
So i'ts actually the timestamp that matters, not the time running on the machine nagios is installed.

Liviu
by li_alm
Fri Oct 06, 2017 10:43 am
Forum: Nagios Log Server
Topic: When does Elasticsearch stop writing to an index?
Replies: 7
Views: 2537

Re: When does Elasticsearch stop writing to an index?

First of all, sorry for the confusion. Yes, ES is the database, and Logstash writes data to ES. Q: "when the day rolls over" - so when the day changes on the machine nagios is running on, I can safely say logstash will no longer write to the "logstash-<previous day>" index? So I ...
by li_alm
Fri Oct 06, 2017 9:30 am
Forum: Nagios Log Server
Topic: When does Elasticsearch stop writing to an index?
Replies: 7
Views: 2537

When does Elasticsearch stop writing to an index?

Hello, I need to write a script that sends queries directly to elasticsearch and I need to process the whole previous day. Example: today = 06.10.2017 yesterday = 05.10.2017 yesterday index = logstash-2017.10.05 At what (minimum) time is it fine to run the script (on 06.10.2017) so that I am sure th...
by li_alm
Wed Oct 04, 2017 2:38 am
Forum: Nagios Log Server
Topic: Match related logs in Nagios Log Server
Replies: 6
Views: 2470

Re: Match related logs in Nagios Log Server

Thank you for you answers. @mcapra: Yes, I also found the aggregations mechanisms. Using aggregations, I can group by messageID and by eventID (using 2 aggregations - 1 inner aggregation) and I can find out the actual count for eventID=101 for every messageID. Then, I need to parse the results (usin...