Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Logs stop coming in

https://support.nagios.com/forum/viewtopic.php?t=30568

Page 1 of 3

Logs stop coming in

Posted: Tue Dec 16, 2014 12:42 pm
by globalgiving
Probably related to my last post about this where I thought there was something wrong with just backups (http://support.nagios.com/forum/viewtop ... 38&t=30496).

First I checked logstash logs:

Code: Select all

{:timestamp=>"2014-12-16T11:16:28.570000-0500", :message=>"Failed to flush outgoing items", :outgoing_count=>5000, :exception=>#<RuntimeError: Non-OK response code from Elasticsearch: 500>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:127:in `bulk_ftw'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:80:in `bulk'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:315:in `flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1339:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:193:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:311:in `receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:86:in `handle'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:78:in `worker_setup'"], :level=>:warn}
{:timestamp=>"2014-12-16T11:19:48.816000-0500", :message=>"Failed to flush outgoing items", :outgoing_count=>5000, :exception=>#<Errno::ECONNRESET: Connection reset by peer - Connection reset by peer>, :backtrace=>["org/jruby/RubyIO.java:3016:in `sysread'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:243:in `read'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/protocol.rb:30:in `read_http_message'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/request.rb:93:in `execute'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:325:in `execute'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:217:in `post!'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:106:in `bulk_ftw'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:80:in `bulk'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:315:in `flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1339:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:193:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:311:in `receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:86:in `handle'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:78:in `worker_setup'"], :level=>:warn}
Then checked the elasticsearch logs and dumped those since the time of the logs stopped coming in. (see attached)

I noticed in the elastic search logs that it said somethign about being out of memory. So I checked the memory usage on the system. Looks ok-ish to me:

Code: Select all

             total       used       free     shared    buffers     cached
Mem:         15898      15734        164          0        155      12785
-/+ buffers/cache:       2793      13105
Swap:         8015          0       8015
While all the memory has been requested, it not actually in use (hence plenty of free -/+ buffers/cache), also no swap used.. so plenty of free.

I restarted elastic search (/etc/init.d/elasticsearch restart) and everything went back to normal. However, I don't want to have to monitor for this and restart elasticsearch all the time.

If there are any other diagnostics I can run, or config information i can give you, i would be happy to.

Re: Logs stop coming in

Posted: Tue Dec 16, 2014 4:37 pm
by emislivec
This was the Java VM running out of heap memory.

You can increase this by setting ES_HEAP_SIZE in /etc/sysconfig/elasticsearch. The general recommendation is to give it half of the available RAM. It looks like you have about 16GB total, so:

Code: Select all

ES_HEAP_SIZE=8g
MAX_LOCKED_MEMORY=unlimited
That last line keeps the ES heap from being swapped if possible (system limits may need to be adjusted to allow this).

Some more info on ES memory tuning: http://www.elasticsearch.org/guide/en/e ... izing.html

Re: Logs stop coming in

Posted: Mon Dec 22, 2014 11:37 am
by myriad
My system was crawling. I think this may have fixed my issues... I hope

Re: Logs stop coming in

Posted: Mon Dec 22, 2014 12:02 pm
by lmiltchev
Let us know if the issue reappears.

Re: Logs stop coming in

Posted: Tue Dec 30, 2014 11:06 am
by myriad
It seemed to fix the issue, but a week later, the server is crawling, and two of my instances are not reporting.

Re: Logs stop coming in

Posted: Tue Dec 30, 2014 11:19 am
by tgriep
Could you post the latest elastic log file and can you run the following command from a shell and post that also?

Code: Select all

df -h

Re: Logs stop coming in

Posted: Tue Dec 30, 2014 11:56 am
by myriad

Code: Select all

[root@logphx ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_oraclelinux-lv_root
                      577G   67G  482G  13% /
tmpfs                 4.0G     0  4.0G   0% /dev/shm
/dev/sda1             485M  153M  307M  34% /boot
/dev/mapper/vg_oraclelinux-lv_home
                      9.5G  150M  8.8G   2% /home

Code: Select all

[root@logcorp ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_oraclelinux-lv_root
                      577G   60G  488G  11% /
tmpfs                 1.5G     0  1.5G   0% /dev/shm
/dev/sda1             485M  153M  307M  34% /boot
/dev/mapper/vg_oraclelinux-lv_home
                      9.5G  150M  8.8G   2% /home

Code: Select all

[root@logonr ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_oraclelinux-lv_root
                      577G   68G  481G  13% /
tmpfs                 4.0G     0  4.0G   0% /dev/shm
/dev/sda1             485M  153M  307M  34% /boot
/dev/mapper/vg_oraclelinux-lv_home
                      9.5G  150M  8.8G   2% /home

Code: Select all

[root@logonr elasticsearch]# tail f1f3c840-af23-497e-9f6a-c213e7e6408f.log.2014-12-29
[2014-12-29 23:59:36,402][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:36,404][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:41,406][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:41,407][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,409][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,411][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,781][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,783][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:51,413][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:51,414][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[root@logonr elasticsearch]#

Code: Select all

[root@logphx elasticsearch]# tail f1f3c840-af23-497e-9f6a-c213e7e6408f.log.2014-12-29
[2014-12-29 18:28:05,684][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:30:05,469][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:31:22,790][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:34:00,285][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:42:05,185][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:42:05,719][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:45:41,537][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 19:26:02,143][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 21:09:11,789][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:16:22,869][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[root@logphx elasticsearch]#

Code: Select all

[root@logcorp elasticsearch]# tail f1f3c840-af23-497e-9f6a-c213e7e6408f.log.2014-12-29
[2014-12-29 20:27:10,294][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 20:46:17,485][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 20:46:19,457][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 21:31:57,464][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 22:19:56,492][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:25:31,777][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:25:39,626][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:26:50,770][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:38:15,203][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:46:23,429][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[root@logcorp elasticsearch]#

Re: Logs stop coming in

Posted: Tue Dec 30, 2014 11:58 am
by myriad
Here's more from the logonr server...

Code: Select all

[2014-12-29 23:56:06,964][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:06,966][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:11,968][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:11,970][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:16,972][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:16,973][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:21,975][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:21,977][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:26,979][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:26,980][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:31,983][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:31,984][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:36,986][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:36,987][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:41,989][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:41,991][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:46,801][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:46,801][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:46,993][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:46,994][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:51,996][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:51,997][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:01,073][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:01,077][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:01,079][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:01,080][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:03,860][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:57:04,000][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:57:06,082][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:06,083][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:11,085][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:11,086][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:16,088][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:16,090][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:21,091][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:21,093][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:26,094][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:26,096][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:31,098][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:31,100][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:36,102][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:36,104][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:41,106][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:41,107][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:46,109][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:46,111][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:46,744][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:46,767][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:51,113][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:51,114][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:01,192][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:01,207][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:01,215][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:01,217][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:02,857][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:58:04,003][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:58:06,218][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:06,220][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:11,222][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:11,224][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:16,226][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:16,228][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:21,230][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:21,231][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:26,233][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:26,235][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:31,237][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:31,238][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:36,240][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:36,242][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:41,244][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:41,246][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:46,247][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:46,249][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:46,746][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:46,841][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:51,250][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:51,252][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:01,348][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:01,369][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:01,376][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:01,377][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:02,859][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:59:03,070][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:59:06,379][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:06,381][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:11,383][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:11,385][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:16,386][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:16,388][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:21,393][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:21,394][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:26,396][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:26,397][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:31,399][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:31,401][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:36,402][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:36,404][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:41,406][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:41,407][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,409][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,411][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,781][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,783][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:51,413][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:51,414][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[root@logonr elasticsearch]#

Re: Logs stop coming in

Posted: Tue Dec 30, 2014 12:51 pm
by tgriep
Can you go to Administration and screen capture your Cluster and Instance status and post them here?

Re: Logs stop coming in

Posted: Tue Dec 30, 2014 4:47 pm
by myriad
Post restart, I got this
http://i.imgur.com/mj2BGnH.png
and
http://i.imgur.com/uaVM2bu.png
All times are UTC-05:00
Page 1 of 3

Powered by phpBB® Forum Software © phpBB Limited