Logs stop coming in

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
User avatar
globalgiving
Posts: 25
Joined: Thu Aug 28, 2014 9:57 am
Location: Washington, DC
Contact:
Contact globalgiving

Logs stop coming in

Post by globalgiving »

Probably related to my last post about this where I thought there was something wrong with just backups (http://support.nagios.com/forum/viewtop ... 38&t=30496).

First I checked logstash logs:

Code: Select all

{:timestamp=>"2014-12-16T11:16:28.570000-0500", :message=>"Failed to flush outgoing items", :outgoing_count=>5000, :exception=>#<RuntimeError: Non-OK response code from Elasticsearch: 500>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:127:in `bulk_ftw'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:80:in `bulk'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:315:in `flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1339:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:193:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:311:in `receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:86:in `handle'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:78:in `worker_setup'"], :level=>:warn}
{:timestamp=>"2014-12-16T11:19:48.816000-0500", :message=>"Failed to flush outgoing items", :outgoing_count=>5000, :exception=>#<Errno::ECONNRESET: Connection reset by peer - Connection reset by peer>, :backtrace=>["org/jruby/RubyIO.java:3016:in `sysread'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:243:in `read'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/protocol.rb:30:in `read_http_message'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/request.rb:93:in `execute'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:325:in `execute'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:217:in `post!'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:106:in `bulk_ftw'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:80:in `bulk'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:315:in `flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1339:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:193:in `buffer_flush'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/elasticsearch.rb:311:in `receive'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:86:in `handle'", "/usr/local/nagioslogserver/logstash/lib/logstash/outputs/base.rb:78:in `worker_setup'"], :level=>:warn}
Then checked the elasticsearch logs and dumped those since the time of the logs stopped coming in. (see attached)

I noticed in the elastic search logs that it said somethign about being out of memory. So I checked the memory usage on the system. Looks ok-ish to me:

Code: Select all

             total       used       free     shared    buffers     cached
Mem:         15898      15734        164          0        155      12785
-/+ buffers/cache:       2793      13105
Swap:         8015          0       8015
While all the memory has been requested, it not actually in use (hence plenty of free -/+ buffers/cache), also no swap used.. so plenty of free.

I restarted elastic search (/etc/init.d/elasticsearch restart) and everything went back to normal. However, I don't want to have to monitor for this and restart elasticsearch all the time.

If there are any other diagnostics I can run, or config information i can give you, i would be happy to.
You do not have the required permissions to view the files attached to this post.
Justin Rupp
Senior Systems Ninja
GlobalGiving Foundation
Top
emislivec
Posts: 52
Joined: Tue Feb 25, 2014 10:06 am

Re: Logs stop coming in

Post by emislivec »

This was the Java VM running out of heap memory.

You can increase this by setting ES_HEAP_SIZE in /etc/sysconfig/elasticsearch. The general recommendation is to give it half of the available RAM. It looks like you have about 16GB total, so:

Code: Select all

ES_HEAP_SIZE=8g
MAX_LOCKED_MEMORY=unlimited
That last line keeps the ES heap from being swapped if possible (system limits may need to be adjusted to allow this).

Some more info on ES memory tuning: http://www.elasticsearch.org/guide/en/e ... izing.html
Top
myriad
Posts: 26
Joined: Tue Dec 02, 2014 1:29 pm

Re: Logs stop coming in

Post by myriad »

My system was crawling. I think this may have fixed my issues... I hope
Top
User avatar
lmiltchev
Former Nagios Staff
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Logs stop coming in

Post by lmiltchev »

Let us know if the issue reappears.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
myriad
Posts: 26
Joined: Tue Dec 02, 2014 1:29 pm

Re: Logs stop coming in

Post by myriad »

It seemed to fix the issue, but a week later, the server is crawling, and two of my instances are not reporting.
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Logs stop coming in

Post by tgriep »

Could you post the latest elastic log file and can you run the following command from a shell and post that also?

Code: Select all

df -h
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
myriad
Posts: 26
Joined: Tue Dec 02, 2014 1:29 pm

Re: Logs stop coming in

Post by myriad »

Code: Select all

[root@logphx ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_oraclelinux-lv_root
                      577G   67G  482G  13% /
tmpfs                 4.0G     0  4.0G   0% /dev/shm
/dev/sda1             485M  153M  307M  34% /boot
/dev/mapper/vg_oraclelinux-lv_home
                      9.5G  150M  8.8G   2% /home

Code: Select all

[root@logcorp ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_oraclelinux-lv_root
                      577G   60G  488G  11% /
tmpfs                 1.5G     0  1.5G   0% /dev/shm
/dev/sda1             485M  153M  307M  34% /boot
/dev/mapper/vg_oraclelinux-lv_home
                      9.5G  150M  8.8G   2% /home

Code: Select all

[root@logonr ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_oraclelinux-lv_root
                      577G   68G  481G  13% /
tmpfs                 4.0G     0  4.0G   0% /dev/shm
/dev/sda1             485M  153M  307M  34% /boot
/dev/mapper/vg_oraclelinux-lv_home
                      9.5G  150M  8.8G   2% /home

Code: Select all

[root@logonr elasticsearch]# tail f1f3c840-af23-497e-9f6a-c213e7e6408f.log.2014-12-29
[2014-12-29 23:59:36,402][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:36,404][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:41,406][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:41,407][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,409][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,411][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,781][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,783][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:51,413][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:51,414][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[root@logonr elasticsearch]#

Code: Select all

[root@logphx elasticsearch]# tail f1f3c840-af23-497e-9f6a-c213e7e6408f.log.2014-12-29
[2014-12-29 18:28:05,684][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:30:05,469][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:31:22,790][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:34:00,285][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:42:05,185][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:42:05,719][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 18:45:41,537][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 19:26:02,143][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 21:09:11,789][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:16:22,869][INFO ][cluster.metadata         ] [9cffbbbd-b73e-4444-acd6-e28302931ccb] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[root@logphx elasticsearch]#

Code: Select all

[root@logcorp elasticsearch]# tail f1f3c840-af23-497e-9f6a-c213e7e6408f.log.2014-12-29
[2014-12-29 20:27:10,294][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 20:46:17,485][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 20:46:19,457][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 21:31:57,464][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 22:19:56,492][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:25:31,777][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:25:39,626][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:26:50,770][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:38:15,203][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[2014-12-29 23:46:23,429][INFO ][cluster.metadata         ] [733d0c27-985e-4744-88dc-30a43a889a68] [logstash-2014.12.30] update_mapping [eventlog] (dynamic)
[root@logcorp elasticsearch]#
Top
myriad
Posts: 26
Joined: Tue Dec 02, 2014 1:29 pm

Re: Logs stop coming in

Post by myriad »

Here's more from the logonr server...

Code: Select all

[2014-12-29 23:56:06,964][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:06,966][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:11,968][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:11,970][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:16,972][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:16,973][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:21,975][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:21,977][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:26,979][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:26,980][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:31,983][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:31,984][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:36,986][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:36,987][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:41,989][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:41,991][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:46,801][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:46,801][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:46,993][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:46,994][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:51,996][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:56:51,997][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:01,073][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:01,077][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:01,079][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:01,080][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:03,860][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:57:04,000][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:57:06,082][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:06,083][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:11,085][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:11,086][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:16,088][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:16,090][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:21,091][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:21,093][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:26,094][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:26,096][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:31,098][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:31,100][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:36,102][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:36,104][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:41,106][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:41,107][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:46,109][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:46,111][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:46,744][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:46,767][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:51,113][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:57:51,114][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:01,192][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:01,207][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:01,215][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:01,217][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:02,857][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:58:04,003][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:58:06,218][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:06,220][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:11,222][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:11,224][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:16,226][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:16,228][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:21,230][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:21,231][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:26,233][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:26,235][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:31,237][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:31,238][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:36,240][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:36,242][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:41,244][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:41,246][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:46,247][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:46,249][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:46,746][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:46,841][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:51,250][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:58:51,252][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:01,348][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:01,369][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:01,376][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:01,377][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:02,859][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:59:03,070][DEBUG][action.index             ] [c05224a9-d045-4794-81c0-52dc5aab030d] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2014-12-29 23:59:06,379][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:06,381][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:11,383][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:11,385][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:16,386][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:16,388][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:21,393][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:21,394][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:26,396][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:26,397][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:31,399][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:31,401][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:36,402][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:36,404][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:41,406][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:41,407][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,409][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,411][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,781][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:46,783][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:51,413][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[2014-12-29 23:59:51,414][DEBUG][action.search.type       ] [c05224a9-d045-4794-81c0-52dc5aab030d] All shards failed for phase: [query_fetch]
[root@logonr elasticsearch]#
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Logs stop coming in

Post by tgriep »

Can you go to Administration and screen capture your Cluster and Instance status and post them here?
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
myriad
Posts: 26
Joined: Tue Dec 02, 2014 1:29 pm

Re: Logs stop coming in

Post by myriad »

Post restart, I got this
http://i.imgur.com/mj2BGnH.png
and
http://i.imgur.com/uaVM2bu.png
Top
Locked

Return to “Nagios Log Server”