Problem index log every day

[scottwilkerson]

Let us know the results of the commands.

Also, when you say it is blank, this makes me wonder if there happen to be any httpd errors of significance

Code: Select all

tail -100 /var/log/httpd/error_log

[chris_Espoir]

Hello,

Here result of commands on all server in the cluster

[root@nagioslog1 ~]# tail -100 /var/log/httpd/error_log
[Sun Jan 04 03:50:02 2015] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 04 03:50:02 2015] [notice] Digest: done
[Sun Jan 04 03:50:02 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations
[Mon Jan 05 08:27:35 2015] [notice] caught SIGTERM, shutting down
[Mon Jan 05 08:28:37 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jan 05 08:28:37 2015] [notice] Digest: generating secret for digest authentication ...
[Mon Jan 05 08:28:37 2015] [notice] Digest: done
[Mon Jan 05 08:28:37 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations
[Tue Jan 06 11:34:02 2015] [error] [client 192.168.9.10] File does not exist: /var/www/html/apple-touch-icon-precomposed.png
[Tue Jan 06 11:34:02 2015] [error] [client 192.168.9.10] File does not exist: /var/www/html/apple-touch-icon.png
[Tue Jan 06 16:41:08 2015] [notice] caught SIGTERM, shutting down
[Tue Jan 06 16:42:12 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Jan 06 16:42:12 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Jan 06 16:42:12 2015] [notice] Digest: done
[Tue Jan 06 16:42:12 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations

[root@nagioslog2 ~]# tail -100 /var/log/httpd/error_log
[Sun Jan 04 03:18:03 2015] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 04 03:18:03 2015] [notice] Digest: done
[Sun Jan 04 03:18:03 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations
[Mon Jan 05 08:27:55 2015] [notice] caught SIGTERM, shutting down
[Mon Jan 05 08:29:01 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jan 05 08:29:01 2015] [notice] Digest: generating secret for digest authentication ...
[Mon Jan 05 08:29:01 2015] [notice] Digest: done
[Mon Jan 05 08:29:01 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations
[Tue Jan 06 16:41:17 2015] [notice] caught SIGTERM, shutting down
[Tue Jan 06 16:42:12 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Jan 06 16:42:12 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Jan 06 16:42:12 2015] [notice] Digest: done
[Tue Jan 06 16:42:12 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations

[root@nagioslog3 ~]# tail -100 /var/log/httpd/error_log
[Sun Jan 04 03:14:03 2015] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 04 03:14:03 2015] [notice] Digest: done
[Sun Jan 04 03:14:03 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations
[Mon Jan 05 08:27:39 2015] [notice] caught SIGTERM, shutting down
[Mon Jan 05 08:28:24 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jan 05 08:28:24 2015] [notice] Digest: generating secret for digest authentication ...
[Mon Jan 05 08:28:24 2015] [notice] Digest: done
[Mon Jan 05 08:28:25 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations
[Tue Jan 06 16:41:11 2015] [notice] caught SIGTERM, shutting down
[Tue Jan 06 16:42:00 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Jan 06 16:42:00 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Jan 06 16:42:00 2015] [notice] Digest: done
[Tue Jan 06 16:42:00 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations

[Nagios Support]

Thanks, chris_Espoir! We will do some digging into this and get back to you within the next 24 hours.

[tmcdonald]

Maybe this is a silly question, but do you have any filters applied underneath the query search bar?

[sreinhardt]

I'd also like to have you apply the fix here: http://support.nagios.com/forum/viewtop ... er#p122363
You may not be seeing crashes, but it seems to be similar issues.

[chris_Espoir]

Maybe this is a silly question, but do you have any filters applied underneath the query search bar?

No question is silly

There is nos filters applied underneath the query search bar.

I'd also like to have you apply the fix here: viewtopic.php?f=38&t=30778&p=122093&hilit=+log+server#p122363
You may not be seeing crashes, but it seems to be similar issues.

I would like, but

You are not authorised to read this forum.

Thanks, chris_Espoir! We will do some digging into this and get back to you within the next 24 hours.

Ok thank you, If you want any information yet, do not hesitate to ask.

[lgroschen]

Here's sreinhardt's post he was reffering to above, try applying this fix:

Looks like you're hitting this guy: https://github.com/elasticsearch/logstash/issues/1604, you should be able to follow the patch here: https://github.com/jordansissel/ruby-ft ... 59eccc18a3. Try replacing theses files with the ones attached to this post:

/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb
/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/pool.rb

Use the attached zip file:

[chris_Espoir]

I install files, wait & see

[sreinhardt]

Sounds great, let us know how it goes!

[chris_Espoir]

Hello,

Bad news, I still have the problem.
Today's index is not even create :/