Some queries never complete

[jpconsilio]

Hi,
I'm not getting any results for certain queries for specific two week period. The page time out before any results are presented.
When I query the same period for events from the same period for my own username, I get results in minutes. For the user of interest: the query never completes. For successful queries of myself I see events only for a three day period, when I know there was activity for every day both weeks. I suspect there was some problem with the log servers during this period.

Occasionally when I sign in to log server, I find that each instance in the two instance cluster in Yellow health status and showing that its opposite is not running the elasticsearch or logstash services with half of the shards

There's about 750GB of data with 224GB available on disk.
Any assistance appreciated.

[cdienger]

Please provide me with screenshots highlighting the query that you are running as well as the results.

Please also PM me a profile from each machine. A can be gathered under Admin > System > System Status > Download System Profile or from the command line with:

Code: Select all

/usr/local/nagioslogserver/scripts/profile.sh

This will create /tmp/system-profile.tar.gz.

Note that this file can be very large and may not be able to be uploaded through the ticketing system. This is usually due to the logs in the Logstash and/or Elasticsearch directories found in it. If it is too large, please open the profile, extract these directories/files and send them separately.