Page 1 of 2
updated to 2.0.2 - http issues
Posted: Wed Jan 24, 2018 4:24 pm
by googol
http server keeps going down on one instance (I monitor it from my network/server monitoring tool)
I try to access the web interface when this alarm goes off, and yep, cant get to the web page, therefore no login.
I reboot the VM to restore http for now. I did the memory changes for 1.0.4, do I have to redo those changes I found in your knowledgebase for 2.0.2 to increase the memory used for elastica or something? I am giving the VM 12GB of RAM.
Re: updated to 2.0.2 - http issues
Posted: Wed Jan 24, 2018 5:37 pm
by npolovenko
Hi,
@googol. How many logs are you monitoring with the log server? Can you upload the acess_log, error_log, and the ssl_error_log from /var/log/ directory? What happens if you run
service httpd status? Also please run a df command to check whether a hard drive is running out of space.
Re: updated to 2.0.2 - http issues
Posted: Wed Jan 24, 2018 6:14 pm
by googol
attached logs for httpd, guessing thats what you wanted
disk space is not full
I'll try the service httpd status next time it dies
Re: updated to 2.0.2 - http issues
Posted: Thu Jan 25, 2018 11:47 am
by npolovenko
@googol, Yes, that's what I meant, sorry. Please open /etc/php.ini and change the:
Code: Select all
memory_limit = 128M
to
memory_limit = 512M
After that please run:
Let me know if that fixed your problem.
Re: updated to 2.0.2 - http issues
Posted: Thu Jan 25, 2018 11:50 am
by googol
not a problem, so basically after every time I update the software, I need to do set these settings?
looks like I set one for logstash, not for php.ini
Re: updated to 2.0.2 - http issues
Posted: Thu Jan 25, 2018 12:00 pm
by googol
actually, memory_limit is already set to 512M in /etc/php.ini
Re: updated to 2.0.2 - http issues
Posted: Thu Jan 25, 2018 2:19 pm
by googol
just happened again, confirmed that php.ini is 512M.
I did service httpd status and it says httpd pid 1656 is running...
but not responding to http requests.
Re: updated to 2.0.2 - http issues
Posted: Thu Jan 25, 2018 4:43 pm
by tmcdonald
I'd like to get some more information about your environment. Please provide the following:
- OS and version?
- If multiple hosts in cluster, is it just this one that is having issues?
- How long does it take on average for the http daemon to die?
- Is there anything else running on this server aside from Log Server?
Re: updated to 2.0.2 - http issues
Posted: Thu Jan 25, 2018 5:31 pm
by googol
This is your OVA (1.0.4 version), upgraded to 2.0.2 using your upgrade script following your steps, I've done yum update to see if updates to packages help.
nothing else installed on this system, dedicated to nagios and only nagios.
I have two instances, one in each datacenter. only my secondary datacenter's nagios instance is having this issue at this time. They both should be the same setup.
uname -r shows 2.6.32-279.11.1.el6.x86_64
nagios login prompt shows CentOS release 6.8 (Final)
havent seen anything to indicate a particular time or time period, or frame why it goes down when it does, but I will try to track that for you.
didnt have this problem on 1.0.4, dunno whats going on
Re: updated to 2.0.2 - http issues
Posted: Fri Jan 26, 2018 2:20 pm
by cdienger
Attached is also a script to gather a profile from the command line. Copy this to the machine and from the command line run:
This will generate a file called system-profile.tar.gz in /tmp. Please PM me this as well as recent log files from /var/log/logstash and /var/log/elasticsearch.