Getting 500 Internal server error

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
Monica7
Posts: 45
Joined: Thu May 19, 2016 4:42 am

Re: Getting 500 Internal server error

Post by Monica7 »

I don't have any other clean system to do nagios log server implementation:(. Is there any way to find out why elastic search and logstash are in exited status instead of running?
Top
Monica7
Posts: 45
Joined: Thu May 19, 2016 4:42 am

Re: Getting 500 Internal server error

Post by Monica7 »

Hi,

I have stopped and started logstash. Immediately if I see the status, it is in running status. But after two minutes if I see the same status, It is in exited status. Please have a look at the below logs.

[root@COGNISRV03 ~]# systemctl stop logstash
[root@COGNISRV03 ~]# systemctl start logstash
[root@COGNISRV03 ~]# systemctl status logstash
logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Thu 2016-06-09 14:14:21 CEST; 6s ago
Process: 11368 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 11432 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
ââ11438 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/lo...
ââ11440 /bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.p...

Jun 09 14:14:20 COGNISRV03 systemd[1]: Starting LSB: Logstash...
Jun 09 14:14:20 COGNISRV03 runuser[11438]: pam_unix(runuser:session): session opened for user root by (uid=0)
Jun 09 14:14:20 COGNISRV03 logstash[11432]: Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environ...server/tmp
Jun 09 14:14:21 COGNISRV03 logstash[11432]: [ OK ]
Jun 09 14:14:21 COGNISRV03 systemd[1]: Started LSB: Logstash.
Hint: Some lines were ellipsized, use -l to show in full.
[root@COGNISRV03 ~]#


After two minutes,

[root@COGNISRV03 ~]# systemctl status logstash
logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (exited) since Thu 2016-06-09 14:14:21 CEST; 2min 39s ago
Process: 11368 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 11432 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)

Jun 09 14:14:20 COGNISRV03 runuser[11438]: pam_unix(runuser:session): session opened for user root by (uid=0)
Jun 09 14:14:20 COGNISRV03 logstash[11432]: Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environ...server/tmp
Jun 09 14:14:21 COGNISRV03 logstash[11432]: [ OK ]
Jun 09 14:14:21 COGNISRV03 systemd[1]: Started LSB: Logstash.
Jun 09 14:14:30 COGNISRV03 logstash[11432]: Jun 09, 2016 2:14:30 PM org.elasticsearch.plugins.PluginsService <init>
Jun 09 14:14:30 COGNISRV03 logstash[11432]: INFO: [J. Jonah Jameson] loaded [], sites []
Jun 09 14:14:32 COGNISRV03 logstash[11432]: Jun 09, 2016 2:14:32 PM org.elasticsearch.plugins.PluginsService <init>
Jun 09 14:14:32 COGNISRV03 logstash[11432]: INFO: [Dansen Macabre] loaded [], sites []
Jun 09 14:14:32 COGNISRV03 logstash[11432]: Jun 09, 2016 2:14:32 PM org.elasticsearch.plugins.PluginsService <init>
Jun 09 14:14:32 COGNISRV03 logstash[11432]: INFO: [Perun] loaded [], sites []Hint: Some lines were ellipsized, use -l to show in full.
[root@COGNISRV03 ~]#


Moreover highlighted lines got added in it.Can you please help me resolve this issue
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: Getting 500 Internal server error

Post by hsmith »

Can you tell me more about the system this is running on?

Do you have an AntiVirus running on it?
Is SELinux running?
What other software is running on here?
Monica7 wrote:I don't have any other clean system to do nagios log server implementation
If this system isn't clean, as stated above, I need to know what's been done with it to figure out what might be happening. There are some systems we can not support, as Log Server was not tested with modifications made to the server.
Former Nagios Employee.
me.
Top
Monica7
Posts: 45
Joined: Thu May 19, 2016 4:42 am

Re: Getting 500 Internal server error

Post by Monica7 »

I don't have clean, minimal system available. Having only one box where I am facing this issue

My Elasticsearch and logstash are in exited status. Does any one aware of how to resolve this issue and to make it running.?
Top
Monica7
Posts: 45
Joined: Thu May 19, 2016 4:42 am

Re: Getting 500 Internal server error

Post by Monica7 »

I don't have clean, minimal system available. Having only one box where I am facing this issue

My Elasticsearch and logstash are in exited status. Does any one aware of how to resolve this issue and to make it running.?
Top
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Contact hsmith

Re: Getting 500 Internal server error

Post by hsmith »

Does anything show up in /var/log/messages after they exit?
Former Nagios Employee.
me.
Top
Monica7
Posts: 45
Joined: Thu May 19, 2016 4:42 am

Re: Getting 500 Internal server error

Post by Monica7 »

Hi,

I am able to make both the elasticsearch and logstash running in Linux box.

[root@COGNISRV03 ~]# service elasticsearch status
elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: active (running) since Mon 2016-06-20 12:12:04 CEST; 1h 3min ago
Process: 59301 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 59342 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
ââ59362 /bin/java -Xms3908m -Xmx3908m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSI...

Jun 20 12:12:04 COGNISRV03 systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Jun 20 12:12:04 COGNISRV03 runuser[59353]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Jun 20 12:12:04 COGNISRV03 runuser[59353]: pam_unix(runuser:session): session closed for user nagios
Jun 20 12:12:04 COGNISRV03 elasticsearch[59342]: Starting elasticsearch: [ OK ]
Jun 20 12:12:04 COGNISRV03 systemd[1]: Started LSB: This service manages the elasticsearch daemon.
[root@COGNISRV03 ~]# service logstash status
Logstash Daemonlogstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Mon 2016-06-20 12:13:13 CEST; 1h 2min ago
Process: 59787 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 60108 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
ââ60124 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/lo...
ââ60126 /bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:+PrintGCDetails -XX:+PrintGCTimeStam...

Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: Retrying request to {}->http://COGNISRV03:9200
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http:...on refused
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: Retrying request to {}->http://COGNISRV03:9200
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http:...on refused
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: Retrying request to {}->http://COGNISRV03:9200
Hint: Some lines were ellipsized, use -l to show in full.
[root@COGNISRV03 ~]#

But I am not able to see the status of elastic search and logstash running in System status in Nagios log server user interface(front end). It keeps on loading. Attached is the screenshot of the screen.
You do not have the required permissions to view the files attached to this post.
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Getting 500 Internal server error

Post by rkennedy »

Do you have a load balancer or anything in front of the machine? What URL are you using to access the machine?

What is the output of nslookup COGNISRV03? I'd like to see how it's resolving itself based on this part -

Code: Select all

Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: Retrying request to {}->http://COGNISRV03:9200
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http:...on refused
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Former Nagios Employee
Top
Locked

Return to “Nagios Log Server”