Nagios Support Forum

I don't have any other clean system to do nagios log server implementation:(. Is there any way to find out why elastic search and logstash are in exited status instead of running?

Hi,

I have stopped and started logstash. Immediately if I see the status, it is in running status. But after two minutes if I see the same status, It is in exited status. Please have a look at the below logs.

[root@COGNISRV03 ~]# systemctl stop logstash
[root@COGNISRV03 ~]# systemctl start logstash
[root@COGNISRV03 ~]# systemctl status logstash
logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Thu 2016-06-09 14:14:21 CEST; 6s ago
Process: 11368 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 11432 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
ââ11438 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/lo...
ââ11440 /bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.p...

Jun 09 14:14:20 COGNISRV03 systemd[1]: Starting LSB: Logstash...
Jun 09 14:14:20 COGNISRV03 runuser[11438]: pam_unix(runuser:session): session opened for user root by (uid=0)
Jun 09 14:14:20 COGNISRV03 logstash[11432]: Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environ...server/tmp
Jun 09 14:14:21 COGNISRV03 logstash[11432]: [ OK ]
Jun 09 14:14:21 COGNISRV03 systemd[1]: Started LSB: Logstash.
Hint: Some lines were ellipsized, use -l to show in full.
[root@COGNISRV03 ~]#


After two minutes,

[root@COGNISRV03 ~]# systemctl status logstash
logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (exited) since Thu 2016-06-09 14:14:21 CEST; 2min 39s ago
Process: 11368 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 11432 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)

Jun 09 14:14:20 COGNISRV03 runuser[11438]: pam_unix(runuser:session): session opened for user root by (uid=0)
Jun 09 14:14:20 COGNISRV03 logstash[11432]: Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environ...server/tmp
Jun 09 14:14:21 COGNISRV03 logstash[11432]: [ OK ]
Jun 09 14:14:21 COGNISRV03 systemd[1]: Started LSB: Logstash.
Jun 09 14:14:30 COGNISRV03 logstash[11432]: Jun 09, 2016 2:14:30 PM org.elasticsearch.plugins.PluginsService <init>
Jun 09 14:14:30 COGNISRV03 logstash[11432]: INFO: [J. Jonah Jameson] loaded [], sites []
Jun 09 14:14:32 COGNISRV03 logstash[11432]: Jun 09, 2016 2:14:32 PM org.elasticsearch.plugins.PluginsService <init>
Jun 09 14:14:32 COGNISRV03 logstash[11432]: INFO: [Dansen Macabre] loaded [], sites []
Jun 09 14:14:32 COGNISRV03 logstash[11432]: Jun 09, 2016 2:14:32 PM org.elasticsearch.plugins.PluginsService <init>
Jun 09 14:14:32 COGNISRV03 logstash[11432]: INFO: [Perun] loaded [], sites []Hint: Some lines were ellipsized, use -l to show in full.
[root@COGNISRV03 ~]#


Moreover highlighted lines got added in it.Can you please help me resolve this issue

Can you tell me more about the system this is running on?

Do you have an AntiVirus running on it?
Is SELinux running?
What other software is running on here?

Monica7 wrote:I don't have any other clean system to do nagios log server implementation

If this system isn't clean, as stated above, I need to know what's been done with it to figure out what might be happening. There are some systems we can not support, as Log Server was not tested with modifications made to the server.

I don't have clean, minimal system available. Having only one box where I am facing this issue

My Elasticsearch and logstash are in exited status. Does any one aware of how to resolve this issue and to make it running.?

I don't have clean, minimal system available. Having only one box where I am facing this issue

My Elasticsearch and logstash are in exited status. Does any one aware of how to resolve this issue and to make it running.?

Does anything show up in /var/log/messages after they exit?

Hi,

I am able to make both the elasticsearch and logstash running in Linux box.

[root@COGNISRV03 ~]# service elasticsearch status
elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: active (running) since Mon 2016-06-20 12:12:04 CEST; 1h 3min ago
Process: 59301 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 59342 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
ââ59362 /bin/java -Xms3908m -Xmx3908m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSI...

Jun 20 12:12:04 COGNISRV03 systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Jun 20 12:12:04 COGNISRV03 runuser[59353]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Jun 20 12:12:04 COGNISRV03 runuser[59353]: pam_unix(runuser:session): session closed for user nagios
Jun 20 12:12:04 COGNISRV03 elasticsearch[59342]: Starting elasticsearch: [ OK ]
Jun 20 12:12:04 COGNISRV03 systemd[1]: Started LSB: This service manages the elasticsearch daemon.
[root@COGNISRV03 ~]# service logstash status
Logstash Daemonlogstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Mon 2016-06-20 12:13:13 CEST; 1h 2min ago
Process: 59787 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 60108 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
ââ60124 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/lo...
ââ60126 /bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:+PrintGCDetails -XX:+PrintGCTimeStam...

Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: Retrying request to {}->http://COGNISRV03:9200
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http:...on refused
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: Retrying request to {}->http://COGNISRV03:9200
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http:...on refused
Jun 20 13:16:09 COGNISRV03 logstash[60108]: Jun 20, 2016 1:16:09 PM org.apache.http.impl.execchain.RetryExec execute
Jun 20 13:16:09 COGNISRV03 logstash[60108]: INFO: Retrying request to {}->http://COGNISRV03:9200
Hint: Some lines were ellipsized, use -l to show in full.
[root@COGNISRV03 ~]#

But I am not able to see the status of elastic search and logstash running in System status in Nagios log server user interface(front end). It keeps on loading. Attached is the screenshot of the screen.

Do you have a load balancer or anything in front of the machine? What URL are you using to access the machine?

What is the output of nslookup COGNISRV03? I'd like to see how it's resolving itself based on this part -