Logstash collector auto stop

[riconeo]

Hi,

I keep having the logstash collector auto stopping.
This is part of the cluster and is working fine until it suddenly got this error.

{:timestamp=>"2015-02-02T06:10:03.485000-0500", :message=>"Error: No config files found: /usr/local/nagioslogserver/logstash/etc/conf.d/*\nCan you make sure this path is a logstash config file?"}
{:timestamp=>"2015-02-02T06:10:03.509000-0500", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}

Any help?
thanks.

[tmcdonald]

Let's take a look at that directory. Please run the following command as root from the Nagios server command line and post the results:

Code: Select all

ls -ld /usr/local/nagioslogserver/logstash/etc/conf.d/
ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/

[riconeo]

[root@sgpnagioslog1 ~]# ls -ld /usr/local/nagioslogserver/logstash/etc/conf.d/
drwxrwxr-x. 2 nagios nagios 4096 Feb 2 15:39 /usr/local/nagioslogserver/logstash/etc/conf.d/


[root@sgpnagioslog1 ~]# ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/
total 12
-rw-r--r-- 1 apache apache 218 Feb 2 15:39 000_inputs.conf
-rw-r--r-- 1 apache apache 978 Feb 2 15:39 500_filters.conf
-rw-r--r-- 1 apache apache 493 Feb 2 15:39 999_outputs.conf

[scottwilkerson]

And to make sure the nagios user can read them

Code: Select all

su nagios -c 'ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/'

If you get the list, you should be able to restart logstash

Code: Select all

service logstash restart

[riconeo]

And to make sure the nagios user can read them

Code: Select all

su nagios -c 'ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/'

If you get the list, you should be able to restart logstash

Code: Select all

service logstash restart

Not sure did I do wrong, after I ran the command, below is the output

"
[root@sgpnagioslog1 ~]# su nagios -c 'ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/'
total 12
-rw-r--r-- 1 apache apache 218 Feb 2 15:39 000_inputs.conf
-rw-r--r-- 1 apache apache 978 Feb 2 15:39 500_filters.conf
-rw-r--r-- 1 apache apache 493 Feb 2 15:39 999_outputs.conf
"

For the 2nd cluster instance, it has same conf.d output but the logstash is not stopping.

"

[scottwilkerson]

On the instance that had it stopped, could you PM me the log from
/var/log/logstash/logstash.log

Thanks

[scottwilkerson]

The file you sent just shows on the 2nd the configs somehow couldn't be read.

Being they are there now lets just start logstash

Code: Select all

service logstash start

[riconeo]

The file you sent just shows on the 2nd the configs somehow couldn't be read.

Being they are there now lets just start logstash

Code: Select all

service logstash start

Hi,

Even after starting the logstash service, the log file is still empty for the faulty cluster instance.
Do you need the working cluster instance?

[tmcdonald]

Have you made any changes to the non-working machine? Is selinux enabled or did you change any passwords or system user information?

[riconeo]

Have you made any changes to the non-working machine? Is selinux enabled or did you change any passwords or system user information?

Hi, only the root password is changed.

Anyway, we have decided to remove the existing 2 cluster instance and set it up from fresh again.
Thanks.