Logstash Daemon dead but pid file exists

[Nags007]

Hi Team,

I have gone thru several threads for the same message, i was unable to get the answer. kindly look into this issue.

We are installing a fresh Nagios Log server downloaded : nagioslogserver-1.4.4.tar.gz on rhel 6.8 santiago.
=============================================================
[root@ NAGIOS-LOG-SOURCE]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.9 (Santiago)
=============================================================
We have installed the Nagios Log server we arefacing the issue of "Logstash Daemon dead but pid file exists".

Please check the attached file, where the red color error message as : logstash is stopped.

[root@nagioslogserver]# service logstash start

/usr/local/nagioslogserver/logstash/etc/conf.d
Starting Logstash Daemon: 54579
[ OK ]
==============================================

[root@va10dlvpos308 nagioslogserver]# java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

================================================

EMPTY :

/var/log/logstash

[root@ logstash]# ll
total 0
============================================
[root@logserver]# cat /usr/local/nagioslogserver/logstash/etc/conf.d/* | grep -v '^$\|^\s*\#'

input {
xxxxxxxxxxxxxxxx


output {
elasticsearch {
cluster => ' '
host => 'localhost'
document_type => '%{type}'
node_name => ''
protocol => 'transport'
workers => 4
}

============================
[root@nagioslogserver]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 2394/qpidd
tcp 0 0 0.0.0.0:4750 0.0.0.0:* LISTEN 2452/bin/rscd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1830/rpcbind
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2270/sshd
tcp 0 0 127.0.0.1:56823 0.0.0.0:* LISTEN 2770/klzagent
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1926/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2313/sendmail
tcp 0 0 0.0.0.0:55327 0.0.0.0:* LISTEN 1852/rpc.statd
tcp 0 0 :::33088 :::* LISTEN 1852/rpc.statd
tcp 0 0 :::2056 :::* LISTEN 932/java
tcp 0 0 :::5544 :::* LISTEN 932/java
tcp 0 0 :::5672 :::* LISTEN 2394/qpidd
tcp 0 0 :::2057 :::* LISTEN 932/java
tcp 0 0 :::111 :::* LISTEN 1830/rpcbind
tcp 0 0 ::ffff:127.0.0.1:9200 :::* LISTEN 33228/java
tcp 0 0 :::80 :::* LISTEN 31316/httpd
tcp 0 0 :::9300 :::* LISTEN 33228/java
tcp 0 0 :::22 :::* LISTEN 2270/sshd
tcp 0 0 ::1:631 :::* LISTEN 1926/cupsd
tcp 0 0 :::3515 :::* LISTEN 932/java

==========================================


[root@nagioslogserver]# service elasticsearch status
elasticsearch (pid 33228) is running...
=================================

[root@logserver init.d]# ls -ltr log*
-rwxr-x--- 1 root root 2920 May 10 03:47 logstashbkp
-rwxrwxr-x 1 root root 2959 May 10 08:19 logstash

[root@logserver init.d]# pwd
/etc/init.d
==============================

[root@ init.d]# ls -l /usr/local/nagioslogserver/logstash/etc/conf.d
total 12
-rwxrwxrwx 1 apache apache 636 Apr 28 03:30 000_inputs.conf
-rwxrwxrwx 1 apache apache 987 Apr 28 03:30 500_filters.conf
-rwxrwxrwx 1 apache apache 501 May 10 05:02 999_outputs.conf
================================================================


[root@ conf.d]# cat 999_outputs.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Fri, 28 Apr 2017 03:30:02 -0400
#

#
# Required output for Nagios Log Server
#

output {
elasticsearch {
cluster => ' '
host => 'localhost'
document_type => '%{type}'
node_name => ''
protocol => 'transport'
workers => 4
}
}

#
# Global outputs
#

# Local outputs
#

============================

plase check on this..

Thanks & Regards
Nag.

[cdienger]

What does the disk usage look like if you run "df -h" ?

Try:

service logstash stop
service logstash status ----make sure the service is stopped before the next step
rm /var/run/logstash/*
service logstash start

Attach a copy of the /var/log/logstash/logstash.log if the above doesn't help.

[Nags007]

Hi,

Please note that: ( no log file )

1) There is no log file present in var/log/logstash folder.

2) below disk available details.

[root@NAGIOSLOGSERVER]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
15G 5.3G 8.6G 38% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 194M 97M 87M 53% /boot
/dev/mapper/rootvg-home_lv
2.0G 70M 1.9G 4% /local_home
/dev/mapper/rootvg-opt_lv
6.0G 266M 5.4G 5% /opt
/dev/mapper/rootvg-tmp_lv
2.0G 1.1G 858M 56% /tmp
/dev/mapper/rootvg-usr_lv
2.0G 563M 1.4G 30% /usr/local
/dev/mapper/rootvg-var_lv
2.0G 855M 1.1G 45% /var
/dev/mapper/rootvg-history_lv
2.0G 68M 1.9G 4% /history_logs
/dev/mapper/rootvg-itm_lv
3.0G 362M 2.5G 13% /opt/IBM/ITM
/dev/mapper/appvg-apps_lv
247G 188M 234G 1% /apps

===================================================================

[root@NAGIOSLOGSERVER]# service logstash stop
/usr/local/nagioslogserver/logstash/etc/conf.d
Stopping Logstash Daemon: [FAILED]

[root@NAGIOSLOGSERVER]# service logstash status
/usr/local/nagioslogserver/logstash/etc/conf.d
Logstash Daemon is stopped


[root@NAGIOSLOGSERVER]# service logstash start
/usr/local/nagioslogserver/logstash/etc/conf.d
Starting Logstash Daemon: 59493
[ OK ]
[root@NAGIOSLOGSERVER]# cd /var/log/logstash

==================================================================

[root@logstash]# ls *.log
ls: cannot access *.log: No such file or directory

[root@logstash]# ll
total 0

=======================================================================

we are still facing the same issue of redcolor for logstash/ Logstash daemon dead..

Thank you,

[cdienger]

Please generate and PM a system profile when you have the chance. The profile can be generated under Administration > System Status.

[dwhitfield]

Also, please send it to me, as Craig will be out of the office tomorrow.

[Nags007]

Hi ,

Please find the Nagios Log Administration > System Status. details:-

[root@NagiosLogserver sysprofile]# ll
total 380
-rw-r--r-- 1 apache apache 210607 May 11 23:22 cluster.txt
-rw-r--r-- 1 apache apache 0 May 11 23:22 firewalld.txt
-rw-r--r-- 1 apache apache 1116 May 11 23:22 indices-status.txt
-rw-r--r-- 1 apache apache 0 May 11 23:22 iptables.txt
-rw-r--r-- 1 apache apache 1195 May 11 23:22 jvm-status.txt
-rw-r--r-- 1 apache apache 9825 May 11 23:22 logstash-confd.txt
-rw-r--r-- 1 apache apache 2350 May 11 23:22 logstash-plugins.txt
-rw-r--r-- 1 apache apache 206 May 11 23:22 masters.txt
-rw-r--r-- 1 apache apache 230 May 11 23:22 memory.txt
-rw-r--r-- 1 apache apache 31749 May 11 23:22 netstat.txt
-rw-r--r-- 1 apache apache 543 May 11 23:22 network.txt
-rw-r--r-- 1 apache apache 254 May 11 23:22 nodes.txt
-rw-r--r-- 1 apache apache 41 May 11 23:22 pending-tasks.txt
-rw-r--r-- 1 apache apache 160 May 11 23:22 plugins.txt
-rw-r--r-- 1 apache apache 18416 May 11 23:22 psaux.txt
-rw-r--r-- 1 apache apache 19749 May 11 23:22 recovery.txt
-rw-r--r-- 1 apache apache 25720 May 11 23:22 shard-health.txt
-rw-r--r-- 1 apache apache 17544 May 11 23:22 shard-status.txt
================================================================

NagiosLogserver$ cat pending-tasks.txt
insertOrder timeInQueue priority source


cat nodes.txt
=============
host ip heap.percent ram.percent load node.role master name
Logserver xx.yy.zz.aa 3 63 0.08 d * 1b3456dd-0xxxxxxxaaadfdfdfdfd

========================================================================================
[root@NagiosLogserver - sysprofile]# cat jvm-status.txt
{
"cluster_name" : "",
"nodes" : {
"ID" : {
"name" : "2b5708dd-0de2-492b-9e53-b25eef8665df",
"transport_address" : "inet[/aa.bbb.ccc.dd:9300]",
"host" : "NagiosLogserver",
"ip" : "xx.yy.zz.bb",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[localhost/127.0.0.1:9200]",
"attributes" : {
"max_local_storage_nodes" : "1"
},
"jvm" : {
"pid" : 33228,
"version" : "1.8.0_131",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "25.131-b11",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1494401772281,
"mem" : {
"heap_init_in_bytes" : 8359247872,
"heap_max_in_bytes" : 8324382720,
"non_heap_init_in_bytes" : 2555904,
"non_heap_max_in_bytes" : 0,
"direct_max_in_bytes" : 8324382720
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Metaspace", "Compressed Class Space", "Par Eden Space", "Par Survivor Space", "CMS Old Gen" ]
}
}
}
}
============================================================================

[root@NagiosLogserver sysprofile]# cat logstash-confd.txt
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Logstash Configuration File
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Dynamically created by Nagios Log Server
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Created Fri, 28 Apr 2017 03:30:02 -0400
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Global inputs
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:input {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: syslog {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: type => 'syslog'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: port => 5544
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: tcp {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: type => 'eventlog'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: port => 3515
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: codec => json {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: charset => 'CP1252'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: tcp {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: type => 'import_raw'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: tags => 'import_raw'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: port => 2056
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: tcp {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: type => 'import_json'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: tags => 'import_json'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: port => 2057
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: codec => json
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:}
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Local inputs
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Logstash Configuration File
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Dynamically created by Nagios Log Server
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Created Fri, 28 Apr 2017 03:30:02 -0400
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Global filters
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:filter {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: if [program] == 'apache_access' {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: grok {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: match => [ 'message', '%{COMBINEDAPACHELOG}']
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: date {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: mutate {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: replace => [ 'type', 'apache_access' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: convert => [ 'bytes', 'integer' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: convert => [ 'response', 'integer' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: if [program] == 'apache_error' {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: grok {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: mutate {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: replace => [ 'type', 'apache_error' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:}
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Local filters
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Logstash Configuration File
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Dynamically created by Nagios Log Server
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Created Fri, 28 Apr 2017 03:30:02 -0400
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Required output for Nagios Log Server
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:output {
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf: elasticsearch {
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf: cluster => 'xyz'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf: host => 'localhost'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf: document_type => '%{type}'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf: node_name => ''
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf: protocol => 'transport'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf: workers => 4
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf: }
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:}
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Global outputs
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Local outputs
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:

===========================================================================================================

[root@NagiosLogserver sysprofile]# cat logstash-plugins.txt

logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-oldlogstashjson
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-anonymize
logstash-filter-checksum
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-dns
logstash-filter-drop
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-multiline
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-eventlog
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-imap
logstash-input-irc
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-xmpp
logstash-input-zeromq
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-elasticsearch_http
logstash-output-email
logstash-output-exec
logstash-output-file
logstash-output-ganglia
logstash-output-gelf
logstash-output-graphite
logstash-output-hipchat
logstash-output-http
logstash-output-irc
logstash-output-juggernaut
logstash-output-kafka
logstash-output-lumberjack
logstash-output-nagios
logstash-output-nagios_nsca
logstash-output-null
logstash-output-opentsdb
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-xmpp
logstash-output-zeromq
logstash-patterns-core

=============================================
[root@NagiosLogserver sysprofile]# cat memory.txt
total used free shared buffers cached
Mem: 15943 11228 4715 0 446 723
-/+ buffers/cache: 10057 5885
Swap: 2047 52 1995
==============================================

[root@NagiosLogserver sysprofile]# cat indices-status.txt
=======================================================
yellow open logstash-2017.05.02 5 1 36428 0 5.8mb 5.8mb
yellow open logstash-2017.05.05 5 1 16875 0 3.4mb 3.4mb
yellow open logstash-2017.05.08 5 1 18979 0 3.7mb 3.7mb
yellow open logstash-2017.05.06 5 1 7277 0 1.3mb 1.3mb
yellow open nagioslogserver_log 5 1 59 0 129.8kb 129.8kb
yellow open logstash-2017.05.10 5 1 120604 0 19.7mb 19.7mb
yellow open logstash-2017.05.12 5 1 37018 0 14.5mb 14.5mb
yellow open logstash-2017.04.28 5 1 4178 0 652.3kb 652.3kb
yellow open logstash-2017.05.01 5 1 2507 0 738.5kb 738.5kb
yellow open nagioslogserver 1 1 40 4 74.6kb 74.6kb
yellow open logstash-2017.04.30 5 1 12465 0 2mb 2mb
yellow open logstash-2017.05.07 5 1 7557 0 1.5mb 1.5mb
yellow open logstash-2017.05.03 5 1 37241 0 5.7mb 5.7mb
yellow open logstash-2017.05.11 5 1 178467 0 31.2mb 31.2mb
yellow open kibana-int 5 1 9 0 89.3kb 89.3kb
yellow open logstash-2017.04.29 5 1 8588 0 1.3mb 1.3mb
yellow open logstash-2017.05.04 5 1 27948 0 4.9mb 4.9mb
yellow open logstash-2017.05.09 5 1 26394 0 4.7mb 4.7mb

=========================================================

Please let me know if you need more information.


Thank you,
Nagesh.

[tgriep]

Lets see if the nagios user account is not expired and if the logstash daemon is running.
Login as root on the nagios server, run the following and post the output.

Code: Select all

chage -l nagios
ps -ef |grep log

Thanks.

[Nags007]

[root@NagiosLogserver]#
======================================================================
[root@NagiosLogserver]# chage -l nagios
============================================
Last password change : Mar 21, 2017
Password expires : May 05, 2017
Password inactive : never
Account expires : never
Minimum number of days between password change : 7
Maximum number of days between password change : 45
Number of days of warning before password expires : 7
====================================================

[root@NagiosLogserver tmp]# ps -ef |grep log
===================================================
root 2195 1 0 Apr14 ? 00:00:00 /usr/sbin/mcelog --daemon
root 2360 1 0 Apr14 ? 00:00:00 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages
root 13720 1 0 Apr28 ? 00:01:31 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
nagios 33228 1 2 May10 ? 03:16:15 /usr/bin/java -Xms7971m -Xmx7971m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=452cd649-0824-4d2a-bdf2-904f7e2cfc92 -Des.node.name=2b5708dd-0de2-492b-9e53-b25eef8665df -Des.discovery.zen.ping.unicast.hosts=localhost -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.6.0.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch

root 45185 1 0 Apr16 ? 00:43:30 /usr/bin/perl -w /opt/tail2syslog/tail2syslog.pl -i rh -a -f authpriv.notice -c /opt/tail2syslog/qradar.cfg -u -p 517
root 45737 43566 0 01:27 pts/1 00:00:00 grep log
root 59952 1 1 May12 ? 01:15:52 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d/ -vv
=========================
Thank you,

[tgriep]

It looks like the nagios user account's password was expired. That account cannot have it's password expire because the system uses that account to run processes automatically and when it expires, it cannot do so.
Login as root and run the following to fix the issue.

Code: Select all

passwd --delete nagios
chage -I -1 -m 0 -M 99999 -E -1 nagios

Then try and restart the logstash daemon and see if that fixes the issue.

[Nags007]

root@Logserver ~]# passwd --delete nagios
Removing password for user nagios.
passwd: Success
[root@Logserver ~]# chage -I -1 -m 0 -M 99999 -E -1 nagios
==============================================================

we have the same issue of error..


root@NagiosLog server ~]# service logstash status
/usr/local/nagioslogserver/logstash/etc/conf.d
Logstash Daemon (pid 33592) is running...


[root@NagiosLogserver ~]# service logstash status
/usr/local/nagioslogserver/logstash/etc/conf.d
Logstash Daemon dead but pid file exists


[root@NagiosLogserver ~]# service logstash restart
/usr/local/nagioslogserver/logstash/etc/conf.d
Restarting Logstash Daemon: [FAILED]
33745
[ OK ]
[root@NagiosLogserver ~]# service logstash status
/usr/local/nagioslogserver/logstash/etc/conf.d
Logstash Daemon (pid 33745) is running...

[root@NagiosLogserver ~]# service logstash status
/usr/local/nagioslogserver/logstash/etc/conf.d
Logstash Daemon dead but pid file exists

[root@NagiosLogserver ~]#
==============================================================