Logstash service will not stay on

[cgutierr]

As I am evaluating your product on a classified network I am unable to copy the output and transfer it here. I would have to type everything out...

[jolson]

I imagine that would be time consuming, but it's hard for us to help without seeing that kind of output. With that being said, the following might help you in your troubleshooting endeavors:

The cron daemon is supposed to launch two processes every minute - jobs and poller. These two processes are details in the output of cat /etc/cron.d/nagioslogserver and the jobs are logged to tail /var/log/cron. Be sure that the cron-jobs are running every minutes as they are supposed to.

My output so that you can compare with your own:

Code: Select all

[root@localhost ~]#     ls -ld /usr/local/nagioslogserver/logstash/etc/conf.d
drwxrwxr-x. 2 nagios nagios 4096 Dec 22 15:06 /usr/local/nagioslogserver/logstash/etc/conf.d
[root@localhost ~]#     egrep "nag|apache" /etc/group
apache:x:48:nagios
nagios:x:500:nagios,apache
[root@localhost ~]#     egrep "nag|apache" /etc/passwd
    chage -l nagios
apache:x:48:48:Apache:/var/www:/sbin/nologin
nagios:x:500:100::/home/nagios:/bin/bash
[root@localhost ~]#     chage -l nagios
Last password change                                    : Dec 21, 2015
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7
[root@localhost ~]#     service crond status
    tail /var/log/cron
    cat /etc/cron.d/nagioslogservercrond (pid  1134) is running...
[root@localhost ~]#     tail /var/log/cron
Mar 24 17:50:01 localhost CROND[5645]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:50:01 localhost CROND[5646]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 24 17:51:01 localhost CROND[5750]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:51:01 localhost CROND[5751]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 24 17:52:01 localhost CROND[5853]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:52:01 localhost CROND[5854]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 24 17:53:01 localhost CROND[5957]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:53:01 localhost CROND[5958]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 24 17:54:01 localhost CROND[6091]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:54:01 localhost CROND[6092]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
[root@localhost ~]#     cat /etc/cron.d/nagioslogserver
# /etc/cron.d/nagioslogserver: crontab fragment for nagioslogserver

* * * * * nagios /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
* * * * * nagios /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
[root@localhost ~]#     tail /var/log/secure
Mar 24 17:54:17 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:17 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:27 localhost sudo:   apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:27 localhost sudo:   apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:32 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:32 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:40 localhost sudo:   apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:40 localhost sudo:   apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:47 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:47 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status

The above was run on a working server.

[cgutierr]

Everything is pretty much the same but the only thing I see that is different is this:

Yours:
[root@localhost ~]# tail /var/log/secure
Mar 24 17:54:17 localhost sudo: nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:17 localhost sudo: nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:27 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:27 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status

Mine:
[root@localhost ~]# tail /var/log/secure
Mar 24 17:54:17 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:17 localhost sudo: apache: TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:27 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:27 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status

You have a combination of nagios and apache while I just have apache. It is running every minute as it should be.

[jolson]

Good catch. If you look at the entirety of your /var/log/secure file, does the nagios user show up, or is it always apache? cat /var/log/secure Also, check out your /etc/sudoers file and ensure that it matches mine:

Mine (near the bottom):

Code: Select all

User_Alias      NAGIOSLOGSERVER=nagios
User_Alias      NAGIOSLOGSERVERWEB=apache
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/change_timezone.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/get_logstash_ports.sh

[cgutierr]

No, the nagios user never shows up, it is always apache. The /etc/sudoers file does match.

[hsmith]

What's do nagios and apache look like in /etc/groups and /etc/passwd?

[cgutierr]

[root@localhost ~]# egrep "nag|apache" /etc/group
apache48:nagios
nagios504:nagios,apache

[root@localhost ~]# egrep "nag|apache" /etc/passwd
apache48:48:Apache:/var/www:/sbin/nologin
nagios504:/home/nagios:/bin/bash

[cgutierr]

One thing that weird is that when I install NLS the permissions on /var/www/html/nagioslogserver are incorrect. I had to run a chmod -R 755 on the entire directory to bring up the webpage. Also, the "verify" button on the global configuration page will pop a permissions error, so I chmod -R 755 on the entire /usr/local/nagioslogserver directory to make sure everything is accessible while troublshooting. I am thinking it is during the initial installation where it sets all the permissions on the files is the problem. Is there some way we can verify that configuration file in the install package?

[cgutierr]

Ok, from going over the fullinstall script and setting the correct ownerships and permissions to the /var/www/html/nagioslogserver and /usr/local/nagioslogserver directories, I was able to get the logstash config files to populate correctly, however, I am still unable to see logs flowing in to the WebUI even though in the Log Server Overview it says I am receiving logs from 4 hosts... These are all Linux hosts as well. Any ideas guys?

[rkennedy]

Can you clarify how you setup the linux hosts to receive logs?

Did you create a custom input for them at all or are they using a built in input? It may be a firewall issue at this point.