Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Logstash service will not stay on

https://support.nagios.com/forum/viewtopic.php?t=37668

Page 2 of 3

Re: Logstash service will not stay on

Posted: Thu Mar 24, 2016 4:50 pm
by cgutierr
As I am evaluating your product on a classified network I am unable to copy the output and transfer it here. I would have to type everything out...

Re: Logstash service will not stay on

Posted: Thu Mar 24, 2016 4:55 pm
by jolson
I imagine that would be time consuming, but it's hard for us to help without seeing that kind of output. With that being said, the following might help you in your troubleshooting endeavors:

The cron daemon is supposed to launch two processes every minute - jobs and poller. These two processes are details in the output of cat /etc/cron.d/nagioslogserver and the jobs are logged to tail /var/log/cron. Be sure that the cron-jobs are running every minutes as they are supposed to.

My output so that you can compare with your own:

Code: Select all

[root@localhost ~]#     ls -ld /usr/local/nagioslogserver/logstash/etc/conf.d
drwxrwxr-x. 2 nagios nagios 4096 Dec 22 15:06 /usr/local/nagioslogserver/logstash/etc/conf.d
[root@localhost ~]#     egrep "nag|apache" /etc/group
apache:x:48:nagios
nagios:x:500:nagios,apache
[root@localhost ~]#     egrep "nag|apache" /etc/passwd
    chage -l nagios
apache:x:48:48:Apache:/var/www:/sbin/nologin
nagios:x:500:100::/home/nagios:/bin/bash
[root@localhost ~]#     chage -l nagios
Last password change                                    : Dec 21, 2015
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7
[root@localhost ~]#     service crond status
    tail /var/log/cron
    cat /etc/cron.d/nagioslogservercrond (pid  1134) is running...
[root@localhost ~]#     tail /var/log/cron
Mar 24 17:50:01 localhost CROND[5645]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:50:01 localhost CROND[5646]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 24 17:51:01 localhost CROND[5750]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:51:01 localhost CROND[5751]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 24 17:52:01 localhost CROND[5853]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:52:01 localhost CROND[5854]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 24 17:53:01 localhost CROND[5957]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:53:01 localhost CROND[5958]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Mar 24 17:54:01 localhost CROND[6091]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Mar 24 17:54:01 localhost CROND[6092]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
[root@localhost ~]#     cat /etc/cron.d/nagioslogserver
# /etc/cron.d/nagioslogserver: crontab fragment for nagioslogserver

* * * * * nagios /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
* * * * * nagios /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
[root@localhost ~]#     tail /var/log/secure
Mar 24 17:54:17 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:17 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:27 localhost sudo:   apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:27 localhost sudo:   apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:32 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:32 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:40 localhost sudo:   apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:40 localhost sudo:   apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:47 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:47 localhost sudo:   nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
The above was run on a working server.

Re: Logstash service will not stay on

Posted: Thu Mar 24, 2016 5:34 pm
by cgutierr
Everything is pretty much the same but the only thing I see that is different is this:

Yours:
[root@localhost ~]# tail /var/log/secure
Mar 24 17:54:17 localhost sudo: nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:17 localhost sudo: nagios : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:27 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:27 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status

Mine:
[root@localhost ~]# tail /var/log/secure
Mar 24 17:54:17 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:17 localhost sudo: apache: TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status
Mar 24 17:54:27 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Mar 24 17:54:27 localhost sudo: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status

You have a combination of nagios and apache while I just have apache. It is running every minute as it should be.

Re: Logstash service will not stay on

Posted: Fri Mar 25, 2016 9:24 am
by jolson
Good catch. If you look at the entirety of your /var/log/secure file, does the nagios user show up, or is it always apache? cat /var/log/secure Also, check out your /etc/sudoers file and ensure that it matches mine:

Mine (near the bottom):

Code: Select all

User_Alias      NAGIOSLOGSERVER=nagios
User_Alias      NAGIOSLOGSERVERWEB=apache
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/change_timezone.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/get_logstash_ports.sh

Re: Logstash service will not stay on

Posted: Fri Mar 25, 2016 9:44 am
by cgutierr
No, the nagios user never shows up, it is always apache. The /etc/sudoers file does match.

Re: Logstash service will not stay on

Posted: Fri Mar 25, 2016 10:38 am
by hsmith
What's do nagios and apache look like in /etc/groups and /etc/passwd?

Re: Logstash service will not stay on

Posted: Fri Mar 25, 2016 11:01 am
by cgutierr
[root@localhost ~]# egrep "nag|apache" /etc/group
apache:x:48:nagios
nagios:x:504:nagios,apache

[root@localhost ~]# egrep "nag|apache" /etc/passwd
apache:x:48:48:Apache:/var/www:/sbin/nologin
nagios:x:504:100::/home/nagios:/bin/bash

Re: Logstash service will not stay on

Posted: Fri Mar 25, 2016 11:22 am
by cgutierr
One thing that weird is that when I install NLS the permissions on /var/www/html/nagioslogserver are incorrect. I had to run a chmod -R 755 on the entire directory to bring up the webpage. Also, the "verify" button on the global configuration page will pop a permissions error, so I chmod -R 755 on the entire /usr/local/nagioslogserver directory to make sure everything is accessible while troublshooting. I am thinking it is during the initial installation where it sets all the permissions on the files is the problem. Is there some way we can verify that configuration file in the install package?

Re: Logstash service will not stay on

Posted: Fri Mar 25, 2016 12:47 pm
by cgutierr
Ok, from going over the fullinstall script and setting the correct ownerships and permissions to the /var/www/html/nagioslogserver and /usr/local/nagioslogserver directories, I was able to get the logstash config files to populate correctly, however, I am still unable to see logs flowing in to the WebUI even though in the Log Server Overview it says I am receiving logs from 4 hosts... These are all Linux hosts as well. Any ideas guys?

Re: Logstash service will not stay on

Posted: Fri Mar 25, 2016 12:50 pm
by rkennedy
Can you clarify how you setup the linux hosts to receive logs?

Did you create a custom input for them at all or are they using a built in input? It may be a firewall issue at this point.
All times are UTC-05:00
Page 2 of 3

Powered by phpBB® Forum Software © phpBB Limited