LogServer not listening on defined input ports?

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
jalfaro
Posts: 22
Joined: Wed Apr 05, 2017 9:50 am

LogServer not listening on defined input ports?

Post by jalfaro »

I've got a 2 node log server cluster setup and everything on the surface appears to be working correctly. I would however expect the input ports to show up on a netstat and I'm not seeing any of the input ports (5544, or 3515) in the netstat output - what is the best way to go about ensuring that logstash is in fact listening on these ports?

netstat output:

Code: Select all

[root@naw02logserver01 config]# netstat -antp | egrep '(3515)|(5544)'
[root@naw02logserver01 config]#
The logstash.log file contains only error messages:

Code: Select all

[root@naw02logserver01 config]# tail /var/log/logstash/logstash.log
{:timestamp=>"2017-07-02T00:08:41.199000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-07-02T00:08:41.199000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-07-02T00:08:41.199000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-07-02T00:08:41.199000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
I found this when googling the error on the elasticsearch site:
I found the problem, here is the solution:

1) on ES config, need to define the cluster name (default is "elasticsearch" )
2) on LS config file, add cluster definition as the above the example:
output {
elasticsearch
{ host => "zslesvm09:9300"
cluster => } # cluster_name defined @ ES
stdout { }
}

"cluster" is introduced by the latest LS 1.5.2.

Before we use the default definition that makes the LS cannot find the correct ES.

Thanks.
I found these cluster names in the logstash and elasticsearch configs:

Code: Select all

[root@naw02logserver01 config]# cat elasticsearch.yml | grep cluster.name
cluster.name: nagios_elasticsearch

[root@naw02logserver01 config]# cat ../../logstash/etc/conf.d/999_outputs.conf | grep cluster
        cluster => '53d62043-9568-47de-af43-7e7635380d8a'
I could change the cluster label to match the elasticsearch config of 'nagios_elasticsearch' but I'm not sure if that might break things... Any ideas?
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: LogServer not listening on defined input ports?

Post by tgriep »

In the Logserver GUI, is the logstash service running?
Go to the Administration > System Status menu and verify that the Logstash Collector it running.
Try restarting it to see if it starts to listen on the ports.
Also, check to see if the inputs are active by going to the Global Configuration menu.

Go to the Cluster Status menu in the Administration menu, what is the status showing?

Run this command in a root shell on the server and post the output here.

Code: Select all

ps -ef --cols=300
Be sure to check out our Knowledgebase for helpful articles and solutions!
jalfaro
Posts: 22
Joined: Wed Apr 05, 2017 9:50 am

Re: LogServer not listening on defined input ports?

Post by jalfaro »

tgriep wrote:In the Logserver GUI, is the logstash service running?
Go to the Administration > System Status menu and verify that the Logstash Collector it running.
Try restarting it to see if it starts to listen on the ports.
Also, check to see if the inputs are active by going to the Global Configuration menu.

Go to the Cluster Status menu in the Administration menu, what is the status showing?

Run this command in a root shell on the server and post the output here.

Code: Select all

ps -ef --cols=300
The Administration > System Status menu shows both logstash and elasticsearch are running on both nodes.
The global config shows all configured inputs (including 3515 and 5544) as active.
Restarted the logstash app, still don't see the ports listed in netstat output.

Code: Select all

[root@naw02logserver01 talfaro]# ps -ef --cols=300
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jun15 ?        01:19:59 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root         2     0  0 Jun15 ?        00:00:01 [kthreadd]
root         3     2  0 Jun15 ?        00:00:31 [ksoftirqd/0]
root         7     2  0 Jun15 ?        00:00:22 [migration/0]
root         8     2  0 Jun15 ?        00:00:00 [rcu_bh]
root         9     2  0 Jun15 ?        00:08:46 [rcu_sched]
root        10     2  0 Jun15 ?        00:00:15 [watchdog/0]
root        11     2  0 Jun15 ?        00:00:16 [watchdog/1]
root        12     2  0 Jun15 ?        00:00:23 [migration/1]
root        13     2  0 Jun15 ?        00:00:29 [ksoftirqd/1]
root        17     2  0 Jun15 ?        00:00:00 [kdevtmpfs]
root        18     2  0 Jun15 ?        00:00:00 [netns]
root        19     2  0 Jun15 ?        00:00:03 [khungtaskd]
root        20     2  0 Jun15 ?        00:00:00 [writeback]
root        21     2  0 Jun15 ?        00:00:00 [kintegrityd]
root        22     2  0 Jun15 ?        00:00:00 [bioset]
root        23     2  0 Jun15 ?        00:00:00 [kblockd]
root        24     2  0 Jun15 ?        00:00:00 [md]
root        30     2  0 Jun15 ?        00:00:14 [kswapd0]
root        31     2  0 Jun15 ?        00:00:00 [ksmd]
root        32     2  0 Jun15 ?        00:00:24 [khugepaged]
root        33     2  0 Jun15 ?        00:00:00 [fsnotify_mark]
root        34     2  0 Jun15 ?        00:00:00 [crypto]
root        42     2  0 Jun15 ?        00:00:00 [kthrotld]
root        44     2  0 Jun15 ?        00:00:00 [kmpath_rdacd]
root        45     2  0 Jun15 ?        00:00:00 [kpsmoused]
root        47     2  0 Jun15 ?        00:00:00 [ipv6_addrconf]
root        66     2  0 Jun15 ?        00:00:00 [deferwq]
root       100     2  0 Jun15 ?        00:00:40 [kauditd]
root       280     2  0 Jun15 ?        00:00:00 [ata_sff]
root       281     2  0 Jun15 ?        00:00:00 [scsi_eh_0]
root       282     2  0 Jun15 ?        00:00:00 [scsi_tmf_0]
root       283     2  0 Jun15 ?        00:00:00 [scsi_eh_1]
root       284     2  0 Jun15 ?        00:00:00 [scsi_tmf_1]
root       288     2  0 Jun15 ?        00:00:00 [mpt_poll_0]
root       289     2  0 Jun15 ?        00:00:00 [mpt/0]
root       298     2  0 Jun15 ?        00:00:00 [scsi_eh_2]
root       299     2  0 Jun15 ?        00:00:00 [scsi_tmf_2]
root       300     2  0 Jun15 ?        00:00:00 [ttm_swap]
root       398     2  0 Jun15 ?        00:00:00 [kdmflush]
root       399     2  0 Jun15 ?        00:00:00 [bioset]
root       410     2  0 Jun15 ?        00:00:00 [kdmflush]
root       411     2  0 Jun15 ?        00:00:00 [bioset]
root       424     2  0 Jun15 ?        00:00:00 [xfsalloc]
root       425     2  0 Jun15 ?        00:00:00 [xfs_mru_cache]
root       426     2  0 Jun15 ?        00:00:00 [xfs-buf/dm-0]
root       427     2  0 Jun15 ?        00:00:00 [xfs-data/dm-0]
root       428     2  0 Jun15 ?        00:00:00 [xfs-conv/dm-0]
root       429     2  0 Jun15 ?        00:00:00 [xfs-cil/dm-0]
root       430     2  0 Jun15 ?        00:00:00 [xfs-reclaim/dm-]
root       431     2  0 Jun15 ?        00:00:00 [xfs-log/dm-0]
root       432     2  0 Jun15 ?        00:00:00 [xfs-eofblocks/d]
root       433     2  0 Jun15 ?        00:09:17 [xfsaild/dm-0]
root       501     1  0 Jun15 ?        00:08:55 /usr/lib/systemd/systemd-journald
root       524     1  0 Jun15 ?        00:00:00 /usr/sbin/lvmetad -f
root       527     1  0 Jun15 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root       544     2  0 Jun15 ?        00:00:00 [nfit]
root       621     2  0 Jun15 ?        00:00:00 [xfs-buf/sda1]
root       622     2  0 Jun15 ?        00:00:00 [xfs-data/sda1]
root       623     2  0 Jun15 ?        00:00:00 [xfs-conv/sda1]
root       624     2  0 Jun15 ?        00:00:00 [xfs-cil/sda1]
root       625     2  0 Jun15 ?        00:00:00 [xfs-reclaim/sda]
root       626     2  0 Jun15 ?        00:00:00 [xfs-log/sda1]
root       627     2  0 Jun15 ?        00:00:00 [xfs-eofblocks/s]
root       628     2  0 Jun15 ?        00:00:00 [xfsaild/sda1]
root       630     2  0 Jun15 ?        00:00:00 [kdmflush]
root       631     2  0 Jun15 ?        00:00:00 [bioset]
root       634     2  0 Jun15 ?        00:00:00 [kdmflush]
root       635     2  0 Jun15 ?        00:00:00 [bioset]
root       637     2  0 Jun15 ?        00:00:00 [kdmflush]
root       639     2  0 Jun15 ?        00:00:00 [bioset]
root       641     2  0 Jun15 ?        00:00:00 [kdmflush]
root       643     2  0 Jun15 ?        00:00:00 [bioset]
root       645     2  0 Jun15 ?        00:00:00 [kdmflush]
root       646     2  0 Jun15 ?        00:00:00 [bioset]
root       652     2  0 Jun15 ?        00:00:00 [xfs-buf/dm-4]
root       653     2  0 Jun15 ?        00:00:00 [xfs-data/dm-4]
root       654     2  0 Jun15 ?        00:00:00 [xfs-conv/dm-4]
root       655     2  0 Jun15 ?        00:00:00 [xfs-cil/dm-4]
root       656     2  0 Jun15 ?        00:00:00 [xfs-reclaim/dm-]
root       657     2  0 Jun15 ?        00:00:00 [xfs-log/dm-4]
root       658     2  0 Jun15 ?        00:00:00 [xfs-eofblocks/d]
root       659     2  0 Jun15 ?        00:09:35 [xfsaild/dm-4]
root       663     2  0 Jun15 ?        00:00:00 [xfs-buf/dm-5]
root       664     2  0 Jun15 ?        00:00:00 [xfs-data/dm-5]
root       665     2  0 Jun15 ?        00:00:00 [xfs-conv/dm-5]
root       666     2  0 Jun15 ?        00:00:00 [xfs-cil/dm-5]
root       667     2  0 Jun15 ?        00:00:00 [xfs-reclaim/dm-]
root       668     2  0 Jun15 ?        00:00:00 [xfs-log/dm-5]
root       669     2  0 Jun15 ?        00:00:00 [xfs-eofblocks/d]
root       671     2  0 Jun15 ?        00:09:29 [xfsaild/dm-5]
root       672     2  0 Jun15 ?        00:00:00 [xfs-buf/dm-3]
root       673     2  0 Jun15 ?        00:00:00 [xfs-data/dm-3]
root       674     2  0 Jun15 ?        00:00:00 [xfs-conv/dm-3]
root       675     2  0 Jun15 ?        00:00:00 [xfs-cil/dm-3]
root       676     2  0 Jun15 ?        00:00:00 [xfs-reclaim/dm-]
root       677     2  0 Jun15 ?        00:00:00 [xfs-log/dm-3]
root       678     2  0 Jun15 ?        00:00:00 [xfs-eofblocks/d]
root       679     2  0 Jun15 ?        00:00:00 [xfsaild/dm-3]
root       683     2  0 Jun15 ?        00:00:00 [xfs-buf/dm-6]
root       684     2  0 Jun15 ?        00:00:00 [xfs-data/dm-6]
root       686     2  0 Jun15 ?        00:00:00 [xfs-conv/dm-6]
root       687     2  0 Jun15 ?        00:00:00 [xfs-cil/dm-6]
root       688     2  0 Jun15 ?        00:00:00 [xfs-reclaim/dm-]
root       689     2  0 Jun15 ?        00:00:00 [xfs-log/dm-6]
root       690     2  0 Jun15 ?        00:00:00 [xfs-eofblocks/d]
root       691     2  0 Jun15 ?        00:00:00 [xfsaild/dm-6]
root       694     2  0 Jun15 ?        00:00:00 [xfs-buf/dm-2]
root       695     2  0 Jun15 ?        00:00:00 [xfs-data/dm-2]
root       696     2  0 Jun15 ?        00:00:00 [xfs-conv/dm-2]
root       697     2  0 Jun15 ?        00:00:00 [xfs-cil/dm-2]
root       698     2  0 Jun15 ?        00:00:00 [xfs-reclaim/dm-]
root       699     2  0 Jun15 ?        00:00:00 [xfs-log/dm-2]
root       700     2  0 Jun15 ?        00:00:00 [xfs-eofblocks/d]
root       701     2  0 Jun15 ?        00:02:03 [xfsaild/dm-2]
root       726     1  0 Jun15 ?        00:02:48 /sbin/auditd -n
root       736     1  0 Jun15 ?        00:05:12 /usr/lib/systemd/systemd-logind
polkitd    739     1  0 Jun15 ?        00:02:38 /usr/lib/polkit-1/polkitd --no-debug
root       742     1  0 Jun15 ?        00:02:32 /usr/sbin/irqbalance --foreground
root       743     1  0 Jun15 ?        00:44:51 /usr/bin/vmtoolsd
dbus       745     1  0 Jun15 ?        00:11:40 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       757     1  0 Jun15 ?        00:00:54 /usr/sbin/crond -n
root       763     1  0 Jun15 ?        00:02:35 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
root       764     1  0 Jun15 tty1     00:00:00 /sbin/agetty --noclear tty1 linux
root       769     1  0 Jun15 ?        00:03:23 /usr/sbin/NetworkManager --no-daemon
root      1034     1  0 Jun15 ?        00:00:00 /usr/sbin/sshd -D
root      1035     1  0 Jun15 ?        00:07:54 /usr/bin/python -Es /usr/sbin/tuned -l -P
root      1217     1  0 Jun15 ?        00:00:00 /opt/pbis/sbin/lwsmd --start-as-daemon
root      1232  1217  0 Jun15 ?        00:00:00 lw-container lwreg
root      1328  1217  0 Jun15 ?        00:00:03 lw-container eventlog
root      1398  1217  0 Jun15 ?        00:00:00 lw-container netlogon
root      1450  1217  0 Jun15 ?        00:00:00 lw-container lwio
root      1608  1217  0 Jun15 ?        00:00:03 lw-container lsass
root      1840  1217  0 Jun15 ?        00:00:00 lw-container reapsysl
root      4533     2  0 20:06 ?        00:00:00 [kworker/1:2]
root      5729     2  0 20:13 ?        00:00:00 [kworker/u4:0]
root      5804     2  0 20:14 ?        00:00:00 [kworker/0:1H]
root      8492     2  0 20:30 ?        00:00:00 [kworker/0:4H]
nagios    8775 17425  0 Jul20 ?        00:00:24 /usr/sbin/httpd -DFOREGROUND
nagios    9803 17425  0 Jul19 ?        00:00:48 /usr/sbin/httpd -DFOREGROUND
nagios    9869 17425  0 Jul19 ?        00:00:47 /usr/sbin/httpd -DFOREGROUND
root     10220     2  0 20:41 ?        00:00:00 [kworker/0:0H]
root     10251     2  0 20:41 ?        00:00:00 [kworker/0:0]
root     10677     2  0 20:44 ?        00:00:00 [kworker/1:0]
root     11120     2  0 20:46 ?        00:00:00 [kworker/0:2H]
root     11221     2  0 20:47 ?        00:00:00 [kworker/0:1]
root     11592     2  0 20:49 ?        00:00:00 [kworker/1:1]
root     11626     1  0 Jun15 ?        00:01:58 sendmail: accepting connections
smmsp    11666     1  0 Jun15 ?        00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
nagios   11862     1 48 20:49 ?        00:01:02 java -Xms3911m -Xmx3911m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluste
root     12101     2  0 20:50 ?        00:00:00 [kworker/u4:1]
root     12222  1034  0 20:50 ?        00:00:00 sshd: talfaro [priv]
talfaro  12226 12222  0 20:50 ?        00:00:00 sshd: talfaro@pts/0
talfaro  12227 12226  0 20:50 pts/0    00:00:00 -bash
root     12290 12227  0 20:50 pts/0    00:00:00 su
root     12324 12290  0 20:50 pts/0    00:00:00 bash
root     12372   757  0 20:51 ?        00:00:00 /usr/sbin/CROND -n
root     12373   757  0 20:51 ?        00:00:00 /usr/sbin/CROND -n
nagios   12376 12373  0 20:51 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   12377 12372  0 20:51 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   12378 12376  0 20:51 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   12379 12377  0 20:51 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     12504 12324  0 20:51 pts/0    00:00:00 ps -ef --cols=300
nagios   13618 17425  0 Jul19 ?        00:00:49 /usr/sbin/httpd -DFOREGROUND
nagios   15214 17425  0 Jul18 ?        00:01:13 /usr/sbin/httpd -DFOREGROUND
root     17425     1  0 Jun16 ?        00:03:42 /usr/sbin/httpd -DFOREGROUND
root     17497     1  0 Jun16 ?        00:06:18 /usr/sbin/rsyslogd -n
nagios   18820 17425  0 18:14 ?        00:00:02 /usr/sbin/httpd -DFOREGROUND
nagios   20557 17425  0 Jun28 ?        00:05:52 /usr/sbin/httpd -DFOREGROUND
nagios   20932 17425  0 Jul12 ?        00:02:53 /usr/sbin/httpd -DFOREGROUND
root     21043     1  0 Jun22 ?        00:00:01 rhnsd
root     23789     2  0 18:44 ?        00:00:00 [kworker/0:3H]
root     26936     2  0 19:03 ?        00:00:00 [kworker/u4:2]
nagios   27122 17425  0 Jul12 ?        00:02:53 /usr/sbin/httpd -DFOREGROUND
root     27302     2  0 Jul19 ?        00:00:00 [kworker/1:1H]
nagios   29618 17425  0 19:23 ?        00:00:01 /usr/sbin/httpd -DFOREGROUND
root     29719     2  0 19:24 ?        00:00:00 [kworker/0:2]
root     30153     2  0 Jul19 ?        00:00:03 [kworker/1:3H]
[root@naw02logserver01 talfaro]#
jalfaro
Posts: 22
Joined: Wed Apr 05, 2017 9:50 am

Re: LogServer not listening on defined input ports?

Post by jalfaro »

I also noticed that the ports do appear to be listening on the second node...

Code: Select all

[root@naw02logserver02 talfaro]# netstat -antpu | egrep '(5544)|(3515)'
tcp        0      0 0.0.0.0:5544            0.0.0.0:*               LISTEN      12723/java
tcp        0      0 0.0.0.0:3515            0.0.0.0:*               LISTEN      12723/java
tcp        0      0 127.0.0.1:5544          127.0.0.1:36190         ESTABLISHED 12723/java
tcp        0      0 127.0.0.1:36190         127.0.0.1:5544          ESTABLISHED 1050/rsyslogd
udp        0      0 0.0.0.0:5544            0.0.0.0:*                           12723/java
[root@naw02logserver02 talfaro]#
jalfaro
Posts: 22
Joined: Wed Apr 05, 2017 9:50 am

Re: LogServer not listening on defined input ports?

Post by jalfaro »

A few restarts of logstash on node 1 later...

Code: Select all

[root@naw02logserver01 talfaro]# netstat -antpu | egrep '(5544)|(3515)'
tcp        0      0 0.0.0.0:5544            0.0.0.0:*               LISTEN      22595/java
tcp        0      0 0.0.0.0:3515            0.0.0.0:*               LISTEN      22595/java
tcp        0      0 10.70.50.3:3515         10.70.39.67:51520       ESTABLISHED 22595/java
tcp        0 1800080 127.0.0.1:47770         127.0.0.1:5544          ESTABLISHED 17497/rsyslogd
tcp   189967      0 10.70.50.3:3515         10.70.54.67:59621       ESTABLISHED 22595/java
tcp   373479      0 10.70.50.3:3515         10.70.51.130:49731      ESTABLISHED 22595/java
tcp   297682      0 10.70.50.3:3515         10.70.56.3:51630        ESTABLISHED 22595/java
tcp   937135      0 127.0.0.1:5544          127.0.0.1:47770         ESTABLISHED 22595/java
tcp   248775      0 10.70.50.3:3515         10.70.55.227:50678      ESTABLISHED 22595/java
udp        0      0 0.0.0.0:5544            0.0.0.0:*                           22595/java
[root@naw02logserver01 talfaro]#
So at least the listeners are binding to the ports again, now I'd be curious to know if there's a way to determine why they died to begin with, and if there's a way to prevent it from happening in the future.
User avatar
tacolover101
Posts: 432
Joined: Mon Apr 10, 2017 11:55 am

Re: LogServer not listening on defined input ports?

Post by tacolover101 »

can you post the logstash.log in it's entirety?
jalfaro
Posts: 22
Joined: Wed Apr 05, 2017 9:50 am

Re: LogServer not listening on defined input ports?

Post by jalfaro »

tacolover101 wrote:can you post the logstash.log in it's entirety?
The logstash process has remained stable since the restarts - thanks for the assistance. We can call this issue resolved.
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: LogServer not listening on defined input ports?

Post by tgriep »

OK, thanks for reporting back. I'll lock the post but if you have any questions in the future, feel free to open a new post.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Locked