Page 1 of 1
LogServer not listening on defined input ports?
Posted: Fri Jul 21, 2017 2:42 pm
by jalfaro
I've got a 2 node log server cluster setup and everything on the surface appears to be working correctly. I would however expect the input ports to show up on a netstat and I'm not seeing any of the input ports (5544, or 3515) in the netstat output - what is the best way to go about ensuring that logstash is in fact listening on these ports?
netstat output:
Code: Select all
[root@naw02logserver01 config]# netstat -antp | egrep '(3515)|(5544)'
[root@naw02logserver01 config]#
The logstash.log file contains only error messages:
Code: Select all
[root@naw02logserver01 config]# tail /var/log/logstash/logstash.log
{:timestamp=>"2017-07-02T00:08:41.199000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-07-02T00:08:41.199000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-07-02T00:08:41.199000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2017-07-02T00:08:41.199000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
I found this when googling the error on the elasticsearch site:
I found the problem, here is the solution:
1) on ES config, need to define the cluster name (default is "elasticsearch" )
2) on LS config file, add cluster definition as the above the example:
output {
elasticsearch
{ host => "zslesvm09:9300"
cluster => } # cluster_name defined @ ES
stdout { }
}
"cluster" is introduced by the latest LS 1.5.2.
Before we use the default definition that makes the LS cannot find the correct ES.
Thanks.
I found these cluster names in the logstash and elasticsearch configs:
Code: Select all
[root@naw02logserver01 config]# cat elasticsearch.yml | grep cluster.name
cluster.name: nagios_elasticsearch
[root@naw02logserver01 config]# cat ../../logstash/etc/conf.d/999_outputs.conf | grep cluster
cluster => '53d62043-9568-47de-af43-7e7635380d8a'
I could change the cluster label to match the elasticsearch config of 'nagios_elasticsearch' but I'm not sure if that might break things... Any ideas?
Re: LogServer not listening on defined input ports?
Posted: Fri Jul 21, 2017 3:44 pm
by tgriep
In the Logserver GUI, is the logstash service running?
Go to the Administration > System Status menu and verify that the Logstash Collector it running.
Try restarting it to see if it starts to listen on the ports.
Also, check to see if the inputs are active by going to the Global Configuration menu.
Go to the Cluster Status menu in the Administration menu, what is the status showing?
Run this command in a root shell on the server and post the output here.
Re: LogServer not listening on defined input ports?
Posted: Fri Jul 21, 2017 3:52 pm
by jalfaro
tgriep wrote:In the Logserver GUI, is the logstash service running?
Go to the Administration > System Status menu and verify that the Logstash Collector it running.
Try restarting it to see if it starts to listen on the ports.
Also, check to see if the inputs are active by going to the Global Configuration menu.
Go to the Cluster Status menu in the Administration menu, what is the status showing?
Run this command in a root shell on the server and post the output here.
The Administration > System Status menu shows both logstash and elasticsearch are running on both nodes.
The global config shows all configured inputs (including 3515 and 5544) as active.
Restarted the logstash app, still don't see the ports listed in netstat output.
Code: Select all
[root@naw02logserver01 talfaro]# ps -ef --cols=300
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Jun15 ? 01:19:59 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root 2 0 0 Jun15 ? 00:00:01 [kthreadd]
root 3 2 0 Jun15 ? 00:00:31 [ksoftirqd/0]
root 7 2 0 Jun15 ? 00:00:22 [migration/0]
root 8 2 0 Jun15 ? 00:00:00 [rcu_bh]
root 9 2 0 Jun15 ? 00:08:46 [rcu_sched]
root 10 2 0 Jun15 ? 00:00:15 [watchdog/0]
root 11 2 0 Jun15 ? 00:00:16 [watchdog/1]
root 12 2 0 Jun15 ? 00:00:23 [migration/1]
root 13 2 0 Jun15 ? 00:00:29 [ksoftirqd/1]
root 17 2 0 Jun15 ? 00:00:00 [kdevtmpfs]
root 18 2 0 Jun15 ? 00:00:00 [netns]
root 19 2 0 Jun15 ? 00:00:03 [khungtaskd]
root 20 2 0 Jun15 ? 00:00:00 [writeback]
root 21 2 0 Jun15 ? 00:00:00 [kintegrityd]
root 22 2 0 Jun15 ? 00:00:00 [bioset]
root 23 2 0 Jun15 ? 00:00:00 [kblockd]
root 24 2 0 Jun15 ? 00:00:00 [md]
root 30 2 0 Jun15 ? 00:00:14 [kswapd0]
root 31 2 0 Jun15 ? 00:00:00 [ksmd]
root 32 2 0 Jun15 ? 00:00:24 [khugepaged]
root 33 2 0 Jun15 ? 00:00:00 [fsnotify_mark]
root 34 2 0 Jun15 ? 00:00:00 [crypto]
root 42 2 0 Jun15 ? 00:00:00 [kthrotld]
root 44 2 0 Jun15 ? 00:00:00 [kmpath_rdacd]
root 45 2 0 Jun15 ? 00:00:00 [kpsmoused]
root 47 2 0 Jun15 ? 00:00:00 [ipv6_addrconf]
root 66 2 0 Jun15 ? 00:00:00 [deferwq]
root 100 2 0 Jun15 ? 00:00:40 [kauditd]
root 280 2 0 Jun15 ? 00:00:00 [ata_sff]
root 281 2 0 Jun15 ? 00:00:00 [scsi_eh_0]
root 282 2 0 Jun15 ? 00:00:00 [scsi_tmf_0]
root 283 2 0 Jun15 ? 00:00:00 [scsi_eh_1]
root 284 2 0 Jun15 ? 00:00:00 [scsi_tmf_1]
root 288 2 0 Jun15 ? 00:00:00 [mpt_poll_0]
root 289 2 0 Jun15 ? 00:00:00 [mpt/0]
root 298 2 0 Jun15 ? 00:00:00 [scsi_eh_2]
root 299 2 0 Jun15 ? 00:00:00 [scsi_tmf_2]
root 300 2 0 Jun15 ? 00:00:00 [ttm_swap]
root 398 2 0 Jun15 ? 00:00:00 [kdmflush]
root 399 2 0 Jun15 ? 00:00:00 [bioset]
root 410 2 0 Jun15 ? 00:00:00 [kdmflush]
root 411 2 0 Jun15 ? 00:00:00 [bioset]
root 424 2 0 Jun15 ? 00:00:00 [xfsalloc]
root 425 2 0 Jun15 ? 00:00:00 [xfs_mru_cache]
root 426 2 0 Jun15 ? 00:00:00 [xfs-buf/dm-0]
root 427 2 0 Jun15 ? 00:00:00 [xfs-data/dm-0]
root 428 2 0 Jun15 ? 00:00:00 [xfs-conv/dm-0]
root 429 2 0 Jun15 ? 00:00:00 [xfs-cil/dm-0]
root 430 2 0 Jun15 ? 00:00:00 [xfs-reclaim/dm-]
root 431 2 0 Jun15 ? 00:00:00 [xfs-log/dm-0]
root 432 2 0 Jun15 ? 00:00:00 [xfs-eofblocks/d]
root 433 2 0 Jun15 ? 00:09:17 [xfsaild/dm-0]
root 501 1 0 Jun15 ? 00:08:55 /usr/lib/systemd/systemd-journald
root 524 1 0 Jun15 ? 00:00:00 /usr/sbin/lvmetad -f
root 527 1 0 Jun15 ? 00:00:00 /usr/lib/systemd/systemd-udevd
root 544 2 0 Jun15 ? 00:00:00 [nfit]
root 621 2 0 Jun15 ? 00:00:00 [xfs-buf/sda1]
root 622 2 0 Jun15 ? 00:00:00 [xfs-data/sda1]
root 623 2 0 Jun15 ? 00:00:00 [xfs-conv/sda1]
root 624 2 0 Jun15 ? 00:00:00 [xfs-cil/sda1]
root 625 2 0 Jun15 ? 00:00:00 [xfs-reclaim/sda]
root 626 2 0 Jun15 ? 00:00:00 [xfs-log/sda1]
root 627 2 0 Jun15 ? 00:00:00 [xfs-eofblocks/s]
root 628 2 0 Jun15 ? 00:00:00 [xfsaild/sda1]
root 630 2 0 Jun15 ? 00:00:00 [kdmflush]
root 631 2 0 Jun15 ? 00:00:00 [bioset]
root 634 2 0 Jun15 ? 00:00:00 [kdmflush]
root 635 2 0 Jun15 ? 00:00:00 [bioset]
root 637 2 0 Jun15 ? 00:00:00 [kdmflush]
root 639 2 0 Jun15 ? 00:00:00 [bioset]
root 641 2 0 Jun15 ? 00:00:00 [kdmflush]
root 643 2 0 Jun15 ? 00:00:00 [bioset]
root 645 2 0 Jun15 ? 00:00:00 [kdmflush]
root 646 2 0 Jun15 ? 00:00:00 [bioset]
root 652 2 0 Jun15 ? 00:00:00 [xfs-buf/dm-4]
root 653 2 0 Jun15 ? 00:00:00 [xfs-data/dm-4]
root 654 2 0 Jun15 ? 00:00:00 [xfs-conv/dm-4]
root 655 2 0 Jun15 ? 00:00:00 [xfs-cil/dm-4]
root 656 2 0 Jun15 ? 00:00:00 [xfs-reclaim/dm-]
root 657 2 0 Jun15 ? 00:00:00 [xfs-log/dm-4]
root 658 2 0 Jun15 ? 00:00:00 [xfs-eofblocks/d]
root 659 2 0 Jun15 ? 00:09:35 [xfsaild/dm-4]
root 663 2 0 Jun15 ? 00:00:00 [xfs-buf/dm-5]
root 664 2 0 Jun15 ? 00:00:00 [xfs-data/dm-5]
root 665 2 0 Jun15 ? 00:00:00 [xfs-conv/dm-5]
root 666 2 0 Jun15 ? 00:00:00 [xfs-cil/dm-5]
root 667 2 0 Jun15 ? 00:00:00 [xfs-reclaim/dm-]
root 668 2 0 Jun15 ? 00:00:00 [xfs-log/dm-5]
root 669 2 0 Jun15 ? 00:00:00 [xfs-eofblocks/d]
root 671 2 0 Jun15 ? 00:09:29 [xfsaild/dm-5]
root 672 2 0 Jun15 ? 00:00:00 [xfs-buf/dm-3]
root 673 2 0 Jun15 ? 00:00:00 [xfs-data/dm-3]
root 674 2 0 Jun15 ? 00:00:00 [xfs-conv/dm-3]
root 675 2 0 Jun15 ? 00:00:00 [xfs-cil/dm-3]
root 676 2 0 Jun15 ? 00:00:00 [xfs-reclaim/dm-]
root 677 2 0 Jun15 ? 00:00:00 [xfs-log/dm-3]
root 678 2 0 Jun15 ? 00:00:00 [xfs-eofblocks/d]
root 679 2 0 Jun15 ? 00:00:00 [xfsaild/dm-3]
root 683 2 0 Jun15 ? 00:00:00 [xfs-buf/dm-6]
root 684 2 0 Jun15 ? 00:00:00 [xfs-data/dm-6]
root 686 2 0 Jun15 ? 00:00:00 [xfs-conv/dm-6]
root 687 2 0 Jun15 ? 00:00:00 [xfs-cil/dm-6]
root 688 2 0 Jun15 ? 00:00:00 [xfs-reclaim/dm-]
root 689 2 0 Jun15 ? 00:00:00 [xfs-log/dm-6]
root 690 2 0 Jun15 ? 00:00:00 [xfs-eofblocks/d]
root 691 2 0 Jun15 ? 00:00:00 [xfsaild/dm-6]
root 694 2 0 Jun15 ? 00:00:00 [xfs-buf/dm-2]
root 695 2 0 Jun15 ? 00:00:00 [xfs-data/dm-2]
root 696 2 0 Jun15 ? 00:00:00 [xfs-conv/dm-2]
root 697 2 0 Jun15 ? 00:00:00 [xfs-cil/dm-2]
root 698 2 0 Jun15 ? 00:00:00 [xfs-reclaim/dm-]
root 699 2 0 Jun15 ? 00:00:00 [xfs-log/dm-2]
root 700 2 0 Jun15 ? 00:00:00 [xfs-eofblocks/d]
root 701 2 0 Jun15 ? 00:02:03 [xfsaild/dm-2]
root 726 1 0 Jun15 ? 00:02:48 /sbin/auditd -n
root 736 1 0 Jun15 ? 00:05:12 /usr/lib/systemd/systemd-logind
polkitd 739 1 0 Jun15 ? 00:02:38 /usr/lib/polkit-1/polkitd --no-debug
root 742 1 0 Jun15 ? 00:02:32 /usr/sbin/irqbalance --foreground
root 743 1 0 Jun15 ? 00:44:51 /usr/bin/vmtoolsd
dbus 745 1 0 Jun15 ? 00:11:40 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 757 1 0 Jun15 ? 00:00:54 /usr/sbin/crond -n
root 763 1 0 Jun15 ? 00:02:35 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
root 764 1 0 Jun15 tty1 00:00:00 /sbin/agetty --noclear tty1 linux
root 769 1 0 Jun15 ? 00:03:23 /usr/sbin/NetworkManager --no-daemon
root 1034 1 0 Jun15 ? 00:00:00 /usr/sbin/sshd -D
root 1035 1 0 Jun15 ? 00:07:54 /usr/bin/python -Es /usr/sbin/tuned -l -P
root 1217 1 0 Jun15 ? 00:00:00 /opt/pbis/sbin/lwsmd --start-as-daemon
root 1232 1217 0 Jun15 ? 00:00:00 lw-container lwreg
root 1328 1217 0 Jun15 ? 00:00:03 lw-container eventlog
root 1398 1217 0 Jun15 ? 00:00:00 lw-container netlogon
root 1450 1217 0 Jun15 ? 00:00:00 lw-container lwio
root 1608 1217 0 Jun15 ? 00:00:03 lw-container lsass
root 1840 1217 0 Jun15 ? 00:00:00 lw-container reapsysl
root 4533 2 0 20:06 ? 00:00:00 [kworker/1:2]
root 5729 2 0 20:13 ? 00:00:00 [kworker/u4:0]
root 5804 2 0 20:14 ? 00:00:00 [kworker/0:1H]
root 8492 2 0 20:30 ? 00:00:00 [kworker/0:4H]
nagios 8775 17425 0 Jul20 ? 00:00:24 /usr/sbin/httpd -DFOREGROUND
nagios 9803 17425 0 Jul19 ? 00:00:48 /usr/sbin/httpd -DFOREGROUND
nagios 9869 17425 0 Jul19 ? 00:00:47 /usr/sbin/httpd -DFOREGROUND
root 10220 2 0 20:41 ? 00:00:00 [kworker/0:0H]
root 10251 2 0 20:41 ? 00:00:00 [kworker/0:0]
root 10677 2 0 20:44 ? 00:00:00 [kworker/1:0]
root 11120 2 0 20:46 ? 00:00:00 [kworker/0:2H]
root 11221 2 0 20:47 ? 00:00:00 [kworker/0:1]
root 11592 2 0 20:49 ? 00:00:00 [kworker/1:1]
root 11626 1 0 Jun15 ? 00:01:58 sendmail: accepting connections
smmsp 11666 1 0 Jun15 ? 00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
nagios 11862 1 48 20:49 ? 00:01:02 java -Xms3911m -Xmx3911m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluste
root 12101 2 0 20:50 ? 00:00:00 [kworker/u4:1]
root 12222 1034 0 20:50 ? 00:00:00 sshd: talfaro [priv]
talfaro 12226 12222 0 20:50 ? 00:00:00 sshd: talfaro@pts/0
talfaro 12227 12226 0 20:50 pts/0 00:00:00 -bash
root 12290 12227 0 20:50 pts/0 00:00:00 su
root 12324 12290 0 20:50 pts/0 00:00:00 bash
root 12372 757 0 20:51 ? 00:00:00 /usr/sbin/CROND -n
root 12373 757 0 20:51 ? 00:00:00 /usr/sbin/CROND -n
nagios 12376 12373 0 20:51 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 12377 12372 0 20:51 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 12378 12376 0 20:51 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 12379 12377 0 20:51 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 12504 12324 0 20:51 pts/0 00:00:00 ps -ef --cols=300
nagios 13618 17425 0 Jul19 ? 00:00:49 /usr/sbin/httpd -DFOREGROUND
nagios 15214 17425 0 Jul18 ? 00:01:13 /usr/sbin/httpd -DFOREGROUND
root 17425 1 0 Jun16 ? 00:03:42 /usr/sbin/httpd -DFOREGROUND
root 17497 1 0 Jun16 ? 00:06:18 /usr/sbin/rsyslogd -n
nagios 18820 17425 0 18:14 ? 00:00:02 /usr/sbin/httpd -DFOREGROUND
nagios 20557 17425 0 Jun28 ? 00:05:52 /usr/sbin/httpd -DFOREGROUND
nagios 20932 17425 0 Jul12 ? 00:02:53 /usr/sbin/httpd -DFOREGROUND
root 21043 1 0 Jun22 ? 00:00:01 rhnsd
root 23789 2 0 18:44 ? 00:00:00 [kworker/0:3H]
root 26936 2 0 19:03 ? 00:00:00 [kworker/u4:2]
nagios 27122 17425 0 Jul12 ? 00:02:53 /usr/sbin/httpd -DFOREGROUND
root 27302 2 0 Jul19 ? 00:00:00 [kworker/1:1H]
nagios 29618 17425 0 19:23 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND
root 29719 2 0 19:24 ? 00:00:00 [kworker/0:2]
root 30153 2 0 Jul19 ? 00:00:03 [kworker/1:3H]
[root@naw02logserver01 talfaro]#
Re: LogServer not listening on defined input ports?
Posted: Fri Jul 21, 2017 4:00 pm
by jalfaro
I also noticed that the ports do appear to be listening on the second node...
Code: Select all
[root@naw02logserver02 talfaro]# netstat -antpu | egrep '(5544)|(3515)'
tcp 0 0 0.0.0.0:5544 0.0.0.0:* LISTEN 12723/java
tcp 0 0 0.0.0.0:3515 0.0.0.0:* LISTEN 12723/java
tcp 0 0 127.0.0.1:5544 127.0.0.1:36190 ESTABLISHED 12723/java
tcp 0 0 127.0.0.1:36190 127.0.0.1:5544 ESTABLISHED 1050/rsyslogd
udp 0 0 0.0.0.0:5544 0.0.0.0:* 12723/java
[root@naw02logserver02 talfaro]#
Re: LogServer not listening on defined input ports?
Posted: Fri Jul 21, 2017 4:39 pm
by jalfaro
A few restarts of logstash on node 1 later...
Code: Select all
[root@naw02logserver01 talfaro]# netstat -antpu | egrep '(5544)|(3515)'
tcp 0 0 0.0.0.0:5544 0.0.0.0:* LISTEN 22595/java
tcp 0 0 0.0.0.0:3515 0.0.0.0:* LISTEN 22595/java
tcp 0 0 10.70.50.3:3515 10.70.39.67:51520 ESTABLISHED 22595/java
tcp 0 1800080 127.0.0.1:47770 127.0.0.1:5544 ESTABLISHED 17497/rsyslogd
tcp 189967 0 10.70.50.3:3515 10.70.54.67:59621 ESTABLISHED 22595/java
tcp 373479 0 10.70.50.3:3515 10.70.51.130:49731 ESTABLISHED 22595/java
tcp 297682 0 10.70.50.3:3515 10.70.56.3:51630 ESTABLISHED 22595/java
tcp 937135 0 127.0.0.1:5544 127.0.0.1:47770 ESTABLISHED 22595/java
tcp 248775 0 10.70.50.3:3515 10.70.55.227:50678 ESTABLISHED 22595/java
udp 0 0 0.0.0.0:5544 0.0.0.0:* 22595/java
[root@naw02logserver01 talfaro]#
So at least the listeners are binding to the ports again, now I'd be curious to know if there's a way to determine why they died to begin with, and if there's a way to prevent it from happening in the future.
Re: LogServer not listening on defined input ports?
Posted: Sun Jul 23, 2017 11:09 pm
by tacolover101
can you post the logstash.log in it's entirety?
Re: LogServer not listening on defined input ports?
Posted: Mon Jul 24, 2017 9:39 am
by jalfaro
tacolover101 wrote:can you post the logstash.log in it's entirety?
The logstash process has remained stable since the restarts - thanks for the assistance. We can call this issue resolved.
Re: LogServer not listening on defined input ports?
Posted: Mon Jul 24, 2017 11:26 am
by tgriep
OK, thanks for reporting back. I'll lock the post but if you have any questions in the future, feel free to open a new post.