Trying to send logs from a syslog-ng server

[Box293]

I'm trying to configure syslog-ng to send logs to Nagios Log Server however the Dashboard does not show any logs coming in. TCP Dumps show traffic.

My syslog-ng server has the ip address 10.25.2.1 and has the name dns1.box293.local.

My Log Server has the ip address 10.25.5.80 and has the name lsproduction.box293.local.

The query I am using on the Dashboard is "10.25.2.1"

I don't know where to look on Nagios Log server to see errors.

syslog-ng is installed on ArchLinux.

I don't know if the syslog-ng.conf settings are correct.

Here is the /etc/syslog-ng/syslog-ng.conf file:

Code: Select all

@version: 3.5
@include "scl.conf"
#
# /etc/syslog-ng/syslog-ng.conf
#

options {
  stats_freq (0);
  flush_lines (0);
  time_reopen (10);
  log_fifo_size (10000);
  chain_hostnames (off);
  use_dns (no);
  use_fqdn (no);
  create_dirs (no);
  keep_hostname (yes);
  perm(0640);
  group("log");
};

source src {
  system();
  internal();
};

destination d_authlog { file("/var/log/auth.log"); };
destination d_syslog { file("/var/log/syslog.log"); };
destination d_cron { file("/var/log/crond.log"); };
destination d_daemon { file("/var/log/daemon.log"); };
destination d_kernel { file("/var/log/kernel.log"); };
destination d_lpr { file("/var/log/lpr.log"); };
destination d_user { file("/var/log/user.log"); };
destination d_uucp { file("/var/log/uucp.log"); };
destination d_mail { file("/var/log/mail.log"); };
destination d_news { file("/var/log/news.log"); };
destination d_ppp { file("/var/log/ppp.log"); };
destination d_debug { file("/var/log/debug.log"); };
destination d_messages { file("/var/log/messages.log"); };
destination d_errors { file("/var/log/errors.log"); };
destination d_everything { file("/var/log/everything.log"); };
destination d_iptables { file("/var/log/iptables.log"); };
destination d_acpid { file("/var/log/acpid.log"); };
destination d_console { usertty("root"); };

# Log everything to tty12
destination console_all { file("/dev/tty12"); };

filter f_auth { facility(auth); };
filter f_authpriv { facility(auth, authpriv); };
filter f_syslog { program(syslog-ng); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kernel { facility(kern) and not filter(f_iptables); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };
filter f_ppp { facility(local2); };
filter f_debug { not facility(auth, authpriv, news, mail); };
filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news, cron) and not program(syslog-ng) and not filter(f_iptables); };
filter f_everything { level(debug..emerg) and not facility(auth, authpriv); };
filter f_emergency { level(emerg); };
filter f_info { level(info); };
filter f_notice { level(notice); };
filter f_warn { level(warn); };
filter f_crit { level(crit); };
filter f_err { level(err); };
filter f_iptables { match("IN=" value("MESSAGE")) and match("OUT=" value("MESSAGE")); };
filter f_acpid { program("acpid"); };

log { source(src); filter(f_acpid); destination(d_acpid); };
log { source(src); filter(f_authpriv); destination(d_authlog); };
log { source(src); filter(f_syslog); destination(d_syslog); };
log { source(src); filter(f_cron); destination(d_cron); };
log { source(src); filter(f_daemon); destination(d_daemon); };
log { source(src); filter(f_kernel); destination(d_kernel); };
log { source(src); filter(f_lpr); destination(d_lpr); };
log { source(src); filter(f_mail); destination(d_mail); };
log { source(src); filter(f_news); destination(d_news); };
log { source(src); filter(f_ppp); destination(d_ppp); };
log { source(src); filter(f_user); destination(d_user); };
log { source(src); filter(f_uucp); destination(d_uucp); };
#log { source(src); filter(f_debug); destination(d_debug); };
log { source(src); filter(f_messages); destination(d_messages); };
log { source(src); filter(f_err); destination(d_errors); };
log { source(src); filter(f_emergency); destination(d_console); };
log { source(src); filter(f_everything); destination(d_everything); };
log { source(src); filter(f_iptables); destination(d_iptables); };

# Log everything to tty12
#log { source(src); destination(console_all); };

# Sending logs to lsproduction.box293.local
destination nagios_log_server { tcp("10.25.5.80" port(5544)); };
log { source(src); destination(nagios_log_server); };

Here is the tcp dump running on the syslog-ng server

Code: Select all

tcpdump src host 10.25.2.1 and tcp dst port 5544 and dst host 10.25.5.80

14:10:29.788432 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 441:597, ack 1, win 913, options [nop,nop,TS val 686264687 ecr 2178954247], length 156
14:10:29.807625 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 597:705, ack 1, win 913, options [nop,nop,TS val 686264689 ecr 2178967560], length 108
14:10:29.842713 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 705:792, ack 1, win 913, options [nop,nop,TS val 686264693 ecr 2178967579], length 87
14:10:29.856473 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 792:875, ack 1, win 913, options [nop,nop,TS val 686264694 ecr 2178967614], length 83
14:10:29.884892 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 875:957, ack 1, win 913, options [nop,nop,TS val 686264697 ecr 2178967627], length 82
14:10:30.442093 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 957:1060, ack 1, win 913, options [nop,nop,TS val 686264753 ecr 2178967656], length 103
14:10:30.446752 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 1060:1157, ack 1, win 913, options [nop,nop,TS val 686264753 ecr 2178968213], length 97
14:10:30.552657 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 1157:1233, ack 1, win 913, options [nop,nop,TS val 686264764 ecr 2178968218], length 76
^C
13 packets captured
31 packets received by filter
0 packets dropped by kernel

Here is the tcp dump running on the Nagios Log Server

Code: Select all

tcpdump src host 10.25.2.1 and tcp dst port 5544 and dst host 10.25.5.80

14:10:29.789399 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 1539010253:1539010409, ack 1580047891, win 913, options [nop,nop,TS val 686264687 ecr 2178954247], length 156
14:10:29.808799 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 156:264, ack 1, win 913, options [nop,nop,TS val 686264689 ecr 2178967560], length 108
14:10:29.843643 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 264:351, ack 1, win 913, options [nop,nop,TS val 686264693 ecr 2178967579], length 87
14:10:29.857451 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 351:434, ack 1, win 913, options [nop,nop,TS val 686264694 ecr 2178967614], length 83
14:10:29.885830 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 434:516, ack 1, win 913, options [nop,nop,TS val 686264697 ecr 2178967627], length 82
14:10:30.443336 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 516:619, ack 1, win 913, options [nop,nop,TS val 686264753 ecr 2178967656], length 103
14:10:30.447682 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 619:716, ack 1, win 913, options [nop,nop,TS val 686264753 ecr 2178968213], length 97
14:10:30.553659 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 716:792, ack 1, win 913, options [nop,nop,TS val 686264764 ecr 2178968218], length 76
14:10:35.983842 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 792:863, ack 1, win 913, options [nop,nop,TS val 686265307 ecr 2178968324], length 71
14:10:38.822377 IP dns1.box293.local.59382 > lsproduction.box293.local.5544: Flags [P.], seq 863:946, ack 1, win 913, options [nop,nop,TS val 686265590 ecr 2178973754], length 83
^C
198 packets captured
198 packets received by filter
0 packets dropped by kernel

[tgriep]

Try running these commands and post the output.

Code: Select all

netstat -naop | grep 5544
ps -ef | grep logstash

[Box293]

Well it's the strangest thing, logs are now coming in from that host.

I do remember checking the system status at the time, I had two ticks up on the top next to System Status.

Thread can be closed, but here's the output from those commands anyway.

Code: Select all

netstat -naop | grep 5544
tcp        0      0 :::5544                     :::*                        LISTEN      27641/java          off (0.00/0/0)
tcp        0      0 ::ffff:10.25.5.80:5544      ::ffff:10.25.2.1:54970      ESTABLISHED 27641/java          off (0.00/0/0)
tcp        0      0 ::1:52004                   ::1:5544                    ESTABLISHED 938/rsyslogd        off (0.00/0/0)
tcp        0      0 ::1:5544                    ::1:52004                   ESTABLISHED 27641/java          off (0.00/0/0)
udp        0      0 :::5544                     :::*                                    27641/java          off (0.00/0/0)

Code: Select all

ps -ef | grep logstash
root      8524  8451  0 09:37 pts/0    00:00:00 grep logstash
root     27639     1  0 Jan16 ?        00:00:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log  -w 4 nagios
nagios   27641 27639  0 Jan16 ?        00:44:32 /usr/bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /usr/local/nagioslogserver/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/usr/local/nagioslogserver/logstash/lib /usr/local/nagioslogserver/logstash/lib/logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4