Default syslog port

[jbullock]

I'm having issues ingesting data using UDP 514. I've followed the directions for allowing privileged ports (am I supposed to change LS_Group as well?). I've also installed and ran tcpdump and the host is communicating over 514 but it's not showing up in the Nagios web interface.

My input looks like this:
syslog {
type => 'syslog'
port => 514
}

I have another input that uses the syslog plugin but a different port. This one runs fine:
syslog {
type => 'Firewall'
port => 5544
}

Any thoughts?

[cdienger]

If you run a "netstat -nap" on the NLS command line do you see port 514 is up and listening?

I'd also try running logstash in debug mode to see if there is a problem parsing the data coming in.

Edit /etc/init.d/logstash and change line 64 from:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

to:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"

and restart the service with:

Code: Select all

systemctl daemon-reload
service logstash restart

Let this run just long enough to allow NLS to recieve some events over port 514 and then collect the /var/log/logstash/logstash.log file before reverting the config back.

[jbullock]

I'm not seeing a line with port 514 other than the tcp6 line.

Here is the line in Nagios Configure section:
Logstash is currently collecting locally on: 161.31.27.217 tcp6: 3515, 514, 5544, 2056, 2057 udp6: 5544, 514

[jbullock]

I changed the line to debug and had a test device generate some logs (verified by our current LMS). Here is the tail of logstash.log:

{:timestamp=>"2020-04-24T14:05:45.803000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2020-04-24T14:05:46.564000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2020-04-24T14:05:47.060000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2020-04-24T14:05:47.060000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2020-04-24T14:05:47.931000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2020-04-24T14:51:26.994000-0500", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2020-04-24T14:51:27.048000-0500", :message=>"stopping pipeline", :id=>"main"}
{:timestamp=>"2020-04-24T14:52:27.625000-0500", :message=>"Pipeline main started"}
{:timestamp=>"2020-04-24T14:54:08.538000-0500", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2020-04-24T14:54:08.550000-0500", :message=>"stopping pipeline", :id=>"main"}
^C

[cdienger]

It looks like elasticsearch is down with those entries. Can you PM me the entire log? I'd also like to get a profile. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:

Code: Select all

/usr/local/nagioslogserver/scripts/profile.sh

This will create /tmp/system-profile.tar.gz.

Note that this file can be very large and may not be able to be uploaded through the PM system. This is usually due to the logs in the Logstash and/or Elasticsearch directories found in it. If it is too large, please open the profile, extract these directories/files and send them separately.

[cdienger]

Logstash isn't able to open 514 on the system for some reason. Please send me a copy of /etc/sysconfig/logstash.

[jbullock]

Have you received my logs over PM? No hurry and I appreciate the help , just wanting to see if they are getting thru.

[cdienger]

It didn't appear to come through the first time but it came through the second time and looks good. Please send copies of the logs found under /var/log/logstash/logstash.log.