Localhost logs no longer populating

[euscada]

Hi all,

New Nagios Log Server user.

I just noticed that our localhost Nagios logs are no longer coming into the reports for that host (localhost)

I am unsure what changed and it also appears our Ubuntu VM (running shorewall and postfix) is no longer talking to nagios either.

Lastly noted is our firewall (Palo Alto with Panorama VM) no longer is communicating via UDP to Nagios, I had to create a rule to talk TCP over port 3515.

Any input helps whittle these nuances down!

Thanks much

[455157]

Hi euscada.

Welcome to the forum!

Are you still receiving logs from *any* of your sources? Just wondering if maybe Logstash is not running for some reason. You can check this in Admin-> System Status menu.

Here is a guide on troubleshooting this sort of thing as well:

https://support.nagios.com/kb/article/n ... in-38.html

It is focused on a CentOS source, but may give you some ideas you can use.

[cdienger]

What inputs and protocols are being used to get the data to NLS? Are the appropriate ports open if you run a "netstat -na | grep <input_port>" ?

The default /etc/rsyslog.d/nagioslogserver.conf on the NLS server should be using syslog over tcp, and you should check that is still the case(note the double @ indicate TCP. A single @ means UDP):

Code: Select all

# ### begin forwarding rule ###
#
# NAGIOS LOG SERVER
#
$WorkDirectory /var/lib/rsyslog    # where to place spool files
$ActionQueueFileName fwdRule1      # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g        # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on      # save messages to disk on shutdown
$ActionQueueType LinkedList        # run asynchronously
$ActionResumeRetryCount -1         # infinite retries if host is down
*.* @@192.168.4.82:5544
#
# ### end of the forwarding rule ###

You should also verify the iptables rules to make sure ports are not getting blocked. Run "iptables -L" to see current rules.

[euscada]

Are you still receiving logs from *any* of your sources? Just wondering if maybe Logstash is not running for some reason. You can check this in Admin-> System Status menu.

Hello again, yes, we have communication to various VMs (Ubuntu, Windows, and an ESXi device) and logs are populating.

The strange thing is, I found this area of Nagios (Admin/System Status) and found that the Elasticsearch and Logstash "status" never "refreshes"...All i see is the throbber (spinning circle) spin and spin...It never finishes "thinking"...

Thanks for the link, will check it out.

What inputs and protocols are being used to get the data to NLS?
Our windows machines are using NXlog (assuming json?), linux machines are using rsyslog UDP, esxi box is configured to talk syslog over UDP 5544, Palo Alto firewall using syslog TCP 3515 (This used to be configured UDP 5544 and stopped working) (i hope this is what you were after?)

Are the appropriate ports open if you run a "netstat -na | grep <input_port>" ?

This appears to be fine

The default /etc/rsyslog.d/nagioslogserver.conf on the NLS server should be using syslog over tcp, and you should check that is still the case(note the double @ indicate TCP. A single @ means UDP):

This is the case (double @@ with port 5544)

You should also verify the iptables rules to make sure ports are not getting blocked. Run "iptables -L" to see current rules.

What is strange is that our Ubuntu (shorewall/postfix vm), our Palo Alto Firewall and the localhost logs all stopped talking around the same time....

Thanks for the replies!

[cdienger]

Not being able to apply configuration is likely related. Is disk space okay on the machine if you run a "df -h" ? Are there any errors or warnings logged in /var/log/logstash.log or /var/log/eleasticsearch/<cluster_uuid>.log ? Does the config verify successfully under Configure > Global (All Instances) > Global Config ?

I'd like to review the config(View > All Files Combined) if the above doesn't help resolve the problem.

[euscada]

Hi again,

Disk space is 3% used (this is a fresh install with minimal config)

/var/log/logstash.log does not exist

Here is the output of uuid.log --
(Not an easy way to copy this out, here is a screen shot)





Here is the All Files Combined output --

#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 13 Sep 2018 09:01:11 -0700
#

#
# Global inputs
#

input {
syslog {
type => 'syslog'
port => 5544
}
tcp {
type => 'eventlog'
port => 3515
codec => json
}
tcp {
type => 'import_raw'
tags => 'import_raw'
port => 2056
}
tcp {
type => 'import_json'
tags => 'import_json'
port => 2057
codec => json
}
}

#
# Local inputs
#

#
# Global filters
#

filter {
if [program] == 'apache_access' {
grok {
match => [ 'message', '%{COMBINEDAPACHELOG}']
}
date {
match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
}
mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
}
}

if [program] == 'apache_error' {
grok {
match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
}
mutate {
replace => [ 'type', 'apache_error' ]
}
}
}

#
# Local filters
#

#
# Global outputs
#

#
# Local outputs
#

Thanks again for the help!

[cdienger]

It's important that all sources sending to an input use the same format. Syslog can use different formats and if sources use different formats then this can cause parsing problems.

That said, try restarting the logstash service:

service logstash restart

[euscada]

How do I see the format that nagios needs for syslog?

I am unsure if I can configure the output of some of my syslog formats?

I can see something amiss in that log file stating the timestamp is incorrect, is there a way to pinpoint who is sending the wrong date format?

I have current support with nagios and I have 3 more instances to configure.

Would it be worth a call to the phone support to further investigate the issue?

I restarted the VM and the logstash with no success.

Please advise and thanks much!

[euscada]

I now have the localhost and Ubuntu machine talking to my instance of Nagios.

I had to add this code to the syslog input

Code: Select all

tcp {
    port => 5544
    type => syslog
  }
  udp {
    port => 5544
    type => syslog
  }
 

Any pros or cons to doing this?

I found this solution from other forums posts on here.

[cdienger]

The syslog input can do some parsing on its own to fill fields while tcp and udp would require setting up filters to do this. Setting up custom filters isn't too difficult and is covered in https://support.nagios.com/kb/article/n ... ew-98.html under the "Logstash Example Configuration" section.

It sounds like some of the devices are not using RFC3164 syslog messages which is what the input needs:

https://www.elastic.co/guide/en/logstas ... yslog.html

While I don't see phone support, you can certainly submit a ticket at http://support.nagios.com/tickets/. I would definitely at least post in "Customer Support/Nagios Log Server" instead of the General Support forums going forward as well since posts there will have a higher priority.