Missing Log entries in Nagios Log Server

[comfone]

Hi All
We are using 2 Nagios Log Server Instances Version 1.4.0 (VM appliance).
We are missing some entries in the Nagios Log Server.
I have attached a list. On the left side of the list you see an extract of the Log, which is sent to Nagios Log Server and on the right side the entries received by NLS.
As you can also see, there are missing entries on the NLS.
The Log on server is appended only once an our if everything is ok.
Also attached is the config file.
Can you please help me to troubleshoot this problem.
How can I find out, why not all logs are sent to NLS!
Thank you.

[hsmith]

Is there any information in /var/log/logstash/logstash.log?

[comfone]

Nope. Nothing in /var/log/logstash/logstash.log
Where else can I search/look?
Is there a log on NLS where I can see if "logs" were dropped?

[hsmith]

logstash.log is where you're going to be told whether or not logs are being dropped. How are you validating they were dropped? Are these servers far away from each other? I've not seen behavior where just a few logs are missing.

[comfone]

I have two Server sending similar logs.
As you can see below some entries can be found in NLS other not!?!?
Log from Server A
SSG-RUN-OK 2016-08-09 12:16:01:351 ""
SSG-RUN-OK 2016-08-09 13:16:01:612 "" not in NLS
SSG-RUN-OK 2016-08-09 14:16:01:869 ""
SSG-RUN-OK 2016-08-09 15:16:02:120 "" not in NLS
SSG-RUN-OK 2016-08-09 16:16:02:093 ""
SSG-RUN-OK 2016-08-09 17:16:02:141 ""
SSG-RUN-OK 2016-08-09 18:16:01:587 "" not in NLS
SSG-RUN-OK 2016-08-09 19:16:01:995 ""
SSG-RUN-OK 2016-08-09 20:16:01:882 ""
SSG-RUN-OK 2016-08-09 21:16:01:354 "" not in NLS

Log from Server B
SSG-RUN-OK 2016-08-09 12:15:02:033 ""
SSG-RUN-OK 2016-08-09 13:15:01:344 "" not in NLS
SSG-RUN-OK 2016-08-09 14:15:02:051 ""
SSG-RUN-OK 2016-08-09 15:15:01:224 ""
SSG-RUN-OK 2016-08-09 16:15:01:208 "" not in NLS
SSG-RUN-OK 2016-08-09 17:15:01:953 ""
SSG-RUN-OK 2016-08-09 18:15:01:448 ""
SSG-RUN-OK 2016-08-09 19:15:02:031 "" not in NLS
SSG-RUN-OK 2016-08-09 20:15:01:327 ""
SSG-RUN-OK 2016-08-09 21:15:01:871 "" not in NLS

I have configured the following filter for this logs:

if [program] == 'SSG-STATISTICS' {
mutate {replace => [ 'type', 'SSG-STATISTICS' ]}
grok {
match => { 'message' => [ '\A%{WORD:Loader}-%{WORD:AlarmType}-%{WORD:Data}%{SPACE}%{TIMESTAMP_ISO8601:TimeStamp}%{SPACE}%{QUOTEDSTRING:Info}' ] }

And on both Server the following rsyslog.d config:

# ## Forwarding
$ModLoad imfile
$InputFilePollInterval 10
#$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

# Input for OracleAlertLog
$InputFileName /data/appl/appliss/tdr/nagios/ssg-nagios_statistics.alarm
$InputFileTag SSG-STATISTICS:
$InputFileStateFile nls-state-ROAM3_TEST-SSG-STATISTICS # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
#$InputFilePersistStateInterval 20000
#$InputFileFacility local9
$InputRunFileMonitor

## Forward to Nagios Logserver and then discard.
if $programname == 'SSG-STATISTICS' then @@IP.IP.IP.18:5544
if $programname == 'SSG-STATISTICS' then ~
## ### end of the forwarding rule ###

[hsmith]

Can you provide a screenshot of the query you're using inside of Logserver to find this information out?

[comfone]

The problem is solved.
After having done a tcp dump on the log sending server.
I saw that the logs are not sent, because the session had timed out.
As I mentioned on the beginning. This log is appended only once an hour.
That was the reason of missing entries.