Nagios log server not receiving any logs

[scottwilkerson]

Are you sure there are logs being sent?

[thanigaivel.a]

uploaded the current logstash log

[thanigaivel.a]

It looks again its not receiving logs, and though we provided enough ram and cpu, utilization seems to be very high. Always, cpu is higher than 150%.

[root@usa0300lv6332 logstash]# free -m
total used free shared buff/cache available
Mem: 15900 8883 139 47 6877 6461
Swap: 3999 43 3956
[root@usa0300lv6332 logstash]#

[root@usa0300lv6332 logstash]# top
top - 16:31:18 up 1 day, 27 min, 2 users, load average: 2.32, 2.32, 2.31
Tasks: 288 total, 1 running, 287 sleeping, 0 stopped, 0 zombie
%Cpu(s): 22.3 us, 0.3 sy, 0.0 ni, 77.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16282608 total, 118568 free, 9109212 used, 7054828 buff/cache
KiB Swap: 4095996 total, 4051452 free, 44544 used. 6604452 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1749 nagios 20 0 53.987g 7.410g 138808 S 176.8 47.7 2793:32 java

[scottwilkerson]

While 150% isn't that high is you have several processors, which is causing that, ?

Code: Select all

ps aux| grep java

[mcapra]

I've seen this before:

Code: Select all

MapperParsingException[failed to parse [timestamp]]; nested: MapperParsingException[failed to parse date field [Jun 18 04:50:35], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: \"Jun 18 04:50:35\"];

I'd make absolutely certain the logs you're sending to syslog inputs are RFC3164 compliant. Or, alternatively, send the non-compliant logs to a generic tcp or udp input instead.

[scottwilkerson]

And based on these invalid formats, it is possible that the logs are being put into indexes with the incorrect dates.

Also, the error above is from Jun 18 which would be recorded in the index 3 weeks ago.