Page 3 of 5

Re: Syslog Source Output as JSON Format

Posted: Mon Sep 09, 2019 10:21 pm
by tcsdi
cdienger wrote:Edit /etc/init.d/logstash and change line 64 from:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"
to:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"
and restart Logstash with:

Code: Select all

service logstash restart
Let this run just long enough for netflow data to come in then revert the changes to disable it. This should create a /var/log/logstash/logstash.log file with some more details. Please PM me a copy of this file as well as a profile from Admin > System > System Status > Download System Profile.

Hi @cdienger,

Very sorry for the delay. Please see attached logs that you requested.

Thanks!

Support Edit: Nagios_logs.zip downloaded and shared with team.

Re: Syslog Source Output as JSON Format

Posted: Tue Sep 10, 2019 2:00 pm
by cdienger
It looks like the netflow data is getting logged and then that logged format is getting sent over. Try setting up the input to look like this instead:

Code: Select all

udp{
type=>'nflow'
port=>'port'
}
and then create a filter with just this:

Code: Select all

kv{}
gather the logs again if there are still any issues getting the data into NLS.

Re: Syslog Source Output as JSON Format

Posted: Wed Sep 11, 2019 1:03 am
by tcsdi
Hi Cdienger,

Thank you for your respones

Config was applied:

Input:
Image

filter:
Image

Netflow - nflow can now be seen:
Image

But NetFlow inputs that NLS receives are like this:
Image

Do we need to add Plugins?

Best Regards,
Christian

Re: Syslog Source Output as JSON Format

Posted: Wed Sep 11, 2019 9:47 am
by cdienger
Please provide a screenshot of the events as seen in the dashboard. Make sure the events are expanded so that we can see all the fields of the event.

Re: Syslog Source Output as JSON Format

Posted: Sun Sep 15, 2019 11:17 pm
by tcsdi
cdienger wrote:Please provide a screenshot of the events as seen in the dashboard. Make sure the events are expanded so that we can see all the fields of the event.
Hi @cdienger,

When we checked the server the next day after applying the input filters, NLS stopped working and upon checking the unique hosts went from 30 to 1010.

For now I disabled netflow config on input, filter and output. But after reboot some tabs became unable to access, specially Admin - cluster :(

You can check out the attachments for the screenshots and also the system profile. I don't know what happened to the server anymore, did I do something wrong?

Support Edit: system-profile (2).tar.gz downloaded and shared with team

Re: Syslog Source Output as JSON Format

Posted: Mon Sep 16, 2019 10:32 am
by cdienger
See https://support.nagios.com/kb/article.php?id=132 for increasing the memory allocated to PHP and fix the issue with Admin > Cluster.

Re: Syslog Source Output as JSON Format

Posted: Mon Sep 23, 2019 2:39 am
by tcsdi
Hi Cdienger,

We can enable it after increasing the memory, however, there is still no explanation why we had 1010 sources.
Also, do we have an update on the review of the system profile? It may happen again and Nagios might crash after enabling NetFlow.

Best Regards,

Re: Syslog Source Output as JSON Format

Posted: Mon Sep 23, 2019 2:32 pm
by mbellerue
Looking at your profile, it's pretty likely that there's a field in the netflow data that's overriding the normal host field. Let's change your filter once more.

Code: Select all

kv {
    exclude_keys => [ "host" ]
}
Then go ahead and enable netflow again, let it run for a few minutes and watch to see if the number of unique hosts increases.

Also, if you still have 1000+ hosts, you can get them cleared out. 24 hours after applying the new kv filter, run this command.

Code: Select all

curl -XDELETE localhost:9200/nagioslogserver/cf_option/known_hosts

Re: Syslog Source Output as JSON Format

Posted: Sun Oct 06, 2019 10:32 pm
by tcsdi
Hi Cdienger,

We have enabled NetFlow and are currently monitoring. For clarification, this filter won’t affect the other inputs?

Image Image

For the other issue,
We have tried adjusting the memory for PHP from 128M to 512M. (https://support.nagios.com/kb/article.php?id=132)
We can access Admin but not Admin-> cluster

ADMIN
Image

ADMIN>CLUSTER
Image

Best Regards,
TCSDI

Re: Syslog Source Output as JSON Format

Posted: Mon Oct 07, 2019 1:49 pm
by mbellerue
We have enabled NetFlow and are currently monitoring. For clarification, this filter won’t affect the other inputs?
No, this should just prevent your unique hosts number from expanding the way it did.

Regarding the pages not being displayed, do you have the available memory to bump the PHP memory limit to say 1024?