Syslog Source Output as JSON Format

[tcsdi]

Edit /etc/init.d/logstash and change line 64 from:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

to:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"

and restart Logstash with:

Code: Select all

service logstash restart

Let this run just long enough for netflow data to come in then revert the changes to disable it. This should create a /var/log/logstash/logstash.log file with some more details. Please PM me a copy of this file as well as a profile from Admin > System > System Status > Download System Profile.

Hi @cdienger,

Very sorry for the delay. Please see attached logs that you requested.

Thanks!

Support Edit: Nagios_logs.zip downloaded and shared with team.

[cdienger]

It looks like the netflow data is getting logged and then that logged format is getting sent over. Try setting up the input to look like this instead:

Code: Select all

udp{
type=>'nflow'
port=>'port'
}

and then create a filter with just this:

Code: Select all

kv{}

gather the logs again if there are still any issues getting the data into NLS.

[tcsdi]

Hi Cdienger,

Thank you for your respones

Config was applied:

Input:


filter:


Netflow - nflow can now be seen:


But NetFlow inputs that NLS receives are like this:


Do we need to add Plugins?

Best Regards,
Christian

[cdienger]

Please provide a screenshot of the events as seen in the dashboard. Make sure the events are expanded so that we can see all the fields of the event.

[tcsdi]

Please provide a screenshot of the events as seen in the dashboard. Make sure the events are expanded so that we can see all the fields of the event.

Hi @cdienger,

When we checked the server the next day after applying the input filters, NLS stopped working and upon checking the unique hosts went from 30 to 1010.

For now I disabled netflow config on input, filter and output. But after reboot some tabs became unable to access, specially Admin - cluster

You can check out the attachments for the screenshots and also the system profile. I don't know what happened to the server anymore, did I do something wrong?

Support Edit: system-profile (2).tar.gz downloaded and shared with team

[cdienger]

See https://support.nagios.com/kb/article.php?id=132 for increasing the memory allocated to PHP and fix the issue with Admin > Cluster.

[tcsdi]

Hi Cdienger,

We can enable it after increasing the memory, however, there is still no explanation why we had 1010 sources.
Also, do we have an update on the review of the system profile? It may happen again and Nagios might crash after enabling NetFlow.

Best Regards,

[mbellerue]

Looking at your profile, it's pretty likely that there's a field in the netflow data that's overriding the normal host field. Let's change your filter once more.

Code: Select all

kv {
    exclude_keys => [ "host" ]
}

Then go ahead and enable netflow again, let it run for a few minutes and watch to see if the number of unique hosts increases.

Also, if you still have 1000+ hosts, you can get them cleared out. 24 hours after applying the new kv filter, run this command.

Code: Select all

curl -XDELETE localhost:9200/nagioslogserver/cf_option/known_hosts

[tcsdi]

Hi Cdienger,

We have enabled NetFlow and are currently monitoring. For clarification, this filter won’t affect the other inputs?



For the other issue,
We have tried adjusting the memory for PHP from 128M to 512M. (https://support.nagios.com/kb/article.php?id=132)
We can access Admin but not Admin-> cluster

ADMIN


ADMIN>CLUSTER


Best Regards,
TCSDI

[mbellerue]

We have enabled NetFlow and are currently monitoring. For clarification, this filter won’t affect the other inputs?

No, this should just prevent your unique hosts number from expanding the way it did.

Regarding the pages not being displayed, do you have the available memory to bump the PHP memory limit to say 1024?