kafka integration with NLS

[lukedevon]

Hi,

I am trying to integrate kafka with nls.

kafka {
topic_id => 'topic_XXXX''
zk_connect => 'kafka-ip:2181'
}

2181 is the port zookeeper running on.
9092 is the port Kafka running on.

I have access to NLS from Kafka/ZK server. Able to ping. Which ports needed to be opened in NLS as to capture the Kafka data? I have opened 2181 and 9092 already. But I am still struggling with the input filter.

Can you please help me to do this configuration in nls?

Thank you
Luke.

[cdienger]

There's an extra quote at the end of the topic_id line which will cause the config to fail to apply. It should be:

Code: Select all

kafka {
topic_id => 'topic_XXXX'
zk_connect => 'kafka-ip:2181'
}

With this config, NLS will attempt to connect to the Kafka server over TCP port 2181. The destination 2181 port needs to open and allowed on the Kafka server.

[lukedevon]

Hi,

As per the usual practice, we send logs from external sources to NLS.
In that case, we need to get access from external nodes to NLS. (pushing logs from external nodes to NLS)

But it seems Kafka integration, quite different than the other nodes integration. (NLS need to pull the logs from Kafka node). That's why I think, as you have stated NLS to need to have access to Kafka port 2181.

May I know, is this the general steps to follow when integrating Kafka logs with NLS?

Regards
Luke

[cdienger]

Yes, using this input you need to allow NLS to pull logs from the Kafka server.

I'm not familiar enough with Kafka to know if has a way of pushing the same data, but if it does then you have the option of setting up another input on NLS to accept the data. Note that you may need to create a filter to parse the data properly if you can go this route.

[lukedevon]

Thank you. I will follow the method you have explained. (pull)

[cdienger]

Sounds good. Let us know if you run into any issues that we can help with.

[lukedevon]

Hi

I got the access from nls to Kafka - 2181 port. However ,so far I was not able see the logs in NLS dashboard.I have taken 2 tcpdump from the kafka node and the NLS node. I will PM you the tcpdump and the IP details.

Please help me to understand what I have done wrong here.

Thanks in advance
Luke.

[cdienger]

They seem to be able to communicate. Is did you replace topic_XXXX with a valid topic? Is the topic getting published to while you run the dumps?

[lukedevon]

Hi

topic_ids have configured correctly. But still, I don't see Nagios receiving logs. Can you help me further, please? I am clueless about what was missing in the configuration.

As I netstat in nls, I can see nls_ip:2181 is established with kafka IP.

In Kafka Node;

netstat -nap | grep nls_ip
tcp6 0 0 kafka_ip:2181 nls_ip:35274 ESTABLISHED 769/zookeeper

in NLS;

netstat -nap | grep 2181
tcp6 0 0 nls_ip:35274 kafka_ip:2181 ESTABLISHED 38316/java

#firewall-cmd --list-ports

80/tcp 443/tcp 9300-9400/tcp 3515/tcp 5544/tcp 2056/tcp 2057/tcp 5544/udp 514/udp 514/tcp 2181/tcp


But, when I checked in the dashboard;

Logstash is currently collecting locally on: nls_ip tcp6: 5544, 514udp6: 5544

I dont see the port 2181 is listed. Is it normal or something I have missed in the configurations?


Thanks
Luke.

[scottwilkerson]

Nagios Log Server doesn't listen on 2181, it need to connect to the Kafka server on 2181

Is the firewall on the Kafka server allowing connections on 2181 from the Nagios Log Server IP?