kafka integration with NLS
kafka integration with NLS
Hi,
I am trying to integrate kafka with nls.
kafka {
topic_id => 'topic_XXXX''
zk_connect => 'kafka-ip:2181'
}
2181 is the port zookeeper running on.
9092 is the port Kafka running on.
I have access to NLS from Kafka/ZK server. Able to ping. Which ports needed to be opened in NLS as to capture the Kafka data? I have opened 2181 and 9092 already. But I am still struggling with the input filter.
Can you please help me to do this configuration in nls?
Thank you
Luke.
I am trying to integrate kafka with nls.
kafka {
topic_id => 'topic_XXXX''
zk_connect => 'kafka-ip:2181'
}
2181 is the port zookeeper running on.
9092 is the port Kafka running on.
I have access to NLS from Kafka/ZK server. Able to ping. Which ports needed to be opened in NLS as to capture the Kafka data? I have opened 2181 and 9092 already. But I am still struggling with the input filter.
Can you please help me to do this configuration in nls?
Thank you
Luke.
Re: kafka integration with NLS
There's an extra quote at the end of the topic_id line which will cause the config to fail to apply. It should be:
With this config, NLS will attempt to connect to the Kafka server over TCP port 2181. The destination 2181 port needs to open and allowed on the Kafka server.
Code: Select all
kafka {
topic_id => 'topic_XXXX'
zk_connect => 'kafka-ip:2181'
}
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: kafka integration with NLS
Hi,
As per the usual practice, we send logs from external sources to NLS.
In that case, we need to get access from external nodes to NLS. (pushing logs from external nodes to NLS)
But it seems Kafka integration, quite different than the other nodes integration. (NLS need to pull the logs from Kafka node). That's why I think, as you have stated NLS to need to have access to Kafka port 2181.
May I know, is this the general steps to follow when integrating Kafka logs with NLS?
Regards
Luke
As per the usual practice, we send logs from external sources to NLS.
In that case, we need to get access from external nodes to NLS. (pushing logs from external nodes to NLS)
But it seems Kafka integration, quite different than the other nodes integration. (NLS need to pull the logs from Kafka node). That's why I think, as you have stated NLS to need to have access to Kafka port 2181.
May I know, is this the general steps to follow when integrating Kafka logs with NLS?
Regards
Luke
Re: kafka integration with NLS
Yes, using this input you need to allow NLS to pull logs from the Kafka server.
I'm not familiar enough with Kafka to know if has a way of pushing the same data, but if it does then you have the option of setting up another input on NLS to accept the data. Note that you may need to create a filter to parse the data properly if you can go this route.
I'm not familiar enough with Kafka to know if has a way of pushing the same data, but if it does then you have the option of setting up another input on NLS to accept the data. Note that you may need to create a filter to parse the data properly if you can go this route.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: kafka integration with NLS
Thank you. I will follow the method you have explained. (pull)
Re: kafka integration with NLS
Sounds good. Let us know if you run into any issues that we can help with.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: kafka integration with NLS
Hi
I got the access from nls to Kafka - 2181 port. However ,so far I was not able see the logs in NLS dashboard.I have taken 2 tcpdump from the kafka node and the NLS node. I will PM you the tcpdump and the IP details.
Please help me to understand what I have done wrong here.
Thanks in advance
Luke.
I got the access from nls to Kafka - 2181 port. However ,so far I was not able see the logs in NLS dashboard.I have taken 2 tcpdump from the kafka node and the NLS node. I will PM you the tcpdump and the IP details.
Please help me to understand what I have done wrong here.
Thanks in advance
Luke.
Re: kafka integration with NLS
They seem to be able to communicate. Is did you replace topic_XXXX with a valid topic? Is the topic getting published to while you run the dumps?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: kafka integration with NLS
Hi
topic_ids have configured correctly. But still, I don't see Nagios receiving logs. Can you help me further, please? I am clueless about what was missing in the configuration.
As I netstat in nls, I can see nls_ip:2181 is established with kafka IP.
In Kafka Node;
netstat -nap | grep nls_ip
tcp6 0 0 kafka_ip:2181 nls_ip:35274 ESTABLISHED 769/zookeeper
in NLS;
netstat -nap | grep 2181
tcp6 0 0 nls_ip:35274 kafka_ip:2181 ESTABLISHED 38316/java
#firewall-cmd --list-ports
80/tcp 443/tcp 9300-9400/tcp 3515/tcp 5544/tcp 2056/tcp 2057/tcp 5544/udp 514/udp 514/tcp 2181/tcp
But, when I checked in the dashboard;
Logstash is currently collecting locally on: nls_ip tcp6: 5544, 514udp6: 5544
I dont see the port 2181 is listed. Is it normal or something I have missed in the configurations?
Thanks
Luke.
topic_ids have configured correctly. But still, I don't see Nagios receiving logs. Can you help me further, please? I am clueless about what was missing in the configuration.
As I netstat in nls, I can see nls_ip:2181 is established with kafka IP.
In Kafka Node;
netstat -nap | grep nls_ip
tcp6 0 0 kafka_ip:2181 nls_ip:35274 ESTABLISHED 769/zookeeper
in NLS;
netstat -nap | grep 2181
tcp6 0 0 nls_ip:35274 kafka_ip:2181 ESTABLISHED 38316/java
#firewall-cmd --list-ports
80/tcp 443/tcp 9300-9400/tcp 3515/tcp 5544/tcp 2056/tcp 2057/tcp 5544/udp 514/udp 514/tcp 2181/tcp
But, when I checked in the dashboard;
Logstash is currently collecting locally on: nls_ip tcp6: 5544, 514udp6: 5544
I dont see the port 2181 is listed. Is it normal or something I have missed in the configurations?
Thanks
Luke.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: kafka integration with NLS
Nagios Log Server doesn't listen on 2181, it need to connect to the Kafka server on 2181
Is the firewall on the Kafka server allowing connections on 2181 from the Nagios Log Server IP?
Is the firewall on the Kafka server allowing connections on 2181 from the Nagios Log Server IP?