I am setting up a proof of concept for a client of ours within EC2. I followed the instructions (http://assets.nagios.com/downloads/nagi ... -Cloud.pdf) using the community AMI and I got the server up no problem. Its collecting logs for itself but I setup another host and the logs are not showing up. Well actually 5 cron log messages have come over but there should be a ton of messages.
I made sure the security groups were correct and even temporarily opened all ports just to make sure that was not the problem. I checked the rsyslog config that was created seems correct with the right ip address of the logstash server. I see the log data coming into the server via tcpdump but it never ends up in elasticsearch.
Any ideas?
david
Adding additional logging hosts
Re: Adding additional logging hosts
How long are you waiting? It can take a bit for log entries to be searchable in the dashboards.
Former Nagios employee
Re: Adding additional logging hosts
I got the server setup this morning around 10 and the other host was up around 11 so >4 hrs.
What is wierd is that I have seen 8 entries come in. All of them from cron. I have restarted services to create logs and also used logger.
What is wierd is that I have seen 8 entries come in. All of them from cron. I have restarted services to create logs and also used logger.
- Box293
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
- Contact:
Re: Adding additional logging hosts
Is SELinux enabled on the log server? This can cause issues.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Adding additional logging hosts
That came to my mind too but I have made sure SELinux is disable in /etc/sysconfig/selinux
I have checked the logstash log and the elasticsearch log and I am not seeing anything that sticks out.
I have checked the logstash log and the elasticsearch log and I am not seeing anything that sticks out.
- Box293
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
- Contact:
Re: Adding additional logging hosts
Just to be 100% certain, what does getenforce return when you run it from the cli?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Adding additional logging hosts
Never hurts to triple check
[ec2-user@ip-172-31-2-181 ~]$ getenforce
Disabled
[ec2-user@ip-172-31-2-181 ~]$ getenforce
Disabled
- Box293
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
- Contact:
Re: Adding additional logging hosts
Can you show me a tcpdump on the sending server AND the receiving server.co-dlk wrote:I see the log data coming into the server via tcpdump but it never ends up in elasticsearch.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Adding additional logging hosts
On the client I ran this command:
[ec2-user@ip-172-31-32-119 ~]$ logger foobar
On the log server I seee this:
[ec2-user@ip-172-31-2-181 ~]$ sudo tcpdump host 172.31.32.119 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
20:21:53.851522 IP ip-172-31-32-119.us-west-2.compute.internal.56137 > ip-172-31-2-181.us-west-2.compute.internal.5544: Flags [P.], seq 309816386:309816440, ack 4221555259, win 211, options [nop,nop,TS val 7287704 ecr 12867034], length 54
E..j..@.@..z.. w.....I...wlB...;....#5.....
.o3...U.<13>Nov 11 01:21:53 ip-172-31-32-119 ec2-user: foobar
20:21:53.852178 IP ip-172-31-2-181.us-west-2.compute.internal.5544 > ip-172-31-32-119.us-west-2.compute.internal.56137: Flags [.], ack 54, win 174, options [nop,nop,TS val 12880814 ecr 7287704], length 0
E..4..@.@.+G...... w...I...;.wlx.....m.....
.....o3.
[ec2-user@ip-172-31-32-119 ~]$ logger foobar
On the log server I seee this:
[ec2-user@ip-172-31-2-181 ~]$ sudo tcpdump host 172.31.32.119 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
20:21:53.851522 IP ip-172-31-32-119.us-west-2.compute.internal.56137 > ip-172-31-2-181.us-west-2.compute.internal.5544: Flags [P.], seq 309816386:309816440, ack 4221555259, win 211, options [nop,nop,TS val 7287704 ecr 12867034], length 54
E..j..@.@..z.. w.....I...wlB...;....#5.....
.o3...U.<13>Nov 11 01:21:53 ip-172-31-32-119 ec2-user: foobar
20:21:53.852178 IP ip-172-31-2-181.us-west-2.compute.internal.5544 > ip-172-31-32-119.us-west-2.compute.internal.56137: Flags [.], ack 54, win 174, options [nop,nop,TS val 12880814 ecr 7287704], length 0
E..4..@.@.+G...... w...I...;.wlx.....m.....
.....o3.
Re: Adding additional logging hosts
Hmmm. Not sure what happened but I am starting to see log messages.
I think it may be a security group issue. Need to investigate some more.
I think it may be a security group issue. Need to investigate some more.