Adding additional logging hosts

[co-dlk]

I am setting up a proof of concept for a client of ours within EC2. I followed the instructions (http://assets.nagios.com/downloads/nagi ... -Cloud.pdf) using the community AMI and I got the server up no problem. Its collecting logs for itself but I setup another host and the logs are not showing up. Well actually 5 cron log messages have come over but there should be a ton of messages.

I made sure the security groups were correct and even temporarily opened all ports just to make sure that was not the problem. I checked the rsyslog config that was created seems correct with the right ip address of the logstash server. I see the log data coming into the server via tcpdump but it never ends up in elasticsearch.

Any ideas?

david

[tmcdonald]

How long are you waiting? It can take a bit for log entries to be searchable in the dashboards.

[co-dlk]

I got the server setup this morning around 10 and the other host was up around 11 so >4 hrs.

What is wierd is that I have seen 8 entries come in. All of them from cron. I have restarted services to create logs and also used logger.

[Box293]

Is SELinux enabled on the log server? This can cause issues.

[co-dlk]

That came to my mind too but I have made sure SELinux is disable in /etc/sysconfig/selinux

I have checked the logstash log and the elasticsearch log and I am not seeing anything that sticks out.

[Box293]

Just to be 100% certain, what does getenforce return when you run it from the cli?

[co-dlk]

Never hurts to triple check

[ec2-user@ip-172-31-2-181 ~]$ getenforce
Disabled

[Box293]

I see the log data coming into the server via tcpdump but it never ends up in elasticsearch.

Can you show me a tcpdump on the sending server AND the receiving server.

[co-dlk]

On the client I ran this command:

[ec2-user@ip-172-31-32-119 ~]$ logger foobar

On the log server I seee this:

[ec2-user@ip-172-31-2-181 ~]$ sudo tcpdump host 172.31.32.119 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
20:21:53.851522 IP ip-172-31-32-119.us-west-2.compute.internal.56137 > ip-172-31-2-181.us-west-2.compute.internal.5544: Flags [P.], seq 309816386:309816440, ack 4221555259, win 211, options [nop,nop,TS val 7287704 ecr 12867034], length 54
E..j..@.@..z.. w.....I...wlB...;....#5.....
.o3...U.<13>Nov 11 01:21:53 ip-172-31-32-119 ec2-user: foobar

20:21:53.852178 IP ip-172-31-2-181.us-west-2.compute.internal.5544 > ip-172-31-32-119.us-west-2.compute.internal.56137: Flags [.], ack 54, win 174, options [nop,nop,TS val 12880814 ecr 7287704], length 0
E..4..@.@.+G...... w...I...;.wlx.....m.....
.....o3.

[co-dlk]

Hmmm. Not sure what happened but I am starting to see log messages.

I think it may be a security group issue. Need to investigate some more.