Local File Input not working

[CFT6Server]

I am trying to get file input working on new nodes. It doesn't seem to be grabbing logs in... any ideas?

Input

Code: Select all

file {
  path => "/dns/proxy/*"
  type => "proxylogs"
  tags => "proxylogs"
}

Code: Select all

# ls -l /dns/proxy/
-rwxrwxrwx 1 root root 22161269 Aug 31 12:52 access
-rwxrwxrwx 1 root root 22587529 Aug 31 12:53 access.201508202355
-rwxrwxrwx 1 root root 21305124 Aug 31 12:53 access.201508212355
-rwxrwxrwx 1 root root 19473056 Aug 31 12:53 access.201508222355
-rwxrwxrwx 1 root root 18445437 Aug 31 12:53 access.201508232355
-rwxrwxrwx 1 root root 21925034 Aug 31 12:53 access.201508242355
-rwxrwxrwx 1 root root 22712223 Aug 31 12:53 access.201508252355
-rwxrwxrwx 1 root root 22464164 Aug 31 12:53 access.201508262355

[jolson]

Your input definition and permissions are correct. Anything in the logstash logs?

Code: Select all

cat /var/log/logstash/logstash.log

It might be worth giving the following input a shot - just a hunch:

Code: Select all

file {
  path => "/dns/proxy/access*"
  type => "proxylogs"
  tags => "proxylogs"
}

[CFT6Server]

Input

Code: Select all

file {
  path => "/dns/proxy/access*"
  type => "proxylogs"
  tags => "proxylogs"
}

This did not work and nothing is showing up. There's also no activity on the logstash logs.

[jolson]

What do your directory permissions look like?

Code: Select all

ls -ld /dns
ls -ld /dns/proxy

The 'nagios' user will need appropriate permissions to tunnel through the above directories to the log files in question. I got this working on my test box, so I'm not sure what the problem could be here. Is the configuration certainly in place on the server?

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

[CFT6Server]

So I think the permissions should be fine and I've updated it to nagios:nagios. But no luck....

Code: Select all

#
# Local inputs
#

input {
    file {
      path => ["/dns/proxy/access*"]
      type => "proxylogs"
      tags => "proxylogs"
    }
}

Code: Select all

# ls -ld /dns/
drwxr-xr-x 3 nagios nagios 4096 Sep  6 09:59 /dns/

Code: Select all

]# ls -ld /dns/proxy/
drwxr-xr-x 6 nagios nagios 4096 Aug 31 19:14 /dns/proxy/

[jolson]

Are there any errors reported in your logstash log?

Code: Select all

cat /var/log/logstash/logstash.log

Try appending a new line to one of your logs - I'd like to see if Nagios picks up the append.

Code: Select all

echo "some log line" >> /dns/proxy/access

[krobertson71]

First off, as stated before, the log files are owned by root.

If you change that to nagios:nagios then I doubt the app will be able to now write to those logs.

Also you are using access* which is not a good idea since you are going to grab the same rotated logs over and over. You should just pull 'access'.

Also, since the log rolls, the new access log that gets created is likely to go back to root:root.

Just my two cents. May be way off.

[CFT6Server]

I get what you are referring to, but I have a specific purpose for this. I just have some log files I need to input into LS. so putting in in nagios:nagios is fine as this is a copy of the logs and eliminate any permission issues. Since I am grabbing all the logs, I just wanted to use *.

Now it should be able to import the files if it is new, and once imported, it will know the previous location where the logs are last read and will not try to reimport them into LS. At least that's how I think it works?

[jolson]

At least that's how I think it works?

It's my understanding that Logstash keeps track of which files it has already scanned using the file input - and how far it's scanned into each one.

Have you tried appending the new line and looking for errors? It would also be worth restarting logstash to check for initialization problems.

Code: Select all

service logstash restart
tail /var/log/logstash/logstash.log