Page 1 of 1
Local File Input not working
Posted: Tue Sep 01, 2015 5:23 pm
by CFT6Server
I am trying to get file input working on new nodes. It doesn't seem to be grabbing logs in... any ideas?
Input
Code: Select all
file {
path => "/dns/proxy/*"
type => "proxylogs"
tags => "proxylogs"
}
Code: Select all
# ls -l /dns/proxy/
-rwxrwxrwx 1 root root 22161269 Aug 31 12:52 access
-rwxrwxrwx 1 root root 22587529 Aug 31 12:53 access.201508202355
-rwxrwxrwx 1 root root 21305124 Aug 31 12:53 access.201508212355
-rwxrwxrwx 1 root root 19473056 Aug 31 12:53 access.201508222355
-rwxrwxrwx 1 root root 18445437 Aug 31 12:53 access.201508232355
-rwxrwxrwx 1 root root 21925034 Aug 31 12:53 access.201508242355
-rwxrwxrwx 1 root root 22712223 Aug 31 12:53 access.201508252355
-rwxrwxrwx 1 root root 22464164 Aug 31 12:53 access.201508262355
Re: Local File Input not working
Posted: Wed Sep 02, 2015 10:04 am
by jolson
Your input definition and permissions are correct. Anything in the logstash logs?
Code: Select all
cat /var/log/logstash/logstash.log
It might be worth giving the following input a shot - just a hunch:
Code: Select all
file {
path => "/dns/proxy/access*"
type => "proxylogs"
tags => "proxylogs"
}
Re: Local File Input not working
Posted: Wed Sep 02, 2015 10:58 pm
by CFT6Server
Input
Code: Select all
file {
path => "/dns/proxy/access*"
type => "proxylogs"
tags => "proxylogs"
}
This did not work and nothing is showing up. There's also no activity on the logstash logs.
Re: Local File Input not working
Posted: Thu Sep 03, 2015 11:21 am
by jolson
What do your directory permissions look like?
The 'nagios' user will need appropriate permissions to tunnel through the above directories to the log files in question. I got this working on my test box, so I'm not sure what the problem could be here. Is the configuration certainly in place on the server?
Code: Select all
cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf
Re: Local File Input not working
Posted: Tue Sep 08, 2015 1:59 pm
by CFT6Server
So I think the permissions should be fine and I've updated it to nagios:nagios. But no luck....
Code: Select all
#
# Local inputs
#
input {
file {
path => ["/dns/proxy/access*"]
type => "proxylogs"
tags => "proxylogs"
}
}
Code: Select all
# ls -ld /dns/
drwxr-xr-x 3 nagios nagios 4096 Sep 6 09:59 /dns/
Code: Select all
]# ls -ld /dns/proxy/
drwxr-xr-x 6 nagios nagios 4096 Aug 31 19:14 /dns/proxy/
Re: Local File Input not working
Posted: Tue Sep 08, 2015 3:59 pm
by jolson
Are there any errors reported in your logstash log?
Code: Select all
cat /var/log/logstash/logstash.log
Try appending a new line to one of your logs - I'd like to see if Nagios picks up the append.
Code: Select all
echo "some log line" >> /dns/proxy/access
Re: Local File Input not working
Posted: Wed Sep 09, 2015 3:41 pm
by krobertson71
First off, as stated before, the log files are owned by root.
If you change that to nagios:nagios then I doubt the app will be able to now write to those logs.
Also you are using access* which is not a good idea since you are going to grab the same rotated logs over and over. You should just pull 'access'.
Also, since the log rolls, the new access log that gets created is likely to go back to root:root.
Just my two cents. May be way off.
Re: Local File Input not working
Posted: Wed Sep 09, 2015 3:47 pm
by CFT6Server
I get what you are referring to, but I have a specific purpose for this. I just have some log files I need to input into LS. so putting in in nagios:nagios is fine as this is a copy of the logs and eliminate any permission issues. Since I am grabbing all the logs, I just wanted to use *.
Now it should be able to import the files if it is new, and once imported, it will know the previous location where the logs are last read and will not try to reimport them into LS. At least that's how I think it works?
Re: Local File Input not working
Posted: Thu Sep 10, 2015 11:09 am
by jolson
At least that's how I think it works?
It's my understanding that Logstash keeps track of which files it has already scanned using the file input - and how far it's scanned into each one.
Have you tried appending the new line and looking for errors? It would also be worth restarting logstash to check for initialization problems.
Code: Select all
service logstash restart
tail /var/log/logstash/logstash.log