Page 1 of 4

Logs not reporting again

Posted: Mon Sep 12, 2016 6:53 pm
by kwhogster
Log Serer 1.4.2

restarted the services

Elasticsearch Database [Restart] [Stop]

Logstash Collector [Restart] [Stop]

I show 11 hosts reporting

But I see this when I click on the dashboard

No results There were no results because no indices were found that match your selected time span

Thoughts?


Update

I think after restarting the services it started working again.

I added another host to see and it does not show on the dashboard.

I have 11 hosts now and added one more should have 12 hosts

Is there a way to see what hosts are reporting to logserver?

Re: Logs not reporting again

Posted: Tue Sep 13, 2016 9:28 am
by mcapra
kwhogster wrote:I have 11 hosts now and added one more should have 12 hosts
In terms of how the count is figured, that is the sum of the count of unique hostnames/addresses in the currently open elasticsearch indices. There could be a log source ready to go that simply hasn't sent any logs yet.
Is there a way to see what hosts are reporting to logserver?
There isn't a very good way from the NLS GUI to see which hosts are sending in logs at the moment. When troubleshooting the topic of "is this host sending logs", we usually reference a tcpdump.

Re: Logs not reporting again

Posted: Tue Sep 13, 2016 7:14 pm
by kwhogster
The error is back again

No logs are being report as I posted in my first message

Why dos Logserver stop working like this all the time???

Thank you

Re: Logs not reporting again

Posted: Wed Sep 14, 2016 8:57 am
by mcapra
Can you share the contents of your logstash log (/var/log/logstash/logstash.log) as well as the contents of your elasticsearch logs (/var/log/elasticsearch/*.log)?

Re: Logs not reporting again

Posted: Sat Sep 17, 2016 10:49 am
by kwhogster
Guys

I figured it out

It was a DNS issue.

did a nslookup of this host and it was showing a workstations dns name with the same ip adderess as the log server host

I then went to my dns server and deleted the entries from the A Record and the reverse lookup zones

Now they are reporting

Still would like to know how to check in Log Server which hosts are reporting has to be a config file or something some where

Thoughts

Re: Logs not reporting again

Posted: Sun Sep 18, 2016 10:49 pm
by Box293
Go to Dashboards
Make sure the query is *
Under ALL EVENTS, in the left pane is a Fields list
Click the word host
Now there is a pretty list
Click the Terms drop down list and select Table
Now a "Top 10 terms" table appears
Next to the heading "Top 10 terms in field host" is a compass icon
Drag the compass icon up so it adds it to a new panel
In the new panel, click the gear icon (Configure)
General tab -> change the title to All Hosts
Panel tab -> Change the length from 10 to 1000
Click Save

Now you can save this dashboard so you can view it later.

Re: Logs not reporting again

Posted: Mon Sep 19, 2016 9:12 pm
by kwhogster
Troy

Was going to try your suggestion but now I can not even access the site

I get this



Waiting for Database Startup

It looks like your local elasticsearch service is starting.

Why am I getting this error?

Elasticsearch can take a little while to start up because of it's indexing. This may take a few seconds.

The page will refresh automatically after 5 seconds...


What do I have to restart to clear this

Thanks

Re: Logs not reporting again

Posted: Mon Sep 19, 2016 10:38 pm
by Box293
How much memory does your log server machine have? We find that you need at least 4GB but 8GB is better.

Can you please run these commands:

Code: Select all

tail /var/log/logstash/*.log -n 100 > /tmp/logstash_logs.txt
tail /var/log/elasticsearch/*.log -n 100 > /tmp/elasticsearch_logs.txt
Then upload these files:
/tmp/logstash_logs.txt
/tmp/elasticsearch_logs.txt

Re: Logs not reporting again

Posted: Mon Sep 19, 2016 10:54 pm
by kwhogster
Troy

Running Cent OS
Server has 8GB of Memory


See attached log

the logstash_log file was empty no records

Re: Logs not reporting again

Posted: Tue Sep 20, 2016 1:11 am
by Box293
I suspect you have some failed shards.

What is the output of these commands:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
curl -XGET 'http://localhost:9200/_cat/shards?v'