Nagios Support Forum

Hello LOG support

Logstash Collector service stops working (twice in the past 2 days). It start OK from system status when discovered to be stopped.
How to check what was the reason?

Thanks,
Dimitri

What's the output of a tail -n100 /var/log/logstash/logstash.log command?

hsmith wrote:What's the output of a tail -n100 /var/log/logstash/logstash.log command?

There is none (0 size)

My DB is growing 12-15 GB / day for the past 2 days (use to be 5-20 MB a day before that)
How to check abusing log sources? (sources with largest log sizes)

If there's no logstash log, that concerns me. It would be good to find out why it's stopping. What's the output of an ls -l /var/log/logstash command? Your number one concern is going to be making sure your machine doesn't fill up. You can lose all of your data if that happens.

Also, let's give this command a shot:
This is going to tell us what's taking space up on your system.

hsmith wrote:If there's no logstash log, that concerns me. It would be good to find out why it's stopping. What's the output of an ls -l /var/log/logstash command? Your number one concern is going to be making sure your machine doesn't fill up. You can lose all of your data if that happens.

Also, let's give this command a shot:

Code: Select all

find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

This is going to tell us what's taking space up on your system.

Here is command output, including disk space info above

From the Dashboards page of NLS, click the 'host' field on the left side, and then click the 'Terms' button, select 'Table'.

This will show you which host is sending so many logs.

rkennedy wrote:From the Dashboards page of NLS, click the 'host' field on the left side, and then click the 'Terms' button, select 'Table'.

This will show you which host is sending so many logs.

Thank you, got it

No problem - let us know if you need any further assistance.

rkennedy wrote:No problem - let us know if you need any further assistance.

Found host and dealt with large log file (deleted). No more collector stops.
- is there a practical log size limit when sent to LOG server (or similar issues) contributing to collector stops (reported in the past maybe?)

Shouldn't be. How much RAM does your server have?