Logstash Collector stops working

[dlukinski]

Hello LOG support

Logstash Collector service stops working (twice in the past 2 days). It start OK from system status when discovered to be stopped.
How to check what was the reason?

Thanks,
Dimitri

[hsmith]

What's the output of a tail -n100 /var/log/logstash/logstash.log command?

[dlukinski]

What's the output of a tail -n100 /var/log/logstash/logstash.log command?

There is none (0 size)

My DB is growing 12-15 GB / day for the past 2 days (use to be 5-20 MB a day before that)
How to check abusing log sources? (sources with largest log sizes)

[hsmith]

If there's no logstash log, that concerns me. It would be good to find out why it's stopping. What's the output of an ls -l /var/log/logstash command? Your number one concern is going to be making sure your machine doesn't fill up. You can lose all of your data if that happens.

Also, let's give this command a shot:

Code: Select all

find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

This is going to tell us what's taking space up on your system.

[dlukinski]

If there's no logstash log, that concerns me. It would be good to find out why it's stopping. What's the output of an ls -l /var/log/logstash command? Your number one concern is going to be making sure your machine doesn't fill up. You can lose all of your data if that happens.

Also, let's give this command a shot:

Code: Select all

find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

This is going to tell us what's taking space up on your system.

Here is command output, including disk space info above

[rkennedy]

From the Dashboards page of NLS, click the 'host' field on the left side, and then click the 'Terms' button, select 'Table'.

This will show you which host is sending so many logs.

[dlukinski]

From the Dashboards page of NLS, click the 'host' field on the left side, and then click the 'Terms' button, select 'Table'.

This will show you which host is sending so many logs.

Thank you, got it

[rkennedy]

No problem - let us know if you need any further assistance.

[dlukinski]

No problem - let us know if you need any further assistance.

Found host and dealt with large log file (deleted). No more collector stops.
- is there a practical log size limit when sent to LOG server (or similar issues) contributing to collector stops (reported in the past maybe?)

[hsmith]

Shouldn't be. How much RAM does your server have?

Code: Select all

free -m