Nagios log server disk space fulling

[mejokj]

Hi,

I am facing an issue with nagios log server the log file /var/log/logstash is taking too much size intermediately and server disk space becomes 100 %.

Is there any way to prevent this log file from filling the disk space?

Also Please give the configuration file to change the log file path for /var/log/logstash logs, I need to change log file path to another secondary disk

[cdienger]

/etc/init.d/logstash controls where the logstash log is stored vi the LS_LOG_FILE option(line 41). After making changes to the location, apply them with:

Code: Select all

service logstash restart

These files usually don't get too big unless there is a problem. I'd suggest reviewing the log to see what is causing the to grow so large.

[mejokj]

Hello.

Yesterday I have cleared the logstash.log. upon checking today, it is again growing to 134GB. On checking the log I could see the below log.

So I have opened the index on that date (today). Why this index is not automatically open, any solution?

++++++++++++++++++++++++++++++
"SourceModuleType"=>"im_msvistalog", "message"=>"The handle to an object was closed.\r\n\r\nSubject :\r\n\tSecurity ID:\t\tS-1-5-21-527237240-1303643608-725345543-4384521\r\n\tAccount Name:\t\tsrvadmin\r\n\tAccount Domain:\t\tAEIFFCONET\r\n\tLogon ID:\t\t0x19074675b\r\n\r\nObject:\r\n\tObject Server:\t\tSecurity\r\n\tHandle ID:\t\t0x6bdc\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x4\r\n\tProcess Name:\t\t", "@version"=>"1", "@timestamp"=>"2019-12-17T00:00:03.949Z", "host"=>"10.1.1.14", "port"=>56091, "type"=>"eventlog"}, "type"]}>, @data={"EventTime"=>"2019-12-17 04:01:17", "Hostname"=>"AESHCOFP02.ae.iffconet.com", "Keywords"=>-9214364837600034816, "EventType"=>"AUDIT_SUCCESS", "SeverityValue"=>2, "Severity"=>"INFO", "EventID"=>4658, "SourceName"=>"Microsoft-Windows-Security-Auditing", "ProviderGuid"=>"{54849625-5478-4994-A5BA-3E3B0328C30D}", "Version"=>0, "Task"=>12800, "OpcodeValue"=>0, "RecordNumber"=>2550387309, "ProcessID"=>4, "ThreadID"=>92, "Channel"=>"Security", "Category"=>"File System", "Opcode"=>"Info", "SubjectUserSid"=>"S-1-5-21-527237240-1303643608-725345543-4384521", "SubjectUserName"=>"srvadmin", "SubjectDomainName"=>"AEIFFCONET", "SubjectLogonId"=>"0x19074675b", "ObjectServer"=>"Security", "HandleId"=>"0x6bdc", "EventReceivedTime"=>"2019-12-17 04:01:18", "SourceModuleName"=>"eventlog", "SourceModuleType"=>"im_msvistalog", "message"=>"The handle to an object was closed.\r\n\r\nSubject :\r\n\tSecurity ID:\t\tS-1-5-21-527237240-1303643608-725345543-4384521\r\n\tAccount Name:\t\tsrvadmin\r\n\tAccount Domain:\t\tAEIFFCONET\r\n\tLogon ID:\t\t0x19074675b\r\n\r\nObject:\r\n\tObject Server:\t\tSecurity\r\n\tHandle ID:\t\t0x6bdc\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x4\r\n\tProcess Name:\t\t", "@version"=>"1", "@timestamp"=>"2019-12-17T00:00:03.949Z", "host"=>"10.1.1.14", "port"=>56091, "type"=>"eventlog"}, @metadata_accessors=#<LogStash::Util::Accessors:0x4cafa9a9 @store={}, @lut={}>, @cancelled=false>], :response=>{"index"=>{"_index"=>"logstash-2019.12.17", "_type"=>"eventlog", "_id"=>nil, "status"=>403, "error"=>"IndexClosedException[[logstash-2019.12.17] closed]"}}, :level=>:warn}
++++++++++++++++++++++++++++++++++

[scottwilkerson]

What is the status of the logstash-2019.12.17 index? I see the following in the output

Code: Select all

"IndexClosedException[[logstash-2019.12.17] closed]"

So it appear that the index is closed

[mejokj]

Hello,

The "IndexClosedException[[logstash-2019.12.17] closed]" index is closed state, So I have opened the index by using curl.

But today also the same issue again every day I need to open the index manually. every day morning the current index shows closed in the dashboard it shows an index of the current date is closed, But in the Admin >> index status shows its opening.

Also, no logs are coming to the Nagios log server until it is opened.

[scottwilkerson]

What are the settings on Admin -> Snapshot & Maintenance ?

Also, is the date on the server correct?

[mejokj]

Hello,

Upon checking the date of server date and Nagios date setting are correct.
I have attached the snapshot and maintenance setting. Also, I have sent you the Nagios log server profile separately in your chat.

[scottwilkerson]

Looking through the profile you sent, you have a bunch of indexes with future dates that are already closed.
such as

Code: Select all

logstash-2019.12.20
logstash-2019.12.21

These should either be opened or deleted.

Closing indexes with future dates will cause the issue you are seeing because when that date occurs the logs will not be able to go into the index.

[mejokj]

Hi Team,

Currently we could not see any indexes with future date in closed state. but still the issue persists.

Kindly help us to find the solution for this issue.

Thanks

[scottwilkerson]

Hi Team,

Currently we could not see any indexes with future date in closed state. but still the issue persists.

Kindly help us to find the solution for this issue.

Thanks

Are you still seeing error messages where the index is closed?