Page 1 of 1
Elasticsearch error in logstash.log
Posted: Fri Jan 25, 2019 4:28 am
by nms
Hi,
We noticed the below output when logging in the syslog gui:
2019-01-25_1019.png
When logged in terminal I observed that the root partition was 100% full. I deleted 2018's files -> logstash-2018.xx.xx files from:
Code: Select all
/usr/local/nagioslogserver/elasticsearch/data/9b2ec30e-7553-4aa8-b3b5-660d8a7cf3e1/nodes/0/indices
After i noticed that the syslog was slow takes ages to login. After a total system reboot, the situation was the same, but i noticed that in the /var/log/logstash/logstash.log file I had the following errors:
Code: Select all
tail -f /var/log/logstash/logstash.log
{:timestamp=>"2019-01-25T10:09:06.001000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2019-01-25T10:09:06.073000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2019-01-25T10:09:06.538000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
httpd. logstash and elasiticsearch are all running:
Code: Select all
[root@am1-sha-syslog1-p indices]# service httpd status
httpd (pid 1770) is running...
[root@am1-sha-syslog1-p indices]# service logstash status
Logstash Daemon (pid 1806) is running...
[root@am1-sha-syslog1-p indices]# service elasticsearch status
elasticsearch (pid 1964) is running...
How can I rectify the issue?
Re: Elasticsearch error in logstash.log
Posted: Fri Jan 25, 2019 10:38 am
by cdienger
There's likely some corruption with the configuration indices that hold elasticsearch settings. NLS creates a backup of the configuration that we can try restoring from.
https://assets.nagios.com/downloads/nag ... erver.pdf has some details but basically config backups are stored in /store/backups/nagioslogserver/ and restore_backup.sh can be run to import old configs:
cd /usr/local/nagioslogserver/scripts/
./restore_backup.sh /store/backups/nagioslogserver/nagioslogserver.<timestamp>.tar.gz
To help avoid filling the drive in the future, I would recommend taking a look over our document to move the database to a different partition if the one it is on currently doesn't have enough space:
https://assets.nagios.com/downloads/nag ... Server.pdf
Also these docs on storing older indices on a remote server:
https://assets.nagios.com/downloads/nag ... enance.pdf
https://assets.nagios.com/downloads/nag ... ations.pdf
Re: Elasticsearch error in logstash.log
Posted: Tue Jan 29, 2019 3:57 am
by nms
Hi,
By restoring a previous backup it seems it's fine however I now encountered a very strange problem.
it seems that the user "nagiosadmin" does not exist anymore. We tried to log in with this user and it keeps stating user does not exist.
I tried to simply run a script to reset the password but it still complains that the user does not exist.
Code: Select all
/usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh --password newpassword
2019-01-29_0953.png
Is there a way to re-create the default user without losing anything (i.e. avoid a re-installation?). Am trying to avoid a new installation and restore from backup if this is not the only way.
Rgds,
Matthew
Re: Elasticsearch error in logstash.log
Posted: Tue Jan 29, 2019 3:53 pm
by cdienger
The backup would contain users and credentials so the fact that the nagiosadmin account appears to be missing is a bit worrisome. I would try other backup configs if you have them. Hopefully one of them will be a good one.
Failing that, we can create a new user from the command line:
Code: Select all
curl -XPUT 'http://localhost:9200/nagioslogserver/user/99' -d '{"username":"someuser","password":"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19","auth_type":"admin","email":"[email protected]","language":"default","apiaccess":"1","apikey":"1396e08757545557073844695e5b64caa0bd3ad3","created":"2015-01-23 10:00:00","created_by":0,"default_dashboard":"/dashboard/elasticsearch/default"}'
This will create a user named "someuser" with the password "nagiosls". You can use this account to log in and optionally create the accounts that you need, then delete the "someuser" account as one of the other users when done.
Re: Elasticsearch error in logstash.log
Posted: Wed Jan 30, 2019 5:39 am
by nms
Thanks!
I created the user then added the nagiosadmin user.
Rgds,
Matt
Re: Elasticsearch error in logstash.log
Posted: Wed Jan 30, 2019 2:22 pm
by cdienger
Did you have anything further related to this or are we okay locking this up?
Re: Elasticsearch error in logstash.log
Posted: Thu Jan 31, 2019 5:02 am
by nms
Hi,
Yes, unfortunately, we have. I see logstash not running.
2019-01-31_1052.png
After I perform a restart, it starts but after a while, it goes down again.
2019-01-31_1052_001.png
Profile attached.
Rgds,
Matt
Re: Elasticsearch error in logstash.log
Posted: Thu Jan 31, 2019 3:18 pm
by cdienger
The logstash config files were wiped out. You may have backups of these under Configure > Config Snapshots. If not, then you'll need to create the inputs under Configure > Global (All Instances) > Global Config. There are four inputs by default:
Code: Select all
syslog {
type => 'syslog'
port => 5544
}
Code: Select all
tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}
Customer inputs, filters and outputs would need to be recreated.
Code: Select all
tcp {
type => 'import_raw'
tags => 'import_raw'
port => 2056
}
Code: Select all
tcp {
type => 'import_json'
tags => 'import_json'
port => 2057
codec => json
}
and one filter:
Code: Select all
if [program] == 'apache_access' {
grok {
match => [ 'message', '%{COMBINEDAPACHELOG}']
}
date {
match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
}
mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
}
}
if [program] == 'apache_error' {
grok {
match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
}
mutate {
replace => [ 'type', 'apache_error' ]
}
}
There is an output as well, but that remained intact.
Custom inputs, filters, and outputs would need to be recreated.
Re: Elasticsearch error in logstash.log
Posted: Tue Feb 19, 2019 7:53 am
by nms
Thank you.
I have prepared those filters and all's fine.
You can kindly close the ticket.
Rgds
Re: Elasticsearch error in logstash.log
Posted: Tue Feb 19, 2019 8:10 am
by scottwilkerson
nms wrote:Thank you.
I have prepared those filters and all's fine.
You can kindly close the ticket.
Rgds
Glad it is resolved!\
Locking thread