Elasticsearch error in logstash.log

[nms]

Hi,

We noticed the below output when logging in the syslog gui:

2019-01-25_1019.png

When logged in terminal I observed that the root partition was 100% full. I deleted 2018's files -> logstash-2018.xx.xx files from:

Code: Select all

/usr/local/nagioslogserver/elasticsearch/data/9b2ec30e-7553-4aa8-b3b5-660d8a7cf3e1/nodes/0/indices

After i noticed that the syslog was slow takes ages to login. After a total system reboot, the situation was the same, but i noticed that in the /var/log/logstash/logstash.log file I had the following errors:

Code: Select all

tail -f /var/log/logstash/logstash.log
{:timestamp=>"2019-01-25T10:09:06.001000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2019-01-25T10:09:06.073000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2019-01-25T10:09:06.538000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}

httpd. logstash and elasiticsearch are all running:

Code: Select all

[root@am1-sha-syslog1-p indices]# service httpd status
httpd (pid  1770) is running...
[root@am1-sha-syslog1-p indices]# service logstash status
Logstash Daemon (pid  1806) is running...
[root@am1-sha-syslog1-p indices]# service elasticsearch status
elasticsearch (pid  1964) is running...

How can I rectify the issue?

[cdienger]

There's likely some corruption with the configuration indices that hold elasticsearch settings. NLS creates a backup of the configuration that we can try restoring from. https://assets.nagios.com/downloads/nag ... erver.pdf​ has some details but basically config backups are stored in /store/backups/nagioslogserver/​ and restore_backup.sh can be run to import old configs:

cd /usr/local/nagioslogserver/scripts/
./restore_backup.sh /store/backups/nagioslogserver/nagioslogserver.<timestamp>.tar.gz

To help avoid filling the drive in the future, I would recommend taking a look over our document to move the database to a different partition if the one it is on currently doesn't have enough space:

https://assets.nagios.com/downloads/nag ... Server.pdf

Also these docs on storing older indices on a remote server:

https://assets.nagios.com/downloads/nag ... enance.pdf
https://assets.nagios.com/downloads/nag ... ations.pdf

[nms]

Hi,

By restoring a previous backup it seems it's fine however I now encountered a very strange problem.
it seems that the user "nagiosadmin" does not exist anymore. We tried to log in with this user and it keeps stating user does not exist.
I tried to simply run a script to reset the password but it still complains that the user does not exist.

Code: Select all

/usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh --password newpassword

2019-01-29_0953.png

Is there a way to re-create the default user without losing anything (i.e. avoid a re-installation?). Am trying to avoid a new installation and restore from backup if this is not the only way.

Rgds,

Matthew

[cdienger]

The backup would contain users and credentials so the fact that the nagiosadmin account appears to be missing is a bit worrisome. I would try other backup configs if you have them. Hopefully one of them will be a good one.

Failing that, we can create a new user from the command line:

Code: Select all

curl -XPUT 'http://localhost:9200/nagioslogserver/user/99' -d '{"username":"someuser","password":"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19","auth_type":"admin","email":"user@example.com","language":"default","apiaccess":"1","apikey":"1396e08757545557073844695e5b64caa0bd3ad3","created":"2015-01-23 10:00:00","created_by":0,"default_dashboard":"/dashboard/elasticsearch/default"}'

This will create a user named "someuser" with the password "nagiosls". You can use this account to log in and optionally create the accounts that you need, then delete the "someuser" account as one of the other users when done.

[nms]

Thanks!

I created the user then added the nagiosadmin user.

Rgds,

Matt

[cdienger]

Did you have anything further related to this or are we okay locking this up?

[nms]

Hi,

Yes, unfortunately, we have. I see logstash not running.

2019-01-31_1052.png

After I perform a restart, it starts but after a while, it goes down again.

2019-01-31_1052_001.png

Profile attached.

Rgds,

Matt

[cdienger]

The logstash config files were wiped out. You may have backups of these under Configure > Config Snapshots. If not, then you'll need to create the inputs under Configure > Global (All Instances) > Global Config. There are four inputs by default:

Code: Select all

        syslog {
            type => 'syslog'
            port => 5544
        }

Code: Select all

        tcp {
            type => 'eventlog'
            port => 3515
            codec => json {
                charset => 'CP1252'
            }
        }

Customer inputs, filters and outputs would need to be recreated.

Code: Select all

        tcp {
            type => 'import_raw'
            tags => 'import_raw'
            port => 2056
        }

Code: Select all

		tcp {
            type => 'import_json'
            tags => 'import_json'
            port => 2057
            codec => json
        }

and one filter:

Code: Select all

      if [program] == 'apache_access' {
            grok {
                match => [ 'message', '%{COMBINEDAPACHELOG}']
            }
            date {
                match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
            }
            mutate {
                replace => [ 'type', 'apache_access' ]
                 convert => [ 'bytes', 'integer' ]
                 convert => [ 'response', 'integer' ]
            }
        }

        if [program] == 'apache_error' {
            grok {
                match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
            }
            mutate {
                replace => [ 'type', 'apache_error' ]
            }
        }

There is an output as well, but that remained intact.

Custom inputs, filters, and outputs would need to be recreated.

[nms]

Thank you.

I have prepared those filters and all's fine.

You can kindly close the ticket.

Rgds

[scottwilkerson]

Thank you.

I have prepared those filters and all's fine.

You can kindly close the ticket.

Rgds

Glad it is resolved!\

Locking thread