Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Waiting for ElasticSearch to Start

https://support.nagios.com/forum/viewtopic.php?t=34070

Page 1 of 2

Waiting for ElasticSearch to Start

Posted: Thu Aug 06, 2015 1:24 am
by fluxbox
Hi,

I have the same error stated here: https://support.nagios.com/forum/viewto ... 37&t=33804
Below is the output of the log.

# tail -n50 /var/log/elasticsearch/*.log
==> /var/log/elasticsearch/81453311-bb4c-4ab2-9625-24e452d638ff_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/81453311-bb4c-4ab2-9625-24e452d638ff_index_search_slowlog.log <==

==> /var/log/elasticsearch/81453311-bb4c-4ab2-9625-24e452d638ff.log <==
[2015-08-06 05:30:40,442][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] started
[2015-08-06 05:30:40,469][INFO ][gateway ] [a052f70c-040e-4074-aac7-99273813abed] recovered [0] indices into cluster_state
[2015-08-06 02:05:40,376][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopping ...
[2015-08-06 02:05:40,481][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopped
[2015-08-06 02:05:40,481][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closing ...
[2015-08-06 02:05:40,535][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closed
[2015-08-06 02:05:45,025][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] version[1.6.0], pid[4366], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-08-06 02:05:45,025][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initializing ...
[2015-08-06 02:05:45,088][INFO ][plugins ] [a052f70c-040e-4074-aac7-99273813abed] loaded [knapsack-1.5.2.0-f340ad1], sites []
[2015-08-06 02:05:45,261][INFO ][env ] [a052f70c-040e-4074-aac7-99273813abed] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [95gb], net total_space [98.4gb], types [rootfs]
[2015-08-06 02:05:49,652][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initialized
[2015-08-06 02:05:49,652][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] starting ...
[2015-08-06 02:05:49,830][INFO ][transport ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.162.136.41:9300]}
[2015-08-06 02:05:49,843][INFO ][discovery ] [a052f70c-040e-4074-aac7-99273813abed] 81453311-bb4c-4ab2-9625-24e452d638ff/OMcdTIFcTDyN6RuPRrpdAg
[2015-08-06 02:05:52,878][INFO ][cluster.service ] [a052f70c-040e-4074-aac7-99273813abed] new_master [a052f70c-040e-4074-aac7-99273813abed][OMcdTIFcTDyN6RuPRrpdAg][phailewdlx-clsp][inet[/10.162.136.41:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-08-06 02:05:52,902][INFO ][http ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:1:9200]}, publish_address {inet[localhost/0:0:0:0:0:0:0:1:9200]}
[2015-08-06 02:05:52,902][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] started
[2015-08-06 02:05:52,924][INFO ][gateway ] [a052f70c-040e-4074-aac7-99273813abed] recovered [0] indices into cluster_state
[2015-08-06 02:06:28,933][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopping ...
[2015-08-06 02:06:28,951][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopped
[2015-08-06 02:06:28,951][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closing ...
[2015-08-06 02:06:28,957][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closed
[2015-08-06 02:06:41,745][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] version[1.6.0], pid[4813], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-08-06 02:06:41,745][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initializing ...
[2015-08-06 02:06:41,759][INFO ][plugins ] [a052f70c-040e-4074-aac7-99273813abed] loaded [knapsack-1.5.2.0-f340ad1], sites []
[2015-08-06 02:06:41,813][INFO ][env ] [a052f70c-040e-4074-aac7-99273813abed] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [95gb], net total_space [98.4gb], types [rootfs]
[2015-08-06 02:06:45,101][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initialized
[2015-08-06 02:06:45,101][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] starting ...
[2015-08-06 02:06:45,220][INFO ][transport ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.162.136.41:9300]}
[2015-08-06 02:06:45,234][INFO ][discovery ] [a052f70c-040e-4074-aac7-99273813abed] 81453311-bb4c-4ab2-9625-24e452d638ff/D9GApkNJTCCe1KH99lNq7w
[2015-08-06 02:06:48,359][INFO ][cluster.service ] [a052f70c-040e-4074-aac7-99273813abed] new_master [a052f70c-040e-4074-aac7-99273813abed][D9GApkNJTCCe1KH99lNq7w][phailewdlx-clsp][inet[/10.162.136.41:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-08-06 02:06:48,434][INFO ][http ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:1:9200]}, publish_address {inet[localhost/0:0:0:0:0:0:0:1:9200]}
[2015-08-06 02:06:48,434][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] started
[2015-08-06 02:06:48,466][INFO ][gateway ] [a052f70c-040e-4074-aac7-99273813abed] recovered [0] indices into cluster_state
[2015-08-06 02:12:24,281][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopping ...
[2015-08-06 02:12:24,307][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopped
[2015-08-06 02:12:24,307][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closing ...
[2015-08-06 02:12:24,318][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closed
[2015-08-06 02:12:27,035][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] version[1.6.0], pid[6210], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-08-06 02:12:27,035][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initializing ...
[2015-08-06 02:12:27,048][INFO ][plugins ] [a052f70c-040e-4074-aac7-99273813abed] loaded [knapsack-1.5.2.0-f340ad1], sites []
[2015-08-06 02:12:27,103][INFO ][env ] [a052f70c-040e-4074-aac7-99273813abed] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [95gb], net total_space [98.4gb], types [rootfs]
[2015-08-06 02:12:30,741][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initialized
[2015-08-06 02:12:30,741][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] starting ...
[2015-08-06 02:12:30,859][INFO ][transport ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.162.136.41:9300]}
[2015-08-06 02:12:30,873][INFO ][discovery ] [a052f70c-040e-4074-aac7-99273813abed] 81453311-bb4c-4ab2-9625-24e452d638ff/9awIZTzATxWA2SFcrRfsrw
[2015-08-06 02:12:33,957][INFO ][cluster.service ] [a052f70c-040e-4074-aac7-99273813abed] new_master [a052f70c-040e-4074-aac7-99273813abed][9awIZTzATxWA2SFcrRfsrw][phailewdlx-clsp][inet[/10.162.136.41:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-08-06 02:12:34,000][INFO ][http ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:1:9200]}, publish_address {inet[localhost/0:0:0:0:0:0:0:1:9200]}
[2015-08-06 02:12:34,001][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] started
[2015-08-06 02:12:34,032][INFO ][gateway ] [a052f70c-040e-4074-aac7-99273813abed] recovered [0] indices into cluster_state
[root@phailewdlx-clsp ~]#

Please advise.
Thanks.

Re: Waiting for ElasticSearch to Start

Posted: Thu Aug 06, 2015 9:48 am
by jolson
Your logs look normal to me - what are the symptoms you're experiencing? How many instances are in your cluster, and did you just finish upgrading them?

Let's check on the health of your indices:

Code: Select all

curl 'localhost:9200/_cluster/health?level=indices&pretty'

Re: Waiting for ElasticSearch to Start

Posted: Thu Aug 06, 2015 10:27 pm
by fluxbox
Hi,
I am running the VM instance from NLS download page.

Code: Select all

curl 'localhost:9200/_cluster/health?level=indices&pretty'
{
  "cluster_name" : "81453311-bb4c-4ab2-9625-24e452d638ff",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "indices" : { }
}

Re: Waiting for ElasticSearch to Start

Posted: Fri Aug 07, 2015 9:22 am
by jolson
Alright - is there any chance that you changed the default data directory of Nagios Log Server? According to the output you've just posted, your cluster is missing all of its default indices.

Re: Waiting for ElasticSearch to Start

Posted: Sun Aug 09, 2015 9:48 pm
by fluxbox
Hi,
How do I check? This is the VM image build that I loaded to the VMWARE.
Thanks.

Re: Waiting for ElasticSearch to Start

Posted: Sun Aug 09, 2015 10:19 pm
by Box293
Can you run this command and post the output:

Code: Select all

cat /var/www/html/nagioslogserver/application/config/config.local.php

Re: Waiting for ElasticSearch to Start

Posted: Mon Aug 10, 2015 6:06 pm
by fluxbox
This is the output of the command.

Code: Select all

 cat /var/www/html/nagioslogserver/application/config/config.local.php
<?php
// Local Configuration Options
// - These options are not overwritten during upgrades

$config['charset'] = 'UTF-8';
$config['admin_email'] = 'nagioslogserver@localhost'; // Used as default SEND email if one hasn't been set
$config['dns_server'] = '';

// Make sure this is unique and difficult to guess
$config['encryption_key'] = sha1($_SERVER['HTTP_HOST']);

// Base url of site& media files directory
$config['site_url'] = '//'.$_SERVER['HTTP_HOST'].'/nagioslogserver';
$config['media_url'] = $config['site_url'] .'/media';

// Backend locations
$config['backend_dir'] = '/usr/local/nagioslogserver';
$config['scripts_dir'] = $config['backend_dir'].'/scripts';
$config['node_uuid_file'] = $config['backend_dir'].'/var/node_uuid';
$config['cluster_uuid_file'] = $config['backend_dir'].'/var/cluster_uuid';
$config['hosts_file'] = $config['backend_dir'].'/var/cluster_hosts';

// Backend constants
$node_id = trim(file_get_contents($config['node_uuid_file']));
$cluster_id = trim(file_get_contents($config['cluster_uuid_file']));
define("NODE", $node_id);
define("CLUSTER", $cluster_id);

// User area settings
$config['min_password_length'] = 8;
$config['forgot_password_expiration'] = 0;

Re: Waiting for ElasticSearch to Start

Posted: Mon Aug 10, 2015 7:45 pm
by Box293
Can you also run these commands please:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
df -h
df -i

Re: Waiting for ElasticSearch to Start

Posted: Tue Aug 11, 2015 12:05 am
by fluxbox
Here are the output of the commands.

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 30 Jul 2015 09:14:28 -0400
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => '81453311-bb4c-4ab2-9625-24e452d638ff'
        host => 'localhost'
        document_type => '%{type}'
        node_name => ''
        protocol => 'transport'
        workers => 4
    }
}

#
# Global outputs
#



#
# Local outputs
#

df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           99G  2.5G   95G   3% /
devtmpfs        995M  148K  995M   1% /dev
tmpfs          1003M     0 1003M   0% /dev/shm
/dev/sda1        99G  2.5G   95G   3% /

df -i
Filesystem      Inodes IUsed   IFree IUse% Mounted on
rootfs         6502400 86014 6416386    2% /
devtmpfs        254650   576  254074    1% /dev
tmpfs           256738     1  256737    1% /dev/shm
/dev/sda1      6502400 86014 6416386    2% /

Re: Waiting for ElasticSearch to Start

Posted: Tue Aug 11, 2015 9:58 am
by jolson
How do I check? This is the VM image build that I loaded to the VMWARE.
To check, we can run the following:

Code: Select all

cat /etc/sysconfig/elasticsearch
The following concerns me:
curl 'localhost:9200/_cluster/health?level=indices&pretty'
{
"cluster_name" : "81453311-bb4c-4ab2-9625-24e452d638ff",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"indices" : { }
}
It concerns me because based on the above information, you have *no* indices in your cluster - which means no data. "indices" : { }

I am thinking that there must be something wrong with how elasticsearch is picking up your data. Run the following commands please:

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_uuid
ls -l /usr/local/nagioslogserver/elasticsearch/data/
find /usr/local/nagioslogserver/elasticsearch/data/*/nodes/0/ -maxdepth 1 -exec ls -l "{}" \;
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited