Waiting for ElasticSearch to Start

[fluxbox]

Hi,

I have the same error stated here: https://support.nagios.com/forum/viewto ... 37&t=33804
Below is the output of the log.

# tail -n50 /var/log/elasticsearch/*.log
==> /var/log/elasticsearch/81453311-bb4c-4ab2-9625-24e452d638ff_index_indexing_slowlog.log <==

==> /var/log/elasticsearch/81453311-bb4c-4ab2-9625-24e452d638ff_index_search_slowlog.log <==

==> /var/log/elasticsearch/81453311-bb4c-4ab2-9625-24e452d638ff.log <==
[2015-08-06 05:30:40,442][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] started
[2015-08-06 05:30:40,469][INFO ][gateway ] [a052f70c-040e-4074-aac7-99273813abed] recovered [0] indices into cluster_state
[2015-08-06 02:05:40,376][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopping ...
[2015-08-06 02:05:40,481][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopped
[2015-08-06 02:05:40,481][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closing ...
[2015-08-06 02:05:40,535][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closed
[2015-08-06 02:05:45,025][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] version[1.6.0], pid[4366], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-08-06 02:05:45,025][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initializing ...
[2015-08-06 02:05:45,088][INFO ][plugins ] [a052f70c-040e-4074-aac7-99273813abed] loaded [knapsack-1.5.2.0-f340ad1], sites []
[2015-08-06 02:05:45,261][INFO ][env ] [a052f70c-040e-4074-aac7-99273813abed] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [95gb], net total_space [98.4gb], types [rootfs]
[2015-08-06 02:05:49,652][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initialized
[2015-08-06 02:05:49,652][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] starting ...
[2015-08-06 02:05:49,830][INFO ][transport ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.162.136.41:9300]}
[2015-08-06 02:05:49,843][INFO ][discovery ] [a052f70c-040e-4074-aac7-99273813abed] 81453311-bb4c-4ab2-9625-24e452d638ff/OMcdTIFcTDyN6RuPRrpdAg
[2015-08-06 02:05:52,878][INFO ][cluster.service ] [a052f70c-040e-4074-aac7-99273813abed] new_master [a052f70c-040e-4074-aac7-99273813abed][OMcdTIFcTDyN6RuPRrpdAg][phailewdlx-clsp][inet[/10.162.136.41:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-08-06 02:05:52,902][INFO ][http ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:1:9200]}, publish_address {inet[localhost/0:0:0:0:0:0:0:1:9200]}
[2015-08-06 02:05:52,902][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] started
[2015-08-06 02:05:52,924][INFO ][gateway ] [a052f70c-040e-4074-aac7-99273813abed] recovered [0] indices into cluster_state
[2015-08-06 02:06:28,933][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopping ...
[2015-08-06 02:06:28,951][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopped
[2015-08-06 02:06:28,951][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closing ...
[2015-08-06 02:06:28,957][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closed
[2015-08-06 02:06:41,745][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] version[1.6.0], pid[4813], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-08-06 02:06:41,745][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initializing ...
[2015-08-06 02:06:41,759][INFO ][plugins ] [a052f70c-040e-4074-aac7-99273813abed] loaded [knapsack-1.5.2.0-f340ad1], sites []
[2015-08-06 02:06:41,813][INFO ][env ] [a052f70c-040e-4074-aac7-99273813abed] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [95gb], net total_space [98.4gb], types [rootfs]
[2015-08-06 02:06:45,101][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initialized
[2015-08-06 02:06:45,101][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] starting ...
[2015-08-06 02:06:45,220][INFO ][transport ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.162.136.41:9300]}
[2015-08-06 02:06:45,234][INFO ][discovery ] [a052f70c-040e-4074-aac7-99273813abed] 81453311-bb4c-4ab2-9625-24e452d638ff/D9GApkNJTCCe1KH99lNq7w
[2015-08-06 02:06:48,359][INFO ][cluster.service ] [a052f70c-040e-4074-aac7-99273813abed] new_master [a052f70c-040e-4074-aac7-99273813abed][D9GApkNJTCCe1KH99lNq7w][phailewdlx-clsp][inet[/10.162.136.41:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-08-06 02:06:48,434][INFO ][http ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:1:9200]}, publish_address {inet[localhost/0:0:0:0:0:0:0:1:9200]}
[2015-08-06 02:06:48,434][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] started
[2015-08-06 02:06:48,466][INFO ][gateway ] [a052f70c-040e-4074-aac7-99273813abed] recovered [0] indices into cluster_state
[2015-08-06 02:12:24,281][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopping ...
[2015-08-06 02:12:24,307][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] stopped
[2015-08-06 02:12:24,307][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closing ...
[2015-08-06 02:12:24,318][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] closed
[2015-08-06 02:12:27,035][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] version[1.6.0], pid[6210], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-08-06 02:12:27,035][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initializing ...
[2015-08-06 02:12:27,048][INFO ][plugins ] [a052f70c-040e-4074-aac7-99273813abed] loaded [knapsack-1.5.2.0-f340ad1], sites []
[2015-08-06 02:12:27,103][INFO ][env ] [a052f70c-040e-4074-aac7-99273813abed] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [95gb], net total_space [98.4gb], types [rootfs]
[2015-08-06 02:12:30,741][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] initialized
[2015-08-06 02:12:30,741][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] starting ...
[2015-08-06 02:12:30,859][INFO ][transport ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.162.136.41:9300]}
[2015-08-06 02:12:30,873][INFO ][discovery ] [a052f70c-040e-4074-aac7-99273813abed] 81453311-bb4c-4ab2-9625-24e452d638ff/9awIZTzATxWA2SFcrRfsrw
[2015-08-06 02:12:33,957][INFO ][cluster.service ] [a052f70c-040e-4074-aac7-99273813abed] new_master [a052f70c-040e-4074-aac7-99273813abed][9awIZTzATxWA2SFcrRfsrw][phailewdlx-clsp][inet[/10.162.136.41:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-08-06 02:12:34,000][INFO ][http ] [a052f70c-040e-4074-aac7-99273813abed] bound_address {inet[/0:0:0:0:0:0:0:1:9200]}, publish_address {inet[localhost/0:0:0:0:0:0:0:1:9200]}
[2015-08-06 02:12:34,001][INFO ][node ] [a052f70c-040e-4074-aac7-99273813abed] started
[2015-08-06 02:12:34,032][INFO ][gateway ] [a052f70c-040e-4074-aac7-99273813abed] recovered [0] indices into cluster_state
[root@phailewdlx-clsp ~]#

Please advise.
Thanks.

[jolson]

Your logs look normal to me - what are the symptoms you're experiencing? How many instances are in your cluster, and did you just finish upgrading them?

Let's check on the health of your indices:

Code: Select all

curl 'localhost:9200/_cluster/health?level=indices&pretty'

[fluxbox]

Hi,
I am running the VM instance from NLS download page.

Code: Select all

curl 'localhost:9200/_cluster/health?level=indices&pretty'
{
  "cluster_name" : "81453311-bb4c-4ab2-9625-24e452d638ff",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "indices" : { }
}

[jolson]

Alright - is there any chance that you changed the default data directory of Nagios Log Server? According to the output you've just posted, your cluster is missing all of its default indices.

[fluxbox]

Hi,
How do I check? This is the VM image build that I loaded to the VMWARE.
Thanks.

[Box293]

Can you run this command and post the output:

Code: Select all

cat /var/www/html/nagioslogserver/application/config/config.local.php

[fluxbox]

This is the output of the command.

Code: Select all

 cat /var/www/html/nagioslogserver/application/config/config.local.php
<?php
// Local Configuration Options
// - These options are not overwritten during upgrades

$config['charset'] = 'UTF-8';
$config['admin_email'] = 'nagioslogserver@localhost'; // Used as default SEND email if one hasn't been set
$config['dns_server'] = '';

// Make sure this is unique and difficult to guess
$config['encryption_key'] = sha1($_SERVER['HTTP_HOST']);

// Base url of site& media files directory
$config['site_url'] = '//'.$_SERVER['HTTP_HOST'].'/nagioslogserver';
$config['media_url'] = $config['site_url'] .'/media';

// Backend locations
$config['backend_dir'] = '/usr/local/nagioslogserver';
$config['scripts_dir'] = $config['backend_dir'].'/scripts';
$config['node_uuid_file'] = $config['backend_dir'].'/var/node_uuid';
$config['cluster_uuid_file'] = $config['backend_dir'].'/var/cluster_uuid';
$config['hosts_file'] = $config['backend_dir'].'/var/cluster_hosts';

// Backend constants
$node_id = trim(file_get_contents($config['node_uuid_file']));
$cluster_id = trim(file_get_contents($config['cluster_uuid_file']));
define("NODE", $node_id);
define("CLUSTER", $cluster_id);

// User area settings
$config['min_password_length'] = 8;
$config['forgot_password_expiration'] = 0;

[Box293]

Can you also run these commands please:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
df -h
df -i

[fluxbox]

Here are the output of the commands.

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 30 Jul 2015 09:14:28 -0400
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => '81453311-bb4c-4ab2-9625-24e452d638ff'
        host => 'localhost'
        document_type => '%{type}'
        node_name => ''
        protocol => 'transport'
        workers => 4
    }
}

#
# Global outputs
#



#
# Local outputs
#

df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           99G  2.5G   95G   3% /
devtmpfs        995M  148K  995M   1% /dev
tmpfs          1003M     0 1003M   0% /dev/shm
/dev/sda1        99G  2.5G   95G   3% /

df -i
Filesystem      Inodes IUsed   IFree IUse% Mounted on
rootfs         6502400 86014 6416386    2% /
devtmpfs        254650   576  254074    1% /dev
tmpfs           256738     1  256737    1% /dev/shm
/dev/sda1      6502400 86014 6416386    2% /

[jolson]

How do I check? This is the VM image build that I loaded to the VMWARE.

To check, we can run the following:

Code: Select all

cat /etc/sysconfig/elasticsearch

The following concerns me:

curl 'localhost:9200/_cluster/health?level=indices&pretty'
{
"cluster_name" : "81453311-bb4c-4ab2-9625-24e452d638ff",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"indices" : { }
}

It concerns me because based on the above information, you have *no* indices in your cluster - which means no data. "indices" : { }

I am thinking that there must be something wrong with how elasticsearch is picking up your data. Run the following commands please:

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_uuid
ls -l /usr/local/nagioslogserver/elasticsearch/data/
find /usr/local/nagioslogserver/elasticsearch/data/*/nodes/0/ -maxdepth 1 -exec ls -l "{}" \;