Can't start logstash after 2.1 Upgrade

[stecino]

"version" : "1.6.0",

I wouldn't be concerned with the version of Java in use here - it is the same on my test cluster.

The NLS page reports 2015R1.3

Try running the 'update check' from 'Administration -> Command Subsystem'. Does the version number correct itself?

I am assuming that's the Elastic Search version. Given the upgrade notes, it matches with the version of the ElasticSearch.
JVM has a different field name for version.

I ran the command, but still didn't change

[jolson]

I am assuming that's the Elastic Search version. Given the upgrade notes, it matches with the version of the ElasticSearch.
JVM has a different field name for version.

You are correct.

I ran the command, but still didn't change

When you ran the upgrade script on all of your nodes, are you certain that there were no errors?

When you downloaded the new upgrade script, were you sure to remove any old versions of the script that might have been in your directory? You can check your elasticsearch and logstash versions across your nodes to ensure that they're all up to date:

Code: Select all

/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
/usr/local/nagioslogserver/logstash/bin/logstash --version

You should see output similar to the following:

Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.7.0_75
logstash 1.5.1

[stecino]

I am assuming that's the Elastic Search version. Given the upgrade notes, it matches with the version of the ElasticSearch.
JVM has a different field name for version.

You are correct.

I ran the command, but still didn't change

When you ran the upgrade script on all of your nodes, are you certain that there were no errors?

When you downloaded the new upgrade script, were you sure to remove any old versions of the script that might have been in your directory? You can check your elasticsearch and logstash versions across your nodes to ensure that they're all up to date:

Code: Select all

/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
/usr/local/nagioslogserver/logstash/bin/logstash --version

You should see output similar to the following:

Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.7.0_75
logstash 1.5.1

Nope didn't see any errors

Ran them and they match up

/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.7.0_71
# /usr/local/nagioslogserver/logstash/bin/logstash --version
logstash 1.5.1

[jolson]

Interesting. Are there any other problems with your cluster aside from the version displaying improperly? I'm thinking that your jobs may be stuck - give the following a try:

Any errors in your cron log?

Code: Select all

cat /var/log/cron

Try resetting your jobs and re-running the run_update_check command:
Navigate to 'Administration -> Command Subsystem'
Click 'reset all jobs'
Run your 'run_update_check' command once more.

Does the behavior of NLS change?

[stecino]

Interesting. Are there any other problems with your cluster aside from the version displaying improperly? I'm thinking that your jobs may be stuck - give the following a try:

Any errors in your cron log?

Code: Select all

cat /var/log/cron

Try resetting your jobs and re-running the run_update_check command:
Navigate to 'Administration -> Command Subsystem'
Click 'reset all jobs'
Run your 'run_update_check' command once more.

Does the behavior of NLS change?

I haven't seen any other issues yet.
But here is the snippet

Aug 6 14:20:01 blah CROND[30648]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Aug 6 14:21:01 blah CROND[30741]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:21:01 blah CROND[30743]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:22:01 blah CROND[30832]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:22:01 blah CROND[30833]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:23:02 blah CROND[30962]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:23:02 blah CROND[30964]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:24:01 blah CROND[31114]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:24:01 blah CROND[31115]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:25:01 blah CROND[31206]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:25:01 blah CROND[31207]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:26:01 blah CROND[31298]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:26:01 blah CROND[31299]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:27:01 blah CROND[31400]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:27:01 blah CROND[31401]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)

I will reset the jobs

[jolson]

Let me know what you find out - the cron log looks fine.

[stecino]

Let me know what you find out - the cron log looks fine.

Thanks for all the help, will let you know

[jolson]

No problem - talk to you soon.