Page 3 of 3

Re: Can't start logstash after 2.1 Upgrade

Posted: Thu Aug 06, 2015 12:59 pm
by stecino
jolson wrote:
"version" : "1.6.0",
I wouldn't be concerned with the version of Java in use here - it is the same on my test cluster.
The NLS page reports 2015R1.3
Try running the 'update check' from 'Administration -> Command Subsystem'. Does the version number correct itself?
I am assuming that's the Elastic Search version. Given the upgrade notes, it matches with the version of the ElasticSearch.
JVM has a different field name for version.

I ran the command, but still didn't change

Re: Can't start logstash after 2.1 Upgrade

Posted: Thu Aug 06, 2015 1:17 pm
by jolson
I am assuming that's the Elastic Search version. Given the upgrade notes, it matches with the version of the ElasticSearch.
JVM has a different field name for version.
You are correct.
I ran the command, but still didn't change
When you ran the upgrade script on all of your nodes, are you certain that there were no errors?

When you downloaded the new upgrade script, were you sure to remove any old versions of the script that might have been in your directory? You can check your elasticsearch and logstash versions across your nodes to ensure that they're all up to date:

Code: Select all

/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
/usr/local/nagioslogserver/logstash/bin/logstash --version
You should see output similar to the following:
Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.7.0_75
logstash 1.5.1

Re: Can't start logstash after 2.1 Upgrade

Posted: Thu Aug 06, 2015 2:17 pm
by stecino
jolson wrote:
I am assuming that's the Elastic Search version. Given the upgrade notes, it matches with the version of the ElasticSearch.
JVM has a different field name for version.
You are correct.
I ran the command, but still didn't change
When you ran the upgrade script on all of your nodes, are you certain that there were no errors?

When you downloaded the new upgrade script, were you sure to remove any old versions of the script that might have been in your directory? You can check your elasticsearch and logstash versions across your nodes to ensure that they're all up to date:

Code: Select all

/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
/usr/local/nagioslogserver/logstash/bin/logstash --version
You should see output similar to the following:
Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.7.0_75
logstash 1.5.1
Nope didn't see any errors

Ran them and they match up

/usr/local/nagioslogserver/elasticsearch/bin/elasticsearch -v
Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.7.0_71
# /usr/local/nagioslogserver/logstash/bin/logstash --version
logstash 1.5.1

Re: Can't start logstash after 2.1 Upgrade

Posted: Thu Aug 06, 2015 2:42 pm
by jolson
Interesting. Are there any other problems with your cluster aside from the version displaying improperly? I'm thinking that your jobs may be stuck - give the following a try:

Any errors in your cron log?

Code: Select all

cat /var/log/cron
Try resetting your jobs and re-running the run_update_check command:
Navigate to 'Administration -> Command Subsystem'
Click 'reset all jobs'
Run your 'run_update_check' command once more.

Does the behavior of NLS change?

Re: Can't start logstash after 2.1 Upgrade

Posted: Thu Aug 06, 2015 4:29 pm
by stecino
jolson wrote:Interesting. Are there any other problems with your cluster aside from the version displaying improperly? I'm thinking that your jobs may be stuck - give the following a try:

Any errors in your cron log?

Code: Select all

cat /var/log/cron
Try resetting your jobs and re-running the run_update_check command:
Navigate to 'Administration -> Command Subsystem'
Click 'reset all jobs'
Run your 'run_update_check' command once more.

Does the behavior of NLS change?
I haven't seen any other issues yet.
But here is the snippet

Aug 6 14:20:01 blah CROND[30648]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Aug 6 14:21:01 blah CROND[30741]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:21:01 blah CROND[30743]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:22:01 blah CROND[30832]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:22:01 blah CROND[30833]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:23:02 blah CROND[30962]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:23:02 blah CROND[30964]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:24:01 blah CROND[31114]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:24:01 blah CROND[31115]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:25:01 blah CROND[31206]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:25:01 blah CROND[31207]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:26:01 blah CROND[31298]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)
Aug 6 14:26:01 blah CROND[31299]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:27:01 blah CROND[31400]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1)
Aug 6 14:27:01 blah CROND[31401]: (nagios) CMD (/usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1)

I will reset the jobs

Re: Can't start logstash after 2.1 Upgrade

Posted: Thu Aug 06, 2015 4:31 pm
by jolson
Let me know what you find out - the cron log looks fine.

Re: Can't start logstash after 2.1 Upgrade

Posted: Thu Aug 06, 2015 5:39 pm
by stecino
jolson wrote:Let me know what you find out - the cron log looks fine.
Thanks for all the help, will let you know

Re: Can't start logstash after 2.1 Upgrade

Posted: Fri Aug 07, 2015 9:51 am
by jolson
No problem - talk to you soon.