Page 1 of 3

Nagios log server upgrade fails with website doesn't work

Posted: Wed Feb 21, 2018 3:34 pm
by thanigaivel.a
We are trying to upgrade nagios log server from 1.4.4 to 2.0.4, but after the upgrade all services like logstash. elasticsearch and httpd started. however, web page is not working. Also, not able to provide you profile since website not working. so kindly guide us to fix this.

[root@usa0300lv6332 nagioslogserver]# bash upgrade
ERROR: unable to read system id.
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Loaded plugins: product-id, search-disabled-repos
Nothing to do
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Archive: sourceguardian/ixed4.lin.x86-64.zip
inflating: /usr/lib64/php/modules/ixed.5.4.lin
Sourceguardian extension found for PHP version 5.4
Sourceguardian extension already in php.ini

===========================
Nagios Log Server 2 Upgrade
===========================

Upgrading to version 2 requires active maintenance on your license. Please double check before upgrading.

Once upgraded, the Log Server interface will not be usable until you have activated. You can activate do this directly in the interface after upgrade.

You will need the following information to activate:

- Log Server license key
- Account ID or Token

Do you want to continue? [Y/n] y
Proceeding with upgrade...
Upgrading Elasticsearch...
Stopping elasticsearch (via systemctl): [ OK ]
Starting elasticsearch (via systemctl): [ OK ]
Elasticsearch upgraded OK
Upgrading Logstash...
Applying Nagios patches to Logstash...
Logstash upgraded OK
Upgrading Kibana...
Kibana upgraded OK
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Re-applying logstash configuration...
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Restarting elasticsearch (via systemctl): [ OK ]
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0

Nagios Log Server Upgrade Complete!

You can access the Nagios Log Server web interface by visiting:
http://10.15.17.27/nagioslogserver/

[root@usa0300lv6332 nagioslogserver]# service httpd status
Redirecting to /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-02-21 08:13:35 EST; 5min ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 6945 (httpd)
Status: "Total requests: 30; Current requests/sec: 0.4; Current traffic: 1.2KB/sec"
CGroup: /system.slice/httpd.service
├─6945 /usr/sbin/httpd -DFOREGROUND
├─6946 /usr/sbin/httpd -DFOREGROUND
├─6947 /usr/sbin/httpd -DFOREGROUND
├─6948 /usr/sbin/httpd -DFOREGROUND
├─6950 /usr/sbin/httpd -DFOREGROUND
├─6951 /usr/sbin/httpd -DFOREGROUND
├─7288 /usr/sbin/httpd -DFOREGROUND
├─7289 /usr/sbin/httpd -DFOREGROUND
├─7290 /usr/sbin/httpd -DFOREGROUND
├─9049 /usr/sbin/httpd -DFOREGROUND
├─9059 /usr/sbin/httpd -DFOREGROUND
└─9060 /usr/sbin/httpd -DFOREGROUND

Feb 21 08:13:35 usa0300lv6332 systemd[1]: Starting The Apache HTTP Server...
Feb 21 08:13:35 usa0300lv6332 httpd[6945]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.15.17.27. Set the 'ServerName' directive globally to suppress this message
Feb 21 08:13:35 usa0300lv6332 systemd[1]: Started The Apache HTTP Server.
Feb 21 08:14:28 usa0300lv6332 sudo[7261]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Feb 21 08:14:28 usa0300lv6332 sudo[7260]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status


[root@usa0300lv6332 logs]# wget https://usa0300lv6332.sdsp.mc.xerox.com ... logserver/
--2018-02-21 11:03:23-- https://usa0300lv6332.sdsp.mc.xerox.com ... logserver/
Resolving www2.mc.xerox.com (www2.mc.xerox.com)... 13.131.1.242
Connecting to www2.mc.xerox.com (www2.mc.xerox.com)|13.131.1.242|:8000... connected.
Proxy request sent, awaiting response... 500 Internal Server Error
2018-02-21 11:03:23 ERROR 500: Internal Server Error.

[root@usa0300lv6332 logs]# service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (running) since Wed 2018-02-21 08:16:15 EST; 2h 51min ago
Docs: man:systemd-sysv-generator(8)
Process: 7536 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 7570 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─7597 java -Xms6926m -Xmx6926m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicit...

Feb 21 08:16:15 usa0300lv6332 systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Feb 21 08:16:15 usa0300lv6332 elasticsearch[7570]: PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Feb 21 08:16:15 usa0300lv6332 runuser[7588]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Feb 21 08:16:15 usa0300lv6332 runuser[7588]: pam_unix(runuser:session): session closed for user nagios
Feb 21 08:16:15 usa0300lv6332 elasticsearch[7570]: Starting elasticsearch: [ OK ]
Feb 21 08:16:15 usa0300lv6332 systemd[1]: Started LSB: This service manages the elasticsearch daemon.
[root@usa0300lv6332 logs]#
[root@usa0300lv6332 logs]#

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Wed Feb 21, 2018 4:48 pm
by kyang
Did you follow our upgrade documentation? Which one? Or did you upgrade using the UI?

I'm attaching a script that can create a NLS profile through the command.

Simply create a new file to store the script in /tmp

Give it permissions and then execute.

Code: Select all

chmod 755 profile.sh
./profile.sh
Please send me the profile after this.
You may have to use something called FileZilla to get the profile off.

After you PM the profile please update this thread, unless you post the profile on here. Thanks!
profile.sh

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Fri Feb 23, 2018 8:26 am
by thanigaivel.a
kindly the attached profile.

[root@usa0300lv6332 nagios]# ./profile.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 626 0 --:--:-- --:--:-- --:--:-- 632
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 9293 0 --:--:-- --:--:-- --:--:-- 10125
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 3630 0 --:--:-- --:--:-- --:--:-- 3521
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 5237 0 --:--:-- --:--:-- --:--:-- 5400
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 9599 0 --:--:-- --:--:-- --:--:-- 10125
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 3983 0 --:--:-- --:--:-- --:--:-- 4050
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 9735 0 --:--:-- --:--:-- --:--:-- 10125
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 9760 0 --:--:-- --:--:-- --:--:-- 10125
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 3246 0 --:--:-- --:--:-- --:--:-- 3375
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 6264 0 --:--:-- --:--:-- --:--:-- 6750
Gem::LoadError: You have already activated minitar 0.6.1, but your Gemfile requires minitar 0.5.4. Prepending `bundle exec` to your command may solve this.
setup at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/runtime.rb:34
each at org/jruby/RubyArray.java:1613
each at /usr/local/nagioslogserver/logstash/vendor/jruby/lib/ruby/1.9/forwardable.rb:201
setup at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/runtime.rb:19
setup at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler.rb:122
setup! at /usr/local/nagioslogserver/logstash/lib/bootstrap/bundler.rb:64
execute at /usr/local/nagioslogserver/logstash/lib/pluginmanager/list.rb:17
run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
execute at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11
run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132
(root) at /usr/local/nagioslogserver/logstash/lib/pluginmanager/main.rb:43
cp: omitting directory ‘/var/log/logstash’
cp: omitting directory ‘/var/log/elasticsearch’
eno16777984: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.15.17.27 netmask 255.255.255.0 broadcast 10.15.17.255
ether 00:50:56:ae:0e:7b txqueuelen 1000 (Ethernet)
RX packets 36641115 bytes 43240396841 (40.2 GiB)
RX errors 0 dropped 142 overruns 0 frame 0
TX packets 9139677 bytes 621726030 (592.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1 (Local Loopback)
RX packets 2229274 bytes 21027809653 (19.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2229274 bytes 21027809653 (19.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

FirewallD is not running
./profile.sh: line 29: ll: command not found
sysprofile/
sysprofile/masters.txt
sysprofile/nodes.txt
sysprofile/shard-health.txt
sysprofile/shard-status.txt
sysprofile/indices-status.txt
sysprofile/jvm-status.txt
sysprofile/cluster.txt
sysprofile/pending-tasks.txt
sysprofile/recovery.txt
sysprofile/plugins.txt
sysprofile/logstash-plugins.txt
sysprofile/memory.txt
sysprofile/psaux.txt
sysprofile/netstat.txt
sysprofile/disk.txt
sysprofile/network.txt
sysprofile/iptables.txt
sysprofile/firewalld.txt
sysprofile/logstash-confd.txt
sysprofile/httpd-access.log
sysprofile/httpd-error.log
sysprofile/systemlog.txt

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Fri Feb 23, 2018 12:16 pm
by scottwilkerson
Can you post your /etc/php.ini

I believe it contains extra sourceguardian lines
thanigaivel.a wrote:ERROR: unable to read system id.
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Loaded plugins: product-id, search-disabled-repos
Nothing to do
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Archive: sourceguardian/ixed4.lin.x86-64.zip
inflating: /usr/lib64/php/modules/ixed.5.4.lin
Sourceguardian extension found for PHP version 5.4
Sourceguardian extension already in php.ini

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Fri Feb 23, 2018 1:15 pm
by thanigaivel.a
Attached the /etc/php.ini file

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Fri Feb 23, 2018 1:28 pm
by scottwilkerson
This line should be removed from the bottom of the file

Code: Select all

extension=ixed.5.4.lin

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Fri Feb 23, 2018 1:51 pm
by thanigaivel.a
Had removed the last line and restarted the below services, but still website not working. kindly help us on priority to fix the issue.

[root@usa0300lv6332 ~]#
[root@usa0300lv6332 ~]# service elasticsearch restart
Restarting elasticsearch (via systemctl): [ OK ]
[root@usa0300lv6332 ~]# service logstash restart
Restarting logstash (via systemctl): [ OK ]
[root@usa0300lv6332 ~]# service httpd restart
Redirecting to /bin/systemctl restart httpd.service

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Fri Feb 23, 2018 3:28 pm
by scottwilkerson
could you run the following and return the results

Code: Select all

tail -200 /var/log/httpd/error_log

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Mon Feb 26, 2018 9:35 am
by thanigaivel.a
Had uploaded the entire /var/log/httpd directory, kindly check the same and provide your direction to fix the issue.

Re: Nagios log server upgrade fails with website doesn't wor

Posted: Mon Feb 26, 2018 11:08 am
by scottwilkerson
your error logs are showing the following

Code: Select all

[Fri Feb 23 13:47:08.985240 2018] [:error] [pid 105631] [client 10.15.23.164:52764] PHP Fatal error:  require_once(): Failed opening required '/var/www/html/nagioslogserver/system/core/CodeIgniter.php' (include_path='.:/usr/share/pear:/usr/share/php') in /var/www/html/nagioslogserver/www/index.php on line 315
Please share the output of

Code: Select all

ls -al /var/www/html/nagioslogserver/system/core/CodeIgniter.php
ls -al /var/www/html/nagioslogserver/system/
ls -al /var/www/html/nagioslogserver/