Nagios log server upgrade fails with website doesn't work

[thanigaivel.a]

We are trying to upgrade nagios log server from 1.4.4 to 2.0.4, but after the upgrade all services like logstash. elasticsearch and httpd started. however, web page is not working. Also, not able to provide you profile since website not working. so kindly guide us to fix this.

[root@usa0300lv6332 nagioslogserver]# bash upgrade
ERROR: unable to read system id.
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Loaded plugins: product-id, search-disabled-repos
Nothing to do
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Archive: sourceguardian/ixed4.lin.x86-64.zip
inflating: /usr/lib64/php/modules/ixed.5.4.lin
Sourceguardian extension found for PHP version 5.4
Sourceguardian extension already in php.ini

===========================
Nagios Log Server 2 Upgrade
===========================

Upgrading to version 2 requires active maintenance on your license. Please double check before upgrading.

Once upgraded, the Log Server interface will not be usable until you have activated. You can activate do this directly in the interface after upgrade.

You will need the following information to activate:

- Log Server license key
- Account ID or Token

Do you want to continue? [Y/n] y
Proceeding with upgrade...
Upgrading Elasticsearch...
Stopping elasticsearch (via systemctl): [ OK ]
Starting elasticsearch (via systemctl): [ OK ]
Elasticsearch upgraded OK
Upgrading Logstash...
Applying Nagios patches to Logstash...
Logstash upgraded OK
Upgrading Kibana...
Kibana upgraded OK
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Re-applying logstash configuration...
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Restarting elasticsearch (via systemctl): [ OK ]
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0

Nagios Log Server Upgrade Complete!

You can access the Nagios Log Server web interface by visiting:
http://10.15.17.27/nagioslogserver/

[root@usa0300lv6332 nagioslogserver]# service httpd status
Redirecting to /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-02-21 08:13:35 EST; 5min ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 6945 (httpd)
Status: "Total requests: 30; Current requests/sec: 0.4; Current traffic: 1.2KB/sec"
CGroup: /system.slice/httpd.service
├─6945 /usr/sbin/httpd -DFOREGROUND
├─6946 /usr/sbin/httpd -DFOREGROUND
├─6947 /usr/sbin/httpd -DFOREGROUND
├─6948 /usr/sbin/httpd -DFOREGROUND
├─6950 /usr/sbin/httpd -DFOREGROUND
├─6951 /usr/sbin/httpd -DFOREGROUND
├─7288 /usr/sbin/httpd -DFOREGROUND
├─7289 /usr/sbin/httpd -DFOREGROUND
├─7290 /usr/sbin/httpd -DFOREGROUND
├─9049 /usr/sbin/httpd -DFOREGROUND
├─9059 /usr/sbin/httpd -DFOREGROUND
└─9060 /usr/sbin/httpd -DFOREGROUND

Feb 21 08:13:35 usa0300lv6332 systemd[1]: Starting The Apache HTTP Server...
Feb 21 08:13:35 usa0300lv6332 httpd[6945]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.15.17.27. Set the 'ServerName' directive globally to suppress this message
Feb 21 08:13:35 usa0300lv6332 systemd[1]: Started The Apache HTTP Server.
Feb 21 08:14:28 usa0300lv6332 sudo[7261]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch status
Feb 21 08:14:28 usa0300lv6332 sudo[7260]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/logstash status


[root@usa0300lv6332 logs]# wget https://usa0300lv6332.sdsp.mc.xerox.com ... logserver/
--2018-02-21 11:03:23-- https://usa0300lv6332.sdsp.mc.xerox.com ... logserver/
Resolving www2.mc.xerox.com (www2.mc.xerox.com)... 13.131.1.242
Connecting to www2.mc.xerox.com (www2.mc.xerox.com)|13.131.1.242|:8000... connected.
Proxy request sent, awaiting response... 500 Internal Server Error
2018-02-21 11:03:23 ERROR 500: Internal Server Error.

[root@usa0300lv6332 logs]# service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (running) since Wed 2018-02-21 08:16:15 EST; 2h 51min ago
Docs: man:systemd-sysv-generator(8)
Process: 7536 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 7570 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─7597 java -Xms6926m -Xmx6926m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicit...

Feb 21 08:16:15 usa0300lv6332 systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
Feb 21 08:16:15 usa0300lv6332 elasticsearch[7570]: PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Feb 21 08:16:15 usa0300lv6332 runuser[7588]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
Feb 21 08:16:15 usa0300lv6332 runuser[7588]: pam_unix(runuser:session): session closed for user nagios
Feb 21 08:16:15 usa0300lv6332 elasticsearch[7570]: Starting elasticsearch: [ OK ]
Feb 21 08:16:15 usa0300lv6332 systemd[1]: Started LSB: This service manages the elasticsearch daemon.
[root@usa0300lv6332 logs]#
[root@usa0300lv6332 logs]#

[kyang]

Did you follow our upgrade documentation? Which one? Or did you upgrade using the UI?

I'm attaching a script that can create a NLS profile through the command.

Simply create a new file to store the script in /tmp

Give it permissions and then execute.

Code: Select all

chmod 755 profile.sh
./profile.sh

Please send me the profile after this.
You may have to use something called FileZilla to get the profile off.

After you PM the profile please update this thread, unless you post the profile on here. Thanks!

profile.sh

[thanigaivel.a]

kindly the attached profile.

[root@usa0300lv6332 nagios]# ./profile.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 626 0 --:--:-- --:--:-- --:--:-- 632
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 9293 0 --:--:-- --:--:-- --:--:-- 10125
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 3630 0 --:--:-- --:--:-- --:--:-- 3521
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 5237 0 --:--:-- --:--:-- --:--:-- 5400
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 9599 0 --:--:-- --:--:-- --:--:-- 10125
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 3983 0 --:--:-- --:--:-- --:--:-- 4050
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 9735 0 --:--:-- --:--:-- --:--:-- 10125
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 9760 0 --:--:-- --:--:-- --:--:-- 10125
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 3246 0 --:--:-- --:--:-- --:--:-- 3375
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 0 81 0 0 6264 0 --:--:-- --:--:-- --:--:-- 6750
Gem::LoadError: You have already activated minitar 0.6.1, but your Gemfile requires minitar 0.5.4. Prepending `bundle exec` to your command may solve this.
setup at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/runtime.rb:34
each at org/jruby/RubyArray.java:1613
each at /usr/local/nagioslogserver/logstash/vendor/jruby/lib/ruby/1.9/forwardable.rb:201
setup at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/runtime.rb:19
setup at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler.rb:122
setup! at /usr/local/nagioslogserver/logstash/lib/bootstrap/bundler.rb:64
execute at /usr/local/nagioslogserver/logstash/lib/pluginmanager/list.rb:17
run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
execute at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11
run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
run at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132
(root) at /usr/local/nagioslogserver/logstash/lib/pluginmanager/main.rb:43
cp: omitting directory ‘/var/log/logstash’
cp: omitting directory ‘/var/log/elasticsearch’
eno16777984: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.15.17.27 netmask 255.255.255.0 broadcast 10.15.17.255
ether 00:50:56:ae:0e:7b txqueuelen 1000 (Ethernet)
RX packets 36641115 bytes 43240396841 (40.2 GiB)
RX errors 0 dropped 142 overruns 0 frame 0
TX packets 9139677 bytes 621726030 (592.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1 (Local Loopback)
RX packets 2229274 bytes 21027809653 (19.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2229274 bytes 21027809653 (19.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

FirewallD is not running
./profile.sh: line 29: ll: command not found
sysprofile/
sysprofile/masters.txt
sysprofile/nodes.txt
sysprofile/shard-health.txt
sysprofile/shard-status.txt
sysprofile/indices-status.txt
sysprofile/jvm-status.txt
sysprofile/cluster.txt
sysprofile/pending-tasks.txt
sysprofile/recovery.txt
sysprofile/plugins.txt
sysprofile/logstash-plugins.txt
sysprofile/memory.txt
sysprofile/psaux.txt
sysprofile/netstat.txt
sysprofile/disk.txt
sysprofile/network.txt
sysprofile/iptables.txt
sysprofile/firewalld.txt
sysprofile/logstash-confd.txt
sysprofile/httpd-access.log
sysprofile/httpd-error.log
sysprofile/systemlog.txt

[scottwilkerson]

Can you post your /etc/php.ini

I believe it contains extra sourceguardian lines

ERROR: unable to read system id.
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Loaded plugins: product-id, search-disabled-repos
Nothing to do
PHP Warning: Module 'SourceGuardian' already loaded in Unknown on line 0
Archive: sourceguardian/ixed4.lin.x86-64.zip
inflating: /usr/lib64/php/modules/ixed.5.4.lin
Sourceguardian extension found for PHP version 5.4
Sourceguardian extension already in php.ini

[thanigaivel.a]

Attached the /etc/php.ini file

[scottwilkerson]

This line should be removed from the bottom of the file

Code: Select all

extension=ixed.5.4.lin

[thanigaivel.a]

Had removed the last line and restarted the below services, but still website not working. kindly help us on priority to fix the issue.

[root@usa0300lv6332 ~]#
[root@usa0300lv6332 ~]# service elasticsearch restart
Restarting elasticsearch (via systemctl): [ OK ]
[root@usa0300lv6332 ~]# service logstash restart
Restarting logstash (via systemctl): [ OK ]
[root@usa0300lv6332 ~]# service httpd restart
Redirecting to /bin/systemctl restart httpd.service

[scottwilkerson]

could you run the following and return the results

Code: Select all

tail -200 /var/log/httpd/error_log

[thanigaivel.a]

Had uploaded the entire /var/log/httpd directory, kindly check the same and provide your direction to fix the issue.

[scottwilkerson]

your error logs are showing the following

Code: Select all

[Fri Feb 23 13:47:08.985240 2018] [:error] [pid 105631] [client 10.15.23.164:52764] PHP Fatal error:  require_once(): Failed opening required '/var/www/html/nagioslogserver/system/core/CodeIgniter.php' (include_path='.:/usr/share/pear:/usr/share/php') in /var/www/html/nagioslogserver/www/index.php on line 315

Please share the output of

Code: Select all

ls -al /var/www/html/nagioslogserver/system/core/CodeIgniter.php
ls -al /var/www/html/nagioslogserver/system/
ls -al /var/www/html/nagioslogserver/