Upgrading Nagios -- blew up

[gyr]

Hi there:

I am in the process of upgrading NAGIOSXI (2209R1.3g) to the latest, on a 32 bit centos vm. I ran the upgrade script and it ran for a long time bringing down rpms from the different repos compiling and installing. After about 20 min I get this error. I looked through the archives, and no joy. It does not appear to be a proxy error, because It installed for a long time, and the error is 503..

Thanks in advance,

Gene Reynolds

NEW VALUES
--------------
xi_commands_command_id_seq = 2688
xi_events_event_id_seq = 872234
xi_meta_meta_id_seq = 872405
xi_options_option_id_seq = 99
xi_sysstat_sysstat_id_seq = 99
xi_usermeta_usermeta_id_seq = 946
xi_users_user_id_seq = 49

Installing new PNP templates...
Updating init script...
Enabling large install tweaks...
Fixing config file permissions...
Fixing htpasswd permissions...
Checking group memberships...
Installing new XI templates...
URL: http://localhost/nagiosql/index.php
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosql/index.php --no-check-certificate --post-data 'Submit=Login&tfUsername=nagiosxi&tfPassword=l807uv' -O nagiosql.login--2013-12-16 11:45:02-- http://localhost/nagiosql/index.php
Resolving webbalance.inl.gov... xxx
Connecting to webbalance.inl.gov|xxx|:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://xxx [following]
--2013-12-16 11:45:03-- http://xxx/
Connecting to Proxy server|xxx|:8080... connected.
Proxy request sent, awaiting response... 503 Service Unavailable

2013-12-16 11:45:05 ERROR 503: Service Unavailable.

NAGIOSQL LOGIN FAILED!
[root@nagiosxi nagiosxi]#

[sreinhardt]

It looks like you need to set the correct proxy settings for wget and nagios xi. Take a look at this documentation. http://support.nagios.com/wiki/index.ph ... th_Proxies

[gyr]

I think it may be something else. Please look closer at the error. I have been doing proxy work here for years, as has our staff. I/we went through the proxy logs and I can see where it passed the url to my nagiosxi machine. wgetrc, pear and yum, and the system all have our proxy setting, and may be configured correctly. It ran for over 20 min through the proxy doing wgets, yum updates, rpm installs etc. The web server answered with a 503 error which means that it answered. We think anyway. I had to put the old vm back in place as we are down with this install. I will schedule some time take the working vm down, and put this up and we will go over the config files, logs etc and send you an update with all the configurations. Please look at this again, I really appreciate your help.

Thanks again

Gene Reynolds
Senior System Administrator
Idaho National Laboratory
Idaho Falls Idaho

[sreinhardt]

The problem is not that wget, pear and yum have your proxy information, it is that they have not been configured to ignore the proxy for localhost connections. In this case and in many cases throughout XI updates and standard use, such as reporting, wget 127.0.0.1 or localhost is used. If wget is not configured to ignore the proxy, your proxy will not reroute the connection back to the nagios host correctly. You are correct that it got a response, likely from the proxy server on 8080 and not the xi system.

[gyr]

Hi there:

Thanks for helping on this, I finally got to this box and removed the proxy connects as directed
I removed all the proxy setting, and included the no proxy statement(s) to exclude the localhost etc.

Now when I run the upgrade script, I get the following error.

URL: http://localhost/nagiosql/index.php
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosql/index.php --no-check-certificate --post-data 'Submit=Login&tfUsername=nagiosxi&tfPassword=l807uv' -O nagiosql.login--2013-12-23 09:20:40-- http://localhost/nagiosql/index.php
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... failed: Connection refused.
NAGIOSQL LOGIN FAILED!
[root@nagiosxi nagiosxi]# service iptables status

Checking for firewall
URL: http://localhost/nagiosql/index.php
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosql/index.php --no-check-certificate --post-data 'Submit=Login&tfUsername=nagiosxi&tfPassword=l807uv' -O nagiosql.login--2013-12-23 09:20:40-- http://localhost/nagiosql/index.php
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... failed: Connection refused.
NAGIOSQL LOGIN FAILED!

Me checking for firewall
[root@nagiosxi bin]# service iptables status
Firewall is stopped.

Me Looking at netstat does not show anything listening on port 80

tcp 0 0 127.0.0.1:8022 0.0.0.0:* LISTEN 3537/python
udp 0 0 fe80::250:56ff:fea8:eef:123 :::* 3048/ntpd
unix 2 [ ACC ] STREAM LISTENING 8034 2746/audispd /var/run/audispd_events
unix 2 [ ] DGRAM 9980 2934/vasd
unix 3 [ ] STREAM CONNECTED 8025 2744/auditd
unix 3 [ ] STREAM CONNECTED 8024 2746/audispd

Please advise,

Thank you

Gene Reynolds

[sreinhardt]

Good to see that we don't have the 503 any more. Let's check a few other things though:

Code: Select all

getenforce
service httpd status
nmap -p 80,443 localhost

I just wanted to check the selinux status since I noticed auditd, but the current issue since you already checked the firewall, would like be that apache is not started.

[gyr]

Hi again: thank you for the fast reply

Seliux is not running
root@nagiosxi ~]# getenforce
Disabled

I am sure you didnt want all this but here it is anyway.
[root@nagiosxi ~]# netstat -p 80 443
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 localhost.lo:x11-ssh-offset localhost.localdomain:33293 ESTABLISHED 6642/sshd
tcp 0 0 localhost.localdomain:6012 localhost.localdomain:36417 ESTABLISHED 18079/sshd
tcp 0 0 nagiosxi.inl.gov:ssh inl419163.inl.gov:50758 ESTABLISHED 18079/sshd
tcp 0 0 localhost.localdomain:58306 localhost.localdomain:6011 ESTABLISHED 7037/xterm
tcp 0 0 localhost.localdom:postgres localhost.localdomain:51413 TIME_WAIT -
tcp 0 0 nagiosxi.inl.gov:58361 AD3.inel.gov:msft-gc ESTABLISHED 2934/vasd
tcp 0 0 localhost.localdom:postgres localhost.localdomain:51409 TIME_WAIT -
tcp 0 0 localhost.localdom:postgres localhost.localdomain:51408 ESTABLISHED 18846/postgres
tcp 0 0 localhost.localdom:postgres localhost.localdomain:51411 TIME_WAIT -
tcp 0 0 localhost.localdomain:6011 localhost.localdomain:58306 ESTABLISHED 7035/sshd
tcp 0 0 localhost.localdom:postgres localhost.localdomain:51405 ESTABLISHED 18787/postgres
tcp 0 0 localhost.localdom:postgres localhost.localdomain:51404 ESTABLISHED 18784/postgres
tcp 0 0 localhost.localdom:postgres localhost.localdomain:51407 ESTABLISHED 18790/postgres
tcp 0 0 localhost.localdom:postgres localhost.localdomain:51403 ESTABLISHED 18780/postgres
tcp 0 0 nagiosxi.inl.gov:43170 webbalance.inl.gov:webcache ESTABLISHED 17140/python
tcp 0 0 nagiosxi.inl.gov:ssh inl419163.inl.gov:50466 ESTABLISHED 6642/sshd
tcp 0 0 nagiosxi.inl.gov:ssh inl419163.inl.gov:50471 ESTABLISHED 7035/sshd
tcp 0 0 localhost.localdomain:51392 localhost.localdom:postgres TIME_WAIT -
tcp 0 0 localhost.localdomain:51405 localhost.localdom:postgres ESTABLISHED 18765/php
tcp 0 0 localhost.localdomain:51404 localhost.localdom:postgres ESTABLISHED 18763/php
tcp 0 0 localhost.localdomain:51407 localhost.localdom:postgres ESTABLISHED 18761/php
tcp 0 0 localhost.localdomain:51403 localhost.localdom:postgres ESTABLISHED 18757/php
tcp 0 0 localhost.localdomain:51408 localhost.localdom:postgres ESTABLISHED 18759/php
tcp 0 0 localhost.localdomain:36417 localhost.localdomain:6012 ESTABLISHED 18173/xterm
tcp 0 0 localhost.localdomain:33293 localhost.lo:x11-ssh-offset ESTABLISHED 6700/xterm
udp 0 0 localhost.localdomain:54196 localhost.localdomain:54196 ESTABLISHED 3262/postmaster

Apache is running,

[root@nagiosxi ~]# ps -ef | grep -i httpd
root 3328 1 0 08:58 ? 00:00:00 /usr/sbin/httpd
apache 3365 3328 0 08:58 ? 00:00:02 /usr/sbin/httpd
apache 3366 3328 0 08:58 ? 00:00:02 /usr/sbin/httpd
apache 3367 3328 0 08:58 ? 00:00:02 /usr/sbin/httpd
apache 3368 3328 0 08:58 ? 00:00:02 /usr/sbin/httpd
apache 3369 3328 0 08:58 ? 00:00:02 /usr/sbin/httpd
apache 3370 3328 0 08:58 ? 00:00:02 /usr/sbin/httpd
apache 3371 3328 0 08:58 ? 00:00:01 /usr/sbin/httpd
apache 3372 3328 0 08:58 ? 00:00:02 /usr/sbin/httpd
apache 8010 3328 0 09:15 ? 00:00:01 /usr/sbin/httpd
root 19293 18196 0 10:01 pts/2 00:00:00 grep -i httpd

[lmiltchev]

Run the following two commands and show us the output:

Code: Select all

nmap -p 80,443 localhost
ll /usr/local/nagiosxi/scripts/nagiosql

[gyr]

Oh sorry, here you go

[root@nagiosxi conf]# nmap -p 80,443 localhost

Starting Nmap 5.00 ( http://nmap.org ) at 2013-12-23 11:18 MST
Interesting ports on localhost.localdomain (127.0.0.1):
PORT STATE SERVICE
80/tcp closed http
443/tcp open https

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds
[root@nagiosxi conf]#

ll /usr/local/nagiosxi/scripts/nagiosql
ls: /usr/local/nagiosxi/scripts/nagiosql: No such file or directory

find / -name nagiosql
[root@nagiosxi /]# find / -name nagiosql
/etc/nagiosql
/usr/local/nagiosxi/html/includes/components/nagiosql
/usr/admin/gene-upgrade/nagiosxi/subcomponents/nagiosql
/usr/admin/gene-upgrade/nagiosxi/nagiosxi/basedir/html/includes/components/nagiosql
/store/backups/mysql/daily/nagiosql
/store/backups/mysql/weekly/nagiosql
/store/backups/mysql/monthly/nagiosql
/tmp/xicomponents_backup/nagiosql
/var/lib/mysql/nagiosql
/var/www/html/nagiosql

I dont know enough about this, to give you better answers, sorry

[lmiltchev]

I am sorry, I forgot to put the asterisk. Please, run this:

Code: Select all

ll /usr/local/nagiosxi/scripts/nagiosql*

and show the output.

Your port 80 is closed...

80/tcp closed http