Page 2 of 2
Re: Recurrent errors in Apache http log
Posted: Tue May 03, 2016 10:33 am
by vmesquita
The requested file:
Code: Select all
# TYPE DATABASE USER CIDR-ADDRESS METHOD
# "local" is for Unix domain socket connections only
#local all all ident sameuser
local all all trust
# IPv4 local connections:
#host all all 127.0.0.1/32 ident sameuser
host all all 127.0.0.1/32 trust
# IPv6 local connections:
#host all all ::1/128 ident sameuser
host all all ::1/128 trust
There are some comment lines above those which are not relevant.
Re: Recurrent errors in Apache http log
Posted: Tue May 03, 2016 3:58 pm
by ssax
That looks fine, what is the output of these commands:
Also, please attach this file:
Code: Select all
/var/lib/pgsql/data/postgresql.conf
Was this a fresh install of XI 5+ or was it upgraded from a pre-XI 5 (2012,2014, etc) system?
Is everything else running fine, you're just seeing that error in logs?
Re: Recurrent errors in Apache http log
Posted: Tue May 03, 2016 4:27 pm
by vmesquita
Code: Select all
# netstat -ano | grep 5432
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 ::1:5432 :::* LISTEN off (0.00/0/0)
tcp 0 0 ::1:46106 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:45946 ::1:5432 TIME_WAIT timewait (9.03/0/0)
tcp 0 0 ::1:46102 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:46119 ::1:5432 TIME_WAIT timewait (51.41/0/0)
tcp 0 0 ::1:5432 ::1:46105 ESTABLISHED keepalive (7187.42/0/0)
tcp 0 0 ::1:42946 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 39 ::1:5432 ::1:42074 ESTABLISHED on (0.20/0/0)
tcp 0 220 ::1:5432 ::1:43034 ESTABLISHED on (0.19/0/0)
tcp 0 0 ::1:5432 ::1:42946 ESTABLISHED keepalive (6431.04/0/0)
tcp 0 0 ::1:5432 ::1:45047 ESTABLISHED keepalive (6940.91/0/0)
tcp 0 0 ::1:45998 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:43499 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:43192 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:43774 ESTABLISHED keepalive (6621.03/0/0)
tcp 0 0 ::1:5432 ::1:46106 ESTABLISHED keepalive (7187.42/0/0)
tcp 0 0 ::1:46023 ::1:5432 TIME_WAIT timewait (29.26/0/0)
tcp 0 0 ::1:49777 ::1:5432 TIME_WAIT timewait (44.28/0/0)
tcp 0 0 ::1:45864 ::1:5432 TIME_WAIT timewait (48.65/0/0)
tcp 0 0 ::1:45867 ::1:5432 TIME_WAIT timewait (48.64/0/0)
tcp 0 0 ::1:42074 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:43192 ESTABLISHED keepalive (6484.09/0/0)
tcp 0 0 ::1:46110 ::1:5432 TIME_WAIT timewait (48.74/0/0)
tcp 0 0 ::1:42003 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:46105 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:46103 ESTABLISHED keepalive (7187.40/0/0)
tcp 0 0 ::1:42056 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:46088 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:41673 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 100 0 ::1:42064 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:46107 ::1:5432 TIME_WAIT timewait (48.53/0/0)
tcp 0 0 ::1:46112 ::1:5432 TIME_WAIT timewait (48.75/0/0)
tcp 0 0 ::1:5432 ::1:44579 ESTABLISHED keepalive (6821.73/0/0)
tcp 0 0 ::1:5432 ::1:45862 TIME_WAIT timewait (48.49/0/0)
tcp 0 0 ::1:44718 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:46074 TIME_WAIT timewait (39.10/0/0)
tcp 0 0 ::1:5432 ::1:42003 ESTABLISHED keepalive (6226.34/0/0)
tcp 0 0 ::1:5432 ::1:44718 ESTABLISHED keepalive (6850.75/0/0)
tcp 0 0 ::1:5432 ::1:46102 ESTABLISHED keepalive (7187.38/0/0)
tcp 0 161 ::1:5432 ::1:42064 ESTABLISHED on (0.20/0/0)
tcp 0 0 ::1:45047 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:46109 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:43034 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:41673 ESTABLISHED keepalive (6149.29/0/0)
tcp 0 0 ::1:5432 ::1:45983 ESTABLISHED keepalive (7158.95/0/0)
tcp 0 0 ::1:45983 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:43499 ESTABLISHED keepalive (6551.03/0/0)
tcp 0 0 ::1:45868 ::1:5432 TIME_WAIT timewait (49.25/0/0)
tcp 0 0 ::1:46103 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:43774 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:39234 ESTABLISHED keepalive (5568.61/0/0)
tcp 0 0 ::1:44579 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:39234 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:45322 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:42030 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:42030 ESTABLISHED keepalive (6227.77/0/0)
tcp 0 0 ::1:44045 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:45865 ::1:5432 TIME_WAIT timewait (50.87/0/0)
tcp 0 0 ::1:42033 ::1:5432 TIME_WAIT timewait (24.21/0/0)
tcp 0 0 ::1:5432 ::1:46088 ESTABLISHED keepalive (7182.43/0/0)
tcp 0 0 ::1:5432 ::1:46109 ESTABLISHED keepalive (7187.41/0/0)
tcp 0 0 ::1:5432 ::1:45998 ESTABLISHED keepalive (7162.19/0/0)
tcp 0 0 ::1:58236 ::1:5432 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::1:5432 ::1:44045 ESTABLISHED keepalive (6677.36/0/0)
tcp 0 220 ::1:5432 ::1:45322 ESTABLISHED on (0.18/0/0)
tcp 0 0 ::1:5432 ::1:42056 ESTABLISHED keepalive (6229.34/0/0)
tcp 0 0 ::1:46104 ::1:5432 TIME_WAIT timewait (48.45/0/0)
tcp 0 0 ::1:46131 ::1:5432 TIME_WAIT timewait (56.06/0/0)
unix 2 [ ACC ] STREAM LISTENING 20421 /tmp/.s.PGSQL.5432
# sestatus
SELinux status: disabled
Re: Recurrent errors in Apache http log
Posted: Tue May 03, 2016 4:40 pm
by tmcdonald
ssax wrote:Was this a fresh install of XI 5+ or was it upgraded from a pre-XI 5 (2012,2014, etc) system?
Is everything else running fine, you're just seeing that error in logs?
Can you please address the other two questions from
@ssax?
Re: Recurrent errors in Apache http log
Posted: Tue May 03, 2016 4:47 pm
by lmiltchev
Can you run the following commands and show the output?
Code: Select all
df -h
service postgresql stop
killall postgresql
service postgresql start
service postgresql status
tail /var/lib/pgsql/pgstartup.log
ls -lad /var/lib/pgsql
ls -la /var/lib/pgsql
This is a "long shot" but also show the output of the command below:
Code: Select all
ls -la /usr/local/nagiosxi/scripts/nagiosql*
BTW, have you tried running vacuum on postgres?
Code: Select all
echo "vacuum;vacuum analyze;vacuum full;"|psql nagiosxi postgres
Re: Recurrent errors in Apache http log
Posted: Wed May 04, 2016 10:51 am
by vmesquita
So here's the output of the series of commands:
Code: Select all
[root@nagios vmesquita]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_nagios-lv_root
36G 27G 7.6G 78% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sdd1 477M 72M 380M 16% /boot
tmpfs 50M 7.1M 43M 15% /var/nagiosramdisk
[root@nagios vmesquita]# service postgresql stop
Stopping postgresql service: [ OK ]
[root@nagios vmesquita]# killall postgresql
postgresql: no process killed
[root@nagios vmesquita]# service postgresql start
Starting postgresql service: [ OK ]
[root@nagios vmesquita]# service postgresql status
postmaster (pid 18959) is running...
[root@nagios vmesquita]# tail /var/lib/pgsql/pgstartup.log
Success. You can now start the database server using:
/usr/bin/postgres -D /var/lib/pgsql/data
or
/usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start
LOG: could not create IPv6 socket: Address family not supported by protocol
LOG: could not create IPv6 socket: Address family not supported by protocol
could not write to log file: No space left on device
[root@nagios vmesquita]# ls -lad /var/lib/pgsql
drwx------. 4 postgres postgres 4096 May 2 19:01 /var/lib/pgsql
[root@nagios vmesquita]# ls -la /var/lib/pgsql
total 24
drwx------. 4 postgres postgres 4096 May 2 19:01 .
drwxr-xr-x. 33 root root 4096 May 4 03:37 ..
drwx------. 2 postgres postgres 4096 Mar 30 2015 backups
-rw-r--r-- 1 postgres postgres 85 Mar 30 2015 .bash_profile
drwx------. 12 postgres postgres 4096 May 4 12:38 data
-rw-------. 1 postgres postgres 1359 Apr 26 00:00 pgstartup.log
Now the ls output:
Code: Select all
-rwxrwxr-x. 1 nagios nagios 161 May 4 12:08 /usr/local/nagiosxi/scripts/nagiosql.cookies
-rwxrwxr-x. 1 nagios nagios 271666 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_defaults.sql
-rwxrwxr-x. 1 nagios nagios 39900 Jan 28 17:17 /usr/local/nagiosxi/scripts/nagiosql.delete.contact
-rwxrwxr-x. 1 nagios nagios 885 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_contact.php
-rwxrwxr-x. 1 nagios nagios 1927 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_host.php
-rwxrwxr-x. 1 nagios nagios 446 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_object.sh
-rwxrwxr-x. 1 nagios nagios 64566 Oct 26 2015 /usr/local/nagiosxi/scripts/nagiosql.delete.service
-rwxrwxr-x. 1 nagios nagios 2756 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_service.php
-rwxrwxr-x. 1 nagios nagios 41292 Jan 28 17:17 /usr/local/nagiosxi/scripts/nagiosql.delete.timeperiod
-rwxrwxr-x. 1 nagios nagios 886 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_timeperiod.php
-rwxrwxr-x. 1 nagios nagios 7400 Jul 1 2014 /usr/local/nagiosxi/scripts/nagiosql.export.additional
-rwxrwxr-x. 1 nagios nagios 1272 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_exportall.php
-rwxrwxr-x. 1 nagios nagios 15847 May 4 12:08 /usr/local/nagiosxi/scripts/nagiosql.export.monitoring
-rwxrwxr-x. 1 nagios nagios 1286 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_importall.php
-rwxrwxr-x. 1 nagios nagios 81899 May 3 11:50 /usr/local/nagiosxi/scripts/nagiosql.import.monitoring
-rwxrwxr-x. 1 nagios nagios 35807 May 4 12:08 /usr/local/nagiosxi/scripts/nagiosql.login
-rwxrwxr-x. 1 nagios nagios 1791 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_login.php
-rwxrwxr-x. 1 nagios nagios 2248 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_snapshot.sh
-rwxrwxr-x. 1 nagios nagios 200 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_trim_backups.sh
And I didn't try vacuum before, so:
Code: Select all
[root@nagios vmesquita]# echo "vacuum;vacuum analyze;vacuum full;"|psql nagiosxi postgres
VACUUM
VACUUM
VACUUM
It was upgraded from Nagios 2014 and everythingelse is running fine. Sorry I didn't see his questions before.
Apparently after running the commands suggested, the recurrent errors changed to:
Code: Select all
[Wed May 04 12:38:03 2016] [error] [client 172.27.15.207] PHP Notice: Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 249, referer: http://nagios.selic.bc/nagiosxi/includes/components/xicore/status.php?show=hostdetail&host=vx576.selic.bc
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice: Undefined variable: service_ids in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 303, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 303, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice: Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 314, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice: Undefined variable: service_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 318, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice: Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 408, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice: Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 75, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice: Undefined variable: service_ids in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 132, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 132, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice: Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 143, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice: Undefined variable: service_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 147, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:18 2016] [error] [client 172.27.15.145] PHP Notice: Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 75, referer: http://nagios.selic.bc/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:18 2016] [error] [client 172.27.15.145] PHP Notice: Undefined variable: service_ids in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 132, referer: http://nagios.selic.bc/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:18 2016] [error] [client 172.27.15.145] PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 132, referer: http://nagios.selic.bc/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
Re: Recurrent errors in Apache http log
Posted: Wed May 04, 2016 4:40 pm
by tgriep
The new messages are Warning and Notices so they can be ignored.
The connection errors to the Postgres database are the ones that should be gone after the vacuuming, etc.
Re: Recurrent errors in Apache http log
Posted: Thu May 05, 2016 11:04 am
by vmesquita
I'm glad to hear it's not a serious issue. But we would like to clean this up if possible to make debugging of logs easier in the future, afterall every single webpage request generated this warnings. Would this be a bug in Nagios product or it this issue related to the install?
Re: Recurrent errors in Apache http log
Posted: Thu May 05, 2016 12:03 pm
by tgriep
Those messages are not a bug, it is just variables that are setup and not initialized when they are setup. Depending how those variables are used, it could be done that way on purpose.
If you want to suppress those notices, you can edit the /etc/php.ini file and search for the error_reporting option and change it to the following
Code: Select all
error_reporting = E_ALL & ~E_DEPRECATED & ~E_NOTICE
Then restart Apache by running
That should stop those notices.