Recurrent errors in Apache http log

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Recurrent errors in Apache http log

Post by vmesquita »

The requested file:

Code: Select all

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
#local   all         all                               ident sameuser
local    all         all                               trust
# IPv4 local connections:
#host    all         all         127.0.0.1/32          ident sameuser
host    all         all         127.0.0.1/32          trust
# IPv6 local connections:
#host    all         all         ::1/128               ident sameuser
host    all         all         ::1/128               trust
There are some comment lines above those which are not relevant.
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Recurrent errors in Apache http log

Post by ssax »

That looks fine, what is the output of these commands:

Code: Select all

netstat -ano | grep 5432
sestatus
Also, please attach this file:

Code: Select all

/var/lib/pgsql/data/postgresql.conf
Was this a fresh install of XI 5+ or was it upgraded from a pre-XI 5 (2012,2014, etc) system?

Is everything else running fine, you're just seeing that error in logs?
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Recurrent errors in Apache http log

Post by vmesquita »

Code: Select all

# netstat -ano | grep 5432
tcp        0      0 127.0.0.1:5432              0.0.0.0:*                   LISTEN      off (0.00/0/0)
tcp        0      0 ::1:5432                    :::*                        LISTEN      off (0.00/0/0)
tcp        0      0 ::1:46106                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:45946                   ::1:5432                    TIME_WAIT   timewait (9.03/0/0)
tcp        0      0 ::1:46102                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:46119                   ::1:5432                    TIME_WAIT   timewait (51.41/0/0)
tcp        0      0 ::1:5432                    ::1:46105                   ESTABLISHED keepalive (7187.42/0/0)
tcp        0      0 ::1:42946                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0     39 ::1:5432                    ::1:42074                   ESTABLISHED on (0.20/0/0)
tcp        0    220 ::1:5432                    ::1:43034                   ESTABLISHED on (0.19/0/0)
tcp        0      0 ::1:5432                    ::1:42946                   ESTABLISHED keepalive (6431.04/0/0)
tcp        0      0 ::1:5432                    ::1:45047                   ESTABLISHED keepalive (6940.91/0/0)
tcp        0      0 ::1:45998                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:43499                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:43192                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:43774                   ESTABLISHED keepalive (6621.03/0/0)
tcp        0      0 ::1:5432                    ::1:46106                   ESTABLISHED keepalive (7187.42/0/0)
tcp        0      0 ::1:46023                   ::1:5432                    TIME_WAIT   timewait (29.26/0/0)
tcp        0      0 ::1:49777                   ::1:5432                    TIME_WAIT   timewait (44.28/0/0)
tcp        0      0 ::1:45864                   ::1:5432                    TIME_WAIT   timewait (48.65/0/0)
tcp        0      0 ::1:45867                   ::1:5432                    TIME_WAIT   timewait (48.64/0/0)
tcp        0      0 ::1:42074                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:43192                   ESTABLISHED keepalive (6484.09/0/0)
tcp        0      0 ::1:46110                   ::1:5432                    TIME_WAIT   timewait (48.74/0/0)
tcp        0      0 ::1:42003                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:46105                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:46103                   ESTABLISHED keepalive (7187.40/0/0)
tcp        0      0 ::1:42056                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:46088                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:41673                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp      100      0 ::1:42064                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:46107                   ::1:5432                    TIME_WAIT   timewait (48.53/0/0)
tcp        0      0 ::1:46112                   ::1:5432                    TIME_WAIT   timewait (48.75/0/0)
tcp        0      0 ::1:5432                    ::1:44579                   ESTABLISHED keepalive (6821.73/0/0)
tcp        0      0 ::1:5432                    ::1:45862                   TIME_WAIT   timewait (48.49/0/0)
tcp        0      0 ::1:44718                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:46074                   TIME_WAIT   timewait (39.10/0/0)
tcp        0      0 ::1:5432                    ::1:42003                   ESTABLISHED keepalive (6226.34/0/0)
tcp        0      0 ::1:5432                    ::1:44718                   ESTABLISHED keepalive (6850.75/0/0)
tcp        0      0 ::1:5432                    ::1:46102                   ESTABLISHED keepalive (7187.38/0/0)
tcp        0    161 ::1:5432                    ::1:42064                   ESTABLISHED on (0.20/0/0)
tcp        0      0 ::1:45047                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:46109                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:43034                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:41673                   ESTABLISHED keepalive (6149.29/0/0)
tcp        0      0 ::1:5432                    ::1:45983                   ESTABLISHED keepalive (7158.95/0/0)
tcp        0      0 ::1:45983                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:43499                   ESTABLISHED keepalive (6551.03/0/0)
tcp        0      0 ::1:45868                   ::1:5432                    TIME_WAIT   timewait (49.25/0/0)
tcp        0      0 ::1:46103                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:43774                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:39234                   ESTABLISHED keepalive (5568.61/0/0)
tcp        0      0 ::1:44579                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:39234                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:45322                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:42030                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:42030                   ESTABLISHED keepalive (6227.77/0/0)
tcp        0      0 ::1:44045                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:45865                   ::1:5432                    TIME_WAIT   timewait (50.87/0/0)
tcp        0      0 ::1:42033                   ::1:5432                    TIME_WAIT   timewait (24.21/0/0)
tcp        0      0 ::1:5432                    ::1:46088                   ESTABLISHED keepalive (7182.43/0/0)
tcp        0      0 ::1:5432                    ::1:46109                   ESTABLISHED keepalive (7187.41/0/0)
tcp        0      0 ::1:5432                    ::1:45998                   ESTABLISHED keepalive (7162.19/0/0)
tcp        0      0 ::1:58236                   ::1:5432                    ESTABLISHED off (0.00/0/0)
tcp        0      0 ::1:5432                    ::1:44045                   ESTABLISHED keepalive (6677.36/0/0)
tcp        0    220 ::1:5432                    ::1:45322                   ESTABLISHED on (0.18/0/0)
tcp        0      0 ::1:5432                    ::1:42056                   ESTABLISHED keepalive (6229.34/0/0)
tcp        0      0 ::1:46104                   ::1:5432                    TIME_WAIT   timewait (48.45/0/0)
tcp        0      0 ::1:46131                   ::1:5432                    TIME_WAIT   timewait (56.06/0/0)
unix  2      [ ACC ]     STREAM     LISTENING     20421  /tmp/.s.PGSQL.5432

# sestatus
SELinux status:                 disabled
You do not have the required permissions to view the files attached to this post.
Top
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Recurrent errors in Apache http log

Post by tmcdonald »

ssax wrote:Was this a fresh install of XI 5+ or was it upgraded from a pre-XI 5 (2012,2014, etc) system?

Is everything else running fine, you're just seeing that error in logs?
Can you please address the other two questions from @ssax?
Former Nagios employee
Top
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Recurrent errors in Apache http log

Post by lmiltchev »

Can you run the following commands and show the output?

Code: Select all

df -h
service postgresql stop
killall postgresql
service postgresql start
service postgresql status
tail /var/lib/pgsql/pgstartup.log
ls -lad /var/lib/pgsql
ls -la /var/lib/pgsql
This is a "long shot" but also show the output of the command below:

Code: Select all

ls -la /usr/local/nagiosxi/scripts/nagiosql*
BTW, have you tried running vacuum on postgres?

Code: Select all

echo "vacuum;vacuum analyze;vacuum full;"|psql nagiosxi postgres
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Recurrent errors in Apache http log

Post by vmesquita »

So here's the output of the series of commands:

Code: Select all

[root@nagios vmesquita]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_nagios-lv_root
                       36G   27G  7.6G  78% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sdd1             477M   72M  380M  16% /boot
tmpfs                  50M  7.1M   43M  15% /var/nagiosramdisk
[root@nagios vmesquita]# service postgresql stop
Stopping postgresql service:                               [  OK  ]
[root@nagios vmesquita]# killall postgresql
postgresql: no process killed
[root@nagios vmesquita]# service postgresql start
Starting postgresql service:                               [  OK  ]
[root@nagios vmesquita]# service postgresql status
postmaster (pid  18959) is running...
[root@nagios vmesquita]# tail /var/lib/pgsql/pgstartup.log

Success. You can now start the database server using:

    /usr/bin/postgres -D /var/lib/pgsql/data
or
    /usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start

LOG:  could not create IPv6 socket: Address family not supported by protocol
LOG:  could not create IPv6 socket: Address family not supported by protocol
could not write to log file: No space left on device
[root@nagios vmesquita]# ls -lad /var/lib/pgsql
drwx------. 4 postgres postgres 4096 May  2 19:01 /var/lib/pgsql
[root@nagios vmesquita]# ls -la /var/lib/pgsql
total 24
drwx------.  4 postgres postgres 4096 May  2 19:01 .
drwxr-xr-x. 33 root     root     4096 May  4 03:37 ..
drwx------.  2 postgres postgres 4096 Mar 30  2015 backups
-rw-r--r--   1 postgres postgres   85 Mar 30  2015 .bash_profile
drwx------. 12 postgres postgres 4096 May  4 12:38 data
-rw-------.  1 postgres postgres 1359 Apr 26 00:00 pgstartup.log
Now the ls output:

Code: Select all

-rwxrwxr-x. 1 nagios nagios    161 May  4 12:08 /usr/local/nagiosxi/scripts/nagiosql.cookies
-rwxrwxr-x. 1 nagios nagios 271666 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_defaults.sql
-rwxrwxr-x. 1 nagios nagios  39900 Jan 28 17:17 /usr/local/nagiosxi/scripts/nagiosql.delete.contact
-rwxrwxr-x. 1 nagios nagios    885 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_contact.php
-rwxrwxr-x. 1 nagios nagios   1927 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_host.php
-rwxrwxr-x. 1 nagios nagios    446 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_object.sh
-rwxrwxr-x. 1 nagios nagios  64566 Oct 26  2015 /usr/local/nagiosxi/scripts/nagiosql.delete.service
-rwxrwxr-x. 1 nagios nagios   2756 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_service.php
-rwxrwxr-x. 1 nagios nagios  41292 Jan 28 17:17 /usr/local/nagiosxi/scripts/nagiosql.delete.timeperiod
-rwxrwxr-x. 1 nagios nagios    886 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_delete_timeperiod.php
-rwxrwxr-x. 1 nagios nagios   7400 Jul  1  2014 /usr/local/nagiosxi/scripts/nagiosql.export.additional
-rwxrwxr-x. 1 nagios nagios   1272 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_exportall.php
-rwxrwxr-x. 1 nagios nagios  15847 May  4 12:08 /usr/local/nagiosxi/scripts/nagiosql.export.monitoring
-rwxrwxr-x. 1 nagios nagios   1286 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_importall.php
-rwxrwxr-x. 1 nagios nagios  81899 May  3 11:50 /usr/local/nagiosxi/scripts/nagiosql.import.monitoring
-rwxrwxr-x. 1 nagios nagios  35807 May  4 12:08 /usr/local/nagiosxi/scripts/nagiosql.login
-rwxrwxr-x. 1 nagios nagios   1791 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_login.php
-rwxrwxr-x. 1 nagios nagios   2248 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_snapshot.sh
-rwxrwxr-x. 1 nagios nagios    200 Apr 25 18:26 /usr/local/nagiosxi/scripts/nagiosql_trim_backups.sh
And I didn't try vacuum before, so:

Code: Select all

[root@nagios vmesquita]# echo "vacuum;vacuum analyze;vacuum full;"|psql nagiosxi postgres
VACUUM
VACUUM
VACUUM
It was upgraded from Nagios 2014 and everythingelse is running fine. Sorry I didn't see his questions before.

Apparently after running the commands suggested, the recurrent errors changed to:

Code: Select all

[Wed May 04 12:38:03 2016] [error] [client 172.27.15.207] PHP Notice:  Undefined variable: result in /usr/local/nagiosxi/html/includes/db.inc.php on line 249, referer: http://nagios.selic.bc/nagiosxi/includes/components/xicore/status.php?show=hostdetail&host=vx576.selic.bc
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice:  Undefined variable: service_ids in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 303, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 303, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice:  Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 314, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice:  Undefined variable: service_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 318, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice:  Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 408, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice:  Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 75, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice:  Undefined variable: service_ids in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 132, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 132, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice:  Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 143, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:16 2016] [error] [client 172.27.143.15] PHP Notice:  Undefined variable: service_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 147, referer: http://nagios/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:18 2016] [error] [client 172.27.15.145] PHP Notice:  Undefined variable: host_ids_str in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 75, referer: http://nagios.selic.bc/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:18 2016] [error] [client 172.27.15.145] PHP Notice:  Undefined variable: service_ids in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 132, referer: http://nagios.selic.bc/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
[Wed May 04 12:38:18 2016] [error] [client 172.27.15.145] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nocscreen/nocscreenapi.php on line 132, referer: http://nagios.selic.bc/nagiosxi/includes/components/nocscreen/noc.php?host=&hostgroup=Producao&servicegroup=&state=&reporttimesubmitbutton=Update
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Recurrent errors in Apache http log

Post by tgriep »

The new messages are Warning and Notices so they can be ignored.
The connection errors to the Postgres database are the ones that should be gone after the vacuuming, etc.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
vmesquita
Posts: 315
Joined: Fri Aug 10, 2012 12:52 pm

Re: Recurrent errors in Apache http log

Post by vmesquita »

I'm glad to hear it's not a serious issue. But we would like to clean this up if possible to make debugging of logs easier in the future, afterall every single webpage request generated this warnings. Would this be a bug in Nagios product or it this issue related to the install?
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Recurrent errors in Apache http log

Post by tgriep »

Those messages are not a bug, it is just variables that are setup and not initialized when they are setup. Depending how those variables are used, it could be done that way on purpose.
If you want to suppress those notices, you can edit the /etc/php.ini file and search for the error_reporting option and change it to the following

Code: Select all

error_reporting = E_ALL & ~E_DEPRECATED & ~E_NOTICE
Then restart Apache by running

Code: Select all

service httpd restart
That should stop those notices.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
Locked

Return to “Nagios XI”