Runtime Error for host and service notification

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:
Contact dwhitfield

Re: Runtime Error for host and service notification

Post by dwhitfield »

And if you don't want to post here as @tgriep suggested, you can PM it to one of us.
Top
valmberg
Posts: 49
Joined: Fri May 16, 2014 9:43 pm

Re: Runtime Error for host and service notification

Post by valmberg »

Here is processes and sudoers. I will pm the services config file.
You do not have the required permissions to view the files attached to this post.
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Runtime Error for host and service notification

Post by tgriep »

Thanks for the config file it looks good. I looked at an old System profile of yours and found some more things to change as well.

Under this line in the /etc/sudoers file

Code: Select all

#includedir /etc/sudoers.d
Add these 2 lines.

Code: Select all

Defaults:nagios !requiretty
Defaults:apache !requiretty
That should stop the stderr errors in the nagios.log

Then edit this file

Code: Select all

/etc/httpd/conf/httpd.conf
Add the following line to the bottom of the file

Code: Select all

LimitRequestLine 100000
And restart the Apache and nagios processes by running

Code: Select all

service httpd restart
service nagios restart
Can you login to the XI server as root, run the following commands and post the output?

Code: Select all

/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php
echo $?
ls -l /usr/local/nagiosxi/scripts/
Then can you post the following files so we can view them?

Code: Select all

/usr/local/nagiosxi/var/event_handler.log
/usr/local/nagiosxi/var/eventman.log
/usr/local/nagios/etc/commands.cfg
/usr/local/nagios/etc/contacts.cfg
/usr/local/nagios/etc/contactgroups.cfg
Thanks
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
valmberg
Posts: 49
Joined: Fri May 16, 2014 9:43 pm

Re: Runtime Error for host and service notification

Post by valmberg »

Here is the output of the 3 commands:

[root@nagios ~]# /usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php
[root@nagios ~]# echo $?
255
[root@nagios ~]# ls -l /usr/local/nagiosxi/scripts/
total 764
-rwxr-x--- 1 nagios nagios 5673 Dec 16 09:48 backup_xi.sh
-rwxr-x--- 1 nagios nagios 1664 Dec 16 09:48 change_timezone.sh
-rw-r----- 1 nagios nagios 3030 Dec 16 09:48 contact_notification_handler.php
-rwxr-x--- 1 nagios nagios 660 Dec 16 09:48 export_nagiosql.sh
-rwxr-x--- 1 nagios nagios 266 Dec 16 09:48 handle_nagioscore_event.php
-rw-r----- 1 nagios nagios 3317 Dec 16 09:48 handle_nagioscore.inc.php
-rwxr-x--- 1 nagios nagios 267 Dec 16 09:48 handle_nagioscore_notification.php
-rwxr-x--- 1 nagios nagios 425 Dec 16 09:48 import_nagiosql.sh
-rw-r----- 1 nagios nagios 908 Dec 16 09:48 import_xiconfig.php
-rwxr-x--- 1 nagios nagios 93 Dec 16 09:48 kill_rrdtool.sh
-rwxr-x--- 1 nagios nagios 2303 Dec 16 09:48 manage_services.sh
-rwxrwxr-x 1 nagios nagios 161 Dec 20 11:20 nagiosql.cookies
-rwxrwxr-x 1 nagios nagios 271666 Dec 16 09:48 nagiosql_defaults.sql
-rwxrwxr-x 1 nagios nagios 45063 Oct 14 15:58 nagiosql.delete.contact
-rwxrwxr-x 1 nagios nagios 895 Dec 16 09:48 nagiosql_delete_contact.php
-rwxrwxr-x 1 nagios nagios 38094 Aug 22 2014 nagiosql.delete.host
-rwxrwxr-x 1 nagios nagios 1937 Dec 16 09:48 nagiosql_delete_host.php
-rwxrwxr-x 1 nagios nagios 497 Dec 16 09:48 nagiosql_delete_object.sh
-rwxrwxr-x 1 nagios nagios 42400 Aug 29 2014 nagiosql.delete.service
-rwxrwxr-x 1 nagios nagios 2766 Dec 16 09:48 nagiosql_delete_service.php
-rwxrwxr-x 1 nagios nagios 49828 Oct 14 15:58 nagiosql.delete.timeperiod
-rwxrwxr-x 1 nagios nagios 896 Dec 16 09:48 nagiosql_delete_timeperiod.php
-rwxrwxr-x 1 nagios nagios 1272 Dec 16 09:48 nagiosql_exportall.php
-rwxrwxr-x 1 nagios nagios 21799 Dec 20 11:20 nagiosql.export.monitoring
-rwxrwxr-x 1 nagios nagios 1296 Dec 16 09:48 nagiosql_importall.php
-rwxrwxr-x 1 nagios nagios 43227 Dec 20 11:20 nagiosql.import.monitoring
-rwxrwxr-x 1 nagios nagios 37631 Dec 20 11:20 nagiosql.login
-rwxrwxr-x 1 nagios nagios 1681 Dec 16 09:48 nagiosql_login.php
-rwxrwxr-x 1 nagios nagios 2247 Dec 16 09:48 nagiosql_snapshot.sh
-rwxrwxr-x 1 nagios nagios 200 Dec 16 09:48 nagiosql_trim_backups.sh
-rw-r----- 1 nagios nagios 366 Dec 16 09:48 nagiosxi_dbtype.php
-rwxr-x--- 1 nagios nagios 408 Dec 16 09:48 nom_create_nagioscore_checkpoint_cond.sh
-rwxr-x--- 1 nagios nagios 961 Dec 16 09:48 nom_create_nagioscore_checkpoint.sh
-rwxr-x--- 1 nagios nagios 774 Dec 16 09:48 nom_create_nagioscore_errorpoint.sh
-rwxr-x--- 1 nagios nagios 760 Dec 16 09:48 nom_restore_nagioscore_checkpoint.sh
-rwxr-x--- 1 nagios nagios 802 Dec 16 09:48 nom_restore_nagioscore_checkpoint_specific.sh
-rwxr-x--- 1 nagios nagios 2893 Dec 16 09:48 nom_trim_nagioscore_checkpoints.sh
-rwxr-x--- 1 nagios nagios 4621 Dec 16 09:48 parse_core_eventlog.php
-rwxr-x--- 1 nagios nagios 6019 Dec 16 09:48 patch_ndoutils.php
-rwxr-x--- 1 nagios nagios 588 Dec 16 09:48 reconfigure_nagios.sh
-rwxr-x--- 1 nagios nagios 1113 Dec 16 09:48 repair_databases.sh
-rwxr-x--- 1 nagios nagios 1501 Dec 16 09:48 repairmysql.sh
-rwxr-x--- 1 nagios nagios 6798 Apr 24 2015 reset_config_perms
-rw-r----- 1 nagios nagios 280 Apr 24 2015 reset_config_perms.c
-rwsr-xr-x 1 root nagios 1010 Dec 16 09:48 reset_config_perms.sh
-rwxr-x--- 1 nagios nagios 1538 Dec 16 09:48 reset_nagiosadmin_password.php
-rwxr-x--- 1 nagios nagios 1563 Dec 16 09:48 restart_nagios_with_export.sh
-rwxr-x--- 1 nagios nagios 767 Dec 16 09:48 restore_defaults.sh
-rwxr-x--- 1 nagios nagios 9031 Dec 16 09:48 restore_xi.sh
-rwxr-x--- 1 nagios nagios 1847 Dec 16 09:48 send_to_auditlog.php
-rw-r----- 1 nagios nagios 2702 Dec 16 09:48 uninstall_xi.sh
-rwxr-x--- 1 nagios nagios 1207 Dec 16 09:48 unlock_user_account.php
-rwxr-x--- 1 nagios nagios 2681 Dec 16 09:48 upgrade_to_latest.sh


I will pm you the contact.cfg file.
You do not have the required permissions to view the files attached to this post.
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Runtime Error for host and service notification

Post by tgriep »

I am thinking that when the server was upgraded, the postgres table didn't get updated so lets update the database manually and see if it helps.
Login as root and run the following to do that.

Code: Select all

cd /tmp
mkdir temp
cd temp
wget http://assets.nagios.com/downloads/nagiosxi/xi-latest.tar.gz
tar xzf xi-latest.tar.gz
service nagios stop
psql nagiosxi nagiosxi -f /tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql
service postgresql restart
service nagios start
Then post this log file

Code: Select all

/var/lib/pgsql/data/pg_log/postgresql-Tue.log
When you upgraded the server, did you do the manual upgrade process or the upgrade through the Web interface?
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
valmberg
Posts: 49
Joined: Fri May 16, 2014 9:43 pm

Re: Runtime Error for host and service notification

Post by valmberg »

I attempted to update through the web UI, but ran out of disk space. so, I expanded the disk and then went through the manual upgrade. But by then, I think things got a little hashed up. Other than this notification, the server works great.

Here is the output of the schema_01.sql command:
[root@nagios temp]# service nagios stop
Stopping nagios:. done.
[root@nagios temp]# psql nagiosxi nagiosxi -f /tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:2: ERROR: column "api_key" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:3: ERROR: column "api_enabled" of relation "xi_users" already exists
UPDATE 13
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:8: ERROR: column "login_attempts" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:9: ERROR: column "last_attempt" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:10: ERROR: column "last_password_change" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:13: ERROR: column "last_login" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:14: ERROR: column "last_edited" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:15: ERROR: column "last_edited_by" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:16: ERROR: column "created_by" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:17: ERROR: column "created_time" of relation "xi_users" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:24: ERROR: relation "xi_eventqueue_eventqueue_id_seq" already exists
psql:/tmp/temp/nagiosxi/nagiosxi/nagiosxi-db/mods/pgsql/schema_01.sql:33: ERROR: relation "xi_eventqueue" already exists
[root@nagios temp]# service postgresql restart
Stopping postgresql service: [ OK ]
Starting postgresql service: [ OK ]
[root@nagios temp]# service nagios start
Starting nagios: done.







Here is the output of the log file. looks like xi_notifications doesn't exist
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
WARNING: nonstandard use of \\ in a string literal at character 138
HINT: Use the escape string syntax for backslashes, e.g., E'\\'.
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: relation "xi_notifications" does not exist
STATEMENT: VACUUM ANALYZE xi_notifications;
ERROR: column "api_key" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN api_key character varying(128);
ERROR: column "api_enabled" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN api_enabled smallint DEFAULT 0::smallint NOT NULL;
ERROR: column "login_attempts" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN login_attempts smallint DEFAULT 0::smallint NOT NULL;
ERROR: column "last_attempt" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN last_attempt integer DEFAULT 0::integer NOT NUll;
ERROR: column "last_password_change" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN last_password_change integer DEFAULT 0::integer NOT NULL;
ERROR: column "last_login" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN last_login integer DEFAULT 0::integer NOT NUll;
ERROR: column "last_edited" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN last_edited integer DEFAULT 0::integer NOT NUll;
ERROR: column "last_edited_by" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN last_edited_by integer DEFAULT 0::integer NOT NUll;
ERROR: column "created_by" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN created_by integer DEFAULT 0::integer NOT NUll;
ERROR: column "created_time" of relation "xi_users" already exists
STATEMENT: ALTER TABLE xi_users ADD COLUMN created_time integer DEFAULT 0::integer NOT NUll;
ERROR: relation "xi_eventqueue_eventqueue_id_seq" already exists
STATEMENT: CREATE SEQUENCE xi_eventqueue_eventqueue_id_seq
INCREMENT BY 1
NO MAXVALUE
NO MINVALUE
CACHE 1;
ERROR: relation "xi_eventqueue" already exists
STATEMENT: CREATE TABLE xi_eventqueue (
eventqueue_id integer DEFAULT nextval('xi_eventqueue_eventqueue_id_seq'::regclass) NOT NULL,
event_time integer,
event_source smallint,
event_type smallint DEFAULT 0 NOT NULL,
event_meta text
);
LOG: received fast shutdown request
LOG: aborting any active transactions
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
LOG: autovacuum launcher shutting down
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
LOG: shutting down
FATAL: the database system is shutting down
FATAL: the database system is shutting down
FATAL: the database system is shutting down
FATAL: the database system is shutting down
FATAL: the database system is shutting down
FATAL: the database system is shutting down
FATAL: the database system is shutting down
LOG: database system is shut down
LOG: database system was shut down at 2016-12-20 14:37:02 PST
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Runtime Error for host and service notification

Post by tgriep »

Can you run the following as root and post the output?

Code: Select all

echo "\dt+;"|psql nagiosxi nagiosxi
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
valmberg
Posts: 49
Joined: Fri May 16, 2014 9:43 pm

Re: Runtime Error for host and service notification

Post by valmberg »

here is it:

# echo "\dt+;"|psql nagiosxi nagiosxi
List of relations
Schema | Name | Type | Owner | Size | Description
--------+---------------+-------+----------+------------+-------------
public | xi_auditlog | table | nagiosxi | 280 kB |
public | xi_commands | table | nagiosxi | 8192 bytes |
public | xi_eventqueue | table | postgres | 0 bytes |
public | xi_events | table | nagiosxi | 0 bytes |
public | xi_incidents | table | nagiosxi | 0 bytes |
public | xi_meta | table | nagiosxi | 2504 kB |
public | xi_options | table | nagiosxi | 16 kB |
public | xi_sysstat | table | nagiosxi | 32 kB |
public | xi_usermeta | table | nagiosxi | 128 kB |
public | xi_users | table | nagiosxi | 8192 bytes |
(10 rows)
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Runtime Error for host and service notification

Post by tgriep »

Can you edit the following file on your server

Code: Select all

/usr/local/nagiosxi/scripts/handle_nagioscore.inc.php
And add the following command on line 65

Code: Select all

var_dump($statement);
Save the file and run these commands and post the output?

Code: Select all

/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php
echo $?
cd /usr/local/nagiosxi/tools
./fix_postgres_sequences.sh
echo '\d+ xi_eventqueue;' | psql nagiosxi nagiosxi
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
valmberg
Posts: 49
Joined: Fri May 16, 2014 9:43 pm

Re: Runtime Error for host and service notification

Post by valmberg »

Here is the output of the commands:

/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php
You have new mail in /var/spool/mail/root
[root@nagios temp]# echo $?
255
[root@nagios temp]# cd /usr/local/nagiosxi/tools
[root@nagios tools]# ./fix_postgres_sequences.sh
Nagios XI Postgres Database Sequence Information

OLD VALUES
--------------
xi_commands_command_id_seq = 2268
xi_events_event_id_seq = 444165
xi_meta_meta_id_seq = 442393
xi_options_option_id_seq = 197
xi_sysstat_sysstat_id_seq = 101
xi_usermeta_usermeta_id_seq = 1496
xi_users_user_id_seq = 67

NEW VALUES
--------------
xi_commands_command_id_seq = 2268
xi_events_event_id_seq = 444165
xi_meta_meta_id_seq = 442393
xi_options_option_id_seq = 197
xi_sysstat_sysstat_id_seq = 101
xi_usermeta_usermeta_id_seq = 1496
xi_users_user_id_seq = 67


echo '\d+ xi_eventqueue;' | psql nagiosxi nagiosxi Table "public.xi_eventqueue"
Column | Type | Modifiers | Storage | Description
---------------+----------+-----------------------------------------------------------------------+----------+-------------
eventqueue_id | integer | not null default nextval('xi_eventqueue_eventqueue_id_seq'::regclass) | plain |
event_time | integer | | plain |
event_source | smallint | | plain |
event_type | smallint | not null default 0 | plain |
event_meta | text | | extended |
Has OIDs: no

Top
Locked

Return to “Nagios XI”