Acknowledgements and Scheduled Downtime not showing up in XI

[shilton]

New acknowledgements and scheduled downtime are not showing up when created in the XI interface, but when created in the core interface they are showing up for core and XI.

Core version: 4.2.4
NagiosXI version: 5.4.8
catdrnd-nagiosxi 2.6.32-573.22.1.el6.x86_64 x86_64
CentOS release 6.7 (Final)

From reading some other threads, possibly this is relevant:

These snippits are from an acknowledgment made from core that does show up in the XI interface

/usr/local/nagios/var/status.dat snippet:

Code: Select all

status.dat
servicecomment {
        host_name=brhel7aa.csdt.redacted.com
        service_description=Check logged in users
        entry_type=4
        comment_id=2617
        source=0
        persistent=0
        entry_time=1504895134
        expires=0
        expire_time=0
        author=Nagios Administrator
        comment_data=auniquetxtline
        }

/usr/local/nagios/var/retention.dat snippet:

Code: Select all

service {
host_name=brhel7aa.csdt.redacted.com
service_description=Check logged in users
modified_attributes=1
check_command=check_nrpe!check_users!!!!!!!
check_period=24x7
notification_period=24x7
event_handler=
has_been_checked=1
check_execution_time=0.040
check_latency=0.000
check_type=0
current_state=1
last_state=1
last_hard_state=1
last_event_id=340810
current_event_id=340897
current_problem_id=94955
last_problem_id=94925
current_attempt=5
max_attempts=5
normal_check_interval=5.000000
retry_check_interval=1.000000
state_type=1
last_state_change=1504811345
last_hard_state_change=1504811575
last_time_ok=1504811345
last_time_warning=1504893048
last_time_unknown=1496206111
last_time_critical=1496213106
plugin_output=USERS WARNING - 7 users currently logged in
long_plugin_output=
performance_data=users=7;5;10;0
last_check=1504893048
next_check=1504893348
check_options=0
notified_on_unknown=0
notified_on_warning=0
notified_on_critical=0
current_notification_number=0
current_notification_id=2479762
last_notification=0
notifications_enabled=0
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=1
process_performance_data=1
obsess=1
is_flapping=0
percent_state_change=0.00
check_flapping_recovery_notification=0
state_history=2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2
}

[dwhitfield]

What's the output of ls -la /usr/local/nagios/var/rw/?

Also, can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

You can also generate a profile manually using the script at /usr/local/nagiosxi/html/includes/components/profile/getprofile.sh

That should generate a profile in /usr/local/nagiosxi/var/components/ which you can get off the server with an application such as FileZilla.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

[shilton]

What's the output of ls -la /usr/local/nagios/var/rw/?

Also, can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

You can also generate a profile manually using the script at /usr/local/nagiosxi/html/includes/components/profile/getprofile.sh

That should generate a profile in /usr/local/nagiosxi/var/components/ which you can get off the server with an application such as FileZilla.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

Not able to PM yet, "We are sorry, but you are not authorised to use this feature. You may have just registered here and may need to participate more to be able to use this feature."

drwxrwsr-x 2 nagios nagcmd 4096 Sep 8 10:51 .
drwxrwxr-x 6 nagios nagios 4096 Sep 8 14:10 ..
prw-rw---- 1 nagios nagcmd 0 Sep 8 13:25 nagios.cmd
srw-rw---- 1 nagios nagcmd 0 Sep 8 10:51 nagios.qh

[dwhitfield]

Apologies. 2nd post has been approved, so you should be able to PM it now.

UPDATE: Profile received and shared with techs

[shilton]

Profile sent in PM, it is in the Outbox (1), but not showing in Sent messages yet.

[dwhitfield]

You will need to run vacuum on the postgres database. The following commands are different if you are using a version of PostgreSQL before v9. To determine which version you have execute the following command:

Code: Select all

postgres -V

Based on that output, execute the commands specific to your version:

Versions BEFORE 9

Code: Select all

echo "vacuum;vacuum analyze;"|psql nagiosxi postgres
service postgresql restart

Versions 9 onwards

Code: Select all

echo "vacuum;vacuum analyze;vacuum full;"|psql nagiosxi postgres
service postgresql restart

To log in the postgres manually, run:

Code: Select all

psql nagiosxi nagiosxi

To view the tables, run:

Code: Select all

\d

and to exit:

Code: Select all

\q

If you tried to run the vacuum on the posgres or you attempted to log in manually in the database, but you see the following error message:

psql: FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres"
HINT: Stop the postmaster and use a standalone backend to vacuum database "postgres".



You may notice either a high CPU usage for the postmaster process, or a repeated error message in the /var/lib/pgsql/data/pg_log file:

transaction ID wrap limit is 2147484146



You can try to fix the issue by running the following command in the command line:

Important: Run the commands one-by-one (don't run them with one go!)

Versions BEFORE PostgreSQL 9

Code: Select all

service postgresql stop
su postgres
echo "VACUUM;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
postgres -D /var/lib/pgsql/data postgres < /tmp/fix.sql
postgres -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

Note: The commands listed above may not work with some versions of PosgreSQL. If you see the following error:

postgres: invalid argument: "nagiosxi"



You will need to run the following commands instead:

Code: Select all

service postgresql stop
su postgres
echo "VACUUM;" > /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data postgres < /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

Versions 9 Onwards

Code: Select all

service postgresql stop
su postgres
echo "VACUUM FULL;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
postgres -D /var/lib/pgsql/data postgres < /tmp/fix.sql
postgres -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

Note: The commands listed above may not work with some versions of PosgreSQL. If you see the following error:

postgres: invalid argument: "nagiosxi"



You will need to run the following commands instead:

Code: Select all

service postgresql stop
su postgres
echo "VACUUM FULL;" > /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data postgres < /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

[shilton]

You will need to run vacuum on the postgres database. The following commands are different if you are using a version of PostgreSQL before v9. To determine which version you have execute the following command:

Code: Select all

postgres -V

Based on that output, execute the commands specific to your version:

Versions BEFORE 9

Code: Select all

echo "vacuum;vacuum analyze;"|psql nagiosxi postgres
service postgresql restart

Versions 9 onwards

Code: Select all

echo "vacuum;vacuum analyze;vacuum full;"|psql nagiosxi postgres
service postgresql restart

To log in the postgres manually, run:

Code: Select all

psql nagiosxi nagiosxi

To view the tables, run:

Code: Select all

\d

and to exit:

Code: Select all

\q

If you tried to run the vacuum on the posgres or you attempted to log in manually in the database, but you see the following error message:

psql: FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres"
HINT: Stop the postmaster and use a standalone backend to vacuum database "postgres".



You may notice either a high CPU usage for the postmaster process, or a repeated error message in the /var/lib/pgsql/data/pg_log file:

transaction ID wrap limit is 2147484146



You can try to fix the issue by running the following command in the command line:

Important: Run the commands one-by-one (don't run them with one go!)

Versions BEFORE PostgreSQL 9

Code: Select all

service postgresql stop
su postgres
echo "VACUUM;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
postgres -D /var/lib/pgsql/data postgres < /tmp/fix.sql
postgres -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

Note: The commands listed above may not work with some versions of PosgreSQL. If you see the following error:

postgres: invalid argument: "nagiosxi"



You will need to run the following commands instead:

Code: Select all

service postgresql stop
su postgres
echo "VACUUM;" > /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data postgres < /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

Versions 9 Onwards

Code: Select all

service postgresql stop
su postgres
echo "VACUUM FULL;" > /tmp/fix.sql
postgres -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
postgres -D /var/lib/pgsql/data postgres < /tmp/fix.sql
postgres -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

Note: The commands listed above may not work with some versions of PosgreSQL. If you see the following error:

postgres: invalid argument: "nagiosxi"



You will need to run the following commands instead:

Code: Select all

service postgresql stop
su postgres
echo "VACUUM FULL;" > /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data nagiosxi < /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data postgres < /tmp/fix.sql
postgres --single -D /var/lib/pgsql/data template1 < /tmp/fix.sql
exit
service postgresql start

The vacuum went ok, but did not seem to correct the problem.

I did restart the nagios services and test a second time also.

I tried to schedule a downtime and received this message at the top of that screen:
" One or more scheduled commands could not be sent to Nagios Core."

I tried to set an acknowledgment, the input screen functioned ok with no error messages, but no new acknowledgment showed in the "Acknowledgements and Comments" screen.

The event log did not note any issues related to this either.


# postgres -V
postgres (PostgreSQL) 8.4.20

# echo "vacuum;vacuum analyze;"|psql nagiosxi postgres
VACUUM
VACUUM

# psql nagiosxi nagiosxi
psql (8.4.20)
Type "help" for help.

nagiosxi=> \d
List of relations
Schema | Name | Type | Owner
--------+---------------------------------+----------+----------
public | if_command_id_seq | sequence | nagiosxi
public | if_meta_id_seq | sequence | nagiosxi
public | if_option_id_seq | sequence | nagiosxi
public | if_sysstat_id_seq | sequence | nagiosxi
public | if_user_id_seq | sequence | nagiosxi
public | if_usermeta_id_seq | sequence | nagiosxi
public | xi_auditlog | table | nagiosxi
public | xi_auditlog_auditlog_id_seq | sequence | nagiosxi
public | xi_commands | table | nagiosxi
public | xi_commands_command_id_seq | sequence | nagiosxi
public | xi_eventqueue | table | nagiosxi
public | xi_eventqueue_eventqueue_id_seq | sequence | nagiosxi
public | xi_events | table | nagiosxi
public | xi_events_event_id_seq | sequence | nagiosxi
public | xi_incidents | table | nagiosxi
public | xi_incidents_id_seq | sequence | nagiosxi
public | xi_meta | table | nagiosxi
public | xi_meta_meta_id_seq | sequence | nagiosxi
public | xi_options | table | nagiosxi
public | xi_options_option_id_seq | sequence | nagiosxi
public | xi_sysstat | table | nagiosxi
public | xi_sysstat_sysstat_id_seq | sequence | nagiosxi
public | xi_usermeta | table | nagiosxi
public | xi_usermeta_usermeta_id_seq | sequence | nagiosxi
public | xi_users | table | nagiosxi
public | xi_users_user_id_seq | sequence | nagiosxi
(26 rows)

[dwhitfield]

Please run ll -R /usr/local >> /tmp/llR.txt and attach (or PM) the file that is created. If you PM, make sure you update the thread.

Update: Output received and shared with the support team

[shilton]

ll -R /usr/local >> /tmp/llR.txt output gzipped and attached to a PM.

[tgriep]

I found the statehistory and the logentries MYSQL tables are corrupt and that could cause the issue you are having.
Run the following as root on the Nagios server to repair them.

Code: Select all

mysqlcheck -f -r -u root -pnagiosxi --use_frm --databases nagios nagiosql

Then Acknowledge a message or put something in downtime and see if the GUI shows the status.