OracleServerspace most performance graphs not generating

[gsl_ops_practice]

Hello folks,

More oracle issues, latest version of NagiosXI on a pre-built VM.

When using the Configuration Wizard to add a new Oracle Serverpace check, the wizard completes, I have all the checks enabled and they all return valid results.

Out of all those checks only the first two, Connection Time and Flash Recovery Area Usage are generating performance graphs, all other checks are not. In the advanced tab for all checks I can see that performance data is being generated, but the Performance Graphs tab says "No performance graphs were found for this service. If you have just started monitoring this object then it may take up to 15 minutes for the performance graphs to appear."

I did some digging on this forum and adjusted the varchar2 value in the nagios database to accommodate longer performance strings but this did not help:

echo "ALTER TABLE nagios_servicestatus MODIFY perfdata VARCHAR(65536);"|mysql -pnagiosxi nagios
echo "ALTER TABLE nagios_servicechecks MODIFY perfdata VARCHAR(65536);"|mysql -pnagiosxi nagios
echo "ALTER TABLE nagios_hoststatus MODIFY perfdata VARCHAR(65536);"|mysql -pnagiosxi nagios
echo "ALTER TABLE nagios_hostchecks MODIFY perfdata VARCHAR(65536);"|mysql -pnagiosxi nagios

What else can be done to generate these graphs?

Thanks,
Alex

[lmiltchev]

Go to one of the "problem" services (Advanced tab) and show us a screenshot of the page.

Run the following commands and show us the output:

Code: Select all

ll -d /usr/local/nagios/share/perfdata
ll -d /usr/local/nagios/share/perfdata/<your oracle host>
ll /usr/local/nagios/share/perfdata/<your oracle host>

Have you tried deleting the RRDs and XMLs and waiting for them to get recreated? Did this fix the graphing issue?

[gsl_ops_practice]

Hello,

There were no RRDs and XMLs for the services where graphs are not generated. I am attaching the info you requested below:

$46E63FBBEEA3B56A.jpg

[root@localhost ~]# ll -d /usr/local/nagios/share/perfdata
drwxrwxr-x 30 nagios nagios 4096 Apr 14 00:40 /usr/local/nagios/share/perfdata

[root@localhost ~]# ll -d /usr/local/nagios/share/perfdata/dbserver-1/
drwxrwxr-x 2 nagios nagios 4096 Apr 16 21:15 /usr/local/nagios/share/perfdata/dbserver-1/

[root@localhost ~]# ll /usr/local/nagios/share/perfdata/dbserver-1/
total 16636
-rwxrwxr-x 1 nagios nagios 1534768 Apr 16 21:15 CPU_Stats.rrd
-rw-rw-r-- 1 nagios nagios 4073 Apr 16 21:15 CPU_Stats.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:15 _data_Disk_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2214 Apr 16 21:15 _data_Disk_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:15 __Disk_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2152 Apr 16 21:15 __Disk_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:16 _flash_Disk_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2233 Apr 16 21:16 _flash_Disk_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:16 _home_Disk_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2214 Apr 16 21:16 _home_Disk_Usage.xml
-rwxrwxr-x 1 nagios nagios 1534768 Apr 16 21:16 _HOST_.rrd
-rw-rw-r-- 1 nagios nagios 4000 Apr 16 21:16 _HOST_.xml
-rwxrwxr-x 1 nagios nagios 1151496 Apr 16 21:15 Load.rrd
-rw-rw-r-- 1 nagios nagios 3487 Apr 16 21:15 Load.xml
-rwxrwxr-x 1 nagios nagios 2301312 Apr 16 21:15 Memory_Usage.rrd
-rw-rw-r-- 1 nagios nagios 5414 Apr 16 21:15 Memory_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:16 _ogg_Disk_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2193 Apr 16 21:16 _ogg_Disk_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:16 Open_Files.rrd
-rw-rw-r-- 1 nagios nagios 2170 Apr 16 21:16 Open_Files.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 14 02:37 Oracle_Query_-_Average_Transaction_time_last_2_min.rrd
-rwxrwxr-x 1 nagios nagios 2728 Apr 14 02:37 Oracle_Query_-_Average_Transaction_time_last_2_min.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:09 Oracle_Query_-_Test_Query.rrd
-rw-rw-r-- 1 nagios nagios 2689 Apr 16 21:09 Oracle_Query_-_Test_Query.xml
-rwxrwxr-x 1 nagios nagios 1534768 Apr 16 21:16 Ping.rrd
-rw-rw-r-- 1 nagios nagios 4113 Apr 16 21:16 Ping.xml
-rwxrwxr-x 1 nagios nagios 384952 Mar 13 20:46 SSH.rrd
-rwxrwxr-x 1 nagios nagios 2078 Mar 13 20:46 SSH.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:16 Swap_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2115 Apr 16 21:16 Swap_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Mar 19 00:25 MYDB_Connected_Users.rrd
-rwxrwxr-x 1 nagios nagios 2435 Mar 19 00:25 MYDB_Connected_Users.xml
-rwxrwxr-x 1 nagios nagios 384952 Mar 19 00:23 MYDB_Connection_Time.rrd
-rwxrwxr-x 1 nagios nagios 2447 Mar 19 00:23 MYDB_Connection_Time.xml
-rwxrwxr-x 1 nagios nagios 768224 Mar 19 00:26 MYDB_Flash_Recovery_Area_Usage.rrd
-rwxrwxr-x 1 nagios nagios 3315 Mar 19 00:26 MYDB_Flash_Recovery_Area_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 10 17:11 MYDB_Connected_Users.rrd
-rwxrwxr-x 1 nagios nagios 2465 Apr 10 17:11 MYDB_Connected_Users.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:14 MYDB_Connection_Time.rrd
-rw-rw-r-- 1 nagios nagios 2461 Apr 16 21:14 MYDB_Connection_Time.xml
-rwxrwxr-x 1 nagios nagios 768224 Apr 16 21:15 MYDB_Flash_Recovery_Area_Usage.rrd
-rw-rw-r-- 1 nagios nagios 3330 Apr 16 21:15 MYDB_Flash_Recovery_Area_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:15 Total_Processes.rrd
-rw-rw-r-- 1 nagios nagios 2138 Apr 16 21:15 Total_Processes.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:16 _u01_Disk_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2201 Apr 16 21:16 _u01_Disk_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:16 Users.rrd
-rw-rw-r-- 1 nagios nagios 2058 Apr 16 21:16 Users.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:15 _var_Disk_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2175 Apr 16 21:15 _var_Disk_Usage.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 16 21:15 var_log_Disk_Usage.rrd
-rw-rw-r-- 1 nagios nagios 2200 Apr 16 21:15 var_log_Disk_Usage.xml

[Box293]

The problem lies with the performance data that is being output.

In your screenshot your performance data is:

rollback_segment_wraps=0;1;100$ rollback_segment_wraps_rate=0.00;1;100$

It should be:

rollback_segment_wraps=0;1;100 rollback_segment_wraps_rate=0.00;1;100

It is the $ sign causing the issue. The backend npcd daemon will be discarding this data.

You can confirm this with some logging. Increase the logging verbosity and then take a deeper look into the logs. Follow the FAQ entry below to increase the log level of process_perfdata and npcd:

http://support.nagios.com/wiki/index.ph ... leshooting

Wait 15 - 20 minutes and then get a tail of the logs:

Code: Select all

tail -250 /usr/local/nagios/var/perfdata.log > /tmp/perfdata.txt
tail -250 /usr/local/nagios/var/npcd.log > /tmp/npcd.txt

Send us a copy of /tmp/perfdata.txt and /tmp/npcd.txt

Don't forget to turn down the log level as per the FAQ when you are done!


The 100 is going to be the critical value passed to the plugin or the plugin's default critical value. Either way the plugin is returning 100$ for the critical value.

Can you find this service definition in /usr/local/nagios/var/objects.cache and paste it here in a code block please.

[gsl_ops_practice]

I will try to do increase the logging level and attach the results as I get them.

Can you please validate that this works correctly with the Oracle Serverspace monitoring wizard? I provided the DB credentials, checked all the boxes and accepted all the default settings - the result was only 2 working graphs out of more than a dozen.

Thanks,
Alex

[lmiltchev]

Alex, all of the oracle checks show the performance graphs on my test box. I didn't do anything fancy - just ran the wizard (Oracle Serverspace - ver. 1.5) and accepted defaults. Here's couple of screenshots:

example01.PNG

example02.PNG

[gsl_ops_practice]

Having said that, what can we do to fix this? I am monitoring 5 different DB servers, last thing I want to do is modify each monitor by hand to fix it...any suggestions? I am running 2014R2.6 from the pre-built VM, confirmed my OracleServerspace wizard is also v1.5

[gsl_ops_practice]

perfdata log as requested

Code: Select all

2015-04-17 14:34:18 [28980] [1] Found Performance Data for dbserver-1 / MYDB_Roll_Wraps (rollback_segment_wraps=0;1;100$ rollback_segment_wraps_rate=0.00;
1;100$)
2015-04-17 14:34:18 [28980] [2] No Custom Template found for check_xi_oracleserverspace (/usr/local/nagios/etc/pnp/check_commands/check_xi_oracleserverspace.cfg
)
2015-04-17 14:34:18 [28980] [2] Template is check_xi_oracleserverspace.php
2015-04-17 14:34:18 [28980] [2] No Custom Template found for check_xi_oracleserverspace (/usr/local/nagios/etc/pnp/check_commands/check_xi_oracleserverspace.cfg
)
2015-04-17 14:34:18 [28980] [2] Template is check_xi_oracleserverspace.php
2015-04-17 14:34:18 [28980] [2] No pattern match in function _parse($ )
2015-04-17 14:34:18 [28980] [1] Invalid Perfdata detected
2015-04-17 14:34:18 [28980] [2] Processing Line 32

attaching the npcd log as well

and the service definition:

Code: Select all

define service {
        host_name       dbserver-1
        service_description     MYDB Roll Wraps
        check_period    xi_timeperiod_24x7
        check_command   check_xi_oracleserverspace!--connect 'dbserver-1:1537/MYDB' --username 'user' --password 'password' --mode roll-wraps --warning 1 --critical 100
        contacts        nagiosadmin
        notification_period     xi_timeperiod_24x7
        initial_state   o
        importance      0
        check_interval  2.000000
        retry_interval  1.000000
        max_check_attempts      4
        is_volatile     0
        parallelize_check       1
        active_checks_enabled   1
        passive_checks_enabled  1
        obsess  1
        event_handler_enabled   1
        low_flap_threshold      0.000000
        high_flap_threshold     0.000000
        flap_detection_enabled  1
        flap_detection_options  a
        freshness_threshold     0
        check_freshness 0
        notification_options    a
        notifications_enabled   0
        notification_interval   60.000000
        first_notification_delay        0.000000
        stalking_options        n
        process_perf_data       1
        retain_status_information       1
        retain_nonstatus_information    1
        _XIWIZARD       oracleserverspace
        }

[gsl_ops_practice]

I just deleted all the Oracle serverspace monitors from one of our DB hosts, applied the config and re-did the Oracle Serverspace wizard, with all default values. I am seeing exactly the same problem as before. Attaching screenshots. Any ideas how we can fix this?

Not sure if this is relevant, I got the 2014R2.5 VM OVF and then did an online upgrade to 2.6.

[lmiltchev]

I am still not sure why you are getting a "$" in the perfdata output but this is clearly the issue that we need to fix.

2015-04-17 14:34:18 [28980] [2] No pattern match in function _parse($ )
2015-04-17 14:34:18 [28980] [1] Invalid Perfdata detected

I noticed that you have the RRDs/XLMs "doubled" for these three checks "Connected Users", "Connection Time", and "Flash Recovery Area Usage"... identical names with different timestamps. I don't see the RRDs/XMLs for the rest of the services. They should have been created by now...

What is the version of the "check_oracle_health" plugin that you are using?

Code: Select all

/usr/local/nagios/libexec/check_oracle_health -V

If you don't care about historical data, you could try deleting the RRDs/XMLs and wait for them to be recreated.

Can you show us the actual command run from the command line along with the output of it? Hide sensitive info.

Example:

Code: Select all

/usr/bin/env LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client/lib ORACLE_HOME=/usr/lib/oracle/11.2/client /usr/local/nagios/libexec/check_oracle_health --connect 'x.x.x.x:1521/orcl' --username 'username' --password 'password' --mode roll-wraps --warning 1 --critical 100
OK - Rollback segment wraps 0.01/sec | rollback_segment_wraps=1;1;100 rollback_segment_wraps_rate=0.01;1;100

Show us the "check_xi_oracleserverspace" command definition.