Adding Attachment / Large output to notification

[BanditBBS]

Need: We need to either add an attachment ot make a service capable or receiving hundreds(2000 perhaps) of lines of information.

Reason: Monitoring hundreds of DBs and DB servers. We check cpu stats every 10 minutes, with retry set to 10 minutes and max attempts to 4. So basically, we allow the CPU to "cook" for 30 minutes before a notification is sent. When its time for that notification I also want to run another check and find out the top CPU using Oracle processes and then run some SQL commands that will then output all kind of data. I then need to get that data to the same people getting the CPU check.

Example data(repeated for the top 3 or 4 CPU using processes):

Code: Select all

Enter Unix process id: 2857
=====================================================================
1 sessions were found with 2857  as their unix process id.
=====================================================================
SID/Serial  : 2285,33237
Foreground  : PID: 10136:7564 - w3wp.exe
Shadow      : PID: 2857 - [email protected]
Terminal    : SERVERNAME/ UNKNOWN
OS User     : NETWORK SERVICE on ITCONVERGENCE\SERVERNAME
Ora User    : CUST
Details     :  - w3wp.exe
Status Flags: ACTIVE DEDICATED USER
Tran Active : NONE
Login Time  : Tue 01:44:10
Last Call   : Tue 01:44:10 -    616.2 min
Lock/ Latch : NONE/ NONE
Latch Spin  : NONE
Current SQL statement:
        select COUNT(a."POSITIONS") as "Count of Positions",a."EMPLOYEE_
        NUMBER" as "Emp Number",a."EMPLOYEE_FULL_NAME_S" as "Employee Na
        me",(concat(concat(substr(a."VAR1", 1, 3),'-'),a."LOCATIONS")) a
        s "Location Info",COUNT(*) as "NWQ_AGG_SET_COUNT_SPECIAL_COL" FR
        OM "CUST"."ITC_NTX_CALLREPORT_V" a WHERE (a."VAR1" LIKE 'Sale%'
        and a."CONNECT_DATE">=(sysdate-90) and a."WORK_TELEPHONE"<>'0')
        group by a."EMPLOYEE_NUMBER",a."EMPLOYEE_FULL_NAME_S",(concat(co
        ncat(substr(a."VAR1", 1, 3),'-'),a."LOCATIONS")) order by 3
Previous SQL statement:
        SELECT TYP.SYSTEM_PERSON_TYPE FROM PER_PERSON_TYPES_TL TTL, PER_
        PERSON_TYPES TYP, PER_PERSON_TYPE_USAGES_F PTU WHERE TTL.LANGUAG
        E = USERENV ('LANG') AND TTL.PERSON_TYPE_ID = TYP.PERSON_TYPE_ID
         AND TYP.SYSTEM_PERSON_TYPE IN ('APL', 'EMP', 'EX_APL', 'EX_EMP'
        , 'CWK', 'EX_CWK', 'OTHER') AND TYP.PERSON_TYPE_ID = PTU.PERSON_
        TYPE_ID AND :B2 BETWEEN PTU.EFFECTIVE_START_DATE AND PTU.EFFECTI
        VE_END_DATE AND PTU.PERSON_ID = :B1 ORDER BY DECODE (TYP.SYSTEM_
        PERSON_TYPE, 'EMP', 1, 'CWK', 2, 'APL', 3, 'EX_EMP', 4, 'EX_CWK'
        , 5, 'EX_APL', 6, 7 )
Session Waits:
        WAITED SHORT TIME: latch: cache buffers chains
Locks:
        TYPE=AE H: S R: NONE - ID1=4977973 ID2=0
=====================================================================
1 sessions were found with 2857  as their unix process id.
Please scroll up to see details of all the sessions.
=====================================================================

My Idea:
Use an event handler that runs when the CPU check is run. Set the proper conditions to only run the SQL when it is the 4th check or higher. Then have it send the results as a passive result to a service based on the hostanme passed to the event handler. Then that service sends out a notification. That service needs to be able to receive hundreds of lines of data though.

So does that sound doable? Hints on how to allow that service to get all that information? Do I need to extend the field in the database like I have in the past for other stuff?

[tmcdonald]

Going your route, you would likely need to expand the database columns, but even that might not be enough if a message is sufficiently long. The route I would take would be to take the output of the event handler, or the check if you want to go that route, and save it to a file, then send that file off somewhere to a directory where it can be referenced/linked to in the notification.

[BanditBBS]

I like your idea better than mine, but I also just thought of an even cooler idea

cpu check is bad, kicks off an event handler that runs, gets the information it gets and then sets that to the CPU service objectnotes. I already have the object notes component installed, so now just trying to figure out how to access and use its functions from a plugin/eventhandler.

Hints welcomed, lol.

[ssax]

This should get you started.

Command:

Code: Select all

$USER1$/object_notes_handler.php '$HOSTNAME$' '$SERVICEDESC$'

/usr/local/nagios/libexec/object_notes_handler.php

Code: Select all

#!/usr/bin/php
<?php
include_once('/usr/local/nagiosxi/html/includes/components/objectnotes/objectnotes.inc.php');

$hostname = $argv[1];
$servicename = $argv[2];
$notes = "WHATEVER YOU WANT TO PUT IN HERE!\n";

objectnotes_component_set_service_notes("$hostname", "$servicename", "$notes");

exit(0);
?>

Let us know the results.

[ssax]

Also, you should be able to convert the script located here to php so that you can acheive your 4th+ check logic:

Code: Select all

https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/eventhandlers.html

[BanditBBS]

You sir, are awesome!

I'll be able to tweak this method to do what I want no problem!

When using the php code you supplied, it works, but I get this error every time:

Code: Select all

PHP Notice:  CCM Config File: /usr/local/nagiosxi/etc/components/ccm_config.inc.php is not writable by apache! in /usr/local/nagiosxi/html/includes/components/ccm/ccm.inc.php on line 123

[ssax]

Awesome, please post the output of these commands:

Code: Select all

ls -l /usr/local/nagiosxi/etc/components/ccm_config.inc.php
ls -ld /usr/local/nagiosxi/etc/components

Here's mine (I don't get that message):

Code: Select all

[root@ssc66xid ~]# ls -l /usr/local/nagiosxi/etc/components/ccm_config.inc.php
-rw-r--r-- 1 apache nagios 644 Jun 21 09:56 /usr/local/nagiosxi/etc/components/ccm_config.inc.php

[root@ssc66xid ~]# ls -ld /usr/local/nagiosxi/etc/components
drwsrwsr-x 3 apache nagios 4096 Mar  3 16:49 /usr/local/nagiosxi/etc/components

If they are different, run these commands and see if it resolves it for you:

Code: Select all

chown apache.nagios /usr/local/nagiosxi/etc/components/ccm_config.inc.php
chmod 644 /usr/local/nagiosxi/etc/components/ccm_config.inc.php
chown apache.nagios /usr/local/nagiosxi/etc/components
chmod 6775 /usr/local/nagiosxi/etc/components

If not, post the output of these commands:

Code: Select all

grep nag /etc/group
grep "User \|Group " /etc/httpd/conf/httpd.conf

Thank you

[BanditBBS]

My files are the same, what about you running this, is yours the same?

Code: Select all

ls -l /usr/local/nagiosxi/etc/components/ccm_config.inc.php
-rw-r--r-- 1 apache nagios 646 Jun 16 15:34 /usr/local/nagiosxi/etc/components/ccm_config.inc.php

Here is output to the other commands:

Code: Select all

[root@iss-chi-nag05 ~]# grep nag /etc/group
nagios:x:500:nagios,apache,snmptt,jclark
nagcmd:x:501:nagios,apache,snmptt
[root@iss-chi-nag05 ~]# grep "User \|Group " /etc/httpd/conf/httpd.conf
#  . On SCO (ODT 3) use "User nouser" and "Group nogroup".
#  when the value of (unsigned)Group is above 60000;
#  don't use Group #-1 on these systems!
User apache
Group apache

[ssax]

Looks good, what do the upper directories show?

Code: Select all

ls -ld /usr/local/nagiosxi/etc/components
ls -ld /usr/local/nagiosxi/etc
ls -ld /usr/local/nagiosxi

Here's mine:

Code: Select all

[root@ssc66xid ~]# ls -ld /usr/local/nagiosxi/etc/components
drwsrwsr-x 3 apache nagios 4096 Mar  3 16:49 /usr/local/nagiosxi/etc/components

[root@ssc66xid ~]# ls -ld /usr/local/nagiosxi/etc
drwxr-xr-x 3 nagios nagios 4096 Mar  3 16:48 /usr/local/nagiosxi/etc

[root@ssc66xid ~]# ls -ld /usr/local/nagiosxi
drwxr-xr-x 10 nagios nagios 4096 Mar  3 16:48 /usr/local/nagiosxi

[BanditBBS]

Mine are identical.

But really, your file was the same as mine?

As a test, I did:

Code: Select all

chmod 666 /usr/local/nagiosxi/etc/components/ccm_config.inc.php

Re-ran the script and I did not get the error any longer.

Which makes no sense since its owner is apache and was 644 and the error said apache couldn't write to it.