Email notifications stopped

[Fred Kroeger]

Picked up a strange problem this morning where Nagios notifications shows that an email notification was sent but there is no corresponding entry in the mail log.
In fact, there are 10 events where no email delivery has ocurred. Coincidentally (maybe/maybe not) the emails started being delivered again after I made a change and applied the configuration.
Normally , I have no problems with email notifications - but this is hard to quantify as the only way I know, is when people complain that they haven't received an email. The only reason it was picked up today was that a user logged in and found several Critical events for which no email was received.

I know it is hard to track down after the fact, but is there any situation in Nagios where a monitor goes Warning/Critical , the Notfications page shows that an email was sent, but the command to email the event doesn't run ? And then it all starts working again after Nagios is restarted? The worrying part is that the time period where emails were not sent is quite large, almost 4hours - between 04:36 and 08:25 this morning. 08:25 was when the config change was applied, with an email notification being sent two minutes later at 08:27.

(Running NagiosXI 5.2.9)

thanks... Fred

[rkennedy]

Can you PM over your profile for us to look at? (Admin -> System Profile -> Download Profile)

Also, from the Manage Email Settings page, are you using sendmail, or SMTP? If you're using sendmail, can you also attach your /var/log/maillog file?

EDIT: Files reiceved.

For reference -

Green lines indicate matching entries in maillog
Red lines indicate no match found in maillog

[Fred Kroeger]

Have PM'd profile, maillog and excerpt from Nagios Notifications list highlighting missing emails

Fred

[tgriep]

Thanks for the profile. I looked at is and the maillog file and couldn't find any errors in them.
We would have to see more log files to see if there were any errors.
Can you post your full /var/log/messages file from that time period?
Also, can you post the nagios log archive file for that time period as well?
That file should be in the /usr/local/nagios/var/archives folder.

[Fred Kroeger]

I may need to escalate this. I have discovered the same problem on a Nagios server at a different site.
There is a gap in emails recieved between Fri 5 Aug to Tue 9 Aug. The emails started being received again straight after I applied a Config Change on 9 Aug.
The Notifications page shows that Nagios sent emails for all the events during that window.
The server maillogs do not have any corresponding entries to indicate that an email was sent from Nagios. The maillogs do show that other emails were sent during that time - Nagios runs an email delivery check. It is just the Notifications that are not being sent.

So to me it is indicating clearly that Nagios is not sending out notifications. And when it starts again it is right after applying a configuration.

I have tracked down wproc error messges in the nagios logs in the archives as you suggested. I will PM you the files for 5th & 9th and for today.
I had a look at todays log and once again I can these wproc errors and then they stop after applying a config.

Ths is an excerpt of what I see:

Code: Select all

[1465216979] wproc: NOTIFY job 8918 from worker Core Worker 46372 is a non-check helper but exited with return code 1
[1465216979] wproc:   host=xx; service=NetApp Failed Disks; contact=SM9
[1465216979] wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1465216979] wproc:   stderr line 01: /root/sent: Permission denied (errno = 13)
[1465216979] wproc:   stdout line 01: Could not send the message.
[1465216979] wproc: NOTIFY job 8918 from worker Core Worker 46373 is a non-check helper but exited with return code 1
[1465216979] wproc:   host=xx; service=NetApp Failed Disks; contact=Storage Admins
[1465216979] wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1465216979] wproc:   stderr line 01: /root/sent: Permission denied (errno = 13)
[1465216979] wproc:   stdout line 01: Could not send the message.
[1465228029] wproc: NOTIFY job 47163 from worker Core Worker 46374 is a non-check helper but exited with return code 1
[1465228029] wproc:   host=xx; service=NetApp Failed Disks; contact=Storage Admins
[1465228029] wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1465228029] wproc:   stderr line 01: /root/sent: Permission denied (errno = 13)
[1465228029] wproc:   stdout line 01: Could not send the message.
[1465228029] wproc: NOTIFY job 47162 from worker Core Worker 46373 is a non-check helper but exited with return code 1
[1465228029] wproc:   host=xx; service=NetApp Failed Disks; contact=SM9
[1465228029] wproc:   early_timeout=0; exited_ok=1; wait_status=256; error_code=0;
[1465228029] wproc:   stderr line 01: /root/sent: Permission denied (errno = 13)
[1465228029] wproc:   stdout line 01: Could not send the message.

[Box293]

What is the output of:

Code: Select all

ps -ef | grep nagios.cfg

[tgriep]

These errors looks like the cause of the emails not being sent.

Code: Select all

[1465228029] wproc:   stderr line 01: /root/sent: Permission denied (errno = 13)
[1465216979] wproc:   stdout line 01: Could not send the message.

We would have to see how your notification command is setup and also the script that send the email.
When the notifications are sent, the scripts are run as the nagios user but the error shows that it is trying to save the sent email in the /root folder.
If that is fixed, then the emails should be sent.

[Fred Kroeger]

The email notifications *do* work normally. If what you are saying is correct, then they wouldn't be working. So I'm not sure that this is the solution.
Plus when they do stop working, they start working again after I apply a config. I'm guessing that if I restarted Nagios manually that would do the same.
I wll PM you the Notification command.

[tgriep]

A guess would be that if one on the notifications hangs causing the rest to fail and then restarting the nagios process clears the hung notification command which causes it to start working again.
Can you post how the notification command is defined as well and the output of this command run in a root shell on the server?

Code: Select all

ps -ef --cols=300

Thanks

[Fred Kroeger]

I removed the /root/sent file on both servers, so that whatever was trying to write to it wouldn't have a problem. Interesting to note that this file hasn't been recreated again. Also that error message hasn't re-appeared since the removal of that file. I will trawl through the email logs today and see if we hadn't any missing emails.

This is the command to email a Host event

Code: Select all

/usr/bin/printf "%b" "***** Nagios Monitor XI Alert *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nCategory: $HOSTALIAS$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\nMonitored by: http://asg-nagios/nagiosxi\n" | /usr/bin/mutt -s "** $NOTIFICATIONTYPE$   Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" -F /usr/local/etc/Nagios.muttrc $CONTACTEMAIL$

Code: Select all

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Apr13 ?        00:00:05 /sbin/init
root         2     0  0 Apr13 ?        00:00:00 [kthreadd]
root         3     2  0 Apr13 ?        06:39:44 [migration/0]
root         4     2  0 Apr13 ?        04:49:23 [ksoftirqd/0]
root         5     2  0 Apr13 ?        00:00:00 [migration/0]
root         6     2  0 Apr13 ?        00:35:26 [watchdog/0]
root         7     2  0 Apr13 ?        00:35:51 [migration/1]
root         8     2  0 Apr13 ?        00:00:00 [migration/1]
root         9     2  0 Apr13 ?        00:29:12 [ksoftirqd/1]
root        10     2  0 Apr13 ?        00:02:38 [watchdog/1]
root        11     2  0 Apr13 ?        00:56:07 [migration/2]
root        12     2  0 Apr13 ?        00:00:00 [migration/2]
root        13     2  0 Apr13 ?        00:49:14 [ksoftirqd/2]
root        14     2  0 Apr13 ?        00:06:16 [watchdog/2]
root        15     2  0 Apr13 ?        04:29:11 [migration/3]
root        16     2  0 Apr13 ?        00:00:00 [migration/3]
root        17     2  0 Apr13 ?        02:21:12 [ksoftirqd/3]
root        18     2  0 Apr13 ?        00:27:05 [watchdog/3]
root        19     2  0 Apr13 ?        03:20:32 [events/0]
root        20     2  0 Apr13 ?        02:07:37 [events/1]
root        21     2  0 Apr13 ?        00:27:59 [events/2]
root        22     2  0 Apr13 ?        02:16:59 [events/3]
root        23     2  0 Apr13 ?        00:00:00 [cgroup]
root        24     2  0 Apr13 ?        00:04:07 [khelper]
root        25     2  0 Apr13 ?        00:00:00 [netns]
root        26     2  0 Apr13 ?        00:00:00 [async/mgr]
root        27     2  0 Apr13 ?        00:00:00 [pm]
root        28     2  0 Apr13 ?        00:02:04 [sync_supers]
root        29     2  0 Apr13 ?        00:03:32 [bdi-default]
root        30     2  0 Apr13 ?        00:00:00 [kintegrityd/0]
root        31     2  0 Apr13 ?        00:00:00 [kintegrityd/1]
root        32     2  0 Apr13 ?        00:00:00 [kintegrityd/2]
root        33     2  0 Apr13 ?        00:00:00 [kintegrityd/3]
root        34     2  0 Apr13 ?        08:01:06 [kblockd/0]
root        35     2  0 Apr13 ?        00:45:27 [kblockd/1]
root        36     2  0 Apr13 ?        02:35:40 [kblockd/2]
root        37     2  0 Apr13 ?        04:50:50 [kblockd/3]
root        38     2  0 Apr13 ?        00:00:00 [kacpid]
root        39     2  0 Apr13 ?        00:00:00 [kacpi_notify]
root        40     2  0 Apr13 ?        00:00:00 [kacpi_hotplug]
root        41     2  0 Apr13 ?        00:00:00 [ata/0]
root        42     2  0 Apr13 ?        00:00:00 [ata/1]
root        43     2  0 Apr13 ?        00:00:00 [ata/2]
root        44     2  0 Apr13 ?        00:00:00 [ata/3]
root        45     2  0 Apr13 ?        00:00:00 [ata_aux]
root        46     2  0 Apr13 ?        00:00:00 [ksuspend_usbd]
root        47     2  0 Apr13 ?        00:00:00 [khubd]
root        48     2  0 Apr13 ?        00:00:00 [kseriod]
root        49     2  0 Apr13 ?        00:00:00 [md/0]
root        50     2  0 Apr13 ?        00:00:00 [md/1]
root        51     2  0 Apr13 ?        00:00:00 [md/2]
root        52     2  0 Apr13 ?        00:00:00 [md/3]
root        53     2  0 Apr13 ?        00:00:00 [md_misc/0]
root        54     2  0 Apr13 ?        00:00:00 [md_misc/1]
root        55     2  0 Apr13 ?        00:00:00 [md_misc/2]
root        56     2  0 Apr13 ?        00:00:00 [md_misc/3]
root        57     2  0 Apr13 ?        00:00:13 [khungtaskd]
root        58     2  0 Apr13 ?        00:28:23 [kswapd0]
root        59     2  0 Apr13 ?        00:00:00 [ksmd]
root        60     2  0 Apr13 ?        00:20:05 [khugepaged]
root        61     2  0 Apr13 ?        00:00:00 [aio/0]
root        62     2  0 Apr13 ?        00:00:00 [aio/1]
root        63     2  0 Apr13 ?        00:00:00 [aio/2]
root        64     2  0 Apr13 ?        00:00:00 [aio/3]
root        65     2  0 Apr13 ?        00:00:00 [crypto/0]
root        66     2  0 Apr13 ?        00:00:00 [crypto/1]
root        67     2  0 Apr13 ?        00:00:00 [crypto/2]
root        68     2  0 Apr13 ?        00:00:00 [crypto/3]
root        73     2  0 Apr13 ?        00:00:00 [kthrotld/0]
root        74     2  0 Apr13 ?        00:00:00 [kthrotld/1]
root        75     2  0 Apr13 ?        00:00:00 [kthrotld/2]
root        76     2  0 Apr13 ?        00:00:00 [kthrotld/3]
root        77     2  0 Apr13 ?        00:00:00 [pciehpd]
root        79     2  0 Apr13 ?        00:00:00 [kpsmoused]
root        80     2  0 Apr13 ?        00:00:00 [usbhid_resumer]
root       111     2  0 Apr13 ?        00:00:00 [kstriped]
root       365     2  0 Apr13 ?        00:00:00 [scsi_eh_0]
root       366     2  0 Apr13 ?        00:00:00 [scsi_eh_1]
root       376     2  0 Apr13 ?        00:32:34 [mpt_poll_0]
root       377     2  0 Apr13 ?        00:00:00 [mpt/0]
root       378     2  0 Apr13 ?        00:00:00 [scsi_eh_2]
root       439     2  0 Apr13 ?        00:00:00 [kdmflush]
root       441     2  0 Apr13 ?        00:00:00 [kdmflush]
root       462     2  0 Apr13 ?        07:56:08 [flush-253:1]
root       463     2  0 Apr13 ?        02:49:59 [jbd2/dm-1-8]
root       464     2  0 Apr13 ?        00:00:00 [ext4-dio-unwrit]
root       465     2  0 Apr13 ?        00:00:00 [ext4-dio-unwrit]
root       466     2  0 Apr13 ?        00:00:00 [ext4-dio-unwrit]
root       467     2  0 Apr13 ?        00:00:00 [ext4-dio-unwrit]
apache     530  5752  3 04:25 ?        00:06:15 /usr/sbin/httpd
postgres   540  5654  0 04:25 ?        00:00:14 postgres: nagiosxi nagiosxi 127.0.0.1(50857) idle
root       547     1  0 Apr13 ?        00:00:00 /sbin/udevd -d
root       735     2  0 Apr13 ?        00:05:50 [vmmemctl]
root       975     2  0 Apr13 ?        00:00:00 [kdmflush]
root       987     2  0 Apr13 ?        1-03:19:25 [flush-253:2]
root      1011     2  0 Apr13 ?        02:28:20 [jbd2/dm-2-8]
root      1012     2  0 Apr13 ?        00:00:00 [ext4-dio-unwrit]
root      1013     2  0 Apr13 ?        00:00:00 [ext4-dio-unwrit]
root      1014     2  0 Apr13 ?        00:00:00 [ext4-dio-unwrit]
root      1015     2  0 Apr13 ?        00:00:00 [ext4-dio-unwrit]
root      1118     2  0 Apr13 ?        00:07:20 [kauditd]
root      1509     1  0 Apr13 ?        04:46:50 /usr/sbin/vmtoolsd
root      1522     1  0 Apr13 ?        00:24:22 auditd
root      1541     1  0 Apr13 ?        00:48:26 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
root      1578     1  0 Apr13 ?        00:20:13 irqbalance
rpc       1592     1  0 Apr13 ?        00:00:40 rpcbind
root      1614     2  0 Apr13 ?        00:00:00 [rpciod/0]
root      1615     2  0 Apr13 ?        00:00:00 [rpciod/1]
root      1616     2  0 Apr13 ?        00:00:00 [rpciod/2]
root      1617     2  0 Apr13 ?        00:00:00 [rpciod/3]
rpcuser   1623     1  0 Apr13 ?        00:00:00 rpc.statd -p 662 -o 2020
dbus      1743     1  0 Apr13 ?        00:00:02 dbus-daemon --system
root      1767     1  0 Apr13 ?        00:00:00 /usr/sbin/acpid
68        1776     1  0 Apr13 ?        00:01:44 hald
root      1777  1776  0 Apr13 ?        00:00:00 hald-runner
root      1805  1777  0 Apr13 ?        00:00:00 hald-addon-input: Listening on /dev/input/event2 /dev/input/event0
68        1817  1777  0 Apr13 ?        00:00:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root      1856     1  0 Apr13 ?        00:00:00 rpc.rquotad -p 875
root      1860     1  0 Apr13 ?        00:00:00 rpc.mountd -p 892
root      1909     1  0 Apr13 ?        00:00:00 rpc.idmapd
root      1916     2  0 Apr13 ?        00:00:00 [lockd]
root      1917     2  0 Apr13 ?        00:00:11 [nfsd4]
root      1918     2  0 Apr13 ?        00:00:00 [nfsd4_callbacks]
root      1919     2  0 Apr13 ?        00:00:00 [nfsd]
root      1920     2  0 Apr13 ?        00:00:00 [nfsd]
root      1921     2  0 Apr13 ?        00:00:00 [nfsd]
root      1922     2  0 Apr13 ?        00:00:00 [nfsd]
root      1923     2  0 Apr13 ?        00:00:00 [nfsd]
root      1924     2  0 Apr13 ?        00:00:00 [nfsd]
root      1925     2  0 Apr13 ?        00:00:01 [nfsd]
root      1926     2  0 Apr13 ?        00:00:00 [nfsd]
root      1940     1  0 Apr13 ?        00:00:00 /usr/sbin/sshd
ntp       1982     1  0 Apr13 ?        00:00:18 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
gearmand  2206     1  2 Apr13 ?        2-16:31:55 /usr/sbin/gearmand -d -L 10.133.1.37
root      2290     1  0 Apr13 ?        00:03:05 /usr/libexec/postfix/master
postfix   2297  2290  0 Apr13 ?        00:01:27 qmgr -l -t fifo -u
root      2314     1  0 Apr13 ?        00:00:00 /usr/sbin/abrtd
root      2322     1  0 Apr13 ?        00:11:00 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages
nagios    2370     1  0 Apr13 ?        00:32:47 /usr/bin/mod_gearman_worker -d --config=/etc/mod_gearman/mod_gearman_worker.conf --pidfile=/var/mod_gearman/mod_gearman_worker.pid
nagios    2400     1  0 Apr13 ?        00:23:05 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
root      2410     1  0 Apr13 ?        00:00:00 /usr/sbin/atd
root      2419     1  0 Apr13 ?        00:00:00 /bin/sh /opt/OV/HPBsmIntNagios/genint/bin/genintd.sh /opt/OV/HPBsmIntNagios/genint/conf/nagios_server.conf
root      2429  2419  1 Apr13 ?        1-15:58:46 /opt/OV/HPBsmIntNagios/genint/bin/genintd /opt/OV/HPBsmIntNagios/genint/conf/nagios_server.conf
ajaxterm  2455     1  0 Apr13 ?        01:39:18 python /usr/share/ajaxterm/ajaxterm.py --daemon --port=8022 --uid=ajaxterm
root      2531     1  0 Apr13 tty1     00:00:00 /sbin/mingetty /dev/tty1
root      2533     1  0 Apr13 tty2     00:00:00 /sbin/mingetty /dev/tty2
root      2535     1  0 Apr13 tty3     00:00:00 /sbin/mingetty /dev/tty3
root      2537     1  0 Apr13 tty4     00:00:00 /sbin/mingetty /dev/tty4
root      2540     1  0 Apr13 tty5     00:00:00 /sbin/mingetty /dev/tty5
root      2544   547  0 Apr13 ?        00:00:00 /sbin/udevd -d
root      2545     1  0 Apr13 tty6     00:00:00 /sbin/mingetty /dev/tty6
apache    3201  5752  2 07:00 ?        00:00:33 /usr/sbin/httpd
postgres  3291  5654  0 07:00 ?        00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(32779) idle
apache    3324  5752  2 07:00 ?        00:00:32 /usr/sbin/httpd
postgres  3458  5654  0 07:00 ?        00:00:01 postgres: nagiosxi nagiosxi 127.0.0.1(32813) idle
apache    4002  5752  3 04:26 ?        00:05:49 /usr/sbin/httpd
postgres  4346  5654  0 04:26 ?        00:00:14 postgres: nagiosxi nagiosxi 127.0.0.1(51848) idle
postfix   4641  2290  0 06:19 ?        00:00:00 pickup -l -t fifo -u
root      5112   547  0 Jun14 ?        00:00:00 /sbin/udevd -d
postgres  5654     1  0 Jun24 ?        00:11:01 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
root      5752     1  0 Jun24 ?        00:06:59 /usr/sbin/httpd
postgres  5837  5654  0 Jun24 ?        00:06:32 postgres: logger process
postgres  5842  5654  0 Jun24 ?        00:36:42 postgres: writer process
postgres  5843  5654  0 Jun24 ?        00:28:28 postgres: wal writer process
postgres  5844  5654  0 Jun24 ?        00:10:20 postgres: autovacuum launcher process
postgres  5845  5654  0 Jun24 ?        00:22:44 postgres: stats collector process
root      6157  1940  0 Aug12 ?        00:00:00 sshd: knagaraaj [priv]
606       6185  6157  0 Aug12 ?        00:00:03 sshd: knagaraaj@pts/2
606       6188  6185  0 Aug12 pts/2    00:00:00 -bash
root      6485  6188  0 Aug12 pts/2    00:00:00 sudo su -
root      6599  6485  0 Aug12 pts/2    00:00:00 su -
root      6634  6599  0 Aug12 pts/2    00:00:00 -bash
root     12442     1  0 Jul19 ?        00:03:05 xinetd -stayalive -pidfile /var/run/xinetd.pid
apache   12551  5752  2 06:36 ?        00:01:21 /usr/sbin/httpd
postgres 12588  5654  0 06:36 ?        00:00:02 postgres: nagiosxi nagiosxi 127.0.0.1(46322) idle
apache   14400  5752  2 06:37 ?        00:01:08 /usr/sbin/httpd
root     14404  2429  0 07:05 ?        00:00:00 [sh] <defunct>
postgres 14498  5654  0 06:37 ?        00:00:02 postgres: nagiosxi nagiosxi 127.0.0.1(46911) idle
nagios   15215 12442  0 Aug03 ?        00:00:00 nsca -c /usr/local/nagios/etc/nsca.cfg --inetd
nagios   15216 12442  0 Aug03 ?        00:00:00 nsca -c /usr/local/nagios/etc/nsca.cfg --inetd
apache   19851  5752  3 05:57 ?        00:02:53 /usr/sbin/httpd
apache   20160  5752  3 05:01 ?        00:04:55 /usr/sbin/httpd
postgres 20321  5654  0 05:57 ?        00:00:06 postgres: nagiosxi nagiosxi 127.0.0.1(50988) idle
postgres 20702  5654  0 05:01 ?        00:00:11 postgres: nagiosxi nagiosxi 127.0.0.1(44958) idle
root     20786  1940  0 07:22 ?        00:00:00 sshd: fkroeger [priv]
fkroeger 21307 20786  0 07:22 ?        00:00:00 sshd: fkroeger@pts/0
fkroeger 21309 21307  0 07:22 pts/0    00:00:00 -bash
root     21421 21309  0 07:22 pts/0    00:00:00 sudo su -
root     21583 21421  0 07:22 pts/0    00:00:00 su -
root     21585 21583  0 07:22 pts/0    00:00:00 -bash
root     24004     1  0 Jul15 ?        00:02:36 crond
apache   24041  5752  3 04:21 ?        00:06:27 /usr/sbin/httpd
postgres 24113  5654  0 04:21 ?        00:00:14 postgres: nagiosxi nagiosxi 127.0.0.1(48494) idle
nagios   25352     1  4 04:50 ?        00:07:27 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   25354 25352  0 04:50 ?        00:00:14 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   25355 25352  0 04:50 ?        00:00:13 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   25356 25352  0 04:50 ?        00:00:12 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   25357 25352  0 04:50 ?        00:00:12 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   25358 25352  0 04:50 ?        00:00:12 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   25359 25352  0 04:50 ?        00:00:13 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   25361 32536  0 04:50 ?        00:00:28 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   25362 25361 10 04:50 ?        00:16:44 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   25365 25352  0 04:50 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
apache   25769  5752  3 05:04 ?        00:04:43 /usr/sbin/httpd
postgres 26394  5654  0 05:04 ?        00:00:11 postgres: nagiosxi nagiosxi 127.0.0.1(46531) idle
nagios   28062  2370  0 07:25 ?        00:00:00 /usr/bin/mod_gearman_worker -d --config=/etc/mod_gearman/mod_gearman_worker.conf --pidfile=/var/mod_gearman/mod_gearman_worker.pid
apache   28151  5752  3 04:23 ?        00:06:13 /usr/sbin/httpd
postgres 28183  5654  0 04:23 ?        00:00:14 postgres: nagiosxi nagiosxi 127.0.0.1(49605) idle
root     30354 24004  0 07:26 ?        00:00:00 CROND
root     30355 24004  0 07:26 ?        00:00:00 CROND
root     30356 24004  0 07:26 ?        00:00:00 CROND
root     30357 24004  0 07:26 ?        00:00:00 CROND
root     30358 24004  0 07:26 ?        00:00:00 CROND
nagios   30360 30357  0 07:26 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios   30362 30354  0 07:26 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios   30364 30356  0 07:26 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios   30365 30360  0 07:26 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios   30367 30362  0 07:26 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios   30368 30364  0 07:26 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios   30369 30358  0 07:26 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios   30371 30369  0 07:26 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios   30372 30355  0 07:26 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios   30374 30372  0 07:26 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
postgres 30382  5654  0 07:26 ?        00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(48430) idle
postgres 30385  5654  0 07:26 ?        00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(48431) idle
postgres 30402  5654  0 07:26 ?        00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(48435) idle
postgres 30408  5654  0 07:26 ?        00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(48436) idle
postgres 30419  5654  0 07:26 ?        00:00:00 postgres: nagiosxi nagiosxi 127.0.0.1(48438) idle
apache   30516  5752  3 04:24 ?        00:06:11 /usr/sbin/httpd
nagios   30549  2370  0 07:26 ?        00:00:00 /usr/bin/mod_gearman_worker -d --config=/etc/mod_gearman/mod_gearman_worker.conf --pidfile=/var/mod_gearman/mod_gearman_worker.pid
postgres 30629  5654  0 04:24 ?        00:00:14 postgres: nagiosxi nagiosxi 127.0.0.1(50261) idle
nagios   31096  2370  0 07:26 ?        00:00:00 /usr/bin/mod_gearman_worker -d --config=/etc/mod_gearman/mod_gearman_worker.conf --pidfile=/var/mod_gearman/mod_gearman_worker.pid
nagios   31097  2370  0 07:26 ?        00:00:00 /usr/bin/mod_gearman_worker -d --config=/etc/mod_gearman/mod_gearman_worker.conf --pidfile=/var/mod_gearman/mod_gearman_worker.pid
nagios   31265  2370  0 07:26 ?        00:00:00 /usr/bin/mod_gearman_worker -d --config=/etc/mod_gearman/mod_gearman_worker.conf --pidfile=/var/mod_gearman/mod_gearman_worker.pid
nagios   31435  2370  0 07:26 ?        00:00:00 /usr/bin/mod_gearman_worker -d --config=/etc/mod_gearman/mod_gearman_worker.conf --pidfile=/var/mod_gearman/mod_gearman_worker.pid
nagios   31487 25355  0 07:26 ?        00:00:00 /usr/local/nagios/libexec/check_nrpe -H 10.33.0.63 -u -t 45 -c check_cpu_stats -a -i 15 -w 95 -c 98
nagios   31842 25355  0 07:26 ?        00:00:00 /usr/local/nagios/libexec/check_nrpe -H 10.35.141.62 -u -t 45 -c check_cpu_stats -a -i 10 -w 95 -c 99
nagios   31865 25359  0 07:26 ?        00:00:00 /usr/local/nagios/libexec/check_nrpe -H 10.35.50.60 -u -t 45 -c check_cpu_stats -a -i 10 -w 95 -c 99
nagios   31915 25355  0 07:26 ?        00:00:00 /usr/local/nagios/libexec/check_nrpe -H 10.35.190.60 -u -t 45 -c check_init_service -a xinetd
nagios   31920 25358  0 07:26 ?        00:00:00 /usr/local/nagios/libexec/check_nrpe -H 10.35.190.50 -u -t 45 -c check_cpu_stats -a -i 10 -w 98 -c 99
root     31921 21585  0 07:26 pts/0    00:00:00 ps -ef --cols=300
apache   32292  5752  3 04:25 ?        00:05:45 /usr/sbin/httpd
root     32345     1  0 Jun24 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql    32471 32345 12 Jun24 ?        6-10:07:22 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306
nagios   32536     1  0 Jun24 ?        00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
postgres 32543  5654  0 04:25 ?        00:00:14 postgres: nagiosxi nagiosxi 127.0.0.1(50752) idle