Page 3 of 5
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 10:01 am
by jolson
Mitch,
We have made some headway on this bug - please see my post here for more details:
http://support.nagios.com/forum/viewtop ... 8&start=20
What's interesting is that you're experiencing a similar problem with a 1-node cluster. I am not convinced that your problem is the same as the one in the other thread.
I would like you to open two terminals to your Nagios Log Server and perform the following test.
On terminal 1, run:
Code: Select all
watch -n 1 "curl -s -XPOST 'http://localhost:9200/_export/state'"
Note the amount of states - we want to start at 0. If you have more than 0 states, please restart elasticsearch to clear them out.
On terminal 2, run:
Code: Select all
/usr/local/nagioslogserver/scripts/create_backup.sh
Please watch the 'states' - does your backup complete properly? What if you run it 2 or 3 times?
If the backups run fine from the command line as root, try switching to 'nagios' on terminal 2 and running them as that user:
Code: Select all
su - nagios
/usr/local/nagioslogserver/scripts/create_backup.sh
Is the result any different?
What if you launch the 'backups' command via the command subsystem?
2015-04-30 10_00_55-Command Subsystem • Nagios Log Server.png
Let us know. Thanks!
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 2:14 pm
by GhostRider2110
Not sure if it matters, but my Command Subsystem screen looks a little different.
CommandSub.png
Will work on your requests...
See-ya
Mitch
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 2:20 pm
by jolson
Mitch,
Could you press the 'reset all jobs' button please? That might actually be the root cause of what you're experiencing. The jobs are in an incorrect format. After resetting your jobs, do they appear properly?
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 2:27 pm
by GhostRider2110
State was 1 on first look. Restarted elasticsearch, which in turn cleared out all the stuck backups.
Ran backups several times as root and user nagios and all worked like a champ. Could not get them to hang....
See-ya
Mitch
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 2:39 pm
by jolson
I'm not sure why your command subsystem is jumbled, but a 'reset all jobs' should resolve that. As for backups not getting stuck, I'll take that as a good sign. Do they get stuck if run from the command subsys?
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 3:32 pm
by GhostRider2110
Ran the reset and now they look find. Ran the backup from there three times all success. Go figure. --Mitch
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 3:36 pm
by jolson
Mitch,
I've never associated stuck backup states with the jobs processes before - I will make a mental note to check on that in the future. I'll keep this thread open in case the problem resurfaces. There is a related bug that this thread helped uncover, so we have that to thank you for
Thanks again,
Jesse
P.S. Feel free to open a new thread regarding your backup repository - though it's possible that the job reset fixed that as well.
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 3:42 pm
by GhostRider2110
Thanks, I'll keep an eye on it over next couple and days and report back Monday.
See-ya
Mitch
Re: Multiple (40+) poller cron jobs running
Posted: Thu Apr 30, 2015 3:59 pm
by jolson
Sounds good - let us know.
Re: Multiple (40+) poller cron jobs running
Posted: Mon May 11, 2015 6:58 am
by GhostRider2110
Seeing problem again. Getting this alert from Nagios:
Code: Select all
Nagios has detected a problem with this service.
Notification Type: PROBLEM
Service: Cron Scheduling Daemon
Host: igaNagiosLS
Address: iganagioslog.iga.local
State: CRITICAL
Info:
crond dead but subsys locked
Date/Time: 2015-05-10 17:28:31
And have several hung/stuck poller/jobs processes:
Code: Select all
[root@IGAnagioslog ~]# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 May08 ? 00:00:01 /sbin/init
root 2 0 0 May08 ? 00:00:00 [kthreadd]
root 3 2 0 May08 ? 00:00:01 [migration/0]
root 4 2 0 May08 ? 00:00:11 [ksoftirqd/0]
root 5 2 0 May08 ? 00:00:00 [migration/0]
root 6 2 0 May08 ? 00:00:00 [watchdog/0]
root 7 2 0 May08 ? 00:00:01 [migration/1]
root 8 2 0 May08 ? 00:00:00 [migration/1]
root 9 2 0 May08 ? 00:00:18 [ksoftirqd/1]
root 10 2 0 May08 ? 00:00:00 [watchdog/1]
root 11 2 0 May08 ? 00:00:01 [migration/2]
root 12 2 0 May08 ? 00:00:00 [migration/2]
root 13 2 0 May08 ? 00:00:13 [ksoftirqd/2]
root 14 2 0 May08 ? 00:00:00 [watchdog/2]
root 15 2 0 May08 ? 00:00:01 [migration/3]
root 16 2 0 May08 ? 00:00:00 [migration/3]
root 17 2 0 May08 ? 00:00:11 [ksoftirqd/3]
root 18 2 0 May08 ? 00:00:00 [watchdog/3]
root 19 2 0 May08 ? 00:00:10 [events/0]
root 20 2 0 May08 ? 00:00:14 [events/1]
root 21 2 0 May08 ? 00:01:30 [events/2]
root 22 2 0 May08 ? 00:00:18 [events/3]
root 23 2 0 May08 ? 00:00:00 [cgroup]
root 24 2 0 May08 ? 00:00:00 [khelper]
root 25 2 0 May08 ? 00:00:00 [netns]
root 26 2 0 May08 ? 00:00:00 [async/mgr]
root 27 2 0 May08 ? 00:00:00 [pm]
root 28 2 0 May08 ? 00:00:00 [sync_supers]
root 29 2 0 May08 ? 00:00:00 [bdi-default]
root 30 2 0 May08 ? 00:00:00 [kintegrityd/0]
root 31 2 0 May08 ? 00:00:00 [kintegrityd/1]
root 32 2 0 May08 ? 00:00:00 [kintegrityd/2]
root 33 2 0 May08 ? 00:00:00 [kintegrityd/3]
root 34 2 0 May08 ? 00:00:03 [kblockd/0]
root 35 2 0 May08 ? 00:00:03 [kblockd/1]
root 36 2 0 May08 ? 00:00:03 [kblockd/2]
root 37 2 0 May08 ? 00:00:03 [kblockd/3]
root 38 2 0 May08 ? 00:00:00 [kacpid]
root 39 2 0 May08 ? 00:00:00 [kacpi_notify]
root 40 2 0 May08 ? 00:00:00 [kacpi_hotplug]
root 41 2 0 May08 ? 00:00:00 [ata/0]
root 42 2 0 May08 ? 00:00:00 [ata/1]
root 43 2 0 May08 ? 00:00:00 [ata/2]
root 44 2 0 May08 ? 00:00:00 [ata/3]
root 45 2 0 May08 ? 00:00:00 [ata_aux]
root 46 2 0 May08 ? 00:00:00 [ksuspend_usbd]
root 47 2 0 May08 ? 00:00:00 [khubd]
root 48 2 0 May08 ? 00:00:00 [kseriod]
root 49 2 0 May08 ? 00:00:00 [md/0]
root 50 2 0 May08 ? 00:00:00 [md/1]
root 51 2 0 May08 ? 00:00:00 [md/2]
root 52 2 0 May08 ? 00:00:00 [md/3]
root 53 2 0 May08 ? 00:00:00 [md_misc/0]
root 54 2 0 May08 ? 00:00:00 [md_misc/1]
root 55 2 0 May08 ? 00:00:00 [md_misc/2]
root 56 2 0 May08 ? 00:00:00 [md_misc/3]
root 57 2 0 May08 ? 00:00:00 [khungtaskd]
root 58 2 0 May08 ? 00:00:43 [kswapd0]
root 59 2 0 May08 ? 00:00:00 [ksmd]
root 60 2 0 May08 ? 00:00:03 [khugepaged]
root 61 2 0 May08 ? 00:00:00 [aio/0]
root 62 2 0 May08 ? 00:00:00 [aio/1]
root 63 2 0 May08 ? 00:00:00 [aio/2]
root 64 2 0 May08 ? 00:00:00 [aio/3]
root 65 2 0 May08 ? 00:00:00 [crypto/0]
root 66 2 0 May08 ? 00:00:00 [crypto/1]
root 67 2 0 May08 ? 00:00:00 [crypto/2]
root 68 2 0 May08 ? 00:00:00 [crypto/3]
root 73 2 0 May08 ? 00:00:00 [kthrotld/0]
root 74 2 0 May08 ? 00:00:00 [kthrotld/1]
root 75 2 0 May08 ? 00:00:00 [kthrotld/2]
root 76 2 0 May08 ? 00:00:00 [kthrotld/3]
root 77 2 0 May08 ? 00:00:00 [pciehpd]
root 79 2 0 May08 ? 00:00:00 [kpsmoused]
root 80 2 0 May08 ? 00:00:00 [usbhid_resumer]
root 322 2 0 May08 ? 00:00:00 [scsi_eh_0]
root 323 2 0 May08 ? 00:00:00 [scsi_eh_1]
root 329 2 0 May08 ? 00:00:03 [mpt_poll_0]
root 330 2 0 May08 ? 00:00:00 [mpt/0]
root 331 2 0 May08 ? 00:00:00 [scsi_eh_2]
root 387 2 0 May08 ? 00:00:07 [flush-8:0]
root 408 2 0 May08 ? 00:00:10 [jbd2/sda1-8]
root 409 2 0 May08 ? 00:00:00 [ext4-dio-unwrit]
root 410 2 0 May08 ? 00:00:00 [ext4-dio-unwrit]
root 411 2 0 May08 ? 00:00:00 [ext4-dio-unwrit]
root 412 2 0 May08 ? 00:00:00 [ext4-dio-unwrit]
root 488 1 0 May08 ? 00:00:00 /sbin/udevd -d
root 791 2 0 May08 ? 00:00:03 [vmmemctl]
root 1069 2 0 May08 ? 00:00:00 [kstriped]
root 1072 2 0 May08 ? 00:00:00 [kdmflush]
root 1075 2 0 May08 ? 00:00:00 [kdmflush]
root 1102 2 0 May08 ? 00:00:00 [xfs_mru_cache]
root 1103 2 0 May08 ? 00:00:05 [xfslogd/0]
root 1104 2 0 May08 ? 00:00:04 [xfslogd/1]
root 1105 2 0 May08 ? 00:00:05 [xfslogd/2]
root 1106 2 0 May08 ? 00:00:05 [xfslogd/3]
root 1107 2 0 May08 ? 00:00:04 [xfsdatad/0]
root 1108 2 0 May08 ? 00:00:04 [xfsdatad/1]
root 1109 2 0 May08 ? 00:00:03 [xfsdatad/2]
root 1110 2 0 May08 ? 00:00:05 [xfsdatad/3]
root 1111 2 0 May08 ? 00:00:00 [xfsconvertd/0]
root 1112 2 0 May08 ? 00:00:00 [xfsconvertd/1]
root 1113 2 0 May08 ? 00:00:00 [xfsconvertd/2]
root 1114 2 0 May08 ? 00:00:00 [xfsconvertd/3]
root 1115 2 0 May08 ? 00:00:19 [xfsbufd/dm-0]
root 1116 2 0 May08 ? 00:00:09 [xfssyncd/dm-0]
root 1117 2 0 May08 ? 00:00:06 [xfsaild/dm-0]
root 1118 2 0 May08 ? 00:00:00 [xfsbufd/dm-1]
root 1119 2 0 May08 ? 00:00:00 [xfssyncd/dm-1]
root 1120 2 0 May08 ? 00:00:00 [xfsaild/dm-1]
root 1154 2 0 May08 ? 00:00:02 [kauditd]
root 1362 1 0 May08 ? 00:00:08 auditd
root 1382 1 0 May08 ? 00:00:09 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
dbus 1396 1 0 May08 ? 00:00:00 dbus-daemon --system
68 1428 1 0 May08 ? 00:00:01 hald
root 1429 1428 0 May08 ? 00:00:00 hald-runner
root 1477 1429 0 May08 ? 00:00:00 hald-addon-input: Listening on /dev/input/event2 /dev/input/event0
68 1485 1429 0 May08 ? 00:00:00 hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event
root 1505 1 0 May08 ? 00:00:00 /usr/sbin/sshd
root 1514 1 0 May08 ? 00:00:05 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp 1523 1 0 May08 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root 1565 1 0 May08 ? 00:00:06 sendmail: accepting connections
smmsp 1574 1 0 May08 ? 00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 1589 1 0 May08 ? 00:03:22 /usr/bin/vmtoolsd
root 1610 1 0 May08 ? 00:00:02 crond
root 1704 1 0 May08 tty1 00:00:00 /sbin/mingetty /dev/tty1
root 1706 1 0 May08 tty2 00:00:00 /sbin/mingetty /dev/tty2
root 1708 1 0 May08 tty3 00:00:00 /sbin/mingetty /dev/tty3
root 1710 1 0 May08 tty4 00:00:00 /sbin/mingetty /dev/tty4
root 1712 1 0 May08 tty5 00:00:00 /sbin/mingetty /dev/tty5
root 1714 1 0 May08 tty6 00:00:00 /sbin/mingetty /dev/tty6
root 1718 488 0 May08 ? 00:00:00 /sbin/udevd -d
root 1719 488 0 May08 ? 00:00:00 /sbin/udevd -d
root 1720 1505 0 May08 ? 00:00:17 sshd: root@pts/0
root 1722 1720 0 May08 pts/0 00:00:00 -bash
root 4575 1 0 May08 pts/0 00:00:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/log
nagios 4577 4575 13 May08 ? 09:17:59 /usr/bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSIni
nagios 4647 1 99 May08 ? 3-02:07:47 /usr/bin/java -Xms2g -Xmx2g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:
root 7129 1610 0 May10 ? 00:00:00 CROND
nagios 7131 7129 0 May10 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 7133 7131 0 May10 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 7160 7133 0 May10 ? 00:00:05 /bin/sh /usr/local/nagioslogserver/scripts/create_backup.sh
root 27845 1 0 May08 ? 00:00:06 /usr/sbin/httpd
root 41356 1610 0 May09 ? 00:00:00 CROND
nagios 41358 41356 0 May09 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 41360 41358 0 May09 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 41551 41360 0 May09 ? 00:00:18 /bin/sh /usr/local/nagioslogserver/scripts/create_backup.sh
apache 42887 27845 0 07:20 ? 00:00:00 /usr/sbin/httpd
apache 43904 27845 0 07:25 ? 00:00:00 /usr/sbin/httpd
apache 43908 27845 0 07:25 ? 00:00:00 /usr/sbin/httpd
apache 43909 27845 0 07:25 ? 00:00:00 /usr/sbin/httpd
apache 44289 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44296 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44297 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44342 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44375 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44376 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44380 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44381 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44382 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
apache 44383 27845 0 07:27 ? 00:00:00 /usr/sbin/httpd
root 45086 1610 0 07:31 ? 00:00:00 CROND
nagios 45088 45086 0 07:31 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 45090 45088 0 07:31 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 45304 1610 0 07:32 ? 00:00:00 CROND
nagios 45307 45304 0 07:32 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 45308 45307 0 07:32 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 45507 1610 0 07:33 ? 00:00:00 CROND
root 45508 1610 0 07:33 ? 00:00:00 CROND
nagios 45509 45507 0 07:33 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 45510 45509 0 07:33 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 45512 45508 0 07:33 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 45513 45512 0 07:33 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 45811 1610 0 07:34 ? 00:00:00 CROND
root 45812 1610 0 07:34 ? 00:00:00 CROND
nagios 45813 45812 0 07:34 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 45814 45813 0 07:34 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 45815 45811 0 07:34 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 45816 45815 0 07:34 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 46010 1610 0 07:35 ? 00:00:00 CROND
root 46011 1610 0 07:35 ? 00:00:00 CROND
nagios 46012 46011 0 07:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 46013 46010 0 07:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 46014 46012 0 07:35 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 46015 46013 0 07:35 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 46128 1610 0 07:36 ? 00:00:00 CROND
root 46129 1610 0 07:36 ? 00:00:00 CROND
nagios 46130 46128 0 07:36 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 46131 46129 0 07:36 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 46132 46131 0 07:36 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 46133 46130 0 07:36 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 46247 1610 0 07:37 ? 00:00:00 CROND
root 46248 1610 0 07:37 ? 00:00:00 CROND
nagios 46249 46247 0 07:37 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 46250 46248 0 07:37 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 46251 46250 0 07:37 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 46252 46249 0 07:37 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 46358 1610 0 07:38 ? 00:00:00 CROND
root 46359 1610 0 07:38 ? 00:00:00 CROND
nagios 46360 46359 0 07:38 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 46361 46358 0 07:38 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 46362 46360 0 07:38 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 46363 46361 0 07:38 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 46557 1610 0 07:39 ? 00:00:00 CROND
root 46558 1610 0 07:39 ? 00:00:00 CROND
nagios 46559 46558 0 07:39 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 46560 46557 0 07:39 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 46561 46559 0 07:39 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 46562 46560 0 07:39 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 46653 1610 0 07:40 ? 00:00:00 CROND
root 46654 1610 0 07:40 ? 00:00:00 CROND
nagios 46657 46654 0 07:40 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 46658 46653 0 07:40 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 46659 46657 0 07:40 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 46660 46658 0 07:40 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 46737 2 0 07:40 ? 00:00:00 [flush-253:0]
root 46773 1610 0 07:41 ? 00:00:00 CROND
root 46774 1610 0 07:41 ? 00:00:00 CROND
nagios 46775 46773 0 07:41 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 46776 46774 0 07:41 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 46777 46775 0 07:41 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 46778 46776 0 07:41 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 46782 41551 0 07:41 ? 00:00:00 curl -s -XPOST http://localhost:9200/_export/state
root 46892 1610 0 07:42 ? 00:00:00 CROND
root 46893 1610 0 07:42 ? 00:00:00 CROND
nagios 46894 46892 0 07:42 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 46895 46893 0 07:42 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 46896 46894 0 07:42 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 46897 46895 0 07:42 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 47012 1610 0 07:43 ? 00:00:00 CROND
root 47013 1610 0 07:43 ? 00:00:00 CROND
nagios 47014 47012 0 07:43 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 47015 47013 0 07:43 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 47016 47014 0 07:43 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 47017 47015 0 07:43 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 47220 1610 0 07:44 ? 00:00:00 CROND
root 47221 1610 0 07:44 ? 00:00:00 CROND
nagios 47222 47220 0 07:44 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 47223 47221 0 07:44 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 47224 47223 0 07:44 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 47225 47222 0 07:44 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 47342 7160 0 07:45 ? 00:00:00 curl -s -XPOST http://localhost:9200/_export/state
root 47346 1610 0 07:45 ? 00:00:00 CROND
root 47347 1610 0 07:45 ? 00:00:00 CROND
nagios 47348 47347 0 07:45 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 47349 47346 0 07:45 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 47350 47348 0 07:45 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios 47351 47349 0 07:45 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root 47473 1610 0 07:46 ? 00:00:00 CROND
root 47474 1610 0 07:46 ? 00:00:00 CROND
nagios 47475 47474 0 07:46 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 47476 47473 0 07:46 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 47477 47476 0 07:46 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 47478 47475 0 07:46 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 47587 1610 0 07:47 ? 00:00:00 CROND
root 47588 1610 0 07:47 ? 00:00:00 CROND
nagios 47589 47587 0 07:47 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 47590 47589 0 07:47 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 47591 47588 0 07:47 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 47592 47591 0 07:47 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 47696 1610 0 07:48 ? 00:00:00 CROND
root 47697 1610 0 07:48 ? 00:00:00 CROND
nagios 47698 47696 0 07:48 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 47699 47697 0 07:48 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 47700 47698 0 07:48 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 47701 47699 0 07:48 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 47896 1610 0 07:49 ? 00:00:00 CROND
root 47897 1610 0 07:49 ? 00:00:00 CROND
nagios 47898 47897 0 07:49 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios 47899 47896 0 07:49 ? 00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios 47900 47899 0 07:49 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios 47901 47898 0 07:49 ? 00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root 47909 1722 0 07:49 pts/0 00:00:00 ps -ef
apache 55102 27845 0 May10 ? 00:00:02 /usr/sbin/httpd
apache 55103 27845 0 May10 ? 00:00:02 /usr/sbin/httpd
apache 55105 27845 0 May10 ? 00:00:02 /usr/sbin/httpd
apache 55106 27845 0 May10 ? 00:00:02 /usr/sbin/httpd
apache 55108 27845 0 May10 ? 00:00:02 /usr/sbin/httpd
apache 55109 27845 0 May10 ? 00:00:02 /usr/sbin/httpd
Nothing in the /usr/local/nagioslogserver/var/jobs.log or /usr/local/nagioslogserver/var/poller.log files.
Code: Select all
[root@IGAnagioslog ~]# tail -f /usr/local/nagioslogserver/var/poller.log /usr/local/nagioslogserver/var/jobs.log
==> /usr/local/nagioslogserver/var/poller.log <==
==> /usr/local/nagioslogserver/var/jobs.log <==
==> /usr/local/nagioslogserver/var/poller.log <==
==> /usr/local/nagioslogserver/var/jobs.log <==
NLS web interface is not responding. Yet load is nill.
Code: Select all
[root@IGAnagioslog ~]# top
top - 07:57:07 up 2 days, 19:36, 1 user, load average: 1.20, 1.45, 1.43
Tasks: 324 total, 1 running, 323 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.1%us, 0.2%sy, 5.1%ni, 45.1%id, 24.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4049240k total, 3937488k used, 111752k free, 208k buffers
Swap: 26226072k total, 166384k used, 26059688k free, 121424k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4647 nagios 20 0 78.5g 2.5g 113m S 100.2 64.2 4455:48 java
4577 nagios 39 19 3226m 652m 11m S 20.6 16.5 559:46.52 java
Have not seen any problems on the clients yet. Am looking on them now to see if there is a problem caused by the NLS issue.
See-ya
Mitch