Multiple (40+) poller cron jobs running

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Multiple (40+) poller cron jobs running

Post by jolson »

Mitch,

We have made some headway on this bug - please see my post here for more details: http://support.nagios.com/forum/viewtop ... 8&start=20

What's interesting is that you're experiencing a similar problem with a 1-node cluster. I am not convinced that your problem is the same as the one in the other thread.

I would like you to open two terminals to your Nagios Log Server and perform the following test.

On terminal 1, run:

Code: Select all

watch -n 1 "curl -s -XPOST 'http://localhost:9200/_export/state'"
Note the amount of states - we want to start at 0. If you have more than 0 states, please restart elasticsearch to clear them out.

On terminal 2, run:

Code: Select all

/usr/local/nagioslogserver/scripts/create_backup.sh
Please watch the 'states' - does your backup complete properly? What if you run it 2 or 3 times?
If the backups run fine from the command line as root, try switching to 'nagios' on terminal 2 and running them as that user:

Code: Select all

su - nagios
/usr/local/nagioslogserver/scripts/create_backup.sh
Is the result any different?

What if you launch the 'backups' command via the command subsystem?
2015-04-30 10_00_55-Command Subsystem • Nagios Log Server.png
Let us know. Thanks!
You do not have the required permissions to view the files attached to this post.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Multiple (40+) poller cron jobs running

Post by GhostRider2110 »

Not sure if it matters, but my Command Subsystem screen looks a little different.
CommandSub.png
Will work on your requests...

See-ya
Mitch
You do not have the required permissions to view the files attached to this post.
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Multiple (40+) poller cron jobs running

Post by jolson »

Mitch,

Could you press the 'reset all jobs' button please? That might actually be the root cause of what you're experiencing. The jobs are in an incorrect format. After resetting your jobs, do they appear properly?
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Multiple (40+) poller cron jobs running

Post by GhostRider2110 »

State was 1 on first look. Restarted elasticsearch, which in turn cleared out all the stuck backups.

Ran backups several times as root and user nagios and all worked like a champ. Could not get them to hang....

See-ya
Mitch
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Multiple (40+) poller cron jobs running

Post by jolson »

I'm not sure why your command subsystem is jumbled, but a 'reset all jobs' should resolve that. As for backups not getting stuck, I'll take that as a good sign. Do they get stuck if run from the command subsys?
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Multiple (40+) poller cron jobs running

Post by GhostRider2110 »

Ran the reset and now they look find. Ran the backup from there three times all success. Go figure. --Mitch
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Multiple (40+) poller cron jobs running

Post by jolson »

Mitch,

I've never associated stuck backup states with the jobs processes before - I will make a mental note to check on that in the future. I'll keep this thread open in case the problem resurfaces. There is a related bug that this thread helped uncover, so we have that to thank you for :D

Thanks again,


Jesse

P.S. Feel free to open a new thread regarding your backup repository - though it's possible that the job reset fixed that as well.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Multiple (40+) poller cron jobs running

Post by GhostRider2110 »

Thanks, I'll keep an eye on it over next couple and days and report back Monday.

See-ya
Mitch
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Multiple (40+) poller cron jobs running

Post by jolson »

Sounds good - let us know.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
GhostRider2110
Posts: 193
Joined: Thu Oct 30, 2014 8:04 am
Location: Indiana
Contact:
Contact GhostRider2110

Re: Multiple (40+) poller cron jobs running

Post by GhostRider2110 »

Seeing problem again. Getting this alert from Nagios:

Code: Select all

Nagios has detected a problem with this service.

Notification Type: PROBLEM

Service: Cron Scheduling Daemon
Host: igaNagiosLS
Address: iganagioslog.iga.local
State: CRITICAL
Info:
crond dead but subsys locked
Date/Time: 2015-05-10 17:28:31
And have several hung/stuck poller/jobs processes:

Code: Select all

[root@IGAnagioslog ~]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 May08 ?        00:00:01 /sbin/init
root         2     0  0 May08 ?        00:00:00 [kthreadd]
root         3     2  0 May08 ?        00:00:01 [migration/0]
root         4     2  0 May08 ?        00:00:11 [ksoftirqd/0]
root         5     2  0 May08 ?        00:00:00 [migration/0]
root         6     2  0 May08 ?        00:00:00 [watchdog/0]
root         7     2  0 May08 ?        00:00:01 [migration/1]
root         8     2  0 May08 ?        00:00:00 [migration/1]
root         9     2  0 May08 ?        00:00:18 [ksoftirqd/1]
root        10     2  0 May08 ?        00:00:00 [watchdog/1]
root        11     2  0 May08 ?        00:00:01 [migration/2]
root        12     2  0 May08 ?        00:00:00 [migration/2]
root        13     2  0 May08 ?        00:00:13 [ksoftirqd/2]
root        14     2  0 May08 ?        00:00:00 [watchdog/2]
root        15     2  0 May08 ?        00:00:01 [migration/3]
root        16     2  0 May08 ?        00:00:00 [migration/3]
root        17     2  0 May08 ?        00:00:11 [ksoftirqd/3]
root        18     2  0 May08 ?        00:00:00 [watchdog/3]
root        19     2  0 May08 ?        00:00:10 [events/0]
root        20     2  0 May08 ?        00:00:14 [events/1]
root        21     2  0 May08 ?        00:01:30 [events/2]
root        22     2  0 May08 ?        00:00:18 [events/3]
root        23     2  0 May08 ?        00:00:00 [cgroup]
root        24     2  0 May08 ?        00:00:00 [khelper]
root        25     2  0 May08 ?        00:00:00 [netns]
root        26     2  0 May08 ?        00:00:00 [async/mgr]
root        27     2  0 May08 ?        00:00:00 [pm]
root        28     2  0 May08 ?        00:00:00 [sync_supers]
root        29     2  0 May08 ?        00:00:00 [bdi-default]
root        30     2  0 May08 ?        00:00:00 [kintegrityd/0]
root        31     2  0 May08 ?        00:00:00 [kintegrityd/1]
root        32     2  0 May08 ?        00:00:00 [kintegrityd/2]
root        33     2  0 May08 ?        00:00:00 [kintegrityd/3]
root        34     2  0 May08 ?        00:00:03 [kblockd/0]
root        35     2  0 May08 ?        00:00:03 [kblockd/1]
root        36     2  0 May08 ?        00:00:03 [kblockd/2]
root        37     2  0 May08 ?        00:00:03 [kblockd/3]
root        38     2  0 May08 ?        00:00:00 [kacpid]
root        39     2  0 May08 ?        00:00:00 [kacpi_notify]
root        40     2  0 May08 ?        00:00:00 [kacpi_hotplug]
root        41     2  0 May08 ?        00:00:00 [ata/0]
root        42     2  0 May08 ?        00:00:00 [ata/1]
root        43     2  0 May08 ?        00:00:00 [ata/2]
root        44     2  0 May08 ?        00:00:00 [ata/3]
root        45     2  0 May08 ?        00:00:00 [ata_aux]
root        46     2  0 May08 ?        00:00:00 [ksuspend_usbd]
root        47     2  0 May08 ?        00:00:00 [khubd]
root        48     2  0 May08 ?        00:00:00 [kseriod]
root        49     2  0 May08 ?        00:00:00 [md/0]
root        50     2  0 May08 ?        00:00:00 [md/1]
root        51     2  0 May08 ?        00:00:00 [md/2]
root        52     2  0 May08 ?        00:00:00 [md/3]
root        53     2  0 May08 ?        00:00:00 [md_misc/0]
root        54     2  0 May08 ?        00:00:00 [md_misc/1]
root        55     2  0 May08 ?        00:00:00 [md_misc/2]
root        56     2  0 May08 ?        00:00:00 [md_misc/3]
root        57     2  0 May08 ?        00:00:00 [khungtaskd]
root        58     2  0 May08 ?        00:00:43 [kswapd0]
root        59     2  0 May08 ?        00:00:00 [ksmd]
root        60     2  0 May08 ?        00:00:03 [khugepaged]
root        61     2  0 May08 ?        00:00:00 [aio/0]
root        62     2  0 May08 ?        00:00:00 [aio/1]
root        63     2  0 May08 ?        00:00:00 [aio/2]
root        64     2  0 May08 ?        00:00:00 [aio/3]
root        65     2  0 May08 ?        00:00:00 [crypto/0]
root        66     2  0 May08 ?        00:00:00 [crypto/1]
root        67     2  0 May08 ?        00:00:00 [crypto/2]
root        68     2  0 May08 ?        00:00:00 [crypto/3]
root        73     2  0 May08 ?        00:00:00 [kthrotld/0]
root        74     2  0 May08 ?        00:00:00 [kthrotld/1]
root        75     2  0 May08 ?        00:00:00 [kthrotld/2]
root        76     2  0 May08 ?        00:00:00 [kthrotld/3]
root        77     2  0 May08 ?        00:00:00 [pciehpd]
root        79     2  0 May08 ?        00:00:00 [kpsmoused]
root        80     2  0 May08 ?        00:00:00 [usbhid_resumer]
root       322     2  0 May08 ?        00:00:00 [scsi_eh_0]
root       323     2  0 May08 ?        00:00:00 [scsi_eh_1]
root       329     2  0 May08 ?        00:00:03 [mpt_poll_0]
root       330     2  0 May08 ?        00:00:00 [mpt/0]
root       331     2  0 May08 ?        00:00:00 [scsi_eh_2]
root       387     2  0 May08 ?        00:00:07 [flush-8:0]
root       408     2  0 May08 ?        00:00:10 [jbd2/sda1-8]
root       409     2  0 May08 ?        00:00:00 [ext4-dio-unwrit]
root       410     2  0 May08 ?        00:00:00 [ext4-dio-unwrit]
root       411     2  0 May08 ?        00:00:00 [ext4-dio-unwrit]
root       412     2  0 May08 ?        00:00:00 [ext4-dio-unwrit]
root       488     1  0 May08 ?        00:00:00 /sbin/udevd -d
root       791     2  0 May08 ?        00:00:03 [vmmemctl]
root      1069     2  0 May08 ?        00:00:00 [kstriped]
root      1072     2  0 May08 ?        00:00:00 [kdmflush]
root      1075     2  0 May08 ?        00:00:00 [kdmflush]
root      1102     2  0 May08 ?        00:00:00 [xfs_mru_cache]
root      1103     2  0 May08 ?        00:00:05 [xfslogd/0]
root      1104     2  0 May08 ?        00:00:04 [xfslogd/1]
root      1105     2  0 May08 ?        00:00:05 [xfslogd/2]
root      1106     2  0 May08 ?        00:00:05 [xfslogd/3]
root      1107     2  0 May08 ?        00:00:04 [xfsdatad/0]
root      1108     2  0 May08 ?        00:00:04 [xfsdatad/1]
root      1109     2  0 May08 ?        00:00:03 [xfsdatad/2]
root      1110     2  0 May08 ?        00:00:05 [xfsdatad/3]
root      1111     2  0 May08 ?        00:00:00 [xfsconvertd/0]
root      1112     2  0 May08 ?        00:00:00 [xfsconvertd/1]
root      1113     2  0 May08 ?        00:00:00 [xfsconvertd/2]
root      1114     2  0 May08 ?        00:00:00 [xfsconvertd/3]
root      1115     2  0 May08 ?        00:00:19 [xfsbufd/dm-0]
root      1116     2  0 May08 ?        00:00:09 [xfssyncd/dm-0]
root      1117     2  0 May08 ?        00:00:06 [xfsaild/dm-0]
root      1118     2  0 May08 ?        00:00:00 [xfsbufd/dm-1]
root      1119     2  0 May08 ?        00:00:00 [xfssyncd/dm-1]
root      1120     2  0 May08 ?        00:00:00 [xfsaild/dm-1]
root      1154     2  0 May08 ?        00:00:02 [kauditd]
root      1362     1  0 May08 ?        00:00:08 auditd
root      1382     1  0 May08 ?        00:00:09 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
dbus      1396     1  0 May08 ?        00:00:00 dbus-daemon --system
68        1428     1  0 May08 ?        00:00:01 hald
root      1429  1428  0 May08 ?        00:00:00 hald-runner
root      1477  1429  0 May08 ?        00:00:00 hald-addon-input: Listening on /dev/input/event2 /dev/input/event0
68        1485  1429  0 May08 ?        00:00:00 hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event
root      1505     1  0 May08 ?        00:00:00 /usr/sbin/sshd
root      1514     1  0 May08 ?        00:00:05 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp       1523     1  0 May08 ?        00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      1565     1  0 May08 ?        00:00:06 sendmail: accepting connections
smmsp     1574     1  0 May08 ?        00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root      1589     1  0 May08 ?        00:03:22 /usr/bin/vmtoolsd
root      1610     1  0 May08 ?        00:00:02 crond
root      1704     1  0 May08 tty1     00:00:00 /sbin/mingetty /dev/tty1
root      1706     1  0 May08 tty2     00:00:00 /sbin/mingetty /dev/tty2
root      1708     1  0 May08 tty3     00:00:00 /sbin/mingetty /dev/tty3
root      1710     1  0 May08 tty4     00:00:00 /sbin/mingetty /dev/tty4
root      1712     1  0 May08 tty5     00:00:00 /sbin/mingetty /dev/tty5
root      1714     1  0 May08 tty6     00:00:00 /sbin/mingetty /dev/tty6
root      1718   488  0 May08 ?        00:00:00 /sbin/udevd -d
root      1719   488  0 May08 ?        00:00:00 /sbin/udevd -d
root      1720  1505  0 May08 ?        00:00:17 sshd: root@pts/0 
root      1722  1720  0 May08 pts/0    00:00:00 -bash
root      4575     1  0 May08 pts/0    00:00:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/log
nagios    4577  4575 13 May08 ?        09:17:59 /usr/bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSIni
nagios    4647     1 99 May08 ?        3-02:07:47 /usr/bin/java -Xms2g -Xmx2g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:
root      7129  1610  0 May10 ?        00:00:00 CROND
nagios    7131  7129  0 May10 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios    7133  7131  0 May10 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios    7160  7133  0 May10 ?        00:00:05 /bin/sh /usr/local/nagioslogserver/scripts/create_backup.sh
root     27845     1  0 May08 ?        00:00:06 /usr/sbin/httpd
root     41356  1610  0 May09 ?        00:00:00 CROND
nagios   41358 41356  0 May09 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   41360 41358  0 May09 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   41551 41360  0 May09 ?        00:00:18 /bin/sh /usr/local/nagioslogserver/scripts/create_backup.sh
apache   42887 27845  0 07:20 ?        00:00:00 /usr/sbin/httpd
apache   43904 27845  0 07:25 ?        00:00:00 /usr/sbin/httpd
apache   43908 27845  0 07:25 ?        00:00:00 /usr/sbin/httpd
apache   43909 27845  0 07:25 ?        00:00:00 /usr/sbin/httpd
apache   44289 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44296 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44297 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44342 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44375 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44376 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44380 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44381 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44382 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
apache   44383 27845  0 07:27 ?        00:00:00 /usr/sbin/httpd
root     45086  1610  0 07:31 ?        00:00:00 CROND
nagios   45088 45086  0 07:31 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   45090 45088  0 07:31 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     45304  1610  0 07:32 ?        00:00:00 CROND
nagios   45307 45304  0 07:32 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   45308 45307  0 07:32 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     45507  1610  0 07:33 ?        00:00:00 CROND
root     45508  1610  0 07:33 ?        00:00:00 CROND
nagios   45509 45507  0 07:33 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   45510 45509  0 07:33 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   45512 45508  0 07:33 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   45513 45512  0 07:33 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     45811  1610  0 07:34 ?        00:00:00 CROND
root     45812  1610  0 07:34 ?        00:00:00 CROND
nagios   45813 45812  0 07:34 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   45814 45813  0 07:34 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   45815 45811  0 07:34 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   45816 45815  0 07:34 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     46010  1610  0 07:35 ?        00:00:00 CROND
root     46011  1610  0 07:35 ?        00:00:00 CROND
nagios   46012 46011  0 07:35 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   46013 46010  0 07:35 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   46014 46012  0 07:35 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   46015 46013  0 07:35 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     46128  1610  0 07:36 ?        00:00:00 CROND
root     46129  1610  0 07:36 ?        00:00:00 CROND
nagios   46130 46128  0 07:36 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   46131 46129  0 07:36 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   46132 46131  0 07:36 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   46133 46130  0 07:36 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     46247  1610  0 07:37 ?        00:00:00 CROND
root     46248  1610  0 07:37 ?        00:00:00 CROND
nagios   46249 46247  0 07:37 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   46250 46248  0 07:37 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   46251 46250  0 07:37 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   46252 46249  0 07:37 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     46358  1610  0 07:38 ?        00:00:00 CROND
root     46359  1610  0 07:38 ?        00:00:00 CROND
nagios   46360 46359  0 07:38 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   46361 46358  0 07:38 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   46362 46360  0 07:38 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   46363 46361  0 07:38 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     46557  1610  0 07:39 ?        00:00:00 CROND
root     46558  1610  0 07:39 ?        00:00:00 CROND
nagios   46559 46558  0 07:39 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   46560 46557  0 07:39 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   46561 46559  0 07:39 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   46562 46560  0 07:39 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     46653  1610  0 07:40 ?        00:00:00 CROND
root     46654  1610  0 07:40 ?        00:00:00 CROND
nagios   46657 46654  0 07:40 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   46658 46653  0 07:40 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   46659 46657  0 07:40 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   46660 46658  0 07:40 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     46737     2  0 07:40 ?        00:00:00 [flush-253:0]
root     46773  1610  0 07:41 ?        00:00:00 CROND
root     46774  1610  0 07:41 ?        00:00:00 CROND
nagios   46775 46773  0 07:41 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   46776 46774  0 07:41 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   46777 46775  0 07:41 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   46778 46776  0 07:41 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   46782 41551  0 07:41 ?        00:00:00 curl -s -XPOST http://localhost:9200/_export/state
root     46892  1610  0 07:42 ?        00:00:00 CROND
root     46893  1610  0 07:42 ?        00:00:00 CROND
nagios   46894 46892  0 07:42 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   46895 46893  0 07:42 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   46896 46894  0 07:42 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   46897 46895  0 07:42 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     47012  1610  0 07:43 ?        00:00:00 CROND
root     47013  1610  0 07:43 ?        00:00:00 CROND
nagios   47014 47012  0 07:43 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   47015 47013  0 07:43 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   47016 47014  0 07:43 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   47017 47015  0 07:43 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     47220  1610  0 07:44 ?        00:00:00 CROND
root     47221  1610  0 07:44 ?        00:00:00 CROND
nagios   47222 47220  0 07:44 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   47223 47221  0 07:44 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   47224 47223  0 07:44 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   47225 47222  0 07:44 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   47342  7160  0 07:45 ?        00:00:00 curl -s -XPOST http://localhost:9200/_export/state
root     47346  1610  0 07:45 ?        00:00:00 CROND
root     47347  1610  0 07:45 ?        00:00:00 CROND
nagios   47348 47347  0 07:45 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   47349 47346  0 07:45 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   47350 47348  0 07:45 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
nagios   47351 47349  0 07:45 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
root     47473  1610  0 07:46 ?        00:00:00 CROND
root     47474  1610  0 07:46 ?        00:00:00 CROND
nagios   47475 47474  0 07:46 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   47476 47473  0 07:46 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   47477 47476  0 07:46 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   47478 47475  0 07:46 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     47587  1610  0 07:47 ?        00:00:00 CROND
root     47588  1610  0 07:47 ?        00:00:00 CROND
nagios   47589 47587  0 07:47 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   47590 47589  0 07:47 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   47591 47588  0 07:47 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   47592 47591  0 07:47 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     47696  1610  0 07:48 ?        00:00:00 CROND
root     47697  1610  0 07:48 ?        00:00:00 CROND
nagios   47698 47696  0 07:48 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   47699 47697  0 07:48 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   47700 47698  0 07:48 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   47701 47699  0 07:48 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     47896  1610  0 07:49 ?        00:00:00 CROND
root     47897  1610  0 07:49 ?        00:00:00 CROND
nagios   47898 47897  0 07:49 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller > /usr/local/nagioslogserver/var/poller.log 2>&1
nagios   47899 47896  0 07:49 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs > /usr/local/nagioslogserver/var/jobs.log 2>&1
nagios   47900 47899  0 07:49 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php jobs
nagios   47901 47898  0 07:49 ?        00:00:00 /usr/bin/php -q /var/www/html/nagioslogserver/www/index.php poller
root     47909  1722  0 07:49 pts/0    00:00:00 ps -ef
apache   55102 27845  0 May10 ?        00:00:02 /usr/sbin/httpd
apache   55103 27845  0 May10 ?        00:00:02 /usr/sbin/httpd
apache   55105 27845  0 May10 ?        00:00:02 /usr/sbin/httpd
apache   55106 27845  0 May10 ?        00:00:02 /usr/sbin/httpd
apache   55108 27845  0 May10 ?        00:00:02 /usr/sbin/httpd
apache   55109 27845  0 May10 ?        00:00:02 /usr/sbin/httpd
Nothing in the /usr/local/nagioslogserver/var/jobs.log or /usr/local/nagioslogserver/var/poller.log files.

Code: Select all

[root@IGAnagioslog ~]# tail -f /usr/local/nagioslogserver/var/poller.log /usr/local/nagioslogserver/var/jobs.log
==> /usr/local/nagioslogserver/var/poller.log <==

==> /usr/local/nagioslogserver/var/jobs.log <==

==> /usr/local/nagioslogserver/var/poller.log <==

==> /usr/local/nagioslogserver/var/jobs.log <==
NLS web interface is not responding. Yet load is nill.

Code: Select all

[root@IGAnagioslog ~]# top

top - 07:57:07 up 2 days, 19:36,  1 user,  load average: 1.20, 1.45, 1.43
Tasks: 324 total,   1 running, 323 sleeping,   0 stopped,   0 zombie
Cpu(s): 25.1%us,  0.2%sy,  5.1%ni, 45.1%id, 24.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4049240k total,  3937488k used,   111752k free,      208k buffers
Swap: 26226072k total,   166384k used, 26059688k free,   121424k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                               
 4647 nagios    20   0 78.5g 2.5g 113m S 100.2 64.2   4455:48 java                                                                                                                                 
 4577 nagios    39  19 3226m 652m  11m S 20.6 16.5 559:46.52 java
Have not seen any problems on the clients yet. Am looking on them now to see if there is a problem caused by the NLS issue.

See-ya
Mitch
Top
Locked

Return to “Nagios Log Server”