Nagios cmdsubsys.log Growing Excessively

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
nortonhealthcare
Posts: 33
Joined: Thu Aug 10, 2017 8:22 am

Re: Nagios cmdsubsys.log Growing Excessively

Post by nortonhealthcare »

Here is the information you requested.

Code: Select all

[root@oplp1521 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  4.9G   46G  10% /
devtmpfs                  24G     0   24G   0% /dev
tmpfs                     24G     0   24G   0% /dev/shm
tmpfs                     24G  2.3G   21G  10% /run
tmpfs                     24G     0   24G   0% /sys/fs/cgroup
/dev/emcpowera2          1.1G  215M  832M  21% /boot
/dev/emcpowera1          192M  9.8M  183M   6% /boot/efi
/dev/mapper/centos-home   57G   33M   57G   1% /home
tmpfs                    4.7G     0  4.7G   0% /run/user/1003
tmpfs                    4.7G     0  4.7G   0% /run/user/1004

Code: Select all

[root@oplp1521 ~]# df -i
Filesystem                Inodes  IUsed    IFree IUse% Mounted on
/dev/mapper/centos-root 26198016 113394 26084622    1% /
devtmpfs                 6052139    612  6051527    1% /dev
tmpfs                    6055592      1  6055591    1% /dev/shm
tmpfs                    6055592   1196  6054396    1% /run
tmpfs                    6055592     16  6055576    1% /sys/fs/cgroup
/dev/emcpowera2           540608     25   540583    1% /boot
/dev/emcpowera1                0      0        0     - /boot/efi
/dev/mapper/centos-home 29734912     37 29734875    1% /home
tmpfs                    6055592      1  6055591    1% /run/user/1003
tmpfs                    6055592      1  6055591    1% /run/user/1004

Code: Select all

[root@oplp1521 ~]# service nagios restart
Restarting nagios (via systemctl):                         [  OK  ]
[root@oplp1521 ~]# service nagios status
nagios is not running

Code: Select all

[root@oplp1521 ~]# service nagios status
nagios is not running
[root@oplp1521 ~]# tail -50 /var/log/messages
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for log data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for system command data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for event handler data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for notification data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for comment data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for downtime data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for flapping data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for program status data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for host status data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for service status data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for adaptive program data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for adaptive host data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for adaptive service data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for external command data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for aggregated status data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for retention data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for contact data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for contact notification data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for acknowledgement data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for state change data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for contact status data'
Sep 28 14:38:35 oplp1521 nagios: ndomod registered for adaptive contact data'
Sep 28 14:38:35 oplp1521 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
Sep 28 14:38:35 oplp1521 nagios: Error: Module loading failed. Aborting.
Sep 28 14:38:35 oplp1521 nagios: ndomod: Shutdown complete.
Sep 28 14:38:35 oplp1521 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
Sep 28 14:38:35 oplp1521 kill: kill: cannot find process ""
Sep 28 14:38:35 oplp1521 systemd: nagios.service: control process exited, code=exited status=1
Sep 28 14:38:37 oplp1521 systemd: Unit nagios.service entered failed state.
Sep 28 14:38:37 oplp1521 systemd: nagios.service failed.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743813 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743813 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743814 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743814 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743815 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743815 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743818 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743818 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743816 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743816 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743819 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743819 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743817 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743817 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743820 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743820 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743821 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743821 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Started Session 743822 of user nagios.
Sep 28 14:39:01 oplp1521 systemd: Starting Session 743822 of user nagios.

Code: Select all

[root@oplp1521 ~]# tail /usr/local/nagios/var/nagios.log
[1538159915] ndomod registered for contact data'
[1538159915] ndomod registered for contact notification data'
[1538159915] ndomod registered for acknowledgement data'
[1538159915] ndomod registered for state change data'
[1538159915] ndomod registered for contact status data'
[1538159915] ndomod registered for adaptive contact data'
[1538159915] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1538159915] Error: Module loading failed. Aborting.
[1538159915] ndomod: Shutdown complete.
[1538159915] Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.

Code: Select all

[root@oplp1521 ~]# ps -ef --cols=300
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Aug09 ?        01:06:19 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root         2     0  0 Aug09 ?        00:00:02 [kthreadd]
root         3     2  0 Aug09 ?        00:00:41 [ksoftirqd/0]
root         5     2  0 Aug09 ?        00:00:00 [kworker/0:0H]
root         8     2  0 Aug09 ?        00:00:04 [migration/0]
root         9     2  0 Aug09 ?        00:00:00 [rcu_bh]
root        10     2  0 Aug09 ?        00:20:26 [rcu_sched]
root        11     2  0 Aug09 ?        00:00:00 [lru-add-drain]
root        12     2  0 Aug09 ?        00:00:17 [watchdog/0]
root        13     2  0 Aug09 ?        00:00:16 [watchdog/1]
root        14     2  0 Aug09 ?        00:00:27 [migration/1]
root        15     2  0 Aug09 ?        00:00:25 [ksoftirqd/1]
root        17     2  0 Aug09 ?        00:00:00 [kworker/1:0H]
root        18     2  0 Aug09 ?        00:00:16 [watchdog/2]
root        19     2  0 Aug09 ?        00:00:08 [migration/2]
root        20     2  0 Aug09 ?        00:00:23 [ksoftirqd/2]
root        22     2  0 Aug09 ?        00:00:00 [kworker/2:0H]
root        23     2  0 Aug09 ?        00:00:17 [watchdog/3]
root        24     2  0 Aug09 ?        00:00:08 [migration/3]
root        25     2  0 Aug09 ?        00:00:19 [ksoftirqd/3]
root        27     2  0 Aug09 ?        00:00:00 [kworker/3:0H]
root        28     2  0 Aug09 ?        00:00:17 [watchdog/4]
root        29     2  0 Aug09 ?        00:00:06 [migration/4]
root        30     2  0 Aug09 ?        00:00:03 [ksoftirqd/4]
root        32     2  0 Aug09 ?        00:00:00 [kworker/4:0H]
root        33     2  0 Aug09 ?        00:00:16 [watchdog/5]
root        34     2  0 Aug09 ?        00:00:04 [migration/5]
root        35     2  0 Aug09 ?        00:00:21 [ksoftirqd/5]
root        37     2  0 Aug09 ?        00:00:00 [kworker/5:0H]
root        38     2  0 Aug09 ?        00:00:17 [watchdog/6]
root        39     2  0 Aug09 ?        00:00:03 [migration/6]
root        40     2  0 Aug09 ?        00:00:07 [ksoftirqd/6]
root        42     2  0 Aug09 ?        00:00:00 [kworker/6:0H]
root        44     2  0 Aug09 ?        00:00:16 [watchdog/7]
root        45     2  0 Aug09 ?        00:00:26 [migration/7]
root        46     2  0 Aug09 ?        00:00:05 [ksoftirqd/7]
root        48     2  0 Aug09 ?        00:00:00 [kworker/7:0H]
root        49     2  0 Aug09 ?        00:00:16 [watchdog/8]
root        50     2  0 Aug09 ?        00:00:08 [migration/8]
root        51     2  0 Aug09 ?        00:00:04 [ksoftirqd/8]
root        53     2  0 Aug09 ?        00:00:00 [kworker/8:0H]
root        54     2  0 Aug09 ?        00:00:16 [watchdog/9]
root        55     2  0 Aug09 ?        00:00:08 [migration/9]
root        56     2  0 Aug09 ?        00:00:03 [ksoftirqd/9]
root        58     2  0 Aug09 ?        00:00:00 [kworker/9:0H]
root        59     2  0 Aug09 ?        00:00:16 [watchdog/10]
root        60     2  0 Aug09 ?        00:00:06 [migration/10]
root        61     2  0 Aug09 ?        00:00:02 [ksoftirqd/10]
root        63     2  0 Aug09 ?        00:00:00 [kworker/10:0H]
root        64     2  0 Aug09 ?        00:00:16 [watchdog/11]
root        65     2  0 Aug09 ?        00:00:04 [migration/11]
root        66     2  0 Aug09 ?        00:00:02 [ksoftirqd/11]
root        68     2  0 Aug09 ?        00:00:00 [kworker/11:0H]
root        71     2  0 Aug09 ?        00:00:00 [kdevtmpfs]
root        72     2  0 Aug09 ?        00:00:00 [netns]
root        73     2  0 Aug09 ?        00:00:03 [khungtaskd]
root        74     2  0 Aug09 ?        00:00:00 [writeback]
root        75     2  0 Aug09 ?        00:00:00 [kintegrityd]
root        76     2  0 Aug09 ?        00:00:00 [bioset]
root        77     2  0 Aug09 ?        00:00:00 [bioset]
root        78     2  0 Aug09 ?        00:00:00 [bioset]
root        79     2  0 Aug09 ?        00:00:00 [kblockd]
root        81     2  0 Aug09 ?        00:00:00 [md]
root        82     2  0 Aug09 ?        00:00:00 [edac-poller]
root        90     2  0 Aug09 ?        00:00:06 [kswapd0]
root        91     2  0 Aug09 ?        00:00:07 [kswapd1]
root        92     2  0 Aug09 ?        00:00:00 [ksmd]
root        93     2  0 Aug09 ?        00:09:16 [khugepaged]
root        94     2  0 Aug09 ?        00:00:00 [crypto]
root       102     2  0 Aug09 ?        00:00:00 [kthrotld]
root       107     2  0 Aug09 ?        00:00:00 [kmpath_rdacd]
root       108     2  0 Aug09 ?        00:00:00 [kaluad]
root       109     2  0 Aug09 ?        00:00:00 [kpsmoused]
root       111     2  0 Aug09 ?        00:00:00 [ipv6_addrconf]
root       127     2  0 Aug09 ?        00:00:00 [deferwq]
root       128     2  0 Aug09 ?        00:01:17 [kworker/u24:1]
root       164     2  0 Aug09 ?        00:02:09 [kauditd]
root       348     2  0 Aug09 ?        00:02:10 [emcpd]
root       349     2  0 Aug09 ?        00:00:04 [emcpdefd]
root       350     2  0 Aug09 ?        00:00:00 [emcprequestd]
root       351     2  0 Aug09 ?        00:00:00 [emcpmiscd]
root       352     2  0 Aug09 ?        00:00:00 [emcpgrioctld]
root       356     2  0 Aug09 ?        00:00:00 [MpxAsyncIoDaemo]
root       357     2  0 Aug09 ?        00:00:00 [MpxResumeIoDaem]
root       358     2  0 Aug09 ?        00:04:13 [MpxPeriodicCall]
root       359     2  0 Aug09 ?        00:00:10 [MpxGrDaemon]
root       360     2  0 Aug09 ?        00:00:00 [MpxProactiveDae]
root       361     2  0 Aug09 ?        00:00:00 [MpxDispatchDaem]
root       362     2  0 Aug09 ?        00:00:00 [MpxTestDaemon]
root       405     2  0 Aug09 ?        00:00:00 [scsi_eh_0]
root       408     2  0 Aug09 ?        00:00:00 [scsi_tmf_0]
root       411     2  0 Aug09 ?        00:00:00 [ata_sff]
root       413     2  0 10:01 ?        00:00:00 [kworker/u25:2]
root       415     2  0 Aug09 ?        00:00:00 [scsi_eh_1]
root       416     2  0 Aug09 ?        00:00:00 [scsi_tmf_1]
root       417     2  0 Aug09 ?        00:00:00 [scsi_eh_2]
root       418     2  0 Aug09 ?        00:00:00 [scsi_tmf_2]
root       419     2  0 Aug09 ?        00:00:00 [scsi_eh_3]
root       420     2  0 Aug09 ?        00:00:00 [scsi_tmf_3]
root       421     2  0 Aug09 ?        00:00:00 [scsi_eh_4]
root       422     2  0 Aug09 ?        00:00:00 [scsi_tmf_4]
root       430     2  0 Aug09 ?        00:00:00 [scsi_eh_5]
root       431     2  0 Aug09 ?        00:00:00 [scsi_tmf_5]
root       432     2  0 Aug09 ?        00:00:00 [scsi_eh_6]
root       433     2  0 Aug09 ?        00:00:00 [scsi_tmf_6]
root       434     2  0 Aug09 ?        00:00:00 [scsi_eh_7]
root       435     2  0 Aug09 ?        00:00:00 [scsi_tmf_7]
root       436     2  0 Aug09 ?        00:00:00 [scsi_eh_8]
root       437     2  0 Aug09 ?        00:00:00 [scsi_tmf_8]
root       438     2  0 Aug09 ?        00:00:00 [scsi_eh_9]
root       439     2  0 Aug09 ?        00:00:00 [scsi_tmf_9]
root       440     2  0 Aug09 ?        00:00:00 [scsi_eh_10]
root       441     2  0 Aug09 ?        00:00:00 [scsi_tmf_10]
root       450     2  0 10:01 ?        00:00:00 [kworker/10:0]
root       459     2  0 Aug09 ?        00:00:00 [scsi_eh_11]
root       460     2  0 Aug09 ?        00:00:00 [scsi_tmf_11]
root       465     2  0 Aug09 ?        00:00:00 [ttm_swap]
root       476     2  0 Aug09 ?        00:00:00 [qla2xxx_11_dpc]
root       477     2  0 Aug09 ?        00:00:00 [scsi_wq_11]
root       478     2  0 Aug09 ?        00:00:00 [scsi_eh_12]
root       479     2  0 Aug09 ?        00:00:00 [scsi_tmf_12]
root       481     2  0 Aug09 ?        00:00:00 [qla2xxx_12_dpc]
root       482     2  0 Aug09 ?        00:00:00 [scsi_wq_12]
root       483     2  0 Aug09 ?        00:00:34 [kworker/0:1H]
root       486     2  0 Aug09 ?        00:00:00 [kworker/u24:2]
root       516     2  0 Aug09 ?        00:00:21 [kworker/6:1H]
root       521     2  0 Aug09 ?        00:00:00 [MpxTestDaemon  ]
root      1302     2  0 14:18 ?        00:00:00 [kworker/0:0]
root      1388     2  0 14:19 ?        00:00:00 [kworker/10:2]
root      1563     2  0 Aug09 ?        00:00:00 [kdmflush]
root      1564     2  0 Aug09 ?        00:00:00 [bioset]
root      1591     2  0 Aug09 ?        00:00:00 [kdmflush]
root      1592     2  0 Aug09 ?        00:00:00 [bioset]
root      1625     2  0 Aug09 ?        00:00:00 [bioset]
root      1626     2  0 Aug09 ?        00:00:00 [xfsalloc]
root      1627     2  0 Aug09 ?        00:00:00 [xfs_mru_cache]
root      1628     2  0 Aug09 ?        00:00:00 [xfs-buf/dm-0]
root      1629     2  0 Aug09 ?        00:00:00 [xfs-data/dm-0]
root      1630     2  0 Aug09 ?        00:00:00 [xfs-conv/dm-0]
root      1631     2  0 Aug09 ?        00:00:00 [xfs-cil/dm-0]
root      1632     2  0 Aug09 ?        00:00:00 [xfs-reclaim/dm-]
root      1633     2  0 Aug09 ?        00:00:00 [xfs-log/dm-0]
root      1634     2  0 Aug09 ?        00:00:00 [xfs-eofblocks/d]
root      1635     2  0 Aug09 ?        00:20:34 [xfsaild/dm-0]
root      1636     2  0 Aug09 ?        00:00:02 [kworker/4:1H]
root      1706     1  0 Aug09 ?        00:21:30 /usr/lib/systemd/systemd-journald
root      1722     1  0 Aug09 ?        00:00:00 /usr/sbin/lvmetad -f
root      1750     1  0 Aug09 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root      1807     2  0 Aug09 ?        00:00:00 [kipmi0]
root      1916     2  0 Aug09 ?        00:00:00 [kvm-irqfd-clean]
root      1923     2  0 Aug09 ?        00:00:00 [xfs-buf/emcpowe]
root      1924     2  0 Aug09 ?        00:00:00 [xfs-data/emcpow]
root      1925     2  0 Aug09 ?        00:00:00 [xfs-conv/emcpow]
root      1926     2  0 Aug09 ?        00:00:00 [xfs-cil/emcpowe]
root      1927     2  0 Aug09 ?        00:00:00 [xfs-reclaim/emc]
root      1928     2  0 Aug09 ?        00:00:00 [xfs-log/emcpowe]
root      1929     2  0 Aug09 ?        00:00:00 [xfs-eofblocks/e]
root      1931     2  0 Aug09 ?        00:00:00 [xfsaild/emcpowe]
root      1944     2  0 Aug09 ?        00:00:08 [kworker/7:1H]
root      1945     2  0 Aug09 ?        00:00:02 [kworker/3:1H]
root      1947     2  0 Aug09 ?        00:00:00 [kdmflush]
root      1948     2  0 Aug09 ?        00:00:00 [bioset]
root      1955     2  0 Aug09 ?        00:00:00 [xfs-buf/dm-2]
root      1956     2  0 Aug09 ?        00:00:00 [xfs-data/dm-2]
root      1957     2  0 Aug09 ?        00:00:00 [xfs-conv/dm-2]
root      1958     2  0 Aug09 ?        00:00:00 [xfs-cil/dm-2]
root      1959     2  0 Aug09 ?        00:00:00 [xfs-reclaim/dm-]
root      1960     2  0 Aug09 ?        00:00:00 [xfs-log/dm-2]
root      1961     2  0 Aug09 ?        00:00:00 [xfs-eofblocks/d]
root      1962     2  0 Aug09 ?        00:00:00 [xfsaild/dm-2]
root      1971     2  0 Aug09 ?        00:00:03 [kworker/2:1H]
root      2008     2  0 Aug09 ?        00:00:02 [kworker/5:1H]
root      2165     1  0 Aug09 ?        00:06:16 /sbin/auditd
root      2188     1  0 Aug09 ?        00:05:24 /sbin/rngd -f
polkitd   2189     1  0 Aug09 ?        00:20:29 /usr/lib/polkit-1/polkitd --no-debug
root      2192     1  0 Aug09 ?        00:00:00 /usr/sbin/smartd -n -q never
root      2194     1  0 Aug09 ?        00:09:50 /usr/sbin/irqbalance --foreground
dbus      2197     1  0 Aug09 ?        00:54:37 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root      2205     1  0 Aug09 ?        00:13:36 /usr/sbin/NetworkManager --no-daemon
libstor+  2207     1  0 Aug09 ?        00:00:10 /usr/bin/lsmd -d
root      2208     1  0 Aug09 ?        00:00:00 /usr/sbin/abrtd -d -s
root      2214     1  0 Aug09 ?        00:00:19 /usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected ernel BUG at list_del corruption list_add corruption do_IRQ: stack overflow: ear stack overflow (cur: eneral protection fault nable to handle kernel oub
root      2215     1  0 Aug09 ?        00:23:03 /usr/lib/systemd/systemd-logind
rpc       2229     1  0 Aug09 ?        00:00:07 /sbin/rpcbind -w
root      2230     1  0 Aug09 ?        00:00:00 /usr/sbin/atd -f
root      2257     1  0 Aug09 tty1     00:00:00 /sbin/agetty --noclear tty1 linux
root      2523     1  0 Aug09 ?        00:05:34 /usr/bin/python -Es /usr/sbin/tuned -l -P
root      2524     1  0 Aug09 ?        00:33:57 /usr/sbin/snmpd -LS0-6d -f
root      2533     1  0 Aug09 ?        00:12:25 /usr/sbin/rsyslogd -n
root      2574     1  0 Aug09 ?        00:00:00 rhnsd
root      2886     2  0 14:22 ?        00:00:00 [kworker/1:2]
root      2888     1  0 Aug09 ?        00:25:11 /sbin/emcp_mond
root      2900     1  0 Aug09 ?        00:00:11 /usr/sbin/sshd -D
root      3478     1  0 Aug09 ?        00:00:13 /usr/libexec/postfix/master -w
postfix   3507  3478  0 Aug09 ?        00:00:02 qmgr -l -t unix -u
root      3543     2  0 10:10 ?        00:00:00 [kworker/4:2]
root      3749     2  0 14:25 ?        00:00:00 [kworker/4:1]
root      3759     2  0 14:25 ?        00:00:00 [kworker/6:1]
root      3985     2  0 13:00 ?        00:00:00 [kworker/9:2]
root      4131     2  0 14:26 ?        00:00:00 [kworker/11:1]
root      4880     2  0 14:28 ?        00:00:00 [kworker/8:0]
root      5068     2  0 14:28 ?        00:00:00 [kworker/1:1]
root      5647     2  0 14:30 ?        00:00:00 [kworker/0:1]
root      6394     2  0 14:32 ?        00:00:00 [kworker/3:0]
root      6693     2  0 13:07 ?        00:00:00 [kworker/6:0]
root      6776     2  0 14:33 ?        00:00:00 [kworker/5:2]
root      7140     2  0 14:34 ?        00:00:00 [kworker/2:0]
root      7863     2  0 14:36 ?        00:00:00 [kworker/11:2]
root      7990     2  0 Aug09 ?        00:00:02 [kworker/8:1H]
apache    8335 14417  1 13:11 ?        00:01:02 /usr/sbin/httpd -DFOREGROUND
root      8343  2900  0 14:37 ?        00:00:00 sshd: ahsxv3nb [priv]
ahsxv3nb  8435  8343  0 14:37 ?        00:00:00 sshd: ahsxv3nb@pts/1
ahsxv3nb  8436  8435  0 14:37 pts/1    00:00:00 -bash
root      8487  8436  0 14:37 pts/1    00:00:00 sudo su -
root      8494  8487  0 14:37 pts/1    00:00:00 su -
root      8495  8494  0 14:37 pts/1    00:00:00 -bash
root      8891     2  0 14:38 ?        00:00:00 [kworker/5:0]
root      8918     2  0 Aug09 ?        00:00:02 [kworker/9:1H]
root      8919     2  0 Aug09 ?        00:00:01 [kworker/10:1H]
root      9068     2  0 14:38 ?        00:00:00 [kworker/1:0]
root      9150     2  0 14:39 ?        00:00:00 [kworker/9:0]
root      9454 14529  0 14:40 ?        00:00:00 /usr/sbin/CROND -n
root      9455 14529  0 14:40 ?        00:00:00 /usr/sbin/CROND -n
root      9457 14529  0 14:40 ?        00:00:00 /usr/sbin/CROND -n
root      9458 14529  0 14:40 ?        00:00:00 /usr/sbin/CROND -n
root      9459 14529  0 14:40 ?        00:00:00 /usr/sbin/CROND -n
nagios    9466  9458  0 14:40 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios    9468  9454  0 14:40 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios    9470  9455  0 14:40 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios    9472  9457  0 14:40 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1
nagios    9474  9459  0 14:40 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios    9476  9468  0 14:40 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios    9477  9470  0 14:40 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios    9479  9474  0 14:40 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios    9482  9472  0 14:40 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios    9483  9466  0 14:40 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
root      9533     2  0 14:40 ?        00:00:00 [kworker/3:2]
root      9837  8495  0 14:40 pts/1    00:00:00 ps -ef --cols=300
root     11313     2  0 Aug09 ?        00:00:12 [kworker/1:1H]
root     11337     2  0 10:33 ?        00:00:00 [kworker/5:1]
root     12742     2  0 Aug09 ?        00:00:01 [kworker/11:1H]
root     13474     2  0 13:24 ?        00:00:02 [kworker/0:2]
root     14417     1  0 10:38 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   14419 14417  0 10:38 ?        00:01:41 /usr/sbin/httpd -DFOREGROUND
apache   14420 14417  0 10:38 ?        00:01:56 /usr/sbin/httpd -DFOREGROUND
apache   14422 14417  0 10:38 ?        00:02:14 /usr/sbin/httpd -DFOREGROUND
apache   14423 14417  0 10:38 ?        00:02:16 /usr/sbin/httpd -DFOREGROUND
apache   14424 14417  0 10:38 ?        00:01:41 /usr/sbin/httpd -DFOREGROUND
nagios   14440     1  0 10:38 ?        00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
root     14456     1  0 Aug10 ?        00:07:29 /usr/sbin/snmptrapd -Ln -f
nagios   14506     1  0 10:38 ?        00:00:00 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
apache   14510 14417  0 10:38 ?        00:01:51 /usr/sbin/httpd -DFOREGROUND
root     14529     1  0 10:38 ?        00:00:00 /usr/sbin/crond -n
apache   14696 14417  0 10:39 ?        00:02:06 /usr/sbin/httpd -DFOREGROUND
apache   14801 14417  0 10:39 ?        00:02:22 /usr/sbin/httpd -DFOREGROUND
ntp      15217     1  0 Aug10 ?        00:00:06 /usr/sbin/ntpd -u ntp:ntp -g
postfix  15289  3478  0 13:29 ?        00:00:00 pickup -l -t unix -u
root     15404     2  0 13:30 ?        00:00:00 [kworker/7:1]
mysql    15985     1  0 10:42 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
mysql    16146 15985  0 10:42 ?        00:01:54 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
root     17120     2  0 13:35 ?        00:00:00 [kworker/8:2]
apache   18008 14417  0 10:46 ?        00:01:58 /usr/sbin/httpd -DFOREGROUND
root     18339     2  0 Sep27 ?        00:00:00 [kworker/u26:0]
root     22596     1  0 Aug16 ?        00:00:37 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt   22597 22596  0 Aug16 ?        00:04:20 /usr/bin/perl /usr/sbin/snmptt --daemon
root     23503     1  0 Aug10 ?        00:00:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
root     24314  2900  0 08:04 ?        00:00:00 sshd: ahsxv3nb [priv]
ahsxv3nb 24394 24314  0 08:04 ?        00:00:00 sshd: ahsxv3nb@pts/0
ahsxv3nb 24395 24394  0 08:04 pts/0    00:00:00 -bash
root     24449 24395  0 08:04 pts/0    00:00:00 sudo su -
root     24452 24449  0 08:04 pts/0    00:00:00 su -
root     24453 24452  0 08:04 pts/0    00:00:00 -bash
shellin+ 24921     1  0 Aug10 ?        00:00:00 /usr/sbin/shellinaboxd -u shellinabox -g shellinabox --cert=/var/lib/shellinabox --port=7878 --disable-ssl-menu -s /:SSH --localhost-only --css white-on-black.css
shellin+ 24924 24921  0 Aug10 ?        00:00:00 /usr/sbin/shellinaboxd -u shellinabox -g shellinabox --cert=/var/lib/shellinabox --port=7878 --disable-ssl-menu -s /:SSH --localhost-only --css white-on-black.css
root     27499     2  0 09:45 ?        00:00:00 [kworker/u25:1]
root     27723     2  0 14:03 ?        00:00:00 [kworker/u26:1]
root     29541     2  0 09:51 ?        00:00:00 [kworker/2:1]
root     30634     2  0 12:45 ?        00:00:00 [kworker/11:0]
root     31071     2  0 12:46 ?        00:00:00 [kworker/3:1]
root     31890     2  0 14:14 ?        00:00:00 [kworker/9:1]
root     32676     2  0 14:16 ?        00:00:00 [kworker/7:2]

Code: Select all

[root@oplp1521 ~]# chage -l nagios
Last password change                                    : Aug 10, 2018
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios cmdsubsys.log Growing Excessively

Post by tgriep »

It looks like the error is with the Broker module.

Code: Select all

[1538159915] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1538159915] Error: Module loading failed. Aborting.

Can you post the following files so we can view them?

Code: Select all

/usr/local/nagios/etc/ndomod.cfg
/usr/local/nagios/etc/nagios.cfg
Then edit the nagios.cfg file and enable debugging by changing this line from

Code: Select all

debug_level=0
to

Code: Select all

debug_level=-1
Save the file. Try to start nagios and post the debug file.

Code: Select all

/usr/local/nagios/var/nagios.debug
Thanks
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
nortonhealthcare
Posts: 33
Joined: Thu Aug 10, 2017 8:22 am

Re: Nagios cmdsubsys.log Growing Excessively

Post by nortonhealthcare »

I have attached the files, made the debugging mode change, and tried to start nagios. Here is the output of the /usr/local/nagios/var/nagios.debug file:

Code: Select all

[root@oplp1521 etc]# cat /usr/local/nagios/var/nagios.debug
[1538162802.200909] [001.0] [pid=27306] clear_volatile_macros_r()
[1538162863.559426] [001.0] [pid=27807] clear_volatile_macros_r()
[1538162898.186145] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.186185] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.186200] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.186211] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.186234] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.189153] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.189537] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.189604] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.189687] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190039] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190115] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190168] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190241] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190297] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190357] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190421] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190472] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190519] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190584] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190643] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190701] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190757] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190811] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190898] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.190911] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191287] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191465] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191484] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191497] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191528] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191547] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191564] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191580] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191597] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191614] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191630] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191646] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191662] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191677] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191694] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191710] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191725] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191740] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191756] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191771] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191808] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191827] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191845] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191861] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191877] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191895] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191907] [064.0] [pid=28081] Module '/usr/local/nagios/bin/ndomod.o' loaded with return code of '0'
[1538162898.191911] [064.0] [pid=28081] nebmodule_deinit() found
[1538162898.191923] [064.1] [pid=28081] Making callbacks (type 2)...
[1538162898.191947] [064.0] [pid=28081] Attempting to unload module '/usr/local/nagios/bin/ndomod.o': flags=1, reason=2
[1538162898.192073] [064.0] [pid=28081] Module '/usr/local/nagios/bin/ndomod.o' unloaded successfully.
[1538162898.192116] [001.0] [pid=28081] clear_volatile_macros_r()
You do not have the required permissions to view the files attached to this post.
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios cmdsubsys.log Growing Excessively

Post by tgriep »

Try commenting out both of the btoker modules.

Code: Select all

# NDOUtils module
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

# mk-livestatus
broker_module=/usr/local/lib/mk-livestatus/livestatus.o /usr/local/nagios/var/rw/live pnp_path=/usr/local/nagios/share/perfdata num_client_threads=30 thread_stack_size=262144
Then see if nagios starts and stays running.

Then put back the ndomod.o back and see if it starts and runs.

Lastly, put back the livestatus module and try it again.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
nortonhealthcare
Posts: 33
Joined: Thu Aug 10, 2017 8:22 am

Re: Nagios cmdsubsys.log Growing Excessively

Post by nortonhealthcare »

That did indeed keep nagios running. Thanks for the help.

Will the /usr/local/nagiosxi/var/cmdsubsys.log growth issue be addressed? I currently have the logging changed to overwrite instead of append in /etc/cron.d/nagiosxi:

Code: Select all

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Nagios cmdsubsys.log Growing Excessively

Post by ssax »

Did you put the ndomod broker back in, restarted the nagios service, and it stayed working?

Code: Select all

# NDOUtils module
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
Technically killing the old process should have fixed it, you can change the cron entry back to >>.
Top
nortonhealthcare
Posts: 33
Joined: Thu Aug 10, 2017 8:22 am

Re: Nagios cmdsubsys.log Growing Excessively

Post by nortonhealthcare »

After double-checking, it appears that uncommenting:

Code: Select all

# broker_module=/usr/local/lib/mk-livestatus/livestatus.o /usr/local/nagios/var/rw/live pnp_path=/usr/local/nagios/share/perfdata num_client_threads=30 thread_stack_size=262144
breaks Nagios. The only way Nagios will stay running is with it commented out. Is livestatus.o an essential part of the software?
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Nagios cmdsubsys.log Growing Excessively

Post by ssax »

livestatus is a 3rd party product, we don't include it or support it, it would have to have been installed by someone on your team at some point.
Top
Locked

Return to “Nagios XI”