All Linux Server CPU Spike at same time

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:

Re: All Linux Server CPU Spike at same time

Post by kwhogster »

on the remote server

/etc/rsyslogd.conf

not found

wrong name or wrong folder?
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: All Linux Server CPU Spike at same time

Post by dwhitfield »

If we ever give you a command and the default path is not found, please run find / -name $nameoffile. Probably you can just edit whatever file it finds, but if it finds more than one or nothing, definitely let us know. There is a caveat to the more than one. If there are two and one of them is in the directory where you extracted Core, then that one can be ignored.

In this case, what does find / -name rsyslogd.conf return?
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:

Re: All Linux Server CPU Spike at same time

Post by kwhogster »

[root@tgcs018 /]# find / -name rsyslogd.conf
[root@tgcs018 /]#


nothing found
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: All Linux Server CPU Spike at same time

Post by dwhitfield »

It's probably /etc/syslogd.conf then, but if not, let us know if that find command finds it. It's not a typo. rsyslog is a different program than syslog.
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:

Re: All Linux Server CPU Spike at same time

Post by kwhogster »

I found this

/etc/rsyslog.conf
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:

Re: All Linux Server CPU Spike at same time

Post by kwhogster »

Ok got the messages log now

ran this on my Nagios server
root@tgcs017:~# /usr/local/nagios/libexec/check_nrpe -H 10.2.8.74
NRPE v2.15


Log from the remote server

Code: Select all

[root@tgcs018 /]# tail -n 100 /var/log/messages
Mar 14 19:51:54 tgcs018 xinetd[20332]: START: nrpe pid=6604 from=::ffff:10.2.8.79
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk]=/usr/local/nagios/libexec/check_asterisk.pl $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_sip]=/usr/local/nagios/libexec/check_sip $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_sip_peers]=sudo /usr/local/nagios/libexec/check_asterisk_sip_peers.sh $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_version]=/usr/local/nagios/libexec/nagisk.pl -c version
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_peers]=/usr/local/nagios/libexec/nagisk.pl -c peers
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_channels]=/usr/local/nagios/libexec/nagisk.pl -c channels
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_zaptel]=/usr/local/nagios/libexec/nagisk.pl -c zaptel
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_span]=/usr/local/nagios/libexec/nagisk.pl -c span -s 1
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_yum]=/usr/local/nagios/libexec/check_yum
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_apt]=/usr/local/nagios/libexec/check_apt
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$
Mar 14 19:51:54 tgcs018 nrpe[6604]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Mar 14 19:51:54 tgcs018 nrpe[6604]: Handling the connection...
Mar 14 19:51:54 tgcs018 nrpe[6604]: Host is asking for command 'check_mem' to be run...
Mar 14 19:51:54 tgcs018 nrpe[6604]: Running command: /usr/local/nagios/libexec/custom_check_mem -n -w 80% -c 90%
Mar 14 19:51:54 tgcs018 nrpe[6604]: Command completed with return code 0 and output:  - 729 / 3791 MB (19%) Free Memory, Used: 3061 MB, Shared: 185 MB, Buffers + Cached: 384 MB | total=3791MB free=729MB used=3061MB shared=185MB buffers_and_cached=384MB
Mar 14 19:51:54 tgcs018 nrpe[6604]: Return Code: 0, Output:  - 729 / 3791 MB (19%) Free Memory, Used: 3061 MB, Shared: 185 MB, Buffers + Cached: 384 MB | total=3791MB free=729MB used=3061MB shared=185MB buffers_and_cached=384MB
Mar 14 19:51:54 tgcs018 xinetd[20332]: EXIT: nrpe status=0 pid=6604 duration=0(sec)
Mar 14 19:52:01 tgcs018 systemd: Started Session 198866 of user nagios.
Mar 14 19:52:01 tgcs018 systemd: Starting Session 198866 of user nagios.
Mar 14 19:52:01 tgcs018 systemd: Started Session 198865 of user nagios.
Mar 14 19:52:01 tgcs018 systemd: Starting Session 198865 of user nagios.
Mar 14 19:52:39 tgcs018 xinetd[20332]: START: nrpe pid=6745 from=::ffff:10.2.8.79
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk]=/usr/local/nagios/libexec/check_asterisk.pl $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_sip]=/usr/local/nagios/libexec/check_sip $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_sip_peers]=sudo /usr/local/nagios/libexec/check_asterisk_sip_peers.sh $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_version]=/usr/local/nagios/libexec/nagisk.pl -c version
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_peers]=/usr/local/nagios/libexec/nagisk.pl -c peers
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_channels]=/usr/local/nagios/libexec/nagisk.pl -c channels
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_zaptel]=/usr/local/nagios/libexec/nagisk.pl -c zaptel
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_span]=/usr/local/nagios/libexec/nagisk.pl -c span -s 1
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_yum]=/usr/local/nagios/libexec/check_yum
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_apt]=/usr/local/nagios/libexec/check_apt
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$
Mar 14 19:52:39 tgcs018 nrpe[6745]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Mar 14 19:52:39 tgcs018 nrpe[6745]: Handling the connection...
Mar 14 19:52:39 tgcs018 nrpe[6745]: Host is asking for command '_NRPE_CHECK' to be run...
Mar 14 19:52:39 tgcs018 nrpe[6745]: Response: NRPE v2.15
Mar 14 19:52:39 tgcs018 nrpe[6745]: Return Code: 0, Output: NRPE v2.15
Mar 14 19:52:39 tgcs018 xinetd[20332]: EXIT: nrpe status=0 pid=6745 duration=0(sec)
Mar 14 19:52:53 tgcs018 xinetd[20332]: START: nrpe pid=6776 from=::ffff:10.2.8.79
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk]=/usr/local/nagios/libexec/check_asterisk.pl $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_sip]=/usr/local/nagios/libexec/check_sip $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_sip_peers]=sudo /usr/local/nagios/libexec/check_asterisk_sip_peers.sh $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_version]=/usr/local/nagios/libexec/nagisk.pl -c version
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_peers]=/usr/local/nagios/libexec/nagisk.pl -c peers
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_channels]=/usr/local/nagios/libexec/nagisk.pl -c channels
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_zaptel]=/usr/local/nagios/libexec/nagisk.pl -c zaptel
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_span]=/usr/local/nagios/libexec/nagisk.pl -c span -s 1
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_yum]=/usr/local/nagios/libexec/check_yum
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_apt]=/usr/local/nagios/libexec/check_apt
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$
Mar 14 19:52:53 tgcs018 nrpe[6776]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Mar 14 19:52:53 tgcs018 nrpe[6776]: Handling the connection...
Mar 14 19:52:53 tgcs018 nrpe[6776]: Host is asking for command 'check_mem' to be run...
Mar 14 19:52:53 tgcs018 nrpe[6776]: Running command: /usr/local/nagios/libexec/custom_check_mem -n -w 80% -c 90%
Mar 14 19:52:53 tgcs018 nrpe[6776]: Command completed with return code 0 and output:  - 727 / 3791 MB (19%) Free Memory, Used: 3063 MB, Shared: 185 MB, Buffers + Cached: 385 MB | total=3791MB free=727MB used=3063MB shared=185MB buffers_and_cached=385MB
Mar 14 19:52:53 tgcs018 nrpe[6776]: Return Code: 0, Output:  - 727 / 3791 MB (19%) Free Memory, Used: 3063 MB, Shared: 185 MB, Buffers + Cached: 385 MB | total=3791MB free=727MB used=3063MB shared=185MB buffers_and_cached=385MB
Mar 14 19:52:53 tgcs018 xinetd[20332]: EXIT: nrpe status=0 pid=6776 duration=0(sec)
Mar 14 19:53:01 tgcs018 systemd: Started Session 198868 of user nagios.
Mar 14 19:53:01 tgcs018 systemd: Starting Session 198868 of user nagios.
Mar 14 19:53:01 tgcs018 systemd: Started Session 198867 of user nagios.
Mar 14 19:53:01 tgcs018 systemd: Starting Session 198867 of user nagios.
Last edited by dwhitfield on Wed Mar 15, 2017 9:01 am, edited 1 time in total.
Reason: code blocks FTW
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: All Linux Server CPU Spike at same time

Post by ssax »

What is the output of this command (run it from the XI server):

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 10.2.8.7 -c check_load -a '-w 5,10,15 -c 6,11,17'
Thank you
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:

Re: All Linux Server CPU Spike at same time

Post by kwhogster »

root@tgcs017:/usr/local/nagios/etc/objects# /usr/local/nagios/libexec/check_nrpe -H 10.2.8.7 -c check_load -a '-w 5,10,15 -c 6,11,17'
CHECK_NRPE: Socket timeout after 10 seconds.

root@tgcs017:/usr/local/nagios/etc/objects# /usr/local/nagios/libexec/check_nrpe -H 10.2.8.7 -t 90s -c check_load -a '-w 5,10,15 -c 6,11,17'
NRPE: Unable to read output
root@tgcs017:/usr/local/nagios/etc/objects# /usr/local/nagios/libexec/check_nrpe -H 10.2.8.7 -c check_load -a '-w 5,10,15 -c 6,11,17'
NRPE: Unable to read output
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: All Linux Server CPU Spike at same time

Post by dwhitfield »

You were using 10.2.8.74 and 10.2.8.79 but now are using 10.2.8.7. What are the IP addresses that are in play here? Is that just a typo? Did you actually make that typo when you ran the command?
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:

Re: All Linux Server CPU Spike at same time

Post by kwhogster »

My Linux hosts are this

10.2.8.74 Cent OS Nagios LogServer

10.2.8.79 Ubuntu Nagios Core 4.1

10.2.8.7 SUSE Enterprise vMA

The above are all VM's on different ESXi 6.0 Hosts

My other Linux is

10.2.8.72 RaspberryPi Test Nagios Core machine
Locked