All Linux Server CPU Spike at same time

An open discussion forum for obtaining help with Nagios Core. Nagios Core users of all experience levels are welcome here. Subforum have been created for the discussion of Nagios Core and Nagios Plugin development.

NOTE: The SourceForge.net mailing lists have been deprecated in favor of this forum in order to expedite support and provide additional features not available on the old mailing list.

Re: All Linux Server CPU Spike at same time

Postby kwhogster » Tue Mar 14, 2017 7:06 pm

on the remote server

/etc/rsyslogd.conf

not found

wrong name or wrong folder?
kwhogster
 
Posts: 383
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA

Re: All Linux Server CPU Spike at same time

Postby dwhitfield » Tue Mar 14, 2017 8:04 pm

If we ever give you a command and the default path is not found, please run find / -name $nameoffile. Probably you can just edit whatever file it finds, but if it finds more than one or nothing, definitely let us know. There is a caveat to the more than one. If there are two and one of them is in the directory where you extracted Core, then that one can be ignored.

In this case, what does find / -name rsyslogd.conf return?
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 3756
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: All Linux Server CPU Spike at same time

Postby kwhogster » Tue Mar 14, 2017 8:19 pm

[root@tgcs018 /]# find / -name rsyslogd.conf
[root@tgcs018 /]#


nothing found
kwhogster
 
Posts: 383
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA

Re: All Linux Server CPU Spike at same time

Postby dwhitfield » Tue Mar 14, 2017 8:34 pm

It's probably /etc/syslogd.conf then, but if not, let us know if that find command finds it. It's not a typo. rsyslog is a different program than syslog.
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 3756
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: All Linux Server CPU Spike at same time

Postby kwhogster » Tue Mar 14, 2017 8:47 pm

I found this

/etc/rsyslog.conf
kwhogster
 
Posts: 383
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA

Re: All Linux Server CPU Spike at same time

Postby kwhogster » Tue Mar 14, 2017 8:56 pm

Ok got the messages log now

ran this on my Nagios server
root@tgcs017:~# /usr/local/nagios/libexec/check_nrpe -H 10.2.8.74
NRPE v2.15


Log from the remote server
Code: Select all
[root@tgcs018 /]# tail -n 100 /var/log/messages
Mar 14 19:51:54 tgcs018 xinetd[20332]: START: nrpe pid=6604 from=::ffff:10.2.8.79
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk]=/usr/local/nagios/libexec/check_asterisk.pl $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_sip]=/usr/local/nagios/libexec/check_sip $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_sip_peers]=sudo /usr/local/nagios/libexec/check_asterisk_sip_peers.sh $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_version]=/usr/local/nagios/libexec/nagisk.pl -c version
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_peers]=/usr/local/nagios/libexec/nagisk.pl -c peers
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_channels]=/usr/local/nagios/libexec/nagisk.pl -c channels
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_zaptel]=/usr/local/nagios/libexec/nagisk.pl -c zaptel
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_asterisk_span]=/usr/local/nagios/libexec/nagisk.pl -c span -s 1
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_yum]=/usr/local/nagios/libexec/check_yum
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_apt]=/usr/local/nagios/libexec/check_apt
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$
Mar 14 19:51:54 tgcs018 nrpe[6604]: Added command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$
Mar 14 19:51:54 tgcs018 nrpe[6604]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Mar 14 19:51:54 tgcs018 nrpe[6604]: Handling the connection...
Mar 14 19:51:54 tgcs018 nrpe[6604]: Host is asking for command 'check_mem' to be run...
Mar 14 19:51:54 tgcs018 nrpe[6604]: Running command: /usr/local/nagios/libexec/custom_check_mem -n -w 80% -c 90%
Mar 14 19:51:54 tgcs018 nrpe[6604]: Command completed with return code 0 and output:  - 729 / 3791 MB (19%) Free Memory, Used: 3061 MB, Shared: 185 MB, Buffers + Cached: 384 MB | total=3791MB free=729MB used=3061MB shared=185MB buffers_and_cached=384MB
Mar 14 19:51:54 tgcs018 nrpe[6604]: Return Code: 0, Output:  - 729 / 3791 MB (19%) Free Memory, Used: 3061 MB, Shared: 185 MB, Buffers + Cached: 384 MB | total=3791MB free=729MB used=3061MB shared=185MB buffers_and_cached=384MB
Mar 14 19:51:54 tgcs018 xinetd[20332]: EXIT: nrpe status=0 pid=6604 duration=0(sec)
Mar 14 19:52:01 tgcs018 systemd: Started Session 198866 of user nagios.
Mar 14 19:52:01 tgcs018 systemd: Starting Session 198866 of user nagios.
Mar 14 19:52:01 tgcs018 systemd: Started Session 198865 of user nagios.
Mar 14 19:52:01 tgcs018 systemd: Starting Session 198865 of user nagios.
Mar 14 19:52:39 tgcs018 xinetd[20332]: START: nrpe pid=6745 from=::ffff:10.2.8.79
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk]=/usr/local/nagios/libexec/check_asterisk.pl $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_sip]=/usr/local/nagios/libexec/check_sip $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_sip_peers]=sudo /usr/local/nagios/libexec/check_asterisk_sip_peers.sh $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_version]=/usr/local/nagios/libexec/nagisk.pl -c version
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_peers]=/usr/local/nagios/libexec/nagisk.pl -c peers
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_channels]=/usr/local/nagios/libexec/nagisk.pl -c channels
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_zaptel]=/usr/local/nagios/libexec/nagisk.pl -c zaptel
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_asterisk_span]=/usr/local/nagios/libexec/nagisk.pl -c span -s 1
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_yum]=/usr/local/nagios/libexec/check_yum
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_apt]=/usr/local/nagios/libexec/check_apt
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$
Mar 14 19:52:39 tgcs018 nrpe[6745]: Added command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$
Mar 14 19:52:39 tgcs018 nrpe[6745]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Mar 14 19:52:39 tgcs018 nrpe[6745]: Handling the connection...
Mar 14 19:52:39 tgcs018 nrpe[6745]: Host is asking for command '_NRPE_CHECK' to be run...
Mar 14 19:52:39 tgcs018 nrpe[6745]: Response: NRPE v2.15
Mar 14 19:52:39 tgcs018 nrpe[6745]: Return Code: 0, Output: NRPE v2.15
Mar 14 19:52:39 tgcs018 xinetd[20332]: EXIT: nrpe status=0 pid=6745 duration=0(sec)
Mar 14 19:52:53 tgcs018 xinetd[20332]: START: nrpe pid=6776 from=::ffff:10.2.8.79
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk]=/usr/local/nagios/libexec/check_asterisk.pl $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_sip]=/usr/local/nagios/libexec/check_sip $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_sip_peers]=sudo /usr/local/nagios/libexec/check_asterisk_sip_peers.sh $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_version]=/usr/local/nagios/libexec/nagisk.pl -c version
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_peers]=/usr/local/nagios/libexec/nagisk.pl -c peers
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_channels]=/usr/local/nagios/libexec/nagisk.pl -c channels
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_zaptel]=/usr/local/nagios/libexec/nagisk.pl -c zaptel
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_asterisk_span]=/usr/local/nagios/libexec/nagisk.pl -c span -s 1
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_init_service]=sudo /usr/local/nagios/libexec/check_init_service $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_load]=/usr/local/nagios/libexec/check_load $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_swap]=/usr/local/nagios/libexec/check_swap $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_cpu_stats]=/usr/local/nagios/libexec/check_cpu_stats.sh $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_mem]=/usr/local/nagios/libexec/custom_check_mem -n $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_yum]=/usr/local/nagios/libexec/check_yum
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_apt]=/usr/local/nagios/libexec/check_apt
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_disk]=/usr/local/nagios/libexec/check_disk $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_ide_smart]=/usr/local/nagios/libexec/check_ide_smart $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_all_procs]=/usr/local/nagios/libexec/custom_check_procs
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_open_files]=/usr/local/nagios/libexec/check_open_files.pl $ARG1$
Mar 14 19:52:53 tgcs018 nrpe[6776]: Added command[check_netstat]=/usr/local/nagios/libexec/check_netstat.pl -p $ARG1$ $ARG2$
Mar 14 19:52:53 tgcs018 nrpe[6776]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Mar 14 19:52:53 tgcs018 nrpe[6776]: Handling the connection...
Mar 14 19:52:53 tgcs018 nrpe[6776]: Host is asking for command 'check_mem' to be run...
Mar 14 19:52:53 tgcs018 nrpe[6776]: Running command: /usr/local/nagios/libexec/custom_check_mem -n -w 80% -c 90%
Mar 14 19:52:53 tgcs018 nrpe[6776]: Command completed with return code 0 and output:  - 727 / 3791 MB (19%) Free Memory, Used: 3063 MB, Shared: 185 MB, Buffers + Cached: 385 MB | total=3791MB free=727MB used=3063MB shared=185MB buffers_and_cached=385MB
Mar 14 19:52:53 tgcs018 nrpe[6776]: Return Code: 0, Output:  - 727 / 3791 MB (19%) Free Memory, Used: 3063 MB, Shared: 185 MB, Buffers + Cached: 385 MB | total=3791MB free=727MB used=3063MB shared=185MB buffers_and_cached=385MB
Mar 14 19:52:53 tgcs018 xinetd[20332]: EXIT: nrpe status=0 pid=6776 duration=0(sec)
Mar 14 19:53:01 tgcs018 systemd: Started Session 198868 of user nagios.
Mar 14 19:53:01 tgcs018 systemd: Starting Session 198868 of user nagios.
Mar 14 19:53:01 tgcs018 systemd: Started Session 198867 of user nagios.
Mar 14 19:53:01 tgcs018 systemd: Starting Session 198867 of user nagios.
Last edited by dwhitfield on Wed Mar 15, 2017 9:01 am, edited 1 time in total.
Reason: code blocks FTW
kwhogster
 
Posts: 383
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA

Re: All Linux Server CPU Spike at same time

Postby ssax » Wed Mar 15, 2017 4:31 pm

What is the output of this command (run it from the XI server):

Code: Select all
/usr/local/nagios/libexec/check_nrpe -H 10.2.8.7 -c check_load -a '-w 5,10,15 -c 6,11,17'


Thank you
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
ssax
Dreams In Code
 
Posts: 2973
Joined: Wed Feb 11, 2015 12:54 pm

Re: All Linux Server CPU Spike at same time

Postby kwhogster » Thu Mar 16, 2017 3:57 pm

root@tgcs017:/usr/local/nagios/etc/objects# /usr/local/nagios/libexec/check_nrpe -H 10.2.8.7 -c check_load -a '-w 5,10,15 -c 6,11,17'
CHECK_NRPE: Socket timeout after 10 seconds.

root@tgcs017:/usr/local/nagios/etc/objects# /usr/local/nagios/libexec/check_nrpe -H 10.2.8.7 -t 90s -c check_load -a '-w 5,10,15 -c 6,11,17'
NRPE: Unable to read output
root@tgcs017:/usr/local/nagios/etc/objects# /usr/local/nagios/libexec/check_nrpe -H 10.2.8.7 -c check_load -a '-w 5,10,15 -c 6,11,17'
NRPE: Unable to read output
kwhogster
 
Posts: 383
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA

Re: All Linux Server CPU Spike at same time

Postby dwhitfield » Fri Mar 17, 2017 1:31 pm

You were using 10.2.8.74 and 10.2.8.79 but now are using 10.2.8.7. What are the IP addresses that are in play here? Is that just a typo? Did you actually make that typo when you ran the command?
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 3756
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: All Linux Server CPU Spike at same time

Postby kwhogster » Fri Mar 17, 2017 1:38 pm

My Linux hosts are this

10.2.8.74 Cent OS Nagios LogServer

10.2.8.79 Ubuntu Nagios Core 4.1

10.2.8.7 SUSE Enterprise vMA

The above are all VM's on different ESXi 6.0 Hosts

My other Linux is

10.2.8.72 RaspberryPi Test Nagios Core machine
kwhogster
 
Posts: 383
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA

PreviousNext

Return to Nagios Core

Who is online

Users browsing this forum: No registered users and 9 guests