Issues following upgrade from Fedora 25 to Fedora 26 - Nagio
Issues following upgrade from Fedora 25 to Fedora 26 - Nagio
Moderator Edit: This thread has been split from another - https://support.nagios.com/forum/viewto ... 35&t=44004
In the future, please create a new thread and link to the old one instead of adding on.
I have upgraded From Fedora 25 to 26 and am now seeing the same problem.
Running from command line as root does not resolve the problem.
check_icmp does work.
In the future, please create a new thread and link to the old one instead of adding on.
I have upgraded From Fedora 25 to 26 and am now seeing the same problem.
Running from command line as root does not resolve the problem.
check_icmp does work.
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
Can you login to the server as root, run the following commands and post the output.
Can you go to the folder where the plugins are located at, run this and post the output.
One thing to try is to recompile the plugins and see if that fixes the issue.
Thanks
Code: Select all
which ping
ping
ping -V
Code: Select all
ls -l check_ping
Thanks
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
Having been moved to a new thread the title might be better as "Issues following upgrade from Fedora 25 to Fedora 26 - Nagios 4.0.8 to 4.3.1"
There are a number of issues that resulted from the upgrade, I was initially trying to keep my response to a single topic/thread.
Historically my system has been running for some time (~ 10 years) and would have been on the early version 3 source, All the software is from Fedora repos with a few plugins from exchange. All functions were working prior to the Fedora 25 to 26 upgrade.
check_ping:
The output you asked for:
/usr/sbin/ping
Usage: ping [-aAbBdDfhLnOqrRUvV64] [-c count] [-i interval] [-I interface]
[-m mark] [-M pmtudisc_option] [-l preload] [-p pattern] [-Q tos]
[-s packetsize] [-S sndbuf] [-t ttl] [-T timestamp_option]
[-w deadline] [-W timeout] [hop1 ...] destination
Usage: ping -6 [-aAbBdDfhLnOqrRUvV] [-c count] [-i interval] [-I interface]
[-l preload] [-m mark] [-M pmtudisc_option]
[-N nodeinfo_option] [-p pattern] [-Q tclass] [-s packetsize]
[-S sndbuf] [-t ttl] [-T timestamp_option] [-w deadline]
[-W timeout] destination
ping utility, iputils-s20161105
-rwxr-xr-x 1 root root 66272 Apr 21 00:13 /usr/lib64/nagios/plugins/check_ping
ping from a shell works as expected, but check_ping always returns
CRITICAL - You need more args!!!
Could not open pipe:
This includes when pinging the localhost on either the loopback or IP address.
My work round has been to use check_icmp as documented elsewhere.
SSL/TLS
I have a large number of services which fail with
CHECK_NRPE: Error - Could not complete SSL handshake with
This appears to be linked with the fact that Fedora 26 has moved to OpenSSL 1.1.0f-fips and that SSLv2 has been completely removed. This has meant that all NRPE connections initially failed.
For most (but not all) of my remote systems running nrpe 3+ I've able to reconfigure SSL/TLS and service checks are working.
For those system on nrpe 2+ (EL6/EL7) I've not found a solution yet.
For any others who are reading following a similar upgrade path, I've also a few other minor issues with check_procs and check_load:
/usr/lib64/nagios/plugins/check_procs -w 10 -c 15 -s Z
Unable to read output
/usr/lib64/nagios/plugins/check_load -w 5.0,4.0,3.0 -c 10.0,6.0,4.0
CRITICAL - You need more args!!!
Error opening
There are a number of issues that resulted from the upgrade, I was initially trying to keep my response to a single topic/thread.
Historically my system has been running for some time (~ 10 years) and would have been on the early version 3 source, All the software is from Fedora repos with a few plugins from exchange. All functions were working prior to the Fedora 25 to 26 upgrade.
check_ping:
The output you asked for:
/usr/sbin/ping
Usage: ping [-aAbBdDfhLnOqrRUvV64] [-c count] [-i interval] [-I interface]
[-m mark] [-M pmtudisc_option] [-l preload] [-p pattern] [-Q tos]
[-s packetsize] [-S sndbuf] [-t ttl] [-T timestamp_option]
[-w deadline] [-W timeout] [hop1 ...] destination
Usage: ping -6 [-aAbBdDfhLnOqrRUvV] [-c count] [-i interval] [-I interface]
[-l preload] [-m mark] [-M pmtudisc_option]
[-N nodeinfo_option] [-p pattern] [-Q tclass] [-s packetsize]
[-S sndbuf] [-t ttl] [-T timestamp_option] [-w deadline]
[-W timeout] destination
ping utility, iputils-s20161105
-rwxr-xr-x 1 root root 66272 Apr 21 00:13 /usr/lib64/nagios/plugins/check_ping
ping from a shell works as expected, but check_ping always returns
CRITICAL - You need more args!!!
Could not open pipe:
This includes when pinging the localhost on either the loopback or IP address.
My work round has been to use check_icmp as documented elsewhere.
SSL/TLS
I have a large number of services which fail with
CHECK_NRPE: Error - Could not complete SSL handshake with
This appears to be linked with the fact that Fedora 26 has moved to OpenSSL 1.1.0f-fips and that SSLv2 has been completely removed. This has meant that all NRPE connections initially failed.
For most (but not all) of my remote systems running nrpe 3+ I've able to reconfigure SSL/TLS and service checks are working.
For those system on nrpe 2+ (EL6/EL7) I've not found a solution yet.
For any others who are reading following a similar upgrade path, I've also a few other minor issues with check_procs and check_load:
/usr/lib64/nagios/plugins/check_procs -w 10 -c 15 -s Z
Unable to read output
/usr/lib64/nagios/plugins/check_load -w 5.0,4.0,3.0 -c 10.0,6.0,4.0
CRITICAL - You need more args!!!
Error opening
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
The check_ping does not have the superuser bit set on the plugin so run the following to fix that.
For the problems running the check_procs and the check_load plugins, are those errors generated when you run them remotely using the check_nrpe plugin or locally like shown in your example?
Can you run the following command on the system that has the errors and post the output here?
Code: Select all
chmod ug+s check_ping
Can you run the following command on the system that has the errors and post the output here?
Code: Select all
/usr/lib64/nagios/plugins/check_procs --help
/usr/lib64/nagios/plugins/check_load --help
ls -l /usr/lib64/nagios/plugins/
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
Setting the super user bit has not resolved this.tgriep wrote:The check_ping does not have the superuser bit set on the plugin so run the following to fix that.
CODE: SELECT ALL
chmod ug+s check_ping
Code: Select all
-rwsr-sr-x 1 root root 66272 Apr 21 00:13 check_ping
./check_ping 127.0.0.1 -w 3000.0,80% -c 5000.0,100% -p 5
CRITICAL - You need more args!!!
Could not open pipe:
Running locally.M3o61CTs wrote:For the problems running the check_procs and the check_load plugins, are those errors generated when you run them remotely using the check_nrpe plugin or locally like shown in your example?
Can you run the following command on the system that has the errors and post the output here?
CODE: SELECT ALL
/usr/lib64/nagios/plugins/check_procs --help
/usr/lib64/nagios/plugins/check_load --help
ls -l /usr/lib64/nagios/plugins/
Re
Code: Select all
/usr/lib64/nagios/plugins/check_procs --help
check_procs v2.2.1 (nagios-plugins 2.2.1)
Copyright (c) 1999 Ethan Galstad <nagios@nagios.org>
Copyright (c) 2000-2014 Nagios Plugin Development Team
<devel@nagios-plugins.org>
Checks all processes and generates WARNING or CRITICAL states if the specified
metric is outside the required threshold ranges. The metric defaults to number
of processes. Search filters can be applied to limit the processes to check.
Usage:
check_procs -w <range> -c <range> [-m metric] [-s state] [-p ppid]
[-u user] [-r rss] [-z vsz] [-P %cpu] [-a argument-array]
[-C command] [-k] [-t timeout] [-v]
Options:
-h, --help
Print detailed help screen
-V, --version
Print version information
--extra-opts=[section][@file]
Read options from an ini file. See
https://www.nagios-plugins.org/doc/extra-opts.html
for usage and examples.
-w, --warning=RANGE
Generate warning state if metric is outside this range
-c, --critical=RANGE
Generate critical state if metric is outside this range
-m, --metric=TYPE
Check thresholds against metric. Valid types:
PROCS - number of processes (default)
VSZ - virtual memory size
RSS - resident set memory size
CPU - percentage CPU
ELAPSED - time elapsed in seconds
-t, --timeout=INTEGER
Seconds before plugin times out (default: 10)
-v, --verbose
Extra information. Up to 3 verbosity levels
-T, --traditional
Filter own process the traditional way by PID instead of /proc/pid/exe
Filters:
-s, --state=STATUSFLAGS
Only scan for processes that have, in the output of `ps`, one or
more of the status flags you specify (for example R, Z, S, RS,
RSZDT, plus others based on the output of your 'ps' command).
-p, --ppid=PPID
Only scan for children of the parent process ID indicated.
-z, --vsz=VSZ
Only scan for processes with VSZ higher than indicated.
-r, --rss=RSS
Only scan for processes with RSS higher than indicated.
-P, --pcpu=PCPU
Only scan for processes with PCPU higher than indicated.
-u, --user=USER
Only scan for processes with user name or ID indicated.
-a, --argument-array=STRING
Only scan for processes with args that contain STRING.
--ereg-argument-array=STRING
Only scan for processes with args that contain the regex STRING.
-C, --command=COMMAND
Only scan for exact matches of COMMAND (without path).
-k, --no-kthreads
Only scan for non kernel threads (works on Linux only).
-g, --cgroup-hierarchy
Only scan for processes belonging to STRING hierarchy (works on Linux only).
RANGEs are specified 'min:max' or 'min:' or ':max' (or 'max'). If
specified 'max:min', a warning status will be generated if the
count is inside the specified range
This plugin checks the number of currently running processes and
generates WARNING or CRITICAL states if the process count is outside
the specified threshold ranges. The process count can be filtered by
process owner, parent process PID, current state (e.g., 'Z'), or may
be the total number of running processes
Examples:
check_procs -w 2:2 -c 2:1024 -C portsentry
Warning if not two processes with command name portsentry.
Critical if < 2 or > 1024 processes
check_procs -c 1:1 -C bind -g /
Critical if not one processes with command name bind belonging to root cgroup.
check_procs -w 10 -a '/usr/local/bin/perl' -u root
Warning alert if > 10 processes with command arguments containing
'/usr/local/bin/perl' and owned by root
check_procs -w 50000 -c 100000 --metric=VSZ
Alert if VSZ of any processes over 50K or 100K
check_procs -w 10 -c 20 --metric=CPU
Alert if CPU of any processes over 10%% or 20%%
Send email to help@nagios-plugins.org if you have questions regarding use
of this software. To submit patches or suggest improvements, send email to
devel@nagios-plugins.org
Code: Select all
/usr/lib64/nagios/plugins/check_load --help
check_load v2.2.1 (nagios-plugins 2.2.1)
Copyright (c) 1999 Felipe Gustavo de Almeida <galmeida@linux.ime.usp.br>
Copyright (c) 1999-2014 Nagios Plugin Development Team
<devel@nagios-plugins.org>
This plugin tests the current system load average.
Usage:
check_load [-r] -w WLOAD1,WLOAD5,WLOAD15 -c CLOAD1,CLOAD5,CLOAD15
Options:
-h, --help
Print detailed help screen
-V, --version
Print version information
--extra-opts=[section][@file]
Read options from an ini file. See
https://www.nagios-plugins.org/doc/extra-opts.html
for usage and examples.
-w, --warning=WLOAD1,WLOAD5,WLOAD15
Exit with WARNING status if load average exceeds WLOADn
-c, --critical=CLOAD1,CLOAD5,CLOAD15
Exit with CRITICAL status if load average exceed CLOADn
the load average format is the same used by "uptime" and "w"
-r, --percpu
Divide the load averages by the number of CPUs (when possible)
Send email to help@nagios-plugins.org if you have questions regarding use
of this software. To submit patches or suggest improvements, send email to
devel@nagios-plugins.org
Code: Select all
ls -l /usr/lib64/nagios/plugins/
total 2732
-rwxr-xr-x 1 root root 4213 Mar 12 2015 check_apc
-rwxr-xr-x 1 root root 2346 Apr 21 00:13 check_breeze
-rwxr-xr-x 1 root root 62544 Apr 21 00:13 check_by_ssh
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_clamd -> check_tcp
-rwxr-xr-x 1 root root 49512 Apr 21 00:13 check_cluster
-rwsr-x--- 1 root nagios 62152 Apr 21 00:13 check_dhcp
-rwxr-xr-x 1 root root 58080 Apr 21 00:13 check_dig
-rwxr-xr-x 1 root root 67048 Apr 21 00:13 check_disk
-rwxr-xr-x 1 root root 9469 Apr 21 00:13 check_disk_smb
-rwxr-xr-x 1 root root 66272 Apr 21 00:13 check_dns
-rwxr-xr-x 1 root root 36624 Jul 12 22:44 check_dummy
-rwxr-xr-x 1 root root 36680 Jul 12 01:10 check_dummy.f25
-rwxr-xr-x 1 root root 36624 Apr 21 00:13 check_dummy.f26
-rwxr-xr-x 1 root root 3860 Apr 21 00:13 check_file_age
-rwxr-xr-x 1 root root 6504 Apr 21 00:13 check_flexlm
-rwsr-x--- 1 root nagios 62168 Apr 21 00:13 check_fping
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_ftp -> check_tcp
-rwxr-xr-x 1 root root 53760 Apr 21 00:13 check_game
-rwxr-xr-x 1 root root 57824 Apr 21 00:13 check_hpjd
-rwxr-xr-x 1 root root 104424 Apr 21 00:13 check_http
-rwsr-x--- 1 root nagios 65800 Apr 21 00:13 check_icmp
-rwsr-x--- 1 root nagios 49712 Apr 21 00:13 check_ide_smart
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_imap -> check_tcp
-rwxr-xr-x 1 root root 6984 Apr 21 00:13 check_ircd
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_jabber -> check_tcp
-rwxr-xr-x 1 root root 58280 Apr 21 00:13 check_ldap
lrwxrwxrwx 1 root root 10 Apr 21 00:13 check_ldaps -> check_ldap
-rwxr-xr-x 1 root root 53552 Apr 21 00:13 check_load
-rwxr-xr-x 1 root root 6669 Apr 21 00:13 check_log
-rwxr-xr-x 1 root root 22733 Apr 21 00:13 check_mailq
-rwxr-xr-x 1 root root 53680 Apr 21 00:13 check_mrtg
-rwxr-xr-x 1 root root 49512 Apr 21 00:13 check_mrtgtraf
-rwxr-xr-x 1 root root 62352 Apr 21 00:13 check_mysql
-rwxr-xr-x 1 root root 58016 Apr 21 00:13 check_mysql_query
-rwxr-xr-x 1 root root 53584 Apr 21 00:13 check_nagios
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_nntp -> check_tcp
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_nntps -> check_tcp
-rwxr-xr-x 1 root root 45488 Jun 15 00:14 check_nrpe
-rwxr-xr-x 1 root root 66144 Apr 21 00:13 check_nt
-rwxr-xr-x 1 root root 66240 Apr 21 00:13 check_ntp
-rwxr-xr-x 1 root root 62360 Apr 21 00:13 check_ntp_peer
-rwxr-xr-x 1 root root 14314 Apr 21 00:13 check_ntp.pl
-rwxr-xr-x 1 root root 62088 Apr 21 00:13 check_ntp_time
-rwxr-xr-x 1 root root 74224 Apr 21 00:13 check_nwstat
-rwxr-xr-x 1 root root 9457 Apr 21 00:13 check_oracle
-rwxr-xr-x 1 root root 57816 Apr 21 00:13 check_overcr
-rwxr-xr-x 1 root root 62344 Apr 21 00:13 check_pgsql
-rwsr-sr-x 1 root root 66272 Apr 21 00:13 check_ping
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_pop -> check_tcp
-rwxr-xr-x 1 root root 66464 Apr 21 00:13 check_procs
-rwxr-xr-x 1 root root 53824 Apr 21 00:13 check_real
-rwxr-xr-x 1 root root 9679 Apr 21 00:13 check_rpc
-rwxr-xr-x 1 root root 1533 Apr 21 00:13 check_sensors
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_simap -> check_tcp
-rwxr-xr-x 1 root root 74880 Apr 21 00:13 check_smtp
-rwxr-xr-x 1 root root 83504 Apr 21 00:13 check_snmp
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_spop -> check_tcp
-rwxr-xr-x 1 root root 53832 Apr 21 00:13 check_ssh
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_ssmtp -> check_tcp
-rwxr-xr-x 1 root root 49448 Apr 21 00:13 check_swap
-rwxr-xr-x 1 root root 70736 Apr 21 00:13 check_tcp
-rwxr-xr-x 1 root root 53768 Apr 21 00:13 check_time
lrwxrwxrwx 1 root root 9 Apr 21 00:13 check_udp -> check_tcp
-rwxr-xr-x 1 root root 62016 Apr 21 00:13 check_ups
-rwxr-xr-x 1 root root 45264 Apr 21 00:13 check_users
-rwxr-xr-x 1 root root 3032 Apr 21 00:13 check_wave
drwxr-xr-x 2 root root 4096 Feb 24 17:59 eventhandlers
-rwxr-xr-x 1 root root 45232 Apr 21 00:13 negate
-rwxr-xr-x 1 root root 45032 Apr 21 00:13 urlize
-rw-r--r-- 1 root root 2088 Apr 21 00:13 utils.pm
-rwxr-xr-x 1 root root 2791 Apr 21 00:13 utils.sh
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
Most likely it's because your system is starved of file-descriptors. You could try either upping the ulimit -n value of Nagios or use check_icmp instead. check_icmp uses a single socket rather than a pipe (two file-descriptors fewer per check). It's also a lot faster, so resources aren't hogged so long.
Either Fedora changed your ulimits, or it's using more than 25.
Either Fedora changed your ulimits, or it's using more than 25.
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
file-descriptors are not something I've looked at much, so I'm not so sure of what I'm looking at but what I see for nagios and nrpe are:
For nrpe these are the same values as for Fedora 25, though I can't confirm for nagios.
Code: Select all
su nagios -s /bin/bash
bash-4.4$ ulimit -Sn
1024
bash-4.4$ ulimit -Hn
4096
lsof | grep nagios | wc -l
162
Code: Select all
su nrpe -s /bin/bash
bash-4.4$ ulimit -Sn
1024
bash-4.4$ ulimit -Hn
4096
lsof | grep nrpe | wc -l
82
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
As a test, open /etc/sysctl.conf
Add the following line:
fs.file-max = 2097152
Then run:
sysctl -p (to apply the change)
Retry check_ping and post the output. Thank you!
Add the following line:
fs.file-max = 2097152
Then run:
sysctl -p (to apply the change)
Retry check_ping and post the output. Thank you!
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
Thank you for the prompts, results below:
Code: Select all
# sysctl -p
fs.file-max = 2097152
# /usr/lib64/nagios/plugins/check_load -w 5.0,4.0,3.0 -c 10.0,6.0,4.0
CRITICAL - You need more args!!!
Error opening
# /usr/lib64/nagios/plugins/check_ping 127.0.0.1 -w 3000.0,80% -c 5000.0,100% -p 5
CRITICAL - You need more args!!!
Could not open pipe:
Re: Issues following upgrade from Fedora 25 to Fedora 26 - N
Next step would probably be to recompile the Nagios plugins. Here are the instructions from the Knowledgebase:
https://support.nagios.com/kb/article/n ... tml#Fedora
Scroll down to "Installing The Nagios Plugins"
The configure scripts do some checking as to what versions of commands (ping, etc.) your system is running.
Once you've recompiled, try running the check_**** commends from the CLI and post the results.
Thank you!
https://support.nagios.com/kb/article/n ... tml#Fedora
Scroll down to "Installing The Nagios Plugins"
The configure scripts do some checking as to what versions of commands (ping, etc.) your system is running.
Once you've recompiled, try running the check_**** commends from the CLI and post the results.
Thank you!