Issues following upgrade from Fedora 25 to Fedora 26 - Nagio

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
M3o61CTs
Posts: 10
Joined: Wed Jul 12, 2017 7:02 pm

Issues following upgrade from Fedora 25 to Fedora 26 - Nagio

Post by M3o61CTs »

Moderator Edit: This thread has been split from another - https://support.nagios.com/forum/viewto ... 35&t=44004
In the future, please create a new thread and link to the old one instead of adding on.


I have upgraded From Fedora 25 to 26 and am now seeing the same problem.

Running from command line as root does not resolve the problem.

check_icmp does work.
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by tgriep »

Can you login to the server as root, run the following commands and post the output.

Code: Select all

which ping
ping
ping -V
Can you go to the folder where the plugins are located at, run this and post the output.

Code: Select all

ls -l check_ping
One thing to try is to recompile the plugins and see if that fixes the issue.
Thanks
Be sure to check out our Knowledgebase for helpful articles and solutions!
M3o61CTs
Posts: 10
Joined: Wed Jul 12, 2017 7:02 pm

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by M3o61CTs »

Having been moved to a new thread the title might be better as "Issues following upgrade from Fedora 25 to Fedora 26 - Nagios 4.0.8 to 4.3.1"

There are a number of issues that resulted from the upgrade, I was initially trying to keep my response to a single topic/thread.

Historically my system has been running for some time (~ 10 years) and would have been on the early version 3 source, All the software is from Fedora repos with a few plugins from exchange. All functions were working prior to the Fedora 25 to 26 upgrade.

check_ping:

The output you asked for:

/usr/sbin/ping

Usage: ping [-aAbBdDfhLnOqrRUvV64] [-c count] [-i interval] [-I interface]
[-m mark] [-M pmtudisc_option] [-l preload] [-p pattern] [-Q tos]
[-s packetsize] [-S sndbuf] [-t ttl] [-T timestamp_option]
[-w deadline] [-W timeout] [hop1 ...] destination
Usage: ping -6 [-aAbBdDfhLnOqrRUvV] [-c count] [-i interval] [-I interface]
[-l preload] [-m mark] [-M pmtudisc_option]
[-N nodeinfo_option] [-p pattern] [-Q tclass] [-s packetsize]
[-S sndbuf] [-t ttl] [-T timestamp_option] [-w deadline]
[-W timeout] destination

ping utility, iputils-s20161105

-rwxr-xr-x 1 root root 66272 Apr 21 00:13 /usr/lib64/nagios/plugins/check_ping


ping from a shell works as expected, but check_ping always returns

CRITICAL - You need more args!!!
Could not open pipe:

This includes when pinging the localhost on either the loopback or IP address.


My work round has been to use check_icmp as documented elsewhere.


SSL/TLS


I have a large number of services which fail with

CHECK_NRPE: Error - Could not complete SSL handshake with


This appears to be linked with the fact that Fedora 26 has moved to OpenSSL 1.1.0f-fips and that SSLv2 has been completely removed. This has meant that all NRPE connections initially failed.

For most (but not all) of my remote systems running nrpe 3+ I've able to reconfigure SSL/TLS and service checks are working.

For those system on nrpe 2+ (EL6/EL7) I've not found a solution yet.



For any others who are reading following a similar upgrade path, I've also a few other minor issues with check_procs and check_load:

/usr/lib64/nagios/plugins/check_procs -w 10 -c 15 -s Z
Unable to read output

/usr/lib64/nagios/plugins/check_load -w 5.0,4.0,3.0 -c 10.0,6.0,4.0
CRITICAL - You need more args!!!
Error opening
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by tgriep »

The check_ping does not have the superuser bit set on the plugin so run the following to fix that.

Code: Select all

chmod ug+s check_ping
For the problems running the check_procs and the check_load plugins, are those errors generated when you run them remotely using the check_nrpe plugin or locally like shown in your example?
Can you run the following command on the system that has the errors and post the output here?

Code: Select all

/usr/lib64/nagios/plugins/check_procs --help
/usr/lib64/nagios/plugins/check_load --help
ls -l /usr/lib64/nagios/plugins/
Be sure to check out our Knowledgebase for helpful articles and solutions!
M3o61CTs
Posts: 10
Joined: Wed Jul 12, 2017 7:02 pm

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by M3o61CTs »

tgriep wrote:The check_ping does not have the superuser bit set on the plugin so run the following to fix that.
CODE: SELECT ALL
chmod ug+s check_ping
Setting the super user bit has not resolved this.

Code: Select all

-rwsr-sr-x 1 root root    66272 Apr 21 00:13 check_ping

./check_ping 127.0.0.1 -w 3000.0,80% -c 5000.0,100% -p 5
CRITICAL - You need more args!!!
Could not open pipe: 

M3o61CTs wrote:For the problems running the check_procs and the check_load plugins, are those errors generated when you run them remotely using the check_nrpe plugin or locally like shown in your example?
Can you run the following command on the system that has the errors and post the output here?
CODE: SELECT ALL
/usr/lib64/nagios/plugins/check_procs --help
/usr/lib64/nagios/plugins/check_load --help
ls -l /usr/lib64/nagios/plugins/
Re
Running locally.

Code: Select all

/usr/lib64/nagios/plugins/check_procs --help
check_procs v2.2.1 (nagios-plugins 2.2.1)
Copyright (c) 1999 Ethan Galstad <nagios@nagios.org>
Copyright (c) 2000-2014 Nagios Plugin Development Team
	<devel@nagios-plugins.org>

Checks all processes and generates WARNING or CRITICAL states if the specified
metric is outside the required threshold ranges. The metric defaults to number
of processes.  Search filters can be applied to limit the processes to check.


Usage:
check_procs -w <range> -c <range> [-m metric] [-s state] [-p ppid]
 [-u user] [-r rss] [-z vsz] [-P %cpu] [-a argument-array]
 [-C command] [-k] [-t timeout] [-v]

Options:
 -h, --help
    Print detailed help screen
 -V, --version
    Print version information
 --extra-opts=[section][@file]
    Read options from an ini file. See
    https://www.nagios-plugins.org/doc/extra-opts.html
    for usage and examples.
 -w, --warning=RANGE
   Generate warning state if metric is outside this range
 -c, --critical=RANGE
   Generate critical state if metric is outside this range
 -m, --metric=TYPE
  Check thresholds against metric. Valid types:
  PROCS   - number of processes (default)
  VSZ     - virtual memory size
  RSS     - resident set memory size
  CPU     - percentage CPU
  ELAPSED - time elapsed in seconds
 -t, --timeout=INTEGER
    Seconds before plugin times out (default: 10)
 -v, --verbose
    Extra information. Up to 3 verbosity levels
 -T, --traditional
   Filter own process the traditional way by PID instead of /proc/pid/exe

Filters:
 -s, --state=STATUSFLAGS
   Only scan for processes that have, in the output of `ps`, one or
   more of the status flags you specify (for example R, Z, S, RS,
   RSZDT, plus others based on the output of your 'ps' command).
 -p, --ppid=PPID
   Only scan for children of the parent process ID indicated.
 -z, --vsz=VSZ
   Only scan for processes with VSZ higher than indicated.
 -r, --rss=RSS
   Only scan for processes with RSS higher than indicated.
 -P, --pcpu=PCPU
   Only scan for processes with PCPU higher than indicated.
 -u, --user=USER
   Only scan for processes with user name or ID indicated.
 -a, --argument-array=STRING
   Only scan for processes with args that contain STRING.
 --ereg-argument-array=STRING
   Only scan for processes with args that contain the regex STRING.
 -C, --command=COMMAND
   Only scan for exact matches of COMMAND (without path).
 -k, --no-kthreads
   Only scan for non kernel threads (works on Linux only).
 -g, --cgroup-hierarchy
   Only scan for processes belonging to STRING hierarchy (works on Linux only).

RANGEs are specified 'min:max' or 'min:' or ':max' (or 'max'). If
specified 'max:min', a warning status will be generated if the
count is inside the specified range

This plugin checks the number of currently running processes and
generates WARNING or CRITICAL states if the process count is outside
the specified threshold ranges. The process count can be filtered by
process owner, parent process PID, current state (e.g., 'Z'), or may
be the total number of running processes

Examples:
 check_procs -w 2:2 -c 2:1024 -C portsentry
  Warning if not two processes with command name portsentry.
  Critical if < 2 or > 1024 processes

 check_procs -c 1:1 -C bind -g /
  Critical if not one processes with command name bind belonging to root cgroup.

 check_procs -w 10 -a '/usr/local/bin/perl' -u root
  Warning alert if > 10 processes with command arguments containing
  '/usr/local/bin/perl' and owned by root

 check_procs -w 50000 -c 100000 --metric=VSZ
  Alert if VSZ of any processes over 50K or 100K

 check_procs -w 10 -c 20 --metric=CPU
  Alert if CPU of any processes over 10%% or 20%%

Send email to help@nagios-plugins.org if you have questions regarding use
of this software. To submit patches or suggest improvements, send email to
devel@nagios-plugins.org

Code: Select all

/usr/lib64/nagios/plugins/check_load --help
check_load v2.2.1 (nagios-plugins 2.2.1)
Copyright (c) 1999 Felipe Gustavo de Almeida <galmeida@linux.ime.usp.br>
Copyright (c) 1999-2014 Nagios Plugin Development Team
	<devel@nagios-plugins.org>

This plugin tests the current system load average.

Usage:
check_load [-r] -w WLOAD1,WLOAD5,WLOAD15 -c CLOAD1,CLOAD5,CLOAD15

Options:
 -h, --help
    Print detailed help screen
 -V, --version
    Print version information
 --extra-opts=[section][@file]
    Read options from an ini file. See
    https://www.nagios-plugins.org/doc/extra-opts.html
    for usage and examples.
 -w, --warning=WLOAD1,WLOAD5,WLOAD15
    Exit with WARNING status if load average exceeds WLOADn
 -c, --critical=CLOAD1,CLOAD5,CLOAD15
    Exit with CRITICAL status if load average exceed CLOADn
    the load average format is the same used by "uptime" and "w"
 -r, --percpu
    Divide the load averages by the number of CPUs (when possible)

Send email to help@nagios-plugins.org if you have questions regarding use
of this software. To submit patches or suggest improvements, send email to
devel@nagios-plugins.org

Code: Select all

ls -l /usr/lib64/nagios/plugins/
total 2732
-rwxr-xr-x 1 root root     4213 Mar 12  2015 check_apc
-rwxr-xr-x 1 root root     2346 Apr 21 00:13 check_breeze
-rwxr-xr-x 1 root root    62544 Apr 21 00:13 check_by_ssh
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_clamd -> check_tcp
-rwxr-xr-x 1 root root    49512 Apr 21 00:13 check_cluster
-rwsr-x--- 1 root nagios  62152 Apr 21 00:13 check_dhcp
-rwxr-xr-x 1 root root    58080 Apr 21 00:13 check_dig
-rwxr-xr-x 1 root root    67048 Apr 21 00:13 check_disk
-rwxr-xr-x 1 root root     9469 Apr 21 00:13 check_disk_smb
-rwxr-xr-x 1 root root    66272 Apr 21 00:13 check_dns
-rwxr-xr-x 1 root root    36624 Jul 12 22:44 check_dummy
-rwxr-xr-x 1 root root    36680 Jul 12 01:10 check_dummy.f25
-rwxr-xr-x 1 root root    36624 Apr 21 00:13 check_dummy.f26
-rwxr-xr-x 1 root root     3860 Apr 21 00:13 check_file_age
-rwxr-xr-x 1 root root     6504 Apr 21 00:13 check_flexlm
-rwsr-x--- 1 root nagios  62168 Apr 21 00:13 check_fping
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_ftp -> check_tcp
-rwxr-xr-x 1 root root    53760 Apr 21 00:13 check_game
-rwxr-xr-x 1 root root    57824 Apr 21 00:13 check_hpjd
-rwxr-xr-x 1 root root   104424 Apr 21 00:13 check_http
-rwsr-x--- 1 root nagios  65800 Apr 21 00:13 check_icmp
-rwsr-x--- 1 root nagios  49712 Apr 21 00:13 check_ide_smart
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_imap -> check_tcp
-rwxr-xr-x 1 root root     6984 Apr 21 00:13 check_ircd
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_jabber -> check_tcp
-rwxr-xr-x 1 root root    58280 Apr 21 00:13 check_ldap
lrwxrwxrwx 1 root root       10 Apr 21 00:13 check_ldaps -> check_ldap
-rwxr-xr-x 1 root root    53552 Apr 21 00:13 check_load
-rwxr-xr-x 1 root root     6669 Apr 21 00:13 check_log
-rwxr-xr-x 1 root root    22733 Apr 21 00:13 check_mailq
-rwxr-xr-x 1 root root    53680 Apr 21 00:13 check_mrtg
-rwxr-xr-x 1 root root    49512 Apr 21 00:13 check_mrtgtraf
-rwxr-xr-x 1 root root    62352 Apr 21 00:13 check_mysql
-rwxr-xr-x 1 root root    58016 Apr 21 00:13 check_mysql_query
-rwxr-xr-x 1 root root    53584 Apr 21 00:13 check_nagios
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_nntp -> check_tcp
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_nntps -> check_tcp
-rwxr-xr-x 1 root root    45488 Jun 15 00:14 check_nrpe
-rwxr-xr-x 1 root root    66144 Apr 21 00:13 check_nt
-rwxr-xr-x 1 root root    66240 Apr 21 00:13 check_ntp
-rwxr-xr-x 1 root root    62360 Apr 21 00:13 check_ntp_peer
-rwxr-xr-x 1 root root    14314 Apr 21 00:13 check_ntp.pl
-rwxr-xr-x 1 root root    62088 Apr 21 00:13 check_ntp_time
-rwxr-xr-x 1 root root    74224 Apr 21 00:13 check_nwstat
-rwxr-xr-x 1 root root     9457 Apr 21 00:13 check_oracle
-rwxr-xr-x 1 root root    57816 Apr 21 00:13 check_overcr
-rwxr-xr-x 1 root root    62344 Apr 21 00:13 check_pgsql
-rwsr-sr-x 1 root root    66272 Apr 21 00:13 check_ping
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_pop -> check_tcp
-rwxr-xr-x 1 root root    66464 Apr 21 00:13 check_procs
-rwxr-xr-x 1 root root    53824 Apr 21 00:13 check_real
-rwxr-xr-x 1 root root     9679 Apr 21 00:13 check_rpc
-rwxr-xr-x 1 root root     1533 Apr 21 00:13 check_sensors
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_simap -> check_tcp
-rwxr-xr-x 1 root root    74880 Apr 21 00:13 check_smtp
-rwxr-xr-x 1 root root    83504 Apr 21 00:13 check_snmp
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_spop -> check_tcp
-rwxr-xr-x 1 root root    53832 Apr 21 00:13 check_ssh
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_ssmtp -> check_tcp
-rwxr-xr-x 1 root root    49448 Apr 21 00:13 check_swap
-rwxr-xr-x 1 root root    70736 Apr 21 00:13 check_tcp
-rwxr-xr-x 1 root root    53768 Apr 21 00:13 check_time
lrwxrwxrwx 1 root root        9 Apr 21 00:13 check_udp -> check_tcp
-rwxr-xr-x 1 root root    62016 Apr 21 00:13 check_ups
-rwxr-xr-x 1 root root    45264 Apr 21 00:13 check_users
-rwxr-xr-x 1 root root     3032 Apr 21 00:13 check_wave
drwxr-xr-x 2 root root     4096 Feb 24 17:59 eventhandlers
-rwxr-xr-x 1 root root    45232 Apr 21 00:13 negate
-rwxr-xr-x 1 root root    45032 Apr 21 00:13 urlize
-rw-r--r-- 1 root root     2088 Apr 21 00:13 utils.pm
-rwxr-xr-x 1 root root     2791 Apr 21 00:13 utils.sh
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by dwhitfield »

Most likely it's because your system is starved of file-descriptors. You could try either upping the ulimit -n value of Nagios or use check_icmp instead. check_icmp uses a single socket rather than a pipe (two file-descriptors fewer per check). It's also a lot faster, so resources aren't hogged so long.

Either Fedora changed your ulimits, or it's using more than 25.
M3o61CTs
Posts: 10
Joined: Wed Jul 12, 2017 7:02 pm

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by M3o61CTs »

file-descriptors are not something I've looked at much, so I'm not so sure of what I'm looking at but what I see for nagios and nrpe are:

Code: Select all

su nagios -s /bin/bash
bash-4.4$ ulimit -Sn
1024
bash-4.4$ ulimit -Hn
4096

lsof | grep nagios | wc -l
162

Code: Select all

su nrpe -s /bin/bash
bash-4.4$ ulimit -Sn
1024
bash-4.4$ ulimit -Hn
4096

lsof | grep nrpe | wc -l
82

For nrpe these are the same values as for Fedora 25, though I can't confirm for nagios.
bolson

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by bolson »

As a test, open /etc/sysctl.conf

Add the following line:

fs.file-max = 2097152

Then run:
sysctl -p (to apply the change)

Retry check_ping and post the output. Thank you!
M3o61CTs
Posts: 10
Joined: Wed Jul 12, 2017 7:02 pm

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by M3o61CTs »

Thank you for the prompts, results below:

Code: Select all

# sysctl -p
fs.file-max = 2097152

# /usr/lib64/nagios/plugins/check_load -w 5.0,4.0,3.0 -c 10.0,6.0,4.0
CRITICAL - You need more args!!!
Error opening 

# /usr/lib64/nagios/plugins/check_ping 127.0.0.1 -w 3000.0,80% -c 5000.0,100% -p 5
CRITICAL - You need more args!!!
Could not open pipe: 
bolson

Re: Issues following upgrade from Fedora 25 to Fedora 26 - N

Post by bolson »

Next step would probably be to recompile the Nagios plugins. Here are the instructions from the Knowledgebase:

https://support.nagios.com/kb/article/n ... tml#Fedora

Scroll down to "Installing The Nagios Plugins"

The configure scripts do some checking as to what versions of commands (ping, etc.) your system is running.

Once you've recompiled, try running the check_**** commends from the CLI and post the results.

Thank you!
Locked