Nagios ssh proxy apt check not working correctly

[nag4csec42]

From the web UI, I run the check command on an Ubuntu 14.04.5 server and in the Nagios UI I am showing no critical updates pending:

[nagios@sec-nagios-001 ~]$ /usr/local/nagios/libexec/check_by_ssh -H 127.0.0.1 -t 40 -C "/usr/local/nagios/libexec/check_apt -U -t 20" -E
APT WARNING: 59 packages available for upgrade (0 critical updates). |available_upgrades=59;;;0 critical_updates=0;;;0

When I run the same command via the shell from the nagios server, I receive the same result above.

Upon login via ssh to the same system, I receive this:
70 packages can be updated.
45 updates are security updates.

Why is there such a difference in packages available to update via Nagios and the Ubuntu server itself?

I am not seeing this on any Ubuntu 16.04 servers using ssh proxy apt checks. However, it does seem to be occurring for the apt checks on the Ubuntu 16.04 servers with passive checks enabled.

Ubuntu 16.04.4 server passive apt check status from Nagios XI web UI:
APT WARNING: 4 packages available for upgrade (0 critical updates).

Ubuntu 16.04.4 server passive apt check ssh shell login status:
55 packages can be updated.
34 updates are security updates.

ncpa version is 2.1.4-1

[kyang]

Could you run the check_apt plugin just from the Ubuntu server? Just to verify the output.

/usr/local/nagios/libexec/check_by_ssh -H 127.0.0.1

Are you using this exact IP? Or did you just edit and changed it?

On the Ubuntu server, run these commands and send us the output.

Code: Select all

apt-get --just-print upgrade 2>&1 | perl -ne 'if (/Inst\s([\w,\-,\d,\.,~,:,\+]+)\s\[([\w,\-,\d,\.,~,:,\+]+)\]\s\(([\w,\-,\d,\.,~,:,\+]+)\)? /i) {print "PROGRAM: $1 INSTALLED: $2 AVAILABLE: $3\n"}' | wc

Code: Select all

apt list --upgradable | wc 

Ubuntu 16.04.4 server passive apt check status from Nagios XI web UI:
APT WARNING: 4 packages available for upgrade (0 critical updates).

Ubuntu 16.04.4 server passive apt check ssh shell login status:
55 packages can be updated.
34 updates are security updates.

ncpa version is 2.1.4-1

For this, you're using it with NCPA correct? What does it show in the NCPA GUI when running this check?

[nag4csec42]

I apologize for my slow response, I have been out of town attending a conference.

Ubuntu 14.04.5 server apt check issue:
I used 127.0.0.1 as an anonymized IP address. The real IP address outputs the same data.

Here is what the OS tells me upon login:

70 packages can be updated.
45 updates are security updates.

The output from the command you suggested to run below is as follows:

Code: Select all

$ /usr/local/nagios/libexec/check_by_ssh -H 127.0.0.1
check_by_ssh: No remotecmd

Here is the output from the entire command on the local server:

Code: Select all

$ /usr/local/nagios/libexec/check_by_ssh -H 127.0.0.1 -t 40 -C "/usr/local/nagios/libexec/check_apt -U -t 20" -E
APT WARNING: 59 packages available for upgrade (0 critical updates). |available_upgrades=59;;;0 critical_updates=0;;;0

The OS on this Ubuntu 14.04.5 server is not asking for a reboot at this time.

Here is the output from the apt-get commands that you asked me to run:

$ apt-get --just-print upgrade 2>&1 | perl -ne 'if (/Inst\s([\w,\-,\d,\.,~,:,\+]+)\s\[([\w,\-,\d,\.,~,:,\+]+)\]\s\(([\w,\-,\d,\.,~,:,\+]+)\)? /i) {print "PROGRAM: $1 INSTALLED: $2 AVAILABLE: $3\n"}' | wc
59 354 4561

apt list --upgradable | wc
$ apt list --upgradable | wc

WARNING: apt does not have a stable CLI interface yet. Use with caution in scripts.

63 373 5936

--------------------------------

Ubuntu 16.04.4 server apt check issue:

Current package status upon login:

78 packages can be updated.
54 updates are security updates.

NCPA Web UI states this for the check_apt command:

APT WARNING: 4 packages available for upgrade (0 critical updates). |available_upgrades=4;;;0 critical_updates=0;;;0

API Endpoint plugin check using check_apt & ''-U -t 20' args:

Code: Select all

APT CRITICAL: 67 packages available for upgrade (43 critical updates). |available_upgrades=67;;;0 critical_updates=43;;;0

Output from the apt-get commands that you asked me to run:

Code: Select all

$ apt-get --just-print upgrade 2>&1 | perl -ne 'if (/Inst\s([\w,\-,\d,\.,~,:,\+]+)\s\[([\w,\-,\d,\.,~,:,\+]+)\]\s\(([\w,\-,\d,\.,~,:,\+]+)\)? /i) {print "PROGRAM: $1 INSTALLED: $2 AVAILABLE: $3\n"}' | wc
     67     402    5469

$ apt list --upgradable | wc

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

     72     427    7527

[kyang]

Thanks,

Could you send me the verbose output when using that plugin.

Code: Select all

/usr/local/nagios/libexec/check_apt -v > /tmp/checkAPT.txt

Let's compare it to a few other commands.

Code: Select all

apt list --upgradable > /tmp/aptList.txt

Code: Select all

apt-get --just-print upgrade 2>&1 | perl -ne 'if (/Inst\s([\w,\-,\d,\.,~,:,\+]+)\s\[([\w,\-,\d,\.,~,:,\+]+)\]\s\(([\w,\-,\d,\.,~,:,\+]+)\)? /i) {print "PROGRAM: $1 INSTALLED: $2 AVAILABLE: $3\n"}' > /tmp/aptGet.txt

After that, please attach all of the text files.

[nag4csec42]

Upon login to the Ubuntu 14.04.5 server:

70 packages can be updated.
45 updates are security updates.

These files are the output from the Ubuntu 14.04.5 server:

aptGet.txt

aptList.txt

checkAPT.txt

[nag4csec42]

Ubuntu 16.04 server login notification:

80 packages can be updated.
55 updates are security updates.

aptGet-16.04.txt

aptList-16.04.txt

checkAPT-16.04.4.txt

[scottwilkerson]

per checkAPT.txt

Code: Select all

apt-get needs root privileges for real execution.

So this should be being called with sudo, for example with check_by_ssh
\

Code: Select all

/usr/local/nagios/libexec/check_by_ssh -H xxx.xx.xxx.xxx -t 40 -C "sudo /usr/local/nagios/libexec/check_apt -U -t 20" -E 

Also, you need to add that script to your sudoers file like so

Code: Select all

Defaults:nagios !requiretty
nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_apt

[nag4csec42]

Thank you for the info.

When I call the check_apt command with sudo from the command line, I do not receive the expected results:

Ubuntu 14.04.5 server:

Upon login:

Code: Select all

70 packages can be updated.
45 updates are security updates.

Code: Select all

$ sudo /usr/local/nagios/libexec/check_apt -U -t 20
APT WARNING: 59 packages available for upgrade (0 critical updates). |available_upgrades=59;;;0 critical_updates=0;;;0

_____________________________________
Ubuntu 16.04.4 server:

Upon login:

Code: Select all

34 packages can be updated.
1 update is a security update.

Code: Select all

$ sudo /usr/local/nagios/libexec/check_apt -U -t 20
APT WARNING: 34 packages available for upgrade (0 critical updates). |available_upgrades=34;;;0 critical_updates=0;;;0

[scottwilkerson]

I'm not sure what is run to get the info you see on login, but you may need to change the flags for different results with the plugin

Here is the manpage
https://nagios-plugins.org/doc/man/check_apt.html

[nag4csec42]

On the Ubuntu 14.04.5 box, I receive this upon login:

70 packages can be updated.
45 updates are security updates.

What flag should I add to get the security updates to show up correctly? If I add the -c flag, I receive this:

$ sudo /usr/local/nagios/libexec/check_apt -U -c -t 20
APT CRITICAL: 59 packages available for upgrade (6 critical updates). |available_upgrades=59;;;0 critical_updates=6;;;0

From a customer's perspective, shouldn't the default output of when I login match the output of the results from check_apt?

At the end of the day, I need Nagios to alert us when there are any critical security updates, and right now it is not doing this for us.

Here is the script that Ubuntu 14.04.5 runs upon login:

Code: Select all

$ cat /usr/lib/update-notifier/update-motd-updates-available
#!/bin/sh -e
#
# helper for update-motd


# poor mans force
if [ "$1" = "--force" ]; then
    NEED_UPDATE_CHECK=yes
fi

# check time when we did the last update check
stamp="/var/lib/update-notifier/updates-available"

# get list dir
StateDir="/var/lib/apt/"
ListDir="lists/"
eval "$(apt-config shell StateDir Dir::State)"
eval "$(apt-config shell ListDir Dir::State::Lists)"

# get sources.list file
EtcDir="etc/apt/"
SourceList="sources.list"
eval "$(apt-config shell EtcDir Dir::Etc)"
eval "$(apt-config shell SourceList Dir::Etc::sourcelist)"

# check if we have a list file or sources.list that needs checking
if [ -e "$stamp" ]; then
    if [ "$(find "/$StateDir/$ListDir" "/$EtcDir/$SourceList" -type f -newer "$stamp" -print -quit)" ]; then
        NEED_UPDATE_CHECK=yes
    fi
else
    if [ "$(find "/$StateDir/$ListDir" "/$EtcDir/$SourceList" -type f -print -quit)" ]; then
        NEED_UPDATE_CHECK=yes
    fi
fi

# output something for update-motd
if [ -n "$NEED_UPDATE_CHECK" ]; then
    {
        echo ""
        /usr/lib/update-notifier/apt-check --human-readable
        echo ""
    } > $stamp
fi

# output what we have (either cached or newly generated)
cat "$stamp"