"No output on stdout" error with check_snmp --rate

[apredeek]

Hi,

I've added a service using the --rate option for check_snmp and am getting this error in XI "(No output on stdout) stderr:". I've also tested from the command line as root, which returns a segmentation fault error. However, when testing the service in the Core Config Manager it works perfectly, creating the temporary file and calculating the rate correctly.

This is the argument string I'm using:
-o jnxCosIfqTxedBytes.749.6.115.105.108.118.101.114 -C <community> -P 2c -u "Bps" -m JUNIPER-COS-MIB --rate

Any assistance would be greatly appreciated.

[rkennedy]

Did it work previously, before adding the --rate part? Does your community string have any special characters in it?

The reason I ask is because usually the 'test check command' isn't nearly as credible as the CLI is. Can you try running the command over the CLI, as the nagios user, and post the full input / output?

[apredeek]

Without the --rate option, it works fine and returns the correct counter data. The community string does indeed have an @ character.

Here is the CLI output running as 'nagios'. The output when running as root is identical.

Code: Select all

[nagios@<hostname> libexec]$ ./check_snmp -H 10.64.0.1 -o jnxCosIfqTxedBytes.749.6.115.105.108.118.101.114 -C <community> -P 2c -u "Bps" -m JUNIPER-COS-MIB --rate
Segmentation fault

[rkennedy]

Perhaps the permissions are set to apache, and not the nagios user as they should be. What is the output of ls -al /usr/local/nagios/var/check_snmp/? This is the folder that contains the temp files --rate will utilize.

[apredeek]

Looks like maybe you're on the right track?

ls -al /usr/local/nagios/var/check_snmp/
total 12
drws------ 2 apache apache 4096 Jul 5 18:26 .
drwxrwxr-x. 7 nagios nagios 4096 Jul 6 12:57 ..

[rkennedy]

Yeah, I think this might be it. Can you adjust the permissions to be nagios:nagios on the check_snmp folder? chown nagios:nagios /usr/local/nagios/var/check_snmp/

Here's what I'm seeing with apache being the owner -

Code: Select all

[root@localhost libexec]# ./check_snmp -H 192.168.x.x -o IF-MIB::ifAdminStatus.1 -C public --rate
SNMP RATE OK - 0 | IF-MIB::ifAdminStatus.1=0
[root@localhost libexec]# su nagios
[nagios@localhost libexec]$ ./check_snmp -H 192.168.x.x -o IF-MIB::ifAdminStatus.1 -C public --rate
Cannot create temporary filename

After changing permissions -

Code: Select all

[root@localhost libexec]# chown nagios:nagios /usr/local/nagios/var/check_snmp/
[root@localhost libexec]# su nagios
[nagios@localhost libexec]$ ./check_snmp -H 192.168.x.x -o IF-MIB::ifAdminStatus.1 -C public --rate
No previous data to calculate rate - assume okay
[nagios@localhost libexec]$ ./chs.1 -C public --rate IF-MIB::ifAdminStatus
SNMP RATE OK - 0 | IF-MIB::ifAdminStatus.1=0

[apredeek]

No such luck. I ran the chown and fixed the ownership of the folder:
ls -al /usr/local/nagios/var/check_snmp/
total 12
drws------ 2 nagios nagios 4096 Jul 5 18:26 .
drwxrwxr-x. 7 nagios nagios 4096 Jul 6 14:41 ..

But I still get the segmentation fault error:
[nagios@<hostname> libexec]$ ./check_snmp -H 10.64.0.1 -o jnxCosIfqTxedBytes.749.6.115.105.108.118.101.114 -C <community> -P 2c -u "Bps" -m JUNIPER-COS-MIB --rate
Segmentation fault

[rkennedy]

Can you show us the permissions on the files as well, not just the . and ..? I'm wondering if it's boiling down to the permissions on theses individual files.

Additionally, when you run ./check_snmp -H 10.64.0.1 -o jnxCosIfqTxedBytes.749.6.115.105.108.118.101.114 -C <community> -P 2c -u "Bps" -m JUNIPER-COS-MIB --rate, can you run 'strace' before it, and post the output? This might help to see where it's getting stuck at. (may have to run yum install strace -y)

Code: Select all

strace ./check_snmp -H 10.64.0.1 -o jnxCosIfqTxedBytes.749.6.115.105.108.118.101.114 -C <community> -P 2c -u "Bps" -m JUNIPER-COS-MIB --rate

[apredeek]

There are no files in the directory. Here's the strace output:

Code: Select all

execve("./check_snmp", ["./check_snmp", "-H", "10.64.0.1", "-o", "jnxCosIfqTxedBytes.749.6.115.105"..., "-C", "<community>", "-P", "2c", "-u", "Bps", "-m", "JUNIPER-COS-MIB", "--rate"], [/* 21 vars */]) = 0
brk(0)                                  = 0x1e12000
fcntl(0, F_GETFD)                       = 0
fcntl(1, F_GETFD)                       = 0
fcntl(2, F_GETFD)                       = 0
access("/etc/suid-debug", F_OK)         = -1 ENOENT (No such file or directory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fdc3e55f000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=39473, ...}) = 0
mmap(NULL, 39473, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fdc3e555000
close(3)                                = 0
open("/lib64/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000^\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=142688, ...}) = 0
mmap(NULL, 2212848, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fdc3e124000
mprotect(0x7fdc3e13b000, 2097152, PROT_NONE) = 0
mmap(0x7fdc3e33b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fdc3e33b000
mmap(0x7fdc3e33d000, 13296, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdc3e33d000
close(3)                                = 0
open("/lib64/libdl.so.2", O_RDONLY)     = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19536, ...}) = 0
mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fdc3df20000
mprotect(0x7fdc3df22000, 2097152, PROT_NONE) = 0
mmap(0x7fdc3e122000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fdc3e122000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY)      = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\356\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1920936, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fdc3e554000
mmap(NULL, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fdc3db8c000
mprotect(0x7fdc3dd16000, 2097152, PROT_NONE) = 0
mmap(0x7fdc3df16000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x7fdc3df16000
mmap(0x7fdc3df1b000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdc3df1b000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fdc3e553000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fdc3e552000
arch_prctl(ARCH_SET_FS, 0x7fdc3e553700) = 0
mprotect(0x7fdc3df16000, 16384, PROT_READ) = 0
mprotect(0x7fdc3e122000, 4096, PROT_READ) = 0
mprotect(0x7fdc3e33b000, 4096, PROT_READ) = 0
mprotect(0x7fdc3e560000, 4096, PROT_READ) = 0
munmap(0x7fdc3e555000, 39473)           = 0
set_tid_address(0x7fdc3e5539d0)         = 19659
set_robust_list(0x7fdc3e5539e0, 24)     = 0
futex(0x7ffea6e13b2c, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ffea6e13b2c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7fdc3e553700) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x7fdc3e129cb0, [], SA_RESTORER|SA_SIGINFO, 0x7fdc3e1337e0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fdc3e129d40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fdc3e1337e0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=10240*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(0)                                  = 0x1e12000
brk(0x1e33000)                          = 0x1e33000
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=99158576, ...}) = 0
mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fdc37cfb000
close(3)                                = 0
gettimeofday({1467837210, 885819}, NULL) = 0
getuid()                                = 0
geteuid()                               = 48
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x30} ---
+++ killed by SIGSEGV +++
Segmentation fault

[rkennedy]

What are the permissions of the folder? ls -al /usr/local/nagios/var/

The strace indicates it's closing right before it should open the file for writing, I'm unsure how it's working from the 'test check command' if there are no files in the /usr/local/nagios/var/check_snmp/ directory, as this is what it uses for a 'temp' folder.

Here is where mine proceeds (on a working system) -

Code: Select all

fstat(3, {st_mode=S_IFREG|0644, st_size=106065056, ...}) = 0
mmap(NULL, 106065056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f467504c000
close(3)                                = 0
getuid()                                = 1000
geteuid()                               = 1000
open("/usr/local/nagios/var/check_snmp/a2cc0764f96824231572a4a3e70aa3adec0f2da8", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0640, st_size=33, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1

Can you try running rm -rf /usr/local/nagios/var/check_snmp* and then as the nagios user, run the command once again? It will then re-create the folder, and possibly be the right permissions this time.

Code: Select all

[nagios@localhost libexec]$ ./check_snmp -H 192.168.x.x -o IF-MIB::ifAdminStatus.1 -C public --rate
SNMP RATE OK - 0 | IF-MIB::ifAdminStatus.1=0
[nagios@localhost libexec]$ rm -rf /usr/local/nagios/var/check_snmp*
[nagios@localhost libexec]$ ./check_snmp -H 192.168.x.x -o IF-MIB::ifAdminStatus.1 -C public --rate
No previous data to calculate rate - assume okay
[nagios@localhost libexec]$ ./check_snmp -H 192.168.x.x -o IF-MIB::ifAdminStatus.1 -C public --rate
SNMP RATE OK - 0 | IF-MIB::ifAdminStatus.1=0

Lastly, what are the permissions on your check_snmp plugin? ls -al /usr/local/nagios/libexec/check_snmp