Page 1 of 3
Intermittent Segmentation Fault with check_vmware_api
Posted: Tue Sep 07, 2021 9:13 am
by Sargento
Hello,
At random we are experiencing segmentation faults while running check_vmware_api. Here is an example..
Code: Select all
/usr/local/nagios/libexec/check_vmware_api.pl -f creds.file -l cpu -w 80 -c 90 -H host -v
CHECK_VMWARE_API.PL OK - cpu usage=1430.00 MHz (3.89%) | cpu_usagemhz=1430.00;80;90 cpu_usage=3.89%;80;90
..great!? Send it into production.. I then run one more test, running the same command, exactly the same, again..
Code: Select all
/usr/local/nagios/libexec/check_vmware_api.pl -f creds.file -l cpu -w 80 -c 90 -H host -v
Segmentation fault
Why could this be? This also occurs while trying to create a session file for this host.
Out of 3 servers I have tested this on this is the only 1 client with this issue.
Thank you,
Rainger
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Tue Sep 07, 2021 4:20 pm
by ssax
What version of the plugin are you running?
Code: Select all
/usr/local/nagios/libexec/check_vmware_api.pl -V
What version of vmware esxi is it running?
Which version of the vmware perl SDK did you install?
What does it show when it segfaults with -vvv --trace=4 on the command?
Code: Select all
/usr/local/nagios/libexec/check_vmware_api.pl -f creds.file -l cpu -w 80 -c 90 -H host -vvv --trace=4
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Tue Sep 07, 2021 4:34 pm
by Sargento
ssax wrote:What version of the plugin are you running?
Code: Select all
/usr/local/nagios/libexec/check_vmware_api.pl -V
What version of vmware esxi is it running?
Which version of the vmware perl SDK did you install?
What does it show when it segfaults with -vvv --trace=4 on the command?
Code: Select all
/usr/local/nagios/libexec/check_vmware_api.pl -f creds.file -l cpu -w 80 -c 90 -H host -vvv --trace=4
API = check_vmware_api.pl 0.7.1
VMWARE ESXI = 6.7
Perl SDK Version = VMware-vSphere-Perl-SDK-7.0.0-17698549.x86_64
Extra module needed after previous issue where no authentication worked = libwww-perl-5.837
When I ran with -vvv --trace=4 I only see
"Segmentation fault". There is no other data put to the console. I verified this on my end a few times.
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Tue Sep 07, 2021 5:08 pm
by ssax
Is this the plugin that has been modified to use the session file?
The only other methods to see where it's failing would be:
Code: Select all
cpan -i Devel::Trace
perl -d:Trace /usr/local/nagios/libexec/check_vmware_api.pl -f creds.file -l cpu -w 80 -c 90 -H host
Or:
Code: Select all
yum install strace
strace -f /usr/local/nagios/libexec/check_vmware_api.pl -f creds.file -l cpu -w 80 -c 90 -H host
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Thu Sep 09, 2021 7:20 am
by Sargento
This is using any authentication. When I use a session file it segfaults, when I use user and pass it segfaults, when I use a user/pass file it segfaults. I've attached the tracefile. What should I be looking for in this file? Again this appears to be the only server in which this issue arises, we can do it on other vmware hosts (probably different release versions) and not have this issue.
I've attached an error and working tracefile. The working one is what we see when the command runs without fault and the error is when we see a segfault. Both of these are the trace from the same command on the same exact server, it's just random when it segfaults.
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Fri Sep 10, 2021 4:09 pm
by ssax
Based on this in the failing one:
Code: Select all
read(5, 0x1ff9453, 5) = -1 ECONNRESET (Connection reset by peer)
Do you have any security software on the system OR in the network path that could be impacting it and flagging it as a threat
sometimes?
That would make a lot of sense and is usually the first thing I suspect when I see connection resets outside of network issues.
A you seeing that on all VMWare checks or just some? All VMWare hosts or just a specific subset?
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Tue Sep 14, 2021 9:24 am
by Sargento
No security software blocking it, just verified. It only seems to affect this one specific ESXi host. I haven't had a single seg fault on any other ESXi host yet, and believe me I tried.
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Tue Sep 14, 2021 5:32 pm
by ssax
Try doing this method:
Code: Select all
cpan -i Devel::Trace
perl -d:Trace /usr/local/nagios/libexec/check_vmware_api.pl -f creds.file -l cpu -w 80 -c 90 -H host
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Wed Sep 15, 2021 8:33 am
by Sargento
Hello,
I've done the perl trace and am reading through the output. I've attached the file below for you to help me through this.
I do really appreciate your help so far.
Re: Intermittent Segmentation Fault with check_vmware_api
Posted: Wed Sep 15, 2021 5:49 pm
by ssax
You received a segfault on that output, right?
What is the output of this command?