Incidents not creating after OS upgrade

[bsivavani]

Hi,

Recently we have upgraded RHEL OS from 6.7 to 6.9, after upgrade we are seeing servicenow incidents not creating. We have observed below error messages in log file.

Apr 17 14:39:55 LXXXXXX nagios: wproc: early_timeout=0; exited_ok=1; wait_status=512; error_code=0;
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 01: Useless use of a variable in void context at /drbd1/nagios/libexec/Eventhandler/sn_ticketer_service_alert.pl line 109.
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 02: Use of uninitialized value in sprintf at /usr/local/share/perl5/SOAP/Lite.pm line 3642.
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 03: SOAP::Transport::HTTP::Client::send_receive: POST https://srv_SOAPimporterNagios:5%7B][email protected]/u_incident_interface.do?SOAP HTTP/1.1
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 04: Accept: text/xml
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 05: Accept: multipart/*
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 06: Accept: application/soap
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 07: Content-Length: 357
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 08: Content-Type: text/xml; charset=utf-8
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 09: SOAPAction: "#"
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 10:
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 11: <?xml version="1.0" encoding="UTF-8"?><soap:Envelope soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap:Body /></soap:Envelope>
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 12: SOAP::Transport::HTTP::Client::send_receive: 500 Can't connect to rwestproxy-neurathdc.rwe.com:8080 (SSL connect attempt failed with unknown errorerror:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol)
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 13: Content-Type: text/plain
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 14: Client-Date: Tue, 17 Apr 2018 12:39:55 GMT
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 15: Client-Warning: Internal response
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 16:
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 17: 500 Can't connect to rwestproxy-neurathdc.rwe.com:8080 (SSL connect attempt failed with unknown errorerror:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol)
Apr 17 14:39:55 LXXXXXX nagios: wproc: stderr line 18: 500 Can't connect to rwestproxy-neurathdc.rwe.com:8080 (SSL connect attempt failed with unknown errorerror:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) at /drbd1/nagios/libexec/Eventhandler/sn_ticketer_service_alert.pl line 475.
Apr 17 15:03:39 LXXXXXX nagios: wproc: stderr line 11: SOAP::Transport::HTTP::Client::send_receive: 500 Can't connect to rwestproxy-neurathdc.rwe.com:8080 (SSL connect attempt failed with unknown errorerror:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol)
Apr 17 15:03:39 LXXXXXX nagios: wproc: stderr line 16: 500 Can't connect to rwestproxy-neurathdc.rwe.com:8080 (SSL connect attempt failed with unknown errorerror:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol)
Apr 17 15:03:39 LXXXXXX nagios: wproc: stderr line 17: 500 Can't connect to rwestproxy-neurathdc.rwe.com:8080 (SSL connect attempt failed with unknown errorerror:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) at /drbd1/nagios/libexec/Eventhandler/sn_ticketer_service_alert.pl line 475.


Kindly let us know if you can help with these error messages.

[cdienger]

It looks like sn_ticketer_service_alert.pl or the system is configured to go through the rwestproxy-neurathdc.rwe.com:8080 and failing. Is this necessary? You can check the system proxy settings following https://support.nagios.com/kb/article/n ... s-147.html. Can rwestproxy-neurathdc.rwe.com be configured to allow these requests?

[bsivavani]

Yes. Proxy configured on the server.

[cdienger]

Can you PM a copy of the sn_ticketer_service_alert.pl script as well as gather a tcpdump catpuring the error? The tcpdump can be run with:

yum -y install tcpdump
tcpdump -s 0 -i any port 8080 -w output.pcap

Let it run while reproducing the error and use CTRL+C to stop it. Gather and provide output.pcap.

[bsivavani]

Please check in PM.

[cdienger]

The script(https://roshamboot.org/retired-project/ ... rvice-now/) doesn't appear to work with proxies with the updates made to the system. This could be due to updates to perl, openssl, or other system packages. Are you getting any meaningful messages if you run the script directly on the command line? If you have another 6.6 machine you can install the script on and run it, I would try that and then try updating just specific packages to see where it eventually breaks to get an idea of where to look. The script mentions that is very simple proxy connection:

Code: Select all

 # Connect to service-now, the proxy connection is very simple and only attempts basic auth.
# Note: Should probably find a way to test this handle... but it's proving difficult... 

I've seen many cases where scripts or programs don't make a rfc compliant proxy request which may be the case here - the tcpdump would help confirm if you can provide that.

[bsivavani]

Hi,

We have recreated the issue in Test server where OS upgraded to 6.9 and able to produce the issue.

Sent tcpdump file. Please check in PM.

[cdienger]

The tcpdump shows the proxy is rejecting the connection with a "400 bad request" message. This usually means the request isn't properly formatted for going through a proxy. The tcpdump that was provided isn't able to be opened in wireshark though so I'm not able to see the actual request.

[bsivavani]

Please check tcpdump output in PM.

If it is not useful, Please let me know your available time, so that I will show you output using Wireshark.

[cdienger]

Please open a ticket at http://support.nagios.com/tickets/, reference this thread, and I will take ownership and provide some next steps.