Nagios Support Forum

Posted: **Fri Oct 16, 2015 7:17 am**

I received this request from a member of our Development team and was wondering if Nagios XI could help out and save the day?

We have an internal web service that occasionally (i.e. very rarely) fails to return the expected result. Thus far we have not been able to reproduce the failure.

I was wondering if we could get a network monitoring program configured to watch this service, and maybe even watch for it to fail. The problem is, it could be months before we get another failure. If we could just capture a rolling 24 hours of traffic to this endpoint, that should let us see the next time it happens.

Back in the day, I used to deploy a dedicated ‘sniffer’ for this kind of thing, but I’m hoping we can just use some software these days.
Do you have any thoughts on doing this?

Test version available at http://<testurl>:8080/SuitabilityServices/FormService.svc

The service is called by multiple systems, but they are only interested in calls from AAEP (internal application) so they will want to filter further to isolate just those exchanges.

Once I have a few examples of the data flowing, they could identify the variance(s) they want to look for. There are multiple methods, but they only want to look at one of them, SubmitNewFormRequest.

Here is the format of the request.

Code: Select all

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:suit="http://ourdomain.net/2011/06/SuitabilityServices">
   <soapenv:Header/>
   <soapenv:Body>
      <suit:SubmitNewFormRequest>
         <!--Optional:-->
         <suit:Form>
            <suit:CompanyCode>?</suit:CompanyCode>
            <suit:ContractId>?</suit:ContractId>
            <!--Optional:-->
            <suit:OwnerName>?</suit:OwnerName>
            <!--Optional:-->
            <suit:OwnerDOB>?</suit:OwnerDOB>
            <!--Optional:-->
            <suit:JointOwner>?</suit:JointOwner>
            <!--Optional:-->
            <suit:JointOwnerDOB>?</suit:JointOwnerDOB>
            <!--Optional:-->
            <suit:ProductName>?</suit:ProductName>
            <!--Optional:-->
            <suit:PlanCode>?</suit:PlanCode>
            <!--Optional:-->
            <suit:ApproxInitialPremium>?</suit:ApproxInitialPremium>
            <!--Optional:-->
            <suit:SurrenderPenaltyPeriod>?</suit:SurrenderPenaltyPeriod>
            <suit:TaxTypeId>?</suit:TaxTypeId>
            <!--Optional:-->
            <suit:StatementUnderstandA>?</suit:StatementUnderstandA>
            <!--Optional:-->
            <suit:StatementUnderstandB>?</suit:StatementUnderstandB>
            <!--Optional:-->
            <suit:StatementUnderstandC>?</suit:StatementUnderstandC>
            <!--Optional:-->
            <suit:StatementUnderstandD>?</suit:StatementUnderstandD>
            <suit:AnnualIncomeRangeId>?</suit:AnnualIncomeRangeId>
            <suit:FinancialExperienceId>?</suit:FinancialExperienceId>
            <suit:RiskToleranceId>?</suit:RiskToleranceId>
            <suit:FederalTaxBracketId>?</suit:FederalTaxBracketId>
            <!--Optional:-->
            <suit:ObjectiveUseIds>
               <!--Zero or more repetitions:-->
               <suit:Id>?</suit:Id>
            </suit:ObjectiveUseIds>
            <!--Optional:-->
            <suit:FundingResourceIds>
               <!--Zero or more repetitions:-->
               <suit:Id>?</suit:Id>
            </suit:FundingResourceIds>
            <!--Optional:-->
            <suit:MakingReplacement>?</suit:MakingReplacement>
            <!--Optional:-->
            <suit:ReplacementReasonIds>
               <!--Zero or more repetitions:-->
               <suit:Id>?</suit:Id>
            </suit:ReplacementReasonIds>
            <!--Optional:-->
            <suit:UnderstandReplacement>?</suit:UnderstandReplacement>
            <!--Optional:-->
            <suit:Replacements>
               <!--Zero or more repetitions:-->
               <suit:Replacement>
                  <!--Optional:-->
                  <suit:OriginalCompanyName>?</suit:OriginalCompanyName>
                  <!--Optional:-->
                  <suit:OriginalProductName>?</suit:OriginalProductName>
                  <!--Optional:-->
                  <suit:FullSurrender>?</suit:FullSurrender>
                  <!--Optional:-->
                  <suit:SurrenderAmount>?</suit:SurrenderAmount>
                  <!--Optional:-->
                  <suit:TimeHeld>?</suit:TimeHeld>
                  <!--Optional:-->
                  <suit:ChargesOrFees>?</suit:ChargesOrFees>
                  <!--Optional:-->
                  <suit:DeathBenefitAmount>?</suit:DeathBenefitAmount>
               </suit:Replacement>
            </suit:Replacements>
            <!--Optional:-->
            <suit:OtherReplacementReason>?</suit:OtherReplacementReason>
            <!--Optional:-->
            <suit:ReplaceOtherContract>?</suit:ReplaceOtherContract>
            <!--Optional:-->
            <suit:ReplaceIncurredCharge>?</suit:ReplaceIncurredCharge>
            <!--Optional:-->
            <suit:ReplaceChargeReason>?</suit:ReplaceChargeReason>
            <!--Optional:-->
            <suit:FinancialEmergency>?</suit:FinancialEmergency>
            <!--Optional:-->
            <suit:FundAccessPlanIds>
               <!--Zero or more repetitions:-->
               <suit:Id>?</suit:Id>
            </suit:FundAccessPlanIds>
            <suit:FundAccessPeriodId>?</suit:FundAccessPeriodId>
            <!--Optional:-->
            <suit:HouseholdNetWorth>?</suit:HouseholdNetWorth>
            <!--Optional:-->
            <suit:HouseholdLiquidAssets>?</suit:HouseholdLiquidAssets>
            <!--Optional:-->
            <suit:LiquidNetWorth>?</suit:LiquidNetWorth>
            <!--Optional:-->
            <suit:LiquidNetWorthPct>?</suit:LiquidNetWorthPct>
            <!--Optional:-->
            <suit:RetirePlanAssets>?</suit:RetirePlanAssets>
            <!--Optional:-->
            <suit:OtherAssetsValue>?</suit:OtherAssetsValue>
            <!--Optional:-->
            <suit:FormReceived>?</suit:FormReceived>
            <!--Optional:-->
            <suit:ReviewedForSuitability>?</suit:ReviewedForSuitability>
            <!--Optional:-->
            <suit:OwnerCertifications>
               <!--Zero or more repetitions:-->
               <suit:OwnerCertification>
                  <!--Optional:-->
                  <suit:OwnerCertify>?</suit:OwnerCertify>
                  <!--Optional:-->
                  <suit:DateSigned>?</suit:DateSigned>
                  <!--Optional:-->
                  <suit:OwnerContactInformation>
                     <!--Optional:-->
                     <suit:Phone>?</suit:Phone>
                     <!--Optional:-->
                     <suit:Cellphone>?</suit:Cellphone>
                     <!--Optional:-->
                     <suit:EmailAddress>?</suit:EmailAddress>
                     <!--Optional:-->
                     <suit:Name>?</suit:Name>
                     <!--Optional:-->
                     <suit:Address1>?</suit:Address1>
                     <!--Optional:-->
                     <suit:Address2>?</suit:Address2>
                     <!--Optional:-->
                     <suit:Address3>?</suit:Address3>
                     <!--Optional:-->
                     <suit:Address4>?</suit:Address4>
                     <!--Optional:-->
                     <suit:City>?</suit:City>
                     <!--Optional:-->
                     <suit:State>?</suit:State>
                     <!--Optional:-->
                     <suit:Zip>?</suit:Zip>
                     <!--Optional:-->
                     <suit:BestContactTime>?</suit:BestContactTime>
                  </suit:OwnerContactInformation>
               </suit:OwnerCertification>
            </suit:OwnerCertifications>
            <!--Optional:-->
            <suit:AgentCertifications>
               <!--Zero or more repetitions:-->
               <suit:AgentCertification>
                  <!--Optional:-->
                  <suit:AgentCertify>?</suit:AgentCertify>
                  <!--Optional:-->
                  <suit:DateSigned>?</suit:DateSigned>
                  <!--Optional:-->
                  <suit:AgentContactInformation>
                     <!--Optional:-->
                     <suit:Phone>?</suit:Phone>
                     <!--Optional:-->
                     <suit:Cellphone>?</suit:Cellphone>
                     <!--Optional:-->
                     <suit:EmailAddress>?</suit:EmailAddress>
                     <!--Optional:-->
                     <suit:Name>?</suit:Name>
                     <!--Optional:-->
                     <suit:Address1>?</suit:Address1>
                     <!--Optional:-->
                     <suit:Address2>?</suit:Address2>
                     <!--Optional:-->
                     <suit:Address3>?</suit:Address3>
                     <!--Optional:-->
                     <suit:Address4>?</suit:Address4>
                     <!--Optional:-->
                     <suit:City>?</suit:City>
                     <!--Optional:-->
                     <suit:State>?</suit:State>
                     <!--Optional:-->
                     <suit:Zip>?</suit:Zip>
                     <!--Optional:-->
                     <suit:BestContactTime>?</suit:BestContactTime>
                  </suit:AgentContactInformation>
               </suit:AgentCertification>
            </suit:AgentCertifications>
            <suit:AppData>
               <!--Optional:-->
               <suit:CompanyCode>?</suit:CompanyCode>
               <!--Optional:-->
               <suit:ContractId>?</suit:ContractId>
               <!--Optional:-->
               <suit:AgentName>?</suit:AgentName>
               <!--Optional:-->
               <suit:AgentId>?</suit:AgentId>
               <!--Optional:-->
               <suit:FirmName>?</suit:FirmName>
               <!--Optional:-->
               <suit:FirmId>?</suit:FirmId>
               <!--Optional:-->
               <suit:ProductName>?</suit:ProductName>
               <!--Optional:-->
               <suit:ProductId>?</suit:ProductId>
               <!--Optional:-->
               <suit:OwnerSSN>?</suit:OwnerSSN>
               <!--Optional:-->
               <suit:OwnerDOB>?</suit:OwnerDOB>
               <!--Optional:-->
               <suit:JointOwnerSSN>?</suit:JointOwnerSSN>
               <!--Optional:-->
               <suit:JointOwnerDOB>?</suit:JointOwnerDOB>
               <!--Optional:-->
               <suit:AppSignDate>?</suit:AppSignDate>
               <!--Optional:-->
               <suit:AppSignState>?</suit:AppSignState>
               <!--Optional:-->
               <suit:AppPremiumAmount>?</suit:AppPremiumAmount>
               <!--Optional:-->
               <suit:TaxTypeId>?</suit:TaxTypeId>
            </suit:AppData>
            <!--Optional:-->
            <suit:StatementUnderstand36Month>?</suit:StatementUnderstand36Month>
            <!--Optional:-->
            <suit:StatementSurrenderCharge>?</suit:StatementSurrenderCharge>
            <!--Optional:-->
            <suit:StatementBenefitLoss>?</suit:StatementBenefitLoss>
            <!--Optional:-->
            <suit:ReverseMortgage>?</suit:ReverseMortgage>
         </suit:Form>
      </suit:SubmitNewFormRequest>
   </soapenv:Body>
</soapenv:Envelope>

I'm not a Developer so if there is additional information needed let me know. Just researching if Nagios can monitor this for them or not.

Posted: **Fri Oct 16, 2015 8:15 am**

If you're really interested in capturing any potential failure, you've no option but to capture every transaction. Whether you call that "sniffing" or not is a matter of semantics. The standard one-minute polling model used by Nagios cannot capture transactional data typically because it only runs once a minute.

What you might consider, if possible, is writing out all relevant information about every transaction to a local file and then point one of Nagios' log parsing utilities at that file.

One other thing you can do is craft "non-invasive" synthetic transactions to be run by Nagios using the check_http plugin (I've not used this, but the docs suggest it would work) or other similar plugins. Your synthetic transaction would be something that doesn't make changes to your system, but that does "exercise" the web service interface (e.g. look up some data). If possible, you could even generate real transactions and then cancel them out.

Ultimately, you have to ask yourself "how" you know you have a failure and then examine ways to instrument it. Your original solution was probably as good as you're going to get. Capturing data like this always comes at the cost of additional load on the monitored host, there's just no way around that.

If you really want to get fancy, check out the pcap libraries and utilities. These can be used to capture packets that are prefiltered on any number of criteria. There's even an rcapd remote capture daemon that can be used by an external client to capture packets from a remote system. Pcap is used by packet analyzers like WireShark. There might even be a Nagios plugin that will utilize the library.

Just had another thought... if you did use the pcap library, or more likely the tcpdump command line utility, and configured it to write properly decoded packets to a local file, then you could use Nagios' log parsing tools to then read that log for a specific pattern. Tcpdump can be used on either "end" of the transaction (client or web service host) or both end simultaneously. It will filter packets by host, TCP port, and many other criteria.

You can find more information at: http://www.tcpdump.org/

Posted: **Fri Oct 16, 2015 2:05 pm**

I have to agree with @mp4783 on this one. Nagios can do a lot of things, but there are better tools for what you are looking to do. Between tcpdump and a special iptables rule that logs all traffic from a certain machine, you should be able to get whatever you want. At that point you can feed the data into Nagios Logserver if you really want to get fancy :)

Posted: **Fri Oct 16, 2015 2:22 pm**

For normal Nagios http tests there is the check_http plugin.

In a quick google search I found something that might do what you want.
http://stackoverflow.com/questions/2337 ... e-response
I hope we are allowed to reference external stuff, if not, I apologize and feel free to delete this post.

Maybe use an external file to save your configuration so the test definition doesn't get too long.
--extra-opts=[section][@file]
Read options from an ini file. See
https://www.nagios-plugins.org/doc/extra-opts.html
for usage and examples.

I hope that provides a viable option for you.

Steve B

Posted: **Fri Oct 16, 2015 2:26 pm**

Thank you everyone for your input. It's valuable and we appreciate it! Let us know what you think @jkinning.

Posted: **Fri Oct 16, 2015 2:29 pm**

Thanks for the feedback! Greatly appreciate it.

Posted: **Fri Oct 16, 2015 2:35 pm**

Let us know if you have any further questions.

Nagios Support Forum

Monitor web transaction

Monitor web transaction

Re: Monitor web transaction

Re: Monitor web transaction

Re: Monitor web transaction

Re: Monitor web transaction

Re: Monitor web transaction

Re: Monitor web transaction