Java memory check

[jkinning]

We recently had a Business Unit that had their application fail due Java not having enough free memory

Code: Select all

WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/12036987
 Exception: System.ServiceModel.ServiceActivationException: The service '/AnnuityBusinessRulesService.svc' cannot be activated due to an exception during compilation.  The exception message is: Memory gates checking failed because the free memory (69607424 bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.. ---> System.InsufficientMemoryException: Memory gates checking failed because the free memory (69607424 bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.
   at System.ServiceModel.Activation.ServiceMemoryGates.Check(Int32 minFreeMemoryPercentage, Boolean throwOnLowMemory, UInt64& availableMemoryBytes)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CheckMemoryCloseIdleServices(EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   --- End of inner exception stack trace ---
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath, EventTraceActivity eventTraceActivity)
 Process Name: w3wp
 Process ID: 4248

I see there are a lot of plugins for Java but I am posting here to see if anyone already is doing so and can provide some advice on setting up such a monitoring check. All our servers are running NRPE or NSClient++. It would have been nice to get notified on this before the application failed.

[ssax]

The error that you are seeing should be mitigated by checking the memory usage of the windows system, are you currently doing that? If so, what did it say during that time?

[jkinning]

Nagios shows nothing for the entire month. I have my check as warning at 90% and critical at 99%.

[ssax]

Are you saying that the memory check is not showing any data/perfdata for the entire month or that it's never showed a problem and the data is there?


Thank you

[jkinning]

Data is there just never showed any issues or that any notifications were dispatched out.

[ssax]

If you look at this post you should be able to get the working set memory and determine that there is a problem if it hits a threshold but I'm not seeing a plugin currently written that would do this.

http://stackoverflow.com/a/21858581

Maybe with counters:

https://blogs.technet.microsoft.com/sam ... n-windows/

Code: Select all

check_nt!COUNTER! -d SHOWALL -l "\\Process(YOURPROCESS)\\Working Set","Memory Working Set for YOURPROCESS is %.f.0","Working Set"

OR

Code: Select all

check_nt!COUNTER! -d SHOWALL -l "\\Process(*)\\Working Set","Memory Working Set is %.f.0","Working Set"

Can you talk with your developers and have them check what is set for this:

Code: Select all

minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element

From the research I've done that is based on total memory of the system so it should have showed the memory usage as high during that timeperiod unless it occurred between when your memory usage checks in XI occurred.