On NLS client rsyslog/python/httpd processing stopped

[cmerchant]

We'll keep this thread open. Let us know what you find out in your development environment. Thanks.

[GhostRider2110]

Just a quick update. We are seeing this error on other systems that utilize python but not to the extent that it locks those systems up. I'm working with out developers to find out why we are only getting a partial stack trace in the logs when this happens and not getting the complete info on calling program/module. I did install on our DEV/TST/QA environments yet par for the course, I have yet to get a complete lockup as happened on the two other systems. Will up date again as I get more info.

See-ya
Mitch

[jolson]

Good to hear from you Mitch - thank you for the update. Let us know - we look forward to your results.

Best,


Jesse

[GhostRider2110]

I'm still waiting on some input from our core development team, but I have seen the issue on 2 of the other environments I installed all the logging on. Log errors are not consistent but the base indicators and cure are the same. Slow login and file access on the client system. Excessive memory uses by the python processes. Restart of rsyslog service clears the problem. As I gather more I will add it. --Mitch

[jolson]

We appreciate your continued input, thank you again.

[GhostRider2110]

Still working with developers to try to get more info on the python/wsgi process. Not sure how much info I can get right now as we are in the peak of processing and sinking time into troubleshooting, since we have a workaround (shut off sending logs to NLS) keeps the systems up. But end of April should be able to get some resources on this issue. Just wanted to update and let you know it's not a dead issue, just idle right now. --Mitch

[tmcdonald]

Always appreciate the updates!

[jolson]

I currently have a thread open about a similar problem - one thing that may be worth looking into is rsyslog queues. I am not an rsyslog guy, but it seems that the queues could be a likely culprit: http://www.rsyslog.com/doc/queues.html.

Again, thank you for the details.

[GhostRider2110]

Thanks, I'll read through that documentation. My gut tells me it is related to this and between rsyslog/httpd/wsgi something is filling up and clogging the works. Since all I have to do to clear the problem is restart rsyslog. I just have to figure out how to get the info to tune it properly. Or tune how python/wsgi is using rsyslog.

See-ya
Mitch

[jolson]

Sounds good Mitch - let us know. Thanks!