Nagios process at 100% CPU, system is crawling

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Nagios process at 100% CPU, system is crawling

Post by scottwilkerson »

consulvation wrote:I had top running when I clicked on the Nagios Service Problems link on the left. It took 1m 15s to load the page.
Can you do this again and after a few seconds of waiting take a screen capture of the top window?
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
consulvation
Posts: 16
Joined: Thu Jul 19, 2018 10:07 pm

Re: Nagios process at 100% CPU, system is crawling

Post by consulvation »

Here you go...
Top
Top
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Nagios process at 100% CPU, system is crawling

Post by scottwilkerson »

Looks like the load has come down considerably, and the nagios process is not in the list of top processes.

The one item listed is a call the the statusjson.cgi which can be intensive if the call is for historical data
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
consulvation
Posts: 16
Joined: Thu Jul 19, 2018 10:07 pm

Re: Nagios process at 100% CPU, system is crawling

Post by consulvation »

Well, yes, the nagios process doesn't appear at all in the top anymore. I suppose making it a deamon helped that. But every call to a cgi starts by taking 100% CPU. It still takes over 1 minute to load any page on the site. It's still unusable. We also stopped getting email notifications. I guess I am wondering what else is wrong because if I stop the nagios process, there is a clear increase in performance even from the command line. Do you have any ideas? This did not happen in v3 by the way, this is strictly new to v4. Thanks for all your time on this, I do appreciate it, but this doesn't seem to be solved.
consulvation
Posts: 16
Joined: Thu Jul 19, 2018 10:07 pm

Re: Nagios process at 100% CPU, system is crawling

Post by consulvation »

Also, I should add that it looks like Nagios is stuck in the past. Looking at the status, the last checks were from 4 days ago. Looking at the scheduling queue, it's from 4 days ago.
2018-07-31_16-22-23.png
2018-07-31_16-22-23.png (15.1 KiB) Viewed 3064 times
2018-07-31_16-22-23.png
2018-07-31_16-22-23.png (15.1 KiB) Viewed 3064 times
Attachments
2018-07-31_16-25-27.png
consulvation
Posts: 16
Joined: Thu Jul 19, 2018 10:07 pm

Re: Nagios process at 100% CPU, system is crawling

Post by consulvation »

Further looking into this, I decided to roll back the changes and move it out of Daemon mode for a second and then I renamed the retention.dat and status.dat files to .old and restarted the nagios process. Those files had gotten to be 2.6GB each. In doing so, the web site is now flying at lightening speeds. I presume this will not last and eventually slow down as the files accumulate. Should they be getting that large?
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Nagios process at 100% CPU, system is crawling

Post by scottwilkerson »

If this is still running the maint version you may have fixed the problem

Their was a bug in 4.4.1 that duplicated comments etc, which could have been causing your issue
consulvation wrote:Also, I should add that it looks like Nagios is stuck in the past. Looking at the status, the last checks were from 4 days ago. Looking at the scheduling queue, it's from 4 days ago.
Is this resolved as well?
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
consulvation
Posts: 16
Joined: Thu Jul 19, 2018 10:07 pm

Re: Nagios process at 100% CPU, system is crawling

Post by consulvation »

It seems to be fine so far. The status.dat and retention.dat files seem to be holding in the 800-900K realm, I will continue to see if it increases significantly. I am still running the maint version, just not as a deamon.

Yes, once I renamed the super large status and retention.dat files, it became current and all host checks are being done, notifications are working again as well.

I presume I should be able to install the next stable release without any problems once it becomes available. Thanks again for your help.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Nagios process at 100% CPU, system is crawling

Post by scottwilkerson »

Great Feel free to open a new issue if this changes.
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Nagios process at 100% CPU, system is crawling

Post by scottwilkerson »

Unlocking per user request
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
Locked