Page 1 of 2
CCM responds with 500 error code on any CCM page
Posted: Mon Jan 04, 2021 12:24 pm
by atremblay
Hey guys,
So my problem is that whenever I load a CCM page I get an HTTP 500 error. It says "... is currently unable to handle the request". This happened after I deleted a Host (not using CCM *tisk on me*) using just the Host page > goto configure tab > click 'Delete Host'. Never done it this way before, but then it said that the job for deleting the host would be queued, and it would be removed soon. Well the host in question is still there so it never finished the job), and now I can't browse any of the CCM pages. They all return the error above.
As I figured this would be a process that is hung up I've basically only done three things so far:
1. Parse some of the nagios logs I know of (like /usr/local/nagios/var/nagios.log ....). [turned up nothing]
2. Restarted some of the core processes, and the server itself eventually.
3. Searched the support boards for someone with a similar issue (I haven't found one exactly like it yet).
Do you guys know what's going on here? Like is there some bug known in the Host Deletion that can hang up CCM like that. And if so, how do I jolt it back to life?
Thanks. And happy new years!
A.
Re: CCM responds with 500 error code on any CCM page
Posted: Mon Jan 04, 2021 2:04 pm
by atremblay
I found a bit more info. I was able to cheat my way onto the "Apply Configuration" page. And when pushing the config it says:
Error: Invalid max_attempts, check_interval, retry_interval, or notification_interval value for service 'Server Drive C: Disk Usage TEST' on host 'OSCRDC01'
Error: Could not register service (config file '/usr/local/nagios/etc/services/COMMON-SERVER TEST.cfg', starting on line 16)
Thing is that I recognize all this EXCEPT the file. That filename COMMON-SERVER TEST.cfg does not exist. Now COMMON-SERVER.cfg does exist, AND it does include the service being mentioned above. Thing is that it's not on line 16, and editing it does nothing. It also already has a valid "use" command with a template which does exist and does have the max_attempts, check_interval, retry_interval, and notification_interval values.
Re: CCM responds with 500 error code on any CCM page
Posted: Tue Jan 05, 2021 10:44 am
by tgriep
Let's see if there are any errors logged to the Apache when trying to access the CCM.
Run this in a root shell.
Then try and access the CCM in the web interface and post the output of the tail command here so we can check it for errors.
After that, run this in the shell which is the command line version of the Apply Config and post the output here.
Code: Select all
su - nagios
cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh
Thanks.
Re: CCM responds with 500 error code on any CCM page
Posted: Tue Jan 05, 2021 11:21 am
by atremblay
Here you go. Thanks. How about that nagiosql failed login?
Code: Select all
==> ssl_access_log <==
10.4.4.231 - - [05/Jan/2021:11:04:09 -0500] "GET /nagiosxi/includes/components/ccm/xi-index.php HTTP/1.1" 200 40281
==> ssl_request_log <==
[05/Jan/2021:11:04:09 -0500] 10.4.4.231 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/includes/components/ccm/xi-index.php HTTP/1.1" 40281
==> ssl_access_log <==
10.4.4.233 - - [05/Jan/2021:11:04:10 -0500] "GET /nagiosxi/includes/components/opscreen/merlin.php HTTP/1.1" 200 27
==> ssl_request_log <==
[05/Jan/2021:11:04:10 -0500] 10.4.4.233 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/includes/components/opscreen/merlin.php HTTP/1.1" 27
==> ssl_error_log <==
[Tue Jan 05 11:04:10.850436 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Notice: Undefined variable: _oA17 in /usr/local/nagiosxi/etc/components/ccm_config.inc.php on line 16, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
[Tue Jan 05 11:04:10.850470 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Notice: Undefined variable: C in /usr/local/nagiosxi/etc/components/ccm_config.inc.php on line 16, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
[Tue Jan 05 11:04:10.865541 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Warning: mysqli::mysqli(): (28000/1045): Access denied for user 'nagiosql'@'localhost' (using password: YES) in /usr/local/nagiosxi/html/includes/components/ccm/classes/db.class.php on line 80, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
[Tue Jan 05 11:04:10.865575 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Warning: mysqli::set_charset(): Couldn't fetch mysqli in /usr/local/nagiosxi/html/includes/components/ccm/classes/db.class.php on line 81, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
[Tue Jan 05 11:04:10.865582 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Warning: mysqli::query(): Couldn't fetch mysqli in /usr/local/nagiosxi/html/includes/components/ccm/classes/db.class.php on line 82, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
[Tue Jan 05 11:04:10.865592 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Warning: mysqli::query(): Couldn't fetch mysqli in /usr/local/nagiosxi/html/includes/components/ccm/classes/db.class.php on line 99, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
[Tue Jan 05 11:04:10.865602 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Warning: db::query(): Property access is not allowed yet in /usr/local/nagiosxi/html/includes/components/ccm/classes/db.class.php on line 101, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
[Tue Jan 05 11:04:10.865607 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Notice: Trying to get property of non-object in /usr/local/nagiosxi/html/includes/components/ccm/classes/db.class.php on line 112, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
[Tue Jan 05 11:04:10.865619 2021] [:error] [pid 8437] [client 10.4.4.231:62378] PHP Fatal error: Call to a member function free() on a non-object in /usr/local/nagiosxi/html/includes/components/ccm/classes/db.class.php on line 117, referer: https://nagios.h2opower.com/nagiosxi/includes/components/ccm/xi-index.php
==> ssl_access_log <==
10.4.4.231 - - [05/Jan/2021:11:04:10 -0500] "GET /nagiosxi/includes/components/ccm/index.php?menu=invisible HTTP/1.1" 500 -
==> ssl_request_log <==
[05/Jan/2021:11:04:10 -0500] 10.4.4.231 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/includes/components/ccm/index.php?menu=invisible HTTP/1.1" -
==> ssl_access_log <==
10.4.4.231 - - [05/Jan/2021:11:04:10 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_pagetop_alert_content_html%22%2C%22args%22%3A%22%22%7D&nsp=ca2565f84dd2b5656cfaf25dbac47a2cd357da6d5ecd2235f015f455d68c3eba HTTP/1.1" 200 1428
==> ssl_request_log <==
[05/Jan/2021:11:04:10 -0500] 10.4.4.231 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_pagetop_alert_content_html%22%2C%22args%22%3A%22%22%7D&nsp=ca2565f84dd2b5656cfaf25dbac47a2cd357da6d5ecd2235f015f455d68c3eba HTTP/1.1" 1428
==> ssl_access_log <==
10.4.4.231 - - [05/Jan/2021:11:04:10 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_tray_alert_html%22%2C%22args%22%3A%22%22%7D&nsp=ca2565f84dd2b5656cfaf25dbac47a2cd357da6d5ecd2235f015f455d68c3eba HTTP/1.1" 200 830
==> ssl_request_log <==
[05/Jan/2021:11:04:10 -0500] 10.4.4.231 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_tray_alert_html%22%2C%22args%22%3A%22%22%7D&nsp=ca2565f84dd2b5656cfaf25dbac47a2cd357da6d5ecd2235f015f455d68c3eba HTTP/1.1" 830
So this might lead in the right direction.
The reconfigure I couldn't perform as I don't have Nagios user password. oddly enough it's not in the "Change Nagios passwords" how to document posted in the support. So I don't think I've ever changed it (new install). Otherwise I don't suppose I can run that as root?
Re: CCM responds with 500 error code on any CCM page
Posted: Tue Jan 05, 2021 3:06 pm
by atremblay
Thank you tgriep. The logs did point out that the nagiosql password was not working. I did change all the passwords following this document:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
The document does work for the other accounts, but doesn't seem to work for changing the nagiosql password. I even just tried running it again to see if I messed up somewhere. So then I ran through the same document changing the password back to what it was before I changed it and now I got CCM back. But it's using an old password which isn't as complex as our new password policy requires.
I'd still like to change the nagiosql password, so if you know which additional steps are needed to change that account, I'd be interested in hearing it.
Re: CCM responds with 500 error code on any CCM page
Posted: Tue Jan 05, 2021 4:27 pm
by tgriep
The nagios user account should not have a password.
If the nagios user account's password is setup to expire, it cannot be set that way. That account cannot have it's password expire because the system uses that account to run processes automatically and when it expires, it cannot do so. Also, the nagios account cannot have a password as that will cause issues.
Login as root and run the following to fix the issue.
Code: Select all
passwd --delete nagios
chage -I -1 -m 0 -M 99999 -E -1 nagios
Change the MYSQL nagiosql password again and do the following.
Go to the Admin > Security Credentials menu in the XI GUI if it is available.
Don't change anything except to click on the Update Credentials button to generate a new ticket.
Either run the reconfigure_nagios.sh script from the previous post or see if you can go to the CCM.
Let me know is this fixes the issue.
If not, what version of Nagios XI is the server running?
Please PM me the following files if they exist on the server so I can check the settings.
Code: Select all
/usr/local/nagiosxi/html/config.inc.php
/var/www/html/nagiosql/config/settings.php
Thanks.
Re: CCM responds with 500 error code on any CCM page
Posted: Wed Jan 06, 2021 11:26 am
by atremblay
Ok so interestingly enough here's what I got.
The nagios account hasn't been changed. This was a brand new system built just a few weeks ago with a fresh NagiosXI install, and then restored from backup from the old system. But when I try to su - nagios with no password, it just gives me invalid login.
The flip side, I tried updating the nagiosql password again [in the mysql db, then in the config.inc.php file], then tried running the Update Credentials like you mentioned. No success. The result was the same that 500 error on CCM pages, and invalid SQL login attempts. So I've reverted back to the old password again. It's working again.
It really does seem like there's a step missing from the Change Default Passwords document.
I'll PM you the config.inc file, but the other folder you mentioned doesn't exist. I'm running 5.5.9. Plan was to upgrade to the latest once I got all this figured out. Perhaps the settings.php file you're describing is only on newer versions.
Re: CCM responds with 500 error code on any CCM page
Posted: Wed Jan 06, 2021 12:32 pm
by tgriep
This file on the server /var/www/html/nagiosql/config/settings.php is used in older versions of XI than what you are running so that is OK that it does not exist.
The invalid login when trying to login as the nagios user should not happen.
Login as root to the server, run the following commands and post the output here.
Code: Select all
chage -l nagios
grep nag /etc/passwd
grep nag /etc/group
It might be the password you are using for the nagiosql MYSQL login. Try using the same password you are using for the other databases.
I went through the document and changed the password and it worked. I did use a simple password for testing so it maybe one of the characters that you used for your password.
Change the password in the MYSQL database and the /usr/local/nagiosxi/html/config.inc.php file and restart Apache.
Logout of XI and re-login in and see if the password is updated in the following file.
Code: Select all
/usr/local/nagiosxi/etc/components/ccm_config.inc.php
If not, then there may be a permission problem that is caused by the issue with the nagios user account.
Re: CCM responds with 500 error code on any CCM page
Posted: Wed Jan 06, 2021 1:14 pm
by tgriep
I just saw that you have a ticket open for this issue. To minimize duplication of efforts, shall we continue on in the ticket?
Re: CCM responds with 500 error code on any CCM page
Posted: Wed Jan 06, 2021 1:23 pm
by atremblay
Yeah sure, this seems like a more unique situation and not necessarily something that will benefit the community or board.