Backend API not working

[rajasegar]

One of our backend API suddenly stopped responding on one of our instance.
Can someone please advice how to troubleshoot this?

Code: Select all

[nagios@nagiosprodxi1 scripts]$ curl -v --noproxy "*" "http://10.17.19.234/nagiosxi/backend/?cmd=getstatehistory&&&search=&reportperiod=last24hours&startdate=&enddate=&hostgroup=&servicegroup=&statetype=both&mode=csv&username=nagiosadmin&ticket=comvge5e"
* About to connect() to 10.17.19.234 port 80 (#0)
*   Trying 10.17.19.234... connected
* Connected to 10.17.19.234 (10.17.19.234) port 80 (#0)
> GET /nagiosxi/backend/?cmd=getstatehistory&&&search=&reportperiod=last24hours&startdate=&enddate=&hostgroup=&servicegroup=&statetype=both&mode=csv&username=nagiosadmin&ticket=comvge5e HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.17.19.234
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 500 Internal Server Error
< Date: Fri, 04 Dec 2015 01:11:58 GMT
< Server: Apache/2.2.15 (Red Hat)
< X-Powered-By: PHP/5.3.3
< Set-Cookie: nagiosxi=midnfci9jqs67llbvf7ln4be62; expires=Fri, 04-Dec-2015 01:41:59 GMT; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Access-Control-Allow-Origin: *
< Content-Length: 39
< Connection: close
< Content-Type: text/xml
<
<?xml version="1.0" encoding="utf-8"?>

[Box293]

I anything logged in httpd/error_log ?

Code: Select all

tail -n 20 /var/log/httpd/error_log

[rajasegar]

I anything logged in httpd/error_log ?

Code: Select all

tail -n 20 /var/log/httpd/error_log

Code: Select all

[Fri Dec 04 06:30:31 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 60428289 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 08:55:18 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59553793 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 08:57:29 2015] [error] [client 10.17.19.234] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59561985 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:09:00 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59555841 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:09:45 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59547649 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:12:22 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59555841 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24

[rajasegar]

Please escalate to your developers to fix this issue below, the whole log file is full of this

Code: Select all

[Fri Dec 04 10:34:55 2015] [error] [client 10.17.44.201] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485, referer: http://10.17.19.234/nagiosxi/help/?xiwindow=api.php
[Fri Dec 04 10:34:56 2015] [error] [client ::1] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485
[Fri Dec 04 10:34:56 2015] [error] [client ::1] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485
[Fri Dec 04 10:35:01 2015] [error] [client ::1] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485
[Fri Dec 04 10:35:22 2015] [error] [client ::1] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485
[Fri Dec 04 10:35:26 2015] [error] [client 10.17.44.201] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485, referer: http://10.17.19.234/nagiosxi/help/?xiwindow=api.php
[Fri Dec 04 10:35:26 2015] [error] [client 10.17.44.201] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485, referer: http://10.17.19.234/nagiosxi/help/?xiwindow=api.php
[Fri Dec 04 10:35:26 2015] [error] [client ::1] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485
[Fri Dec 04 10:35:26 2015] [error] [client ::1] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485

[Box293]

I anything logged in httpd/error_log ?

Code: Select all

tail -n 20 /var/log/httpd/error_log

Code: Select all

[Fri Dec 04 06:30:31 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 60428289 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 08:55:18 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59553793 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 08:57:29 2015] [error] [client 10.17.19.234] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59561985 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:09:00 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59555841 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:09:45 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59547649 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:12:22 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59555841 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24

Can you try editing your /etc/php.ini file and update the memory_limit = 128M value (perhaps double it).
Then service httpd restart

Does this fix the API issue?

[Box293]

Please escalate to your developers to fix this issue below, the whole log file is full of this

Code: Select all

[Fri Dec 04 10:34:55 2015] [error] [client 10.17.44.201] PHP Notice:  Undefined variable: f2 in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 485, referer: http://10.17.19.234/nagiosxi/help/?xiwindow=api.php
xxxxx

Can I get you to open a new thread to address this issue please.
Can you report in there the XI version and the SNMP Trap Sender component version (Admin > System Extensions > Manage Components).

[rajasegar]

I anything logged in httpd/error_log ?

Code: Select all

tail -n 20 /var/log/httpd/error_log

Code: Select all

[Fri Dec 04 06:30:31 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 60428289 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 08:55:18 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59553793 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 08:57:29 2015] [error] [client 10.17.19.234] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59561985 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:09:00 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59555841 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:09:45 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59547649 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24
[Fri Dec 04 09:12:22 2015] [error] [client 10.17.19.235] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 59555841 bytes) in /usr/local/nagiosxi/html/backend/includes/handler-reports.inc.php on line 24

Can you try editing your /etc/php.ini file and update the memory_limit = 128M value (perhaps double it).
Then service httpd restart

Does this fix the API issue?

I truncated the logentries and the statehistory tables and it is back to normal.
Anyway will put in your recommendations also.
Just checked and one of my servers is already at 512M so I made it the same for all.

Will create a new thread for the error messages.

[Box293]

Do I need to restart any service after that change?

Just restarting httpd should be enough.

[rajasegar]

Please close this case. Problem resolved

[rkennedy]

Sounds good, glad to see this resolved! I will now close this thread out, but feel free to open a new one if you ever need assistance in the future.