Unable to Poll/Check live data timeout

[blariv]

hi,

I just built a new fusion system on RHEL7.7, everything installed correctly I then added 2 XI servers. getting this error every time the XI servers are getting polled. I increased the polling settings as recommended in the forums. see attached screenshot for my settings and stats.

Your Nagios Fusion installation is up to date.
Latest Available Version: 4.1.8
Installed Version: 4.1.8
Last Update Check: 2020-04-10 09:33:20


logging enabled:

[2020-04-10 11:53:01] [SYSTEM] [ERROR]: poll_server() unable to poll data for s:NagiosXI NA, u:nagiosadmin, poll:alerts
[2020-04-10 11:53:01] [SYSTEM] [ERROR]: poll_server() CHECK YOUR LIVE_DATA_TIMEOUT SETTINGS. IT MAY NEED INCREASED
[2020-04-10 11:55:09] [SYSTEM] [ERROR]: poll_server() unable to poll data for s:NagiosXI EU, u:nagiosadmin, poll:nagiosxi_bpi
[2020-04-10 11:55:09] [SYSTEM] [ERROR]: poll_server() CHECK YOUR LIVE_DATA_TIMEOUT SETTINGS. IT MAY NEED INCREASED

[tgriep]

What version of XI are the 2 servers running?
Are they configured to use SSL for the GUI?

When you setup the servers in Fusion, did you test to see if the fusion server can authenticate to the XI servers?

The following example is a command you can run as root on the Fusion server to see if it can connect to the remote Nagios server using the fusekey.
Run the following replacing the nagiosxiserver with the XI server's host name and the xxxxxxxxxxxxxxxxxxxxxxxx with the fusion key.

Code: Select all

curl -XGET https://nagiosxiserver/nagiosxi/api/v1/system/status?fusekey=xxxxxxxxxxxxxxxxxxxxxxxx -k -v

Run this command to see how long it takes to transfer 100 records.

Code: Select all

time curl -XGET https://nagiosxiserver/nagiosxi/api/v1/objects/servicestatus?fusekey=xxxxxxxxxxxxxxxxxxx&outputtype=xml&user=nagiosadmin&records=1000:0?brevity=1 -k

Add the output of the curl commands to the post.
If the servers are not using SSL, change https to http and remove the -k option.

[blariv]

here are the outputs from the one w/ the fuse key, the other is being upgraded to the latest version but right now is 2012R2.9


[root@nagiosfusion ~]# curl -XGET https://nagiosxi.na.hasbro.com/nagiosxi ... 54B1AAA9-k -v
* About to connect() to nagiosxi.na.hasbro.com port 443 (#0)
* Trying 10.3.35.139...
* Connected to nagiosxi.na.hasbro.com (10.3.35.139) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=nagiosxi.hasbro.com,OU=Information System,O="Hasbro, Inc.",L=East Providence,ST=Rhode Island,C=US
* start date: Mar 11 19:18:00 2019 GMT
* expire date: Mar 10 19:18:00 2023 GMT
* common name: nagiosxi.hasbro.com
* issuer: CN=Hasbro Ent1 CA,DC=na,DC=hasbro,DC=com
> GET /nagiosxi/api/v1/system/status?fusekey=C9EF8E6AA7D7CE3FD52E4A8B54B1AAA9-k HTTP/1.1
> User-Agent: curl/7.29.0
> Host: nagiosxi.na.hasbro.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 10 Apr 2020 19:15:52 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/5.4.16
< X-Powered-By: PHP/5.4.16
< Access-Control-Allow-Orgin: *
< Access-Control-Allow-Methods: *
< Content-Length: 28
< Content-Type: application/json
<
{"error":"Invalid API Key"}
* Connection #0 to host nagiosxi.na.hasbro.com left intact





[root@nagiosfusion ~]# time curl -XGET https://nagiosxi.na.hasbro.com/nagiosxi ... ?brevity=1 -k
[1] 4314
[2] 4315
[3] 4316
-bash: -k: command not found
[2]- Done outputtype=xml
[3]+ Done user=nagiosadmin
[root@nagiosfusion ~]#
real 0m0.912s
user 0m0.036s
sys 0m0.048s

[1]+ Done time curl -XGET https://nagiosxi.na.hasbro.com/nagiosxi ... 8B54B1AAA9
[root@nagiosfusion ~]#

[tgriep]

The XI version 2012R2.9 is not compatible with the newer version of Fusion so that is the reason that server is not working.
If needs to be 2014R2.7 to use the Core connection setting and XX 5.4 to use the Fuse Key.

The curl commands had some errors in them so try running these modified examples.

Code: Select all

curl -k -v -XGET https://nagiosxi.na.hasbro.com/nagiosxi/api/v1/system/status?fusekey=C9EF8E6AA7D7CE3FD52E4A8B54B1AAA9
time curl -k -v -XGET https://nagiosxi.na.hasbro.com/nagiosxi/api/v1/objects/servicestatus?fusekey=C9EF8E6AA7D7CE3FD52E4A8B54B1AAA9&outputtype=xml&user=nagiosadmin&records=1000:0?brevity=1

[blariv]

ok, not worried about the old version as its being upgraded.

[root@nagiosfusion ~]# curl -k -v -XGET https://nagiosxi.na.hasbro.com/nagiosxi ... 8B54B1AAA9 time
* About to connect() to nagiosxi.na.hasbro.com port 443 (#0)
* Trying 10.3.35.139...
* Connected to nagiosxi.na.hasbro.com (10.3.35.139) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=nagiosxi.hasbro.com,OU=Information System,O="Hasbro, Inc.",L=East Providence,ST=Rhode Island,C=US
* start date: Mar 11 19:18:00 2019 GMT
* expire date: Mar 10 19:18:00 2023 GMT
* common name: nagiosxi.hasbro.com
* issuer: CN=Hasbro Ent1 CA,DC=na,DC=hasbro,DC=com
> GET /nagiosxi/api/v1/system/status?fusekey=C9EF8E6AA7D7CE3FD52E4A8B54B1AAA9 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: nagiosxi.na.hasbro.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 10 Apr 2020 20:20:08 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/5.4.16
< X-Powered-By: PHP/5.4.16
< Access-Control-Allow-Orgin: *
< Access-Control-Allow-Methods: *
< Content-Length: 838
< Content-Type: application/json
<
{"instance_id":"1","instance_name":"localhost","status_update_time":"2020-04-10 16:20:03","program_start_time":"2020-04-10 12:26:07","program_run_time":"14041","program_end_time":"0000-00-00 00:00:00","is_currently_running":"1","process_id":"81914","daemon_mode":"1","last_command_check":"1969-12-31 19:00:00","last_log_rotation":"1969-12-31 19:00:00","notifications_enabled":"1","active_service_checks_enabled":"1","passive_service_checks_enabled":"1","active_host_checks_enabled":"1","passive_host_checks_enabled":"1","event_handlers_enabled":"0","flap_detection_enabled":"0","process_performance_data":"1","obsess_over_hosts":"0","obsess_over_services":"0","modified_host_attributes":"95","modified_service_attributes":"95","global_host_event_handler":"xi_host_event_handler","global_service_event_handler":"xi_service_event_handler"}
* Connection #0 to host nagiosxi.na.hasbro.com left intact
* Could not resolve host: time; Unknown error
* Closing connection 1
curl: (6) Could not resolve host: time; Unknown error


[root@nagiosfusion ~]# curl -k -v -XGET https://nagiosxi.na.hasbro.com/nagiosxi ... ?brevity=1
[1] 12886
[2] 12887
[3] 12888
[root@nagiosfusion ~]# * About to connect() to nagiosxi.na.hasbro.com port 443 (#0)
* Trying 10.3.35.139...
* Connected to nagiosxi.na.hasbro.com (10.3.35.139) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=nagiosxi.hasbro.com,OU=Information System,O="Hasbro, Inc.",L=East Providence,ST=Rhode Island,C=US
* start date: Mar 11 19:18:00 2019 GMT
* expire date: Mar 10 19:18:00 2023 GMT
* common name: nagiosxi.hasbro.com
* issuer: CN=Hasbro Ent1 CA,DC=na,DC=hasbro,DC=com
> GET /nagiosxi/api/v1/objects/servicestatus?fusekey=C9EF8E6AA7D7CE3FD52E4A8B54B1AAA9 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: nagiosxi.na.hasbro.com
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 500 Internal Server Error
< Date: Fri, 10 Apr 2020 20:20:37 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/5.4.16
< X-Powered-By: PHP/5.4.16
< Access-Control-Allow-Orgin: *
< Access-Control-Allow-Methods: *
< Content-Length: 0
< Connection: close
< Content-Type: application/json
<
* Closing connection 0

[1] Done curl -k -v -XGET https://nagiosxi.na.hasbro.com/nagiosxi ... 8B54B1AAA9
[2]- Done outputtype=xml
[3]+ Done user=nagiosadmin
[root@nagiosfusion ~]#

[ssax]

Please attach your /etc/php.ini.

Additionally, send me the full output of this command (as root):
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pfusion in the command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('fusion');" | mysql -h 127.0.0.1 -uroot -pfusion --table​

How many total users do you have in XI?

[blariv]

72 users. nothing has been changed on the base install.

php attached

[root@nagiosfusion ~]# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('fusion');" | mysql -h 10.3.45.200 -uroot -pfusion --table
ERROR 1045 (28000): Access denied for user 'root'@'nagiosfusion.na.hasbro.com' (using password: YES)

[ssax]

First, please edit your /etc/php.ini and change these:

Code: Select all

max_execution_time = 30
max_input_time = 60
; max_input_vars = 1000
memory_limit = 128M

To these:

Code: Select all

max_execution_time = 300
max_input_time = 300
max_input_vars = 100000
memory_limit = 1024M

Then restart apache on the Fusion server:

Code: Select all

service httpd restart

I'd increase them on your XI servers as well if they haven't been yet.

Then test.

Please re-run the command and get it to produce output, you must've changed the root password for mysql, I need to see the output as the size of the tables could affect the system. Please adjust the password and try again.

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('fusion');" | mysql -h X.X.X.X -uroot -pfusion --table

Or try this one:

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('fusion');" | mysql -h X.X.X.X -ufusion -pfusion --table

Thank you

[blariv]

adjusted the settings on fusion and both XI instances.
seems resolved on the fused one, but not on the other. I am fine with that since the other will be upgraded shortly.

you can close this and thanks for the great support.

[root@nagiosfusion ~]# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('fusion');" | mysql -h localhost -ufusion -pfusion --table
+-----------------+------------+
| Table | Size in MB |
+-----------------+------------+
| auth_tokens | 0.03 |
| commands | 0.02 |
| dashboards | 0.02 |
| dashlets | 0.02 |
| dashlets_params | 0.02 |
| log | 1.52 |
| meta | 0.02 |
| options | 0.02 |
| polled_averages | 0.03 |
| polled_data | 0.23 |
| polled_deltas | 0.03 |
| polled_extras | 72.06 |
| polling_lock | 0.02 |
| servers | 0.03 |
| sysstat | 0.03 |
| users | 0.03 |
| users_servers | 0.02 |
| users_sessions | 0.03 |
| views | 0.02 |
+-----------------+------------+

[ssax]

That's great to hear! Locking the thread now.