The Availability Reports generation never ends

[luk206]

Hi,

when try to generate the Availability Reports, the process never ends.
It seem's that the avail.cgi read all logs related from /usr/local/nagios/var/archives/, so, the process don't take the history information from mysql database, only from disk (with the performance problem that this represent).

The size of ours archives log files are about 5-10MB, it seem's not very huge.

There are some configuration for take the history information from mysql, instead from archives files?
There are some configuration to improve the performance of Availability Reports generation time?
There are some log to show if there are some problem with the Availability Reports generation process?

Thanks in advance,

[bwallace]

The availability report calculates values by reading the nagios.log file and their archives, so disk speed is the primary factor in fast reports.
What is the time period you are running the report for?
Does this machine have fast disks, e.g. SSD, RAID Array?

What is the output of the following?

Code: Select all

ls -l /usr/local/nagios/var/nagios.log
du -h /usr/local/nagios/var/archives/
ls /usr/local/nagios/var/archives/|wc -l

[luk206]

Hi again. I attach the outputs:

Code: Select all

ls -l /usr/local/nagios/var/nagios.log
-rw-r--r-- 1 nagios nagios 13103971 Jul 19 13:06 /usr/local/nagios/var/nagios.log

du -h /usr/local/nagios/var/archives/
117M	/usr/local/nagios/var/archives/

ls /usr/local/nagios/var/archives/|wc -l
25

hdparm -Tt /dev/vda
/dev/vda:
 Timing cached reads:   14904 MB in  2.00 seconds = 7463.03 MB/sec
 Timing buffered disk reads: 2352 MB in  3.00 seconds = 783.73 MB/sec

The Availability Report time generation is more than 5 minutes from "This week" period selected, if you select another bigger period, the generation process of reports never ends.

[bwallace]

Hmm...could you generate the never ending report and while that is running, generate a profile of your Nagios XI install and PM that to us so we can dig a bit deeper into this?

You can gather a profile by going to: Admin > System Config > System Profile > Download Profile

[ssax]

In addition to what bwallace is asking for, are you seeing any errors in your /var/log/httpd/error_log or /var/log/httpd/ssl_error_log relating to PHP max execution time or max size being exceeded when you run the report?

Also, please attach your /etc/php.ini.

Thank you

[luk206]

Hmm...could you generate the never ending report and while that is running, generate a profile of your Nagios XI install and PM that to us so we can dig a bit deeper into this?

You can gather a profile by going to: Admin > System Config > System Profile > Download Profile

Hi again,

I'm sorry but I can't send you the profile file by http post, it seem's that there are some problem with the forum form.
So, you can download the profile here: https://drive.google.com/file/d/0B2JPen ... luOVU/view

Thanks in advance.

[luk206]

In addition to what bwallace is asking for, are you seeing any errors in your /var/log/httpd/error_log or /var/log/httpd/ssl_error_log relating to PHP max execution time or max size being exceeded when you run the report?

Also, please attach your /etc/php.ini.

Thank you

Hi again.

I don't see php errors related to the availability report generation.

This is the php.ini:

Code: Select all

[root@centos-nagiosxi-01 scripts]# grep -v ^';' /etc/php.ini | egrep -v '^\s*$'

Thanks in advance.
[PHP]
engine = On
short_open_tag = Off
asp_tags = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 300
max_input_time = 60
memory_limit = 4096M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 8M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 2M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
[CLI Server]
cli_server.color = On
[Date]
date.timezone = UTC
[filter]
[iconv]
[intl]
[sqlite]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
sendmail_path = /usr/sbin/sendmail -t -i
mail.add_x_header = On
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQL]
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 60
mysql.trace_mode = Off
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[Sybase-CT]
sybct.allow_persistent = On
sybct.max_persistent = -1
sybct.max_links = -1
sybct.min_server_severity = 10
sybct.min_client_severity = 10
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.bug_compat_42 = Off
session.bug_compat_warn = Off
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[MSSQL]
mssql.allow_persistent = On
mssql.max_persistent = -1
mssql.max_links = -1
mssql.min_error_severity = 10
mssql.min_message_severity = 10
mssql.compatability_mode = Off
mssql.secure_connection = Off
[Assertion]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[mcrypt]
[dba]
[root@centos-nagiosxi-01 scripts]# 

[luk206]

hi all,

It seems that if the period is not to big, like one week o less, the availability generation process ends, before 5 minutes, and it no consumes more that 4GB (memory_limit) of ram.
But, if the period is bigger than one week, the availability report process consume more than 4GB of ram, and it is closse, and another new availability report process is create, and this last begins from the first archive log file again, and the history is repeated indefinily.

Thanks again.

[tgriep]

Can you try and run the Legacy Availability report and see if the issue is the same?
If that runs quickly, it could some corruption in the MYSQL database and should be repaired.
Take a look at this KB article for instructions on how to repair the database.

[luk206]

Hi again,

I'm sorry but, it seem's that there is not database corruption, I did some repairs and the problem continues.
In fact, the availability reports generation process (avail.cgi) don't use the database, it finds all the history information only from log files, concretatly from log_archive_path.

I think so, because, while the avail.cgi is running, I don't find any "select" or other sql query on mysql database.

Code: Select all

+-------+----------+-----------+----------+---------+------+-------+------------------+----------+
| Id    | User     | Host      | db	  | Command | Time | State | Info             | Progress |
+-------+----------+-----------+----------+---------+------+-------+------------------+----------+
| 53828 | nagiosxi | localhost | nagiosxi | Sleep   | 4340 |	   |                  | 0.000    |
| 53829 | ndoutils | localhost | nagios   | Sleep   | 3056 |	   |                  | 0.000    |
| 53830 | nagiosql | localhost | nagiosql | Sleep   | 4340 |	   |                  | 0.000    |
| 53873 | nagiosxi | localhost | nagiosxi | Sleep   | 4331 |	   |                  | 0.000    |
| 53874 | ndoutils | localhost | nagios   | Sleep   | 4331 |	   |                  | 0.000    |
| 53875 | nagiosql | localhost | nagiosql | Sleep   | 4331 |	   |                  | 0.000    |
| 54038 | nagiosxi | localhost | nagiosxi | Sleep   | 4183 |	   |                  | 0.000    |
| 54039 | ndoutils | localhost | nagios   | Sleep   | 4183 |	   |                  | 0.000    |
| 54040 | nagiosql | localhost | nagiosql | Sleep   | 4183 |	   |                  | 0.000    |
| 54520 | nagiosxi | localhost | nagiosxi | Sleep   | 4333 |	   |                  | 0.000    |
| 54521 | ndoutils | localhost | nagios   | Sleep   | 4333 |	   |                  | 0.000    |
| 54522 | nagiosql | localhost | nagiosql | Sleep   | 4333 |	   |                  | 0.000    |
| 55103 | ndoutils | localhost | nagios   | Sleep   | 2    |	   |                  | 0.000    |
| 55104 | ndoutils | localhost | nagios   | Sleep   | 4197 |	   |                  | 0.000    |
| 55432 | nagiosxi | localhost | nagiosxi | Sleep   | 58   |	   |                  | 0.000    |
| 55433 | ndoutils | localhost | nagios   | Sleep   | 58   |	   |                  | 0.000    |
| 55434 | nagiosql | localhost | nagiosql | Sleep   | 58   |	   |                  | 0.000    |
| 56866 | nagiosxi | localhost | nagiosxi | Sleep   | 53   |	   |                  | 0.000    |
| 56867 | ndoutils | localhost | nagios   | Sleep   | 53   |	   |                  | 0.000    |
| 56868 | nagiosql | localhost | nagiosql | Sleep   | 53   |	   |                  | 0.000    |
| 57250 | nagiosxi | localhost | nagiosxi | Sleep   | 9    |	   |                  | 0.000    |
| 57251 | ndoutils | localhost | nagios   | Sleep   | 9    |	   |                  | 0.000    |
| 57252 | nagiosql | localhost | nagiosql | Sleep   | 9    |	   |                  | 0.000    |
| 57973 | nagiosxi | localhost | nagiosxi | Sleep   | 57   |	   |                  | 0.000    |
| 57974 | ndoutils | localhost | nagios   | Sleep   | 57   |	   |                  | 0.000    |
| 57975 | nagiosql | localhost | nagiosql | Sleep   | 57   |	   |                  | 0.000    |
| 58144 | nagiosxi | localhost | nagiosxi | Sleep   | 52   |	   |                  | 0.000    |
| 58145 | ndoutils | localhost | nagios   | Sleep   | 52   |	   |                  | 0.000    |
| 58146 | nagiosql | localhost | nagiosql | Sleep   | 52   |	   |                  | 0.000    |
| 58147 | nagiosxi | localhost | nagiosxi | Sleep   | 11   |	   |                  | 0.000    |
| 58148 | ndoutils | localhost | nagios   | Sleep   | 11   |	   |                  | 0.000    |
| 58149 | nagiosql | localhost | nagiosql | Sleep   | 11   |	   |                  | 0.000    |
| 58714 | nagiosxi | localhost | nagiosxi | Sleep   | 15   |	   |                  | 0.000    |
| 58715 | ndoutils | localhost | nagios   | Sleep   | 15   |	   |                  | 0.000    |
| 58716 | nagiosql | localhost | nagiosql | Sleep   | 15   |	   |                  | 0.000    |
| 58807 | nagiosxi | localhost | nagiosxi | Sleep   | 97   |	   |                  | 0.000    |
| 58808 | ndoutils | localhost | nagios   | Sleep   | 97   |	   |                  | 0.000    |
| 58809 | nagiosql | localhost | nagiosql | Sleep   | 97   |	   |                  | 0.000    |
| 59596 | nagiosxi | localhost | nagiosxi | Sleep   | 39   |	   |                  | 0.000    |
| 59597 | ndoutils | localhost | nagios   | Sleep   | 39   |	   |                  | 0.000    |
| 59598 | nagiosql | localhost | nagiosql | Sleep   | 39   |	   |                  | 0.000    |
| 59755 | nagiosxi | localhost | nagiosxi | Sleep   | 30   |	   |                  | 0.000    |
| 59756 | ndoutils | localhost | nagios   | Sleep   | 30   |	   |                  | 0.000    |
| 59757 | nagiosql | localhost | nagiosql | Sleep   | 30   |	   |                  | 0.000    |
| 59776 | nagiosxi | localhost | nagiosxi | Sleep   | 11   |	   |                  | 0.000    |
| 59777 | ndoutils | localhost | nagios   | Sleep   | 11   |	   |                  | 0.000    |
| 59778 | nagiosql | localhost | nagiosql | Sleep   | 11   |	   |                  | 0.000    |
| 59779 | nagiosxi | localhost | nagiosxi | Sleep   | 1    |	   |                  | 0.000    |
| 59780 | nagiosxi | localhost | nagiosxi | Sleep   | 1    |	   |                  | 0.000    |
| 59781 | ndoutils | localhost | nagios   | Sleep   | 11   |	   |                  | 0.000    |
| 59782 | ndoutils | localhost | nagios   | Sleep   | 11   |	   |                  | 0.000    |
| 59783 | nagiosql | localhost | nagiosql | Sleep   | 11   |	   |                  | 0.000    |
| 59784 | nagiosql | localhost | nagiosql | Sleep   | 11   |	   |                  | 0.000    |
| 59788 | nagiosxi | localhost | nagiosxi | Sleep   | 2    |	   |                  | 0.000    |
| 59789 | nagiosxi | localhost | nagiosxi | Sleep   | 6    |	   |                  | 0.000    |
| 59790 | ndoutils | localhost | nagios   | Sleep   | 11   |	   |                  | 0.000    |
| 59791 | ndoutils | localhost | nagios   | Sleep   | 11   |	   |                  | 0.000    |
| 59792 | nagiosql | localhost | nagiosql | Sleep   | 11   |	   |                  | 0.000    |
| 59793 | nagiosql | localhost | nagiosql | Sleep   | 11   |	   |                  | 0.000    |
| 59810 | root     | localhost |          | Query   | 0    |	   | show processlist | 0.000    |
+-------+----------+-----------+----------+---------+------+-------+------------------+----------+

Instead, if I find what files use the process avail.cgi while are running, it can see the archives log files.

Code: Select all

COMMAND     PID   USER   FD   TYPE             DEVICE  SIZE/OFF    NODE NAME
avail.cgi 25577 apache  cwd    DIR              253,1	   4096 1828087 /usr/local/nagiosxi/html/reports
avail.cgi 25577 apache  rtd    DIR              253,1	   4096       2 /
avail.cgi 25577 apache  txt    REG              253,1    297752 1838597 /usr/local/nagios/sbin/avail.cgi
avail.cgi 25577 apache  mem    REG              253,1   3633153  132924 /mnt/archives/nagios-07-09-2016-00.log
avail.cgi 25577 apache  mem    REG              253,1 106065056    7324 /usr/lib/locale/locale-archive
avail.cgi 25577 apache  mem    REG              253,1   2112384 1045312 /usr/lib64/libc-2.17.so
avail.cgi 25577 apache  mem    REG              253,1    164440 1044482 /usr/lib64/ld-2.17.so
avail.cgi 25577 apache    0r   CHR                1,3       0t0    1028 /dev/null
avail.cgi 25577 apache    1w  FIFO                0,8       0t0 1200696 pipe
avail.cgi 25577 apache    2w   REG              253,1    254825  396942 /var/log/httpd/error_log
avail.cgi 25577 apache    3r   REG              253,1   3633153  132924 /mnt/archives/nagios-07-09-2016-00.log
avail.cgi 25577 apache   16u  unix 0xffff880408bcecc0       0t0 1104644 socket
avail.cgi 25577 apache   17u  unix 0xffff88035be87440       0t0 1105396 socket
avail.cgi 25577 apache   18u  unix 0xffff88035be86900       0t0 1105397 socket

The legacy availability reports seem's the same problem, the process never ends, but the http answer is one 504,
Gateway Timeout
The gateway did not receive a timely response from the upstream server or application.

Thanks for everybody, but the problem continues.