Export of large queries

[mafamgm]

Hello,

I have to export some large queries (about 500k hits) to CSV. (or plain text)

I already did some changes to php.ini described here (https://support.nagios.com/kb/article/n ... e-611.html - I know its for XI, but it should also work for LS, right?).
But it did not changed the behavior.

So, after I changed the Paging settings (Per Page 250, Page Limit 3000) the little spinning wheel keeps going for some seconds, and then it stopped and the "All Events"-Page is empty with " 0 to 0 of 0 available for paging"
I have observed that about 40k entries are displayed, everything above that will not be displayed.

Do you have any suggestion or idea to achieve this goal?

Thanks,
Marcel

[ssax]

Yes, they should be valid for Log Server as well.

What did you set these to:

Code: Select all

max_execution_time
memory_limit
max_input_vars

Did you restart HTTPD after?

Code: Select all

systemctl restart httpd

Run this tail command as root and leave it running:

Code: Select all

tail -Fn0 /var/log/httpd/error_log /var/log/httpd/ssl_error_log /var/log/elasticticsearch/*.log

Then replicate the issue in the web interface again and after it shows 0, send the full output from the still running tail command above.

Attach the output of these commands:

Code: Select all

free -m
grep HEAP /etc/sysconfig/elasticsearch /etc/default/elasticsearch

Please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button.

[mafamgm]

Code: Select all

max_execution_time = 120
memory_limit = 1024M
max_input_vars = 50000

Code: Select all

[root@log01 ~]# tail -Fn0 /var/log/httpd/error_log /var/log/httpd/ssl_error_log /var/log/elasticsearch/XXXXXXXXXXXXXX/*.log
==> /var/log/httpd/error_log <==

==> /var/log/httpd/ssl_error_log <==
tail: cannot open ‘/var/log/elasticsearch/XXXXXXXXXXXXXXXXXXX/*.log’ for reading: No such file or directory
tail: cannot watch parent directory of ‘/var/log/elasticsearch/XXXXXXXXXXXXXXX/*.log’: No such file or directory
tail: inotify cannot be used, reverting to polling

==> /var/log/httpd/error_log <==
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 43119    0 43119    0     0  34269      0 --:--:--  0:00:01 --:--:-- 34275

==> /var/log/httpd/ssl_error_log <==
[Thu Jul 08 07:45:14.635665 2021] [:error] [pid 13731] [client 10.0.XXX.XXX:65086] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 77 bytes) in /var/www/html/nagioslogserver/application/libraries/Elasticsearch.php on line 0, referer: https://log01.xxxxxxxxxxxxxxx/nagioslogserver/dashboard

Code: Select all

[root@log01 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          63915       35216        1350         217       27348       27839
Swap:          8187          11        8176


[root@log01 ~]# grep HEAP /etc/sysconfig/elasticsearch /etc/default/elasticsearch
/etc/sysconfig/elasticsearch:ES_HEAP_SIZE=$(expr $(free -m|awk '/^Mem:/{print $2}') / 2 )m
/etc/sysconfig/elasticsearch:#ES_HEAP_NEWSIZE=
grep: /etc/default/elasticsearch: No such file or directory

Yes, I restarted the Apache.

Thanks for the commands.
So, PHP needs more RAM, right? Any recommendation or should I try it out?

[ssax]

Change these:

Code: Select all

max_execution_time = 120
memory_limit = 1024M

To these:

Code: Select all

max_execution_time = 300
memory_limit = 4096M

Then restart apache:

Code: Select all

systemctl restart httpd

Then try again, if it fails, run the tail command again and see if it's still failing on the memory_limit.