httpd logs growing heavily

[HAL]

Hi,
PFB snapshot showing huge growth in error and access logs for httpd Server consuming 45 GB size.
[root@oslms018 httpd]# ls -ltrh
total 45G
-rw-r--r-- 1 root root 321 Sep 28 2012 ssl_request_log-20120930
-rw-r--r-- 1 root root 2.0K Sep 28 2012 ssl_error_log-20120930
-rw-r--r-- 1 root root 270 Sep 28 2012 ssl_access_log-20120930
-rw-r--r-- 1 root root 35M Sep 30 2012 access_log-20120930
-rw-r--r-- 1 root root 112M Sep 30 2012 error_log-20120930
-rw-r--r-- 1 root root 0 Oct 7 2012 ssl_error_log
-rw-r--r-- 1 root root 0 Oct 7 2012 error_log
-rw-r--r-- 1 root root 0 Oct 7 2012 access_log
-rw-r--r-- 1 root root 0 Oct 20 2012 ssl_request_log
-rw-r--r-- 1 root root 0 Oct 20 2012 ssl_access_log
-rw-r--r-- 1 root root 1.4M Jan 10 09:47 ssl_error_log-20121007
-rw-r--r-- 1 root root 65K Jan 10 09:47 ssl_access_log-20121020
-rw-r--r-- 1 root root 68K Jan 10 09:47 ssl_request_log-20121020
-rw-r--r-- 1 root root 42G Apr 22 16:26 error_log-20121007
-rw-r--r-- 1 root root 3.2G Apr 22 16:27 access_log-20121007

Seems log rotation/retention of logs is not happening/functional currently, request to guide me in the needful towards setting up the cleanup of old logs.

Thanks,
Sanjeev

[slansing]

Can you please send us your httpd.conf file? You can block out any sensitive information. Also, can you do the following:

Please stop apache, move your error_log-20121007, and access_log-20121007 to a temporary folder, then start apache again. Then lets check and see if those files are re-created in the log directory. They are not the standard log file names "they have an additional notation at the end" so they could very well not be being rotated by the log rotation program. Do you know off hand if this is a pre-configured system for your company that is customized? Or that someone fiddled with the logging and rotation configurations?

[HAL]

Hi,

Please find attached httpd configuration file, its the default one, there is no manipluation done. Even we haven't done anything with regards to rotation of logs, means everything is default, as we installed nagios in our RHEL box.

This Saturday we have maintenance, then I can stop apache and will delete error_log-20121007 and access_log-20121007, hope that should not impact the things and once apache is started again, the error_log-***** and access_log-***** would be created again.

Kindly look in to the requirement, do let me know for any more information in this regard.

[abrist]

The apache access logs usually cannot be rotated while the apache server is running as the apache server keeps the file open. You would have to create a cron job to gracefully shut down the service, move the log, and restart the apache server. See: http://httpd.apache.org/docs/2.2/logs.html#rotation

On even a moderately busy server, the quantity of information stored in the log files is very large. The access log file typically grows 1 MB or more per 10,000 requests. It will consequently be necessary to periodically rotate the log files by moving or deleting the existing logs. This cannot be done while the server is running, because Apache will continue writing to the old log file as long as it holds the file open. Instead, the server must be restarted after the log files are moved or deleted so that it will open new log files.

By using a graceful restart, the server can be instructed to open new log files without losing any existing or pending connections from clients. However, in order to accomplish this, the server must continue to write to the old log files while it finishes serving old requests. It is therefore necessary to wait for some time after the restart before doing any processing on the log files. A typical scenario that simply rotates the logs and compresses the old logs to save space is:

mv access_log access_log.old
mv error_log error_log.old
apachectl graceful
sleep 600
gzip access_log.old error_log.old

[HAL]

Thanks for the suggestion, I will plan to stop the httpd services shortly, as maintenance date is postponed.

[abrist]

No problem, HAL. Was there a common repeated line in error_log? If so, post it here, as you may have a problem with your apache server/XI install.

[HAL]

Hi,
error_log and access_log are of 0 bytes only, as I have posted earlier. PFB the snapshot again.
-rw-r--r-- 1 root root 321 Sep 28 2012 ssl_request_log-20120930
-rw-r--r-- 1 root root 2.0K Sep 28 2012 ssl_error_log-20120930
-rw-r--r-- 1 root root 270 Sep 28 2012 ssl_access_log-20120930
-rw-r--r-- 1 root root 35M Sep 30 2012 access_log-20120930
-rw-r--r-- 1 root root 112M Sep 30 2012 error_log-20120930
-rw-r--r-- 1 root root 0 Oct 7 2012 ssl_error_log
-rw-r--r-- 1 root root 0 Oct 7 2012 error_log
-rw-r--r-- 1 root root 0 Oct 7 2012 access_log
-rw-r--r-- 1 root root 0 Oct 20 2012 ssl_request_log
-rw-r--r-- 1 root root 0 Oct 20 2012 ssl_access_log
-rw-r--r-- 1 root root 1.4M Jan 10 09:47 ssl_error_log-20121007
-rw-r--r-- 1 root root 65K Jan 10 09:47 ssl_access_log-20121020
-rw-r--r-- 1 root root 68K Jan 10 09:47 ssl_request_log-20121020
-rw-r--r-- 1 root root 49G May 23 16:11 error_log-20121007
-rw-r--r-- 1 root root 3.6G May 23 16:11 access_log-20121007

[sreinhardt]

The particular logs you are showing are from mid to late 2012, there should be no real reason to keep them local to the system if at all any longer. However you are correct that it is a little strange that none of the logs that should be active are being used. The apache.conf that you originally posted looks just fine, lets check where else you might be logging to. Please run the following command and return the results.

Code: Select all

grep -R -i "log" /etc/httpd/* 

[HAL]

Hi,
I have attached a portion against command grep -R -i "log" /etc/httpd/*

[slansing]

Do you have a timezone set in your php.ini file? Open the following, and search for "date.timezone":

Code: Select all

vi /etc/php.ini

You must set "date.timezone" to your local timezone, the correct format for your timezone in php is found here:

http://www.php.net/manual/en/datetime.configuration.php