Disk space and permissions issues

[peakyblinder]

Hi,

I've got a disk space issue on our Nagios XI Server, specifically /user/local. It would appear its Nagios related files consuming the space rather than anything else.
I did notice a large syststat.log file.. about 3GB! I have tried to emptying the file with dev null but it would appear I have no permissions to do so. The file is owned by nagios:users (other files are are nagios:nagios) and am unable to su to this account as I cannot put hand to the nagios password.

Log rotate seems to be running, I think, else I'd have numerous large files.

So maybe its the permissions issue I need to sort out first and the log file issue will go away?

Can anyone advise where I start with this please?

I admit, there probably isnt enough information above, sorry.

Thanks,

[dchurch]

You're right, that file (and in fact all those logs) should get auto-truncated daily once it reaches 5 megs.

What's the outputs from the following commands?

Code: Select all

service crond status
ls -la /etc/cron.daily/logrotate
cat /etc/cron.daily/logrotate
ls -la /etc/logrotate.d/nagiosxi
cat /etc/logrotate.d/nagiosxi
ls -la /usr/local/nagiosxi/var

What distro is this running on?

[peakyblinder]

Hi, thanks for replying.

Here are the outputs:

service crond status

Redirecting to /bin/systemctl status crond.service
● crond.service - Command Scheduler
Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-02-06 12:06:54 GMT; 1 weeks 6 days ago
Main PID: 909 (crond)
Tasks: 1
Memory: 3.8M
CGroup: /system.slice/crond.service
└─909 /usr/sbin/crond -n

ls -la /etc/cron.daily/logrotate

-rwx------ 1 root root 219 Jun 15 2018 /etc/cron.daily/logrotate

As sudo:

#!/bin/sh

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

cat /etc/cron.daily/logrotate

cat: /etc/cron.daily/logrotate: Permission denied

As sudo

#!/bin/sh

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

ls -la /etc/logrotate.d/nagiosxi

-rw-r--r--. 1 root root 389 Jun 9 2017 /etc/logrotate.d/nagiosxi

cat /etc/logrotate.d/nagiosxi

/usr/local/nagiosxi/var/*log {
missingok
notifempty
size 5M
rotate 1
compress
}

/usr/local/nagiosxi/var/xidebug.log {
missingok
notifempty
size 100M
create 0660 apache nagios
rotate 1
compress
}

/usr/local/nagiosxi/var/xidebug.log.backtrace {
missingok
notifempty
size 100M
create 0660 apache nagios
rotate 1
compress

ls -la /usr/local/nagiosxi/var

Code: Select all

total 6374132
drwxrwxr-x.  7 nagios nagios       4096 Feb 20 08:14 .
drwxr-xr-x. 10 root   nagios        102 Jul 25  2017 ..
drwxrwxr-x   2 nagios nagios          6 Mar 12  2019 certs
-rw-r--r--   1 nagios nagios  324182714 Feb 20 08:14 cleaner.log
-rw-r--r--   1 nagios nagios   85785185 Feb 20 08:14 cmdsubsys.log
drwsrwsr-x.  3 apache nagios        111 Jul  7  2020 components
-rw-r--r--.  1 nagios nagios          7 Feb 20 08:14 corelog.data
-rw-r--r--.  1 nagios nagios          0 Feb 20 08:14 corelog.diff
-rw-r--r--   1 nagios nagios  625459546 Feb 20 08:10 dbmaint.log
-rw-r--r--   1 nagios nagios   66023649 Feb 20 08:14 deadpool.log
-rw-r--r--   1 nagios users           0 Feb 20 08:14 event_handler.lock
-rw-r--r--   1 nagios nagios  219602990 Feb 20 08:14 event_handler.log
-rw-r--r--   1 nagios nagios  416672673 Feb 20 08:14 eventman.log
-rw-r--r--   1 nagios nagios   22372267 Feb 20 08:14 feedproc.log
drwxrwxr-x   2 nagios nagios         20 Mar 12  2019 keys
-rw-r--r--.  1 nagios nagios       1153 Dec  4 12:05 load_url.log
-rw-r--r--.  1 nagios nagios    3356745 Feb 19 15:30 nom.log
-rw-rw-rw-   1 nagios nagios          0 Mar 12  2019 NXTI_Write_Test
-rw-r--r--   1 nagios nagios   74215485 Feb 20 08:14 perfdataproc.log
-rw-r--r--   1 nagios nagios 1410780362 Feb 20 08:01 recurringdowntime.log
-rw-r--r--.  1 nagios nagios     816929 Mar 12  2019 reportengine.log
-rw-r--r--   1 apache apache      21308 Jun 11  2020 scheduledreporting.log
-rw-r--r--   1 nagios nagios          0 Jun 27  2019 snmptt_service_results.log
drwxrwxr-x.  2 nagios nagios         50 Feb  6 12:06 subsys
-rw-r--r--   1 nagios users  3269822422 Feb 20 08:14 sysstat.log
drwxrwxr-x.  2 apache nagios       4096 Jul  7  2020 upgrades
-rw-rw-r--   1 nagios nagios       2141 Jun 11  2020 wkhtmltox.log
-rw-r--r--.  1 nagios nagios          7 Jun  9  2017 xi-itype
-rw-r--r--.  1 nagios nagios      12119 Jun 27  2019 xi-sys.cfg
-rw-r--r--.  1 nagios nagios         37 Jun  9  2017 xi-uuid
-rw-r--r--.  1 nagios nagios        196 Jun 27  2019 xiversion

Thanks,

[dchurch]

One more:

Code: Select all

sudo cat /etc/anacrontab

[peakyblinder]

Hi,

Sure, no problem:

# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly

[dchurch]

Thanks!

I'm going to assume that wasn't a typo, and that the logrotate entry really is missing the last curly-brace. There are other things missing too such as the "su" lines, but that'll be fixed if you follow this next step.

Edit /etc/logrotate.d/nagiosxi, and replace the whole file with the following:

Code: Select all

/usr/local/nagiosxi/var/*log {
    missingok
    notifempty
    size 5M
    su nagios nagios
    rotate 1
    compress
}

/usr/local/nagiosxi/var/xidebug.log {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    su apache nagios
    rotate 1
    compress
}

/usr/local/nagiosxi/var/xidebug.log.backtrace {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    su apache nagios
    rotate 1
    compress
}

/usr/local/nagiosxi/var/components/auditlog.log {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    rotate 1
    compress
}

Note: Spaces OR tabs are just fine in this file / interchangeable.

[peakyblinder]

Hi,

Yes, it was a typo, I missed off the last curly-brace, my bad.

I ran the logrotate in debug mode to see what would happen if it were run and as I thought, there are some permissions issues.

Code: Select all

reading config file nagiosxi
Allocating hash table for state file, size 15360 B

Handling 4 logs

rotating pattern: /usr/local/nagiosxi/var/*log  forced from command line (1 rotations)
empty log files are not rotated, old logs are removed
switching euid to 1000 and egid to 1001
error: error switching euid to 1000 and egid to 1001: Operation not permitted

rotating pattern: /usr/local/nagiosxi/var/xidebug.log  forced from command line (1 rotations)
empty log files are not rotated, old logs are removed
switching euid to 48 and egid to 1001
error: error switching euid to 48 and egid to 1001: Operation not permitted

rotating pattern: /usr/local/nagiosxi/var/xidebug.log.backtrace  forced from command line (1 rotations)
empty log files are not rotated, old logs are removed
switching euid to 48 and egid to 1001
error: error switching euid to 48 and egid to 1001: Operation not permitted

rotating pattern: /usr/local/nagiosxi/var/components/auditlog.log  forced from command line (1 rotations)
empty log files are not rotated, old logs are removed
considering log /usr/local/nagiosxi/var/components/auditlog.log
  log /usr/local/nagiosxi/var/components/auditlog.log does not exist -- skipping
error: error creating output file /var/lib/logrotate/logrotate.status.tmp: Permission denied

[dchurch]

What's the output from this command?

Code: Select all

ls -la /var/lib/logrotate

There are other things missing too such as the "su" lines, but that'll be fixed if you follow this next step.

Make sure those "su" lines from my post above are in-place. The UID errors, at least, are from the "su" lines not being there. The logrotate config I pasted came from the xi-5.8.1 VMware image. It's what the contents should be.

[peakyblinder]

Hi,

The su lines are definitely there. It was a straight copy and paste job from your post.

The output from:

Code: Select all

ls -la /var/lib/logrotate

Code: Select all

total 8
drwxr-xr-x.  2 root root   30 Feb 23 03:36 .
drwxr-xr-x. 48 root root 4096 Nov 30 08:19 ..
-rw-r--r--   1 root root 2499 Feb 23 03:36 logrotate.status

Thanks,

[dchurch]

I ran the logrotate in debug mode to see what would happen if it were run and as I thought, there are some permissions issues.

Did you run logrotate in debug mode as root or as your unprivileged user? Please run it as root again and post any errors here.