service-perfdata Not Rolling Over

[jimmlegs]

Good Day,

My Nagios XI server recently started performing very slugishly and I think it is related to the logging subsystem. I am receiving the following errors in my nagios.log file:

[1405685353] Unable to determine status of file /usr/local/nagios/var/nagios.log: No such file or directory
[1405685353] SIGXFSZ received because a file's size may have exceeded the file size limits of the filesystem. The largest file checked, '/usr/local/nagios/var/nagios.log', has a size of 2147483647 bytes

and this in my npcd.log file:

[07-18-2014 08:08:57] NPCD: ERROR: Executed command exits with return code '4'
[07-18-2014 08:08:57] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1404393464.perfdata.host'

the service-perfdata file is 2147483647 bytes and it looks like it was created July 10th

I am running Nagios XI 2014R1.2 on CentOS 5.10

Thanks in advance.

[lmiltchev]

If you don't care about the log info you can just clear the nagios.log by running:

Code: Select all

cat /dev/null > /usr/local/nagios/var/nagios.log

If you want to save the data, you can try moving it to "/tmp/" (it may take a long time):

Code: Select all

mv /usr/local/nagios/var/nagios.log /tmp/
service nagios restart

Note: the log should be recreated.

Did you disable log rotation? What is the output of the following command?

Code: Select all

grep log_rotation_method /usr/local/nagios/etc/nagios.cfg

Also run the commands below and show us the output:

Code: Select all

ulimit -a
ls /usr/local/nagios/var/spool/xidpe | wc -l
ls /usr/local/nagios/var/spool/perfdata | wc -l
ls /usr/local/nagios/var/spool/checkresults | wc -l

[jimmlegs]

[root@bos-nagios nagios]# grep log_rotation_method etc/nagios.cfg
log_rotation_method=d

[root@bos-nagios nagios]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64621
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 64621
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[root@bos-nagios nagios]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64621
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 64621
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[root@bos-nagios nagios]# ls /usr/local/nagios/var/spool/xidpe | wc -l
1

[root@bos-nagios nagios]# ls /usr/local/nagios/var/spool/perfdata | wc -l
126659

[root@bos-nagios nagios]# ls /usr/local/nagios/var/spool/checkresults | wc -l
0

Thanks

[tmcdonald]

Usually when the perfdata directory is full like that it is because your npcd is not running, and therefor cannot process the files. Please run the following and post the output:

Code: Select all

w
service npcd status

[jimmlegs]

I didn't want to move the service-perfdata file but ended up doing so and restarting the server. It looks OK (for at least the last 5 minutes)

NPCD running (pid 5041)

Thanks

[tmcdonald]

We'll keep this thread open for a bit in case you run into the same issue again. Also, the "w" is a command and not a typo in my last response - it shows who is logged in as well as the CPU usage.

[jimmlegs]

Thanks. I received the following message while backing up the system: "mysqldump: Got error: 145: Table './nagios/nagios_logentries' is marked as crashed and should be repaired when using LOCK TABLES" so I ran the "repair_databases.sh" script and so for so good.

[tmcdonald]

Guessing you haven't run into any more trouble this last week so I'll be closing this out now.