Nagios Support Forum

Good Day,

My Nagios XI server recently started performing very slugishly and I think it is related to the logging subsystem. I am receiving the following errors in my nagios.log file:

[1405685353] Unable to determine status of file /usr/local/nagios/var/nagios.log: No such file or directory
[1405685353] SIGXFSZ received because a file's size may have exceeded the file size limits of the filesystem. The largest file checked, '/usr/local/nagios/var/nagios.log', has a size of 2147483647 bytes

and this in my npcd.log file:

[07-18-2014 08:08:57] NPCD: ERROR: Executed command exits with return code '4'
[07-18-2014 08:08:57] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1404393464.perfdata.host'

the service-perfdata file is 2147483647 bytes and it looks like it was created July 10th

I am running Nagios XI 2014R1.2 on CentOS 5.10

Thanks in advance.

If you don't care about the log info you can just clear the nagios.log by running:
If you want to save the data, you can try moving it to "/tmp/" (it may take a long time):
Note: the log should be recreated.

Did you disable log rotation? What is the output of the following command?
Also run the commands below and show us the output:

[root@bos-nagios nagios]# grep log_rotation_method etc/nagios.cfg
log_rotation_method=d

[root@bos-nagios nagios]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64621
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 64621
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[root@bos-nagios nagios]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64621
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 64621
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[root@bos-nagios nagios]# ls /usr/local/nagios/var/spool/xidpe | wc -l
1

[root@bos-nagios nagios]# ls /usr/local/nagios/var/spool/perfdata | wc -l
126659

[root@bos-nagios nagios]# ls /usr/local/nagios/var/spool/checkresults | wc -l
0

Thanks

Usually when the perfdata directory is full like that it is because your npcd is not running, and therefor cannot process the files. Please run the following and post the output:

I didn't want to move the service-perfdata file but ended up doing so and restarting the server. It looks OK (for at least the last 5 minutes)

NPCD running (pid 5041)

Thanks

We'll keep this thread open for a bit in case you run into the same issue again. Also, the "w" is a command and not a typo in my last response - it shows who is logged in as well as the CPU usage.

Thanks. I received the following message while backing up the system: "mysqldump: Got error: 145: Table './nagios/nagios_logentries' is marked as crashed and should be repaired when using LOCK TABLES" so I ran the "repair_databases.sh" script and so for so good.

Guessing you haven't run into any more trouble this last week so I'll be closing this out now.