MRTG Permission Denied

[shoreypu]

Our systems team alerted me to the following error this morning after we upgraded our test server to XI v5.75

2020-12-17 10:30:01: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_38677: Permission denied at /usr/bin/mrtg line 1962.
Here is the cron job that were running:
/5 * * * * root LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lib/mrtg/mrtg.lock --confcache-file /var/lib/mrtg/mrtg.ok --user=nagios --group=nagios

I checked our prod box, which sends errors to a log file and were are seeing the same errors. Were still running XI v5.6.13 in production.

Has anyone else run into this problem? Any help on this is greatly appreciated.

Thanks...John

[vtrac]

Hi shoreypu,
1) Please make sure that user "nagios" has write permission for folder "/var/lib/mrtg" as show below:

var-lib-mrtg.png

You can run these commands as root/sudo to fix them:

Code: Select all

chown -R apache.nagios /var/lib/mrtg
chmod -R 664 /var/lib/mrtg

2) Please make sure to remove the "/var/lib/mrtg/mrtg.lock" file if existed.

I looked at line 1962 of "/usr/bin/mrtg" and looks like it is trying to create a $templock file:

Code: Select all

  open(LOCK,">$templock") or die "$NOW: ERROR: Creating templock $templock: $!";

I tested the same command on my Nagios XI using the CLI and also tested by creating a cron job with the exact command and both ran OK.

Also, I checked and my mrtg version is "2.17.4".
Could you please check and see what version of mrtg do you have?

Best Regards,
Vinh

[shoreypu]

Vinh,

Thanks for your reply.

I've confirmed the configurations, as written below, are the same on our primary, failover and test servers.

1) I've confirmed that nagios has write permissions. AS you'll see, you recommended 664, were set to 775.
[root@nagiosqa mrtg]# ls -la
total 4
drwxrwxr-x 2 apache nagios 20 Dec 18 15:45 .
drwxr-xr-x. 42 root root 4096 Dec 4 08:05 ..
-rwxrwxr-x 1 apache nagios 0 Dec 18 15:45 mrtg.ok

2) As you see above,the lock file does not exist.

3) I've confirmed that line 1962 of "/usr/bin/mrtg: is trying to create a $templock file:
open(LOCK,">$templock") or die "$NOW: ERROR: Creating templock $templock: $!";

I also verified that mrtg version is 2.17.4

Thanks...John

[vtrac]

Hi John,
Have you try re-ran the cron job again?

If it is still failing, then could you please try logging in as "nagios" and try create a new file under the "/var/lib/mrtg/" folder as user "nagios".

Regards,
Vinh

[shoreypu]

So the cron job runs every five minutes. I did go in to try and run the job earlier and it ran without error, though I did not see a lock file. I then check the log where I'm outputing errors and there were no errors as of this morning at 1am. Then the errors started again at 10am, which was after I tried my tests. Here's some of the log output.

2020-12-21 00:45:02: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_119498: Permission denied at /usr/bin/mrtg line 1962.
2020-12-21 00:50:01: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_121834: Permission denied at /usr/bin/mrtg line 1962.
2020-12-21 00:55:01: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_123884: Permission denied at /usr/bin/mrtg line 1962.
2020-12-21 01:00:02: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_125873: Permission denied at /usr/bin/mrtg line 1962.
2020-12-21 10:00:01: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_91103: Permission denied at /usr/bin/mrtg line 1962.
2020-12-21 10:05:01: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_93148: Permission denied at /usr/bin/mrtg line 1962.
2020-12-21 10:10:01: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_95187: Permission denied at /usr/bin/mrtg line 1962.
2020-12-21 10:15:01: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_97212: Permission denied at /usr/bin/mrtg line 1962.

[vtrac]

Hi John,
The error said you don't have permission to create the "lock" file under "/var/lib/mrtg".
let restart crond:

Code: Select all

# systemctl restart crond

Can you please try as "nagios", create a empty file under "/var/lib/mrtg":

Code: Select all

# su - nagios
# echo "" > /var/lib/mrtg/test.out

The cron job is called "mrtg", let try changing the "lock" file location to "/tmp":

Code: Select all

# vi /etc/cron.d/mrtg

Code: Select all

*/5 * * * * root LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /tmp/mrtg.lock --confcache-file /var/lib/mrtg/mrtg.ok --user=nagios --group=nagios

Please update the results if moving the "lock" file to "/tmp" solve the issue or not.

Regards,
Vinh

[shoreypu]

Vinh,

As nagios, I was able to write /var/lib/mrtg/test.out.

I then changed the lock file location in the cron job to /tmp/mrtg.lock. Since then, the error has not returned. I also do not see a lock file in /tmp/. I assume there should be a lock file there, correct?

Thanks...John

[vtrac]

Hi John,
Great!! .... so moving the lock file to "/tmp" works.
Let keep its there then.

As to the lock file, its should be removed once the mrtg ran is completed each time.
The reason for the lock file is to prevent two mrtg(s) run at the same time.

Regards,
Vinh

[shoreypu]

Vinh,

This did seem to work for a bit, but now that I'm back to work, I'm seeing a new error. There's no timestamp, so I'm showing you the previous line, which reflects the last date of the error we resolved.

2020-12-23 13:20:01: ERROR: Creating templock /var/lib/mrtg/mrtg.lock_80688: Permission denied at /usr/bin/mrtg line 1962.
ERROR: writing /var/lib/mrtg/mrtg.ok.ok: Permission denied at /usr/bin/../lib/mrtg2/MRTG_lib.pm line 1630.

John

[vtrac]

Hi John,
The error you were getting seems like your mrtg's cron job were not able to create the "mrtg.ok" file after it is completed.

Please make sure "nagios" do have "write" permission under "/var/lib/mrtg" folder.

Can you please run the below command and upload the results:

Code: Select all

# ls -la /var/lib/mrtg

Do you know if there are TWO mrtg jobs scheduled to run every 5 minutes?

Can you please run the below command and upload the results:

Code: Select all

# ls -l /etc/cron.d/

Also, let's try moving the "mrtg.ok" file to "/tmp" as well:

Code: Select all

*/5 * * * * root LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /tmp/mrtg.lock --confcache-file /tmp/mrtg.ok --user=nagios --group=nagios

Regards,
Vinh