Page 1 of 1

permission denied on second /dev/fio device

Posted: Tue May 02, 2017 10:29 am
by newkid
We have 2 partitions that are mounted on /dev/fioa and /dev/fiob respectively.

1. The first partition is reported correctly on the Nagios XI server:

DISK OK - free space: /var/lib/pgsql 1049755 MB (97% inode=99%):

2. The second partition is reporting a permission denied error:

DISK CRITICAL - /var/lib/pgsql/9.6/data/pg_xlog is not accessible: Permission denied


3. If performed on the monitored server itself the command executes properly:

[root@remotehost]# /usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /var/lib/pgsql/9.6/data/pg_xlog
DISK OK - free space: /var/lib/pgsql/9.6/data/pg_xlog 1072496 MB (99% inode=99%);| /var/lib/pgsql/9.6/data/pg_xlog=1062MB;904814;1017916;0;1131018


4. Additional details:
remotehost means centos server being monitored, not the Nagios XI server.

[root@remotehost]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md126p4 202G 129G 63G 68% /
tmpfs 32G 4.0K 32G 1% /dev/shm
/dev/md126p1 969M 57M 863M 7% /boot
/dev/md126p2 7.7G 51M 7.3G 1% /tmp
/dev/fioa 1.1T 24G 1.1T 3% /var/lib/pgsql
/dev/fiob 1.1T 1.1G 1.1T 1% /var/lib/pgsql/9.6/data/pg_xlog

[root@remotehost]# pwd
/var/lib
[root@remotehost]# ls -ld pgsql
drwxr-xr-x 4 postgres postgres 4096 May 1 11:32 pgsql

[root@remotehost]# pwd
/var/lib/pgsql/9.6/data
[root@remotehost]# ls -ld pg_xlog
drwxr-xr-x 4 postgres postgres 4096 Apr 28 16:07 pg_xlog

[root@remotehost]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 232.9G 0 disk
└─md126 9:126 0 221.2G 0 raid1
├─md126p1 259:0 0 1000M 0 md /boot
├─md126p2 259:1 0 7.9G 0 md /tmp
├─md126p3 259:2 0 7.9G 0 md [SWAP]
└─md126p4 259:3 0 204.4G 0 md /
sda 8:0 0 232.9G 0 disk
└─md126 9:126 0 221.2G 0 raid1
├─md126p1 259:0 0 1000M 0 md /boot
├─md126p2 259:1 0 7.9G 0 md /tmp
├─md126p3 259:2 0 7.9G 0 md [SWAP]
└─md126p4 259:3 0 204.4G 0 md /
fioa 252:0 0 1.1T 0 disk /var/lib/pgsql
fiob 252:16 0 1.1T 0 disk /var/lib/pgsql/9.6/data/pg_xlog


Thanks

Re: permission denied on second /dev/fio device

Posted: Tue May 02, 2017 3:33 pm
by tgriep
If you are using the NRPE Agent to monitor that partition, the commands are run as the nagios user account and one of the folders in that path probably doesn't have the read bit set for the other group and that is why the error is generated as the nagios user cannot access the path.
One fix it to verify that the nagios user can access all of the folders in that path by changing the permissions to the folder.
The other option is to edit the /etc/group file and add the nagios user to the postgres group.
After doing one of the above changes, you can test it by changing to the nagios user and running the check_disk command.

Code: Select all

su nagios
/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /var/lib/pgsql/9.6/data/pg_xlog

Re: permission denied on second /dev/fio device

Posted: Wed May 03, 2017 2:52 pm
by newkid
The information did help me however I encountered a new road block: PostgreSQL-9.6 requires permission as "postgres:postgres 700" on /var/lib/pgsql/9.6/data otherwise it won't start. I added a line "check_disk" with visudo :

# NEEDED TO ALLOW NAGIOS TO CHECK SERVICE STATUS
Defaults:nagios !requiretty
nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_init_service
nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_disk

Then made the test to validate sudoers settings:

configuration:
[root@monitored libexec]# ls -ld /var/lib/pgsql/9.6/data
drwx------ 20 postgres postgres 4096 May 3 13:12 /var/lib/pgsql/9.6/data

test without sudo prefix:
=====================
[nagios@monitored libexec]$ /usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /var/lib/pgsql/9.6/data/pg_xlog
DISK CRITICAL - /var/lib/pgsql/9.6/data/pg_xlog is not accessible: Permission denied

test with sudo prefix:
==================
[nagios@monitored libexec]$ sudo /usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /var/lib/pgsql/9.6/data/pg_xlog
DISK OK - free space: /var/lib/pgsql/9.6/data/pg_xlog 1073472 MB (99% inode=99%);| /var/lib/pgsql/9.6/data/pg_xlog=86MB;904814;1017916;0;1131018

After forcing an immediate check on the Nagios XI server the Permission denied is still received....I was under the impression that sudo would have be used somehow for this command on the Nagios XI server:

check_nrpe!check_disk!-a '-w 20% -c 10% -p /var/lib/pgsql/9.6/data/pg_xlog'

So next I edited /usr/local/nagios/etc/nrpe.cfg to uncomment the sudo line below:

# COMMAND PREFIX
# This option allows you to prefix all commands with a user-defined string.
# A space is automatically added between the specified prefix string and the
# command line from the command definition.
command_prefix=/usr/bin/sudo

It fixed the Permission denied but then I started to see "NRPE: Unable to read output" on other items being monitored, I commented out the command_prefix line...one item not monitored is better than numerous not monitored!

xinetd is used on my system (meaning that on my system "service nrpe xxx" is not used but rather "service xinetd xxxx" )

Re: permission denied on second /dev/fio device

Posted: Wed May 03, 2017 3:15 pm
by tgriep
The check_disk command that is defined in the NRPE agent, you would need to add sudo to the command like the example below

Code: Select all

command[check_disk]=sudo /usr/local/nagios/libexec/check_disk $ARG1$ $ARG2$ $ARG3$
The will let the command run as root so it should work for you.

Re: permission denied on second /dev/fio device

Posted: Wed May 03, 2017 5:17 pm
by newkid
yes got it working with this line in /usr/local/nagios/etc/nrpe.cfg:

command[check_pg_xlog]=/usr/bin/sudo /usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /var/lib/pgsql/9.6/data/pg_xlog

and this line in the sudoers

nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_disk

PS: Eventually I need to tackle this line, once I get comfortable with the syntax, the $ARG$ ordering and use is not so obvious to me.

#command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$

Re: permission denied on second /dev/fio device

Posted: Wed May 03, 2017 8:10 pm
by tacolover101
i'll try to help explain the args.

command[check_disk]=/usr/local/nagios/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$

now when you call to check_nrpe you could run it like this -
check_nrpe -H 1.3.3.7 -c check_disk -a warningvalue critvalue '2 -p 1 -b2'

Spaces separate the arg values, and quotes around the p/b flag encapsulate all that to represented as a single value. hope that helps.

Re: permission denied on second /dev/fio device

Posted: Thu May 04, 2017 10:02 am
by tgriep
Take a look at the NRPE Source install, it has some descriptions on the Arguments and how they work.
https://assets.nagios.com/downloads/nag ... ios-XI.pdf