Permission denied errors in my audit logs

[dhatha03]

I am getting hundreds of permission denied messages in my Nagios server audit logs. For example,

node=<nagios server> msg=audit(09/16/2020 10:13:45.634:2983937) exit=-13(Permission denied) auid=nagios uid=nagios gid=nagios comm=snmpget exe=/usr/bin/snmpget
node=<nagios server> msg=audit(09/16/2020 10:13:48.181:2983940) exit=-13(Permission denied) auid=nagios uid=nagios gid=nagios comm=snmpwalk exe=/usr/bin/snmpwalk
node=<nagios server> msg=audit(09/16/2020 10:13:43.828:2983935) exit=-13(Permission denied) auid=nagios uid=nagios gid=nagcmd comm=php exe=/usr/bin/php

File permissions are as follows:
-rwxr-xr-x 1 root root 13920 Mar 22 2017 /usr/bin/snmpget
-rwxr-xr-x 1 root root 16824 Mar 22 2017 /usr/bin/snmpwalk
-rwxr-xr-x 1 root root 3273840 Mar 22 2017 /usr/bin/php

All SNMP checks within NagiosXI are executing and completing as expected, but our log files are filling up quickly with the messages above.

[benjaminsmith]

Hi,

The file permissions look right and are the same as my test server, do you have SELinux enabled?

Code: Select all

getenforce

Also, have you made any changes to the sudoers file, please post the output to the following command.

Code: Select all

cat /etc/suders

Regards,
Benjamin

[dhatha03]

getenforce = Disabled

I cannot include the full sudoers file due to company restrictions, but the Nagios part is below.

User_Alias NAGIOSXI=nagios
User_Alias NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/send_to_nls.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_ssl_config.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

[lmiltchev]

The entries in the sudoers look fine.

Have you modified your system from the "default", which could've caused the issue, e.g. hardening the server, restricting cron to root user only, LDAP sudoers, TCP wrappers, etc.?

Please run the following commands and show the output in code wraps:

Code: Select all

grep requiretty /etc/sudoers
chage nagios -l
chage apache -l
grep nag /etc/group
grep nag /etc/passwd
tail -50 /var/log/cron

Also, PM me your latest profile (Admin > System Config > System Profile > Download Profile).

[dhatha03]

profile.zip sent to you.

Code: Select all

[root@tvm-nagios01 ~]# grep requiretty /etc/sudoers
#Defaults    requiretty
# changed in order to be able to use sudo without a tty. See requiretty above.

Code: Select all

[root@tvm-nagios01 ~]# chage nagios -l
Last password change                                    : Sep 27, 2018
Password expires                                        : Dec 05, 2020
Password inactive                                       : Jan 09, 2021
Account expires                                         : never
Minimum number of days between password change          : 1
Maximum number of days between password change          : 800
Number of days of warning before password expires       : 7

Code: Select all

[root@tvm-nagios01 ~]# chage apache -l
Last password change                                    : Sep 27, 2018
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : -1
Maximum number of days between password change          : -1
Number of days of warning before password expires       : -1

Code: Select all

[root@tvm-nagios01 ~]# grep nag /etc/group
nagcmd:x:500:nagios,apache
nagios:x:501:nagios,apache,snmptt

Code: Select all

[root@tvm-nagios01 ~]# grep nag /etc/passwd
nagios:x:500:500:Nagios Owner Account:/home/nagios:/bin/bash

Code: Select all

[root@tvm-nagios01 ~]# grep nag /etc/passwd
nagios:x:500:500:Nagios Owner Account:/home/nagios:/bin/bash

[root@tvm-nagios01 ~]# tail -50 /var/log/cron
Sep 23 18:49:01 tvm-nagios01 CROND[15343]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Sep 23 18:49:01 tvm-nagios01 CROND[15344]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Sep 23 18:50:01 tvm-nagios01 CROND[15871]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Sep 23 18:50:01 tvm-nagios01 CROND[15873]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Sep 23 18:50:01 tvm-nagios01 CROND[15874]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Sep 23 18:50:02 tvm-nagios01 CROND[15939]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15940]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15942]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php >> /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15938]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Sep 23 18:50:02 tvm-nagios01 CROND[15943]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15944]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15941]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15945]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15947]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15946]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15949]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Sep 23 18:50:02 tvm-nagios01 CROND[15948]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lib/mrtg/mrtg.lock --confcache-file /var/lib/mrtg/mrtg.ok)
Sep 23 18:51:01 tvm-nagios01 CROND[16378]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16381]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16380]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16382]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16379]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16384]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16386]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16383]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16387]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16385]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Sep 23 18:51:01 tvm-nagios01 CROND[16390]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16782]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16783]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16784]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16785]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16786]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16788]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16787]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16790]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16791]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16792]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Sep 23 18:52:01 tvm-nagios01 CROND[16796]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17181]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17183]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17184]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17186]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17187]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17188]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17189]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17192]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17195]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17193]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Sep 23 18:53:01 tvm-nagios01 CROND[17196]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1)

[lmiltchev]

Thank you for the profile! I found these errors in the apache error log:

sh: /usr/local/nagios/var/rw/nagios.cmd: Permission denied
sh: /usr/local/nagios/var/rw/nagios.cmd: Permission denied
sh: /usr/local/nagios/var/rw/nagios.cmd: Permission denied
sh: /usr/local/nagios/var/rw/nagios.cmd: Permission denied

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.

sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified

So, let's check some permissions:

Code: Select all

ls -lad /usr/local/nagios /usr/local/nagios/var/ /usr/local/nagios/var/rw/
ls -la /usr/local/nagios/var/rw/

Also, make sure you have the following line in sudoers:

Code: Select all

Defaults:nagios !requiretty

Can you run the command below and show the output?

Code: Select all

su nagios -c "/usr/bin/php -v"

Have you modified your system from the "default", which could've caused the issue, e.g. hardening the server, restricting cron to root user only, LDAP sudoers, TCP wrappers, etc.?

[dhatha03]

cron is not restricted, selinux is disabled, server does connect to LDAP for local user authentication. I think I might try to rebuild the server and see if that helps. I will backup my existing configuration and then once rebuilt import it and see what happens.

[lmiltchev]

Let us know how it goes. We will keep the topic open for the time being in case you have more questions.

[BackNBlack]

Hi Guys,
We're going to hold off on rebuilding just yet. Thanks for the help, let's see where this new information leads us.

An update, the snmpget and snmpwalk entries have stopped but the php entries are constant and still persist.

I have the answers to your last questions:

Sudoers:

Code: Select all

#Defaults    requiretty
Defaults   !visiblepw

Directory permissions

Code: Select all

ls -lad /usr/local/nagios /usr/local/nagios/var/ /usr/local/nagios/var/rw/
drwxr-xr-x 8 root   root   4096 Sep 27  2018 /usr/local/nagios
drwxrwxr-x 6 nagios nagios 4096 Oct  1 14:16 /usr/local/nagios/var/
drwsr-xr-x 2 nagios nagios 4096 Oct  1 12:04 /usr/local/nagios/var/rw/

PHP version

Code: Select all

su nagios -c "/usr/bin/php -v"
PHP 5.3.3 (cli) (built: Mar 22 2017 12:27:09)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

[lmiltchev]

Your nagios directory is owned by root...

drwxr-xr-x 8 root root 4096 Sep 27 2018 /usr/local/nagios

Change the ownership to nagios.

Code: Select all

chown nagios.nagios /usr/local/nagios

The permissions of the "/usr/local/nagios/var/rw/" directory are also wrong.

drwsr-xr-x 2 nagios nagios 4096 Oct 1 12:04 /usr/local/nagios/var/rw/

They have to be:

drwxrwsr-x nagios nagios

Fix the permissions by running:

Code: Select all

chmod u-s,g+ws /usr/local/nagios/var/rw

Also, add the following line to sudoers:

Code: Select all

Defaults:nagios !requiretty

One last thing - you didn't show us the output of the command below as asked:

Code: Select all

ls -la /usr/local/nagios/var/rw/