New NagiosXI server apply configuration taking hours

[akepley]

I've fixed it and added back a few lines that absolutely do not affect nagios. I have tried the apply again and still no difference. The cmdsubsys.log still is not writing

Code: Select all

[email protected] var# ls -rlt
total 28
-rw-r--r-- 1 nagios nagios  196 Mar  6 09:43 xiversion
drwxr-xr-x 2 nagios nagios 4096 Mar  6 09:43 upgrades
-rw-r--r-- 1 nagios nagios  950 Mar  6 09:43 xi-sys.cfg
-rw-r--r-- 1 nagios nagios   37 Mar  6 09:46 xi-uuid
-rw-r--r-- 1 nagios nagios    7 Mar  6 09:46 xi-itype
drwsrwsr-x 2 apache nagios 4096 Mar 20 16:41 components
-rw------- 1 nagios nagios    0 Mar 20 16:52 cmdsubsys.log
drwxr-xr-x 2 nagios nagios 4096 Mar 20 18:24 subsys
[email protected] var# tail -f cmdsubsys.log

I've also noticed other issues. The component status is showing a lot of offline components (see attached NagiosXI Component status.PNG) and a force immediate check on any host or service results in a prompt (NagiosXI check error.PNG)

[avandemore]

My guess is that you still have sudoers issues. What is your current iteration of it?

[akepley]

Code: Select all

## Sudoers allows particular users to run various commands as
## the root user, without needing the root password.
##
## Examples are provided at the bottom of the file for collections
## of related commands, which can then be delegated out to particular
## users or groups.
##
## This file must be edited with the 'visudo' command.

## Host Aliases
## Groups of machines. You may prefer to use hostnames (perhaps using
## wildcards for entire domains) or IP addresses instead.
# Host_Alias     FILESERVERS = fs1, fs2
# Host_Alias     MAILSERVERS = smtp, smtp2

## User Aliases
## These aren't often necessary, as you can use regular groups
## (ie, from files, LDAP, NIS, etc) in this file - just use %groupname
## rather than USERALIAS
# User_Alias ADMINS = jsmith, mikem


## Command Aliases
## These are groups of related commands...

## Networking
# Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

## Installation and management of software
# Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum

## Services
# Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig

## Updating the locate database
# Cmnd_Alias LOCATE = /usr/bin/updatedb

## Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
# Cmnd_Alias DRIVERS = /sbin/modprobe

# Defaults specification

#
# Refuse to run if unable to disable echo on the tty.
#
Defaults   !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
#
Defaults    always_set_home

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults   env_keep += "HOME"

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
##      user    MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL

## Allows members of the 'sys' group to run networking, software,
## service management apps and more.
# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands
%wheel        ALL=(ALL)       ALL

## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL

## Allows members of the users group to mount and unmount the
## cdrom as root
# %users  ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom

## Allows members of the users group to shutdown this system
# %users  localhost=/sbin/shutdown -h now

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d
cloud-user      ALL=(ALL)       NOPASSWD: ALL
ec2-user        ALL=(ALL)       NOPASSWD: ALL

User_Alias      NAGIOSXI=nagios
User_Alias              NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

[akepley]

Here is the diff. The only differences are the two AWS users that are required to be available and the wheel group which is required in our environment for admin

Code: Select all

[email protected] ~# diff /root/sudoers /etc/sudoers
98c98
< # %wheel        ALL=(ALL)       ALL
---
> %wheel        ALL=(ALL)       ALL
112c112,113
<
---
> cloud-user      ALL=(ALL)       NOPASSWD: ALL
> ec2-user        ALL=(ALL)       NOPASSWD: ALL
146a148
>

[tgriep]

All of the errors in the component status screen is cause by the cron daemon not running on the server.
The cron daemon is needed to run those components and it can be started by running this as root.

Code: Select all

service crond start

It it fails to start, look on the /var/log/cron log file for any errors and post them here so we can further troubleshoot the issue.

[WillemDH]

I feel like this is the same issue I had, see https://support.nagios.com/forum/viewto ... =6&t=42057
Also on fresh Linode server. Never found the solution.. mostly due to lack of time I guess.

[akepley]

Found the issue looking at the cron log. The nagios user was expired. I used chage to disable expiration and restarted the server. The apply configuration now runs quickly.

Only issue I can see now on the component list is the database maintenance, but maybe that is because my db is not local?

[akepley]

So I changed a couple of service configs this morning and applied. While the web page says it was successful, when I go into the service it still has a status of "Not Applied" and NagiosXI has a banner saying I need to apply my configuration.

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
OUTPUT=Starting nagios: done.
RETURNCODE=0

PROCESSED 2 COMMANDS


tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated


.......................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.export.monitoring”

0K .......... .......... . 706K=0.03s

2017-03-22 06:48:08 (706 KB/s) - “nagiosql.export.monitoring” saved [22317]

WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.2.4
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 12-07-2016
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Warning: Duplicate definition found for service 'Ping' on host 'Gateway AS400 System' (config file '/usr/local/nagios/etc/services/Gateway_ASstarting on line 16)
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 1228 services.
Checked 174 hosts.
Checked 18 host groups.
Checked 0 service groups.
Checked 13 contacts.
Checked 7 contact groups.
Checked 130 commands.
Checked 26 time periods.
Checked 336 host escalations.
Checked 1768 service escalations.
Checking for circular paths...
Checked 174 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 26 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
OUTPUT=Starting nagios: done.
RETURNCODE=0

PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.............................^C

[avandemore]

Does the database maintenance still reflect a bad state or is the "Not Applied" the only issue now?

[akepley]

Both are issues. The apply not taking effect is the bigger one for me. I don't know what hte database maintenance does, nor how it works with a remote database.