Auto-discovery error: "XML was not valid"

[mvndnburg]

Here's fstab:

/dev/mapper/vg_hostredacted-lv_root / ext4 defaults 1 1
UUID=0067e743-22a6-4ec2-be0a-cace4xxxxxx /boot ext4 defaults,nodev,noexec,nosuid 1 2
/dev/mapper/vg_hostredacted-lv_home /home ext4 acl,nodev 1 2
/dev/mapper/vg_hostredacted-lv_opt /opt ext4 acl,nodev 1 2
/dev/mapper/vg_hostredacted-lv_tmp /tmp ext4 acl,nodev 1 2
/dev/mapper/vg_hostredacted-lv_usr /usr ext4 acl,nodev 1 2
/dev/mapper/vg_hostredacted-lv_var /var ext4 acl,nodev 1 2
/dev/mapper/vg_hostredacted-lv_var_log /var/log ext4 nodev,nosuid,noexec,acl 1 2
/dev/mapper/vg_hostredacted-lv_var_log_audit /var/log/audit ext4 nodev,nosuid,noexec,acl 1 2
/dev/mapper/vg_hostredacted-lv_swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs nodev,nosuid,noexec,size=90% 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/tmp /var/tmp none bind 0 0
/tmp /var/tmp none bind 0 0

[dwhitfield]

Good news. 5.3.3 came complete with an offline install yesterday: http://repo.nagios.com/

Do you mind giving that a shot? If you don't want to give it a shot, it might be time to open a ticket by emailing [email protected].

[mvndnburg]

Good news. 5.3.3 came complete with an offline install yesterday: http://repo.nagios.com/

Do you mind giving that a shot? If you don't want to give it a shot, it might be time to open a ticket by emailing [email protected].

Good to know there's progress in that area

Being new to Nagios, we have never done an upgrade before. The offline installation required some tweaking to get things right. For example: changed credentials for MySQL, remote MySQL host, create home directory for the nagios user, set SElinux context, etc. etc. And we're not fully done yet (configuring reception of SNMP traps, to just name one item).

The bottom line is: what does an update do to the existing configurations? How are configuration differences managed? WIll we have to redo all that work?

[dwhitfield]

Please note:

This procedure is NOT intended to upgrade a system that was originally installed online or started as a pre-created VM and then moved offline.

That said, you'll see upgrade instructions on page 2 of https://assets.nagios.com/downloads/nag ... onment.pdf

Our backup scripts back up the following:

• Nagios Core files (/usr/local/nagios)
• Nagios XI files (/usr/local/nagiosxi)
• NagiosQL files (/var/www/html/nagiosql and /etc/nagiosql)
• Select Apache config files (in /etc/httpd/conf.d)
• Select logrotate config files (in /etc/logrotate.d)
• Select MySQL databases (nagios and nagiosql)

If you have configurations elsewhere, you will need to back them up separately. Generally speaking, the database should take care of things. We've already got your profile, so if you do run into issues, we should be able to help you get things up and running pretty quickly.

Additional information on the backup/restore process is at https://assets.nagios.com/downloads/nag ... ios-XI.pdf.

Given the US holiday Thursday and us closing early tomorrow, my suggestion would be to wait to try the upgrade on Monday. Ultimately, it's your call. We'll be here another 5.5+ hours today.

[mvndnburg]

Hi,

It's me again.
I performed the upgrade from 5.2.9 from 5.3.3 and it looks like it went without errors.

Now when starting a new Autodiscovery job, it stays 'hanging' in the GUI (little spinner icon) and the following message appeared in the HTTPD error log:

[Fri Dec 16 14:14:12 2016] [error] [client 10.132.164.185] PHP Warning: filemtime(): stat failed for /usr/local/nagiosxi/html/includes/components/autodiscovery/jobs/aSe6AG.out in /usr/local/nagiosxi/html/includes/components/autodiscovery/index.php on line 280, referer: https://<deleted>/nagiosxi/includes/components/autodiscovery/index.php?mode=newjob

No new files are created in that directory (or - they were deleted by the script). The permissions

on the folder were:

[root@deleted jobs]# ls -laZ /usr/local/nagiosxi/html/includes/components/autodiscovery/jobs/
drwxrwx---. nagios nagios system_u:object_r:usr_t:s0 .
drwxr-x---. nagios nagios system_u:object_r:usr_t:s0 ..
-rw-r-----. nagios nagios system_u:object_r:usr_t:s0 b2a27b.out
-rw-r-----. nagios nagios system_u:object_r:usr_t:s0 b2a27b.watch
-rw-r-----. nagios nagios system_u:object_r:usr_t:s0 b2a27b.xml

I changed the SElinux context of the 'autodiscovery' directory recursively, with

chcon -R --reference=/var/www/html /usr/local/nagiosxi/html/includes/components/autodiscovery/jobs/

But that did not solved the problem. The HTTPD error log shows:

[Fri Dec 16 14:14:12 2016] [error] [client 10.132.164.185] PHP Warning: filemtime(): stat failed for /usr/local/nagiosxi/html/includes/components/autodiscovery/jobs/aSe6AG.out in /usr/local/nagiosxi/html/includes/components/autodiscovery/index.php on line 280, referer: https://<deleted>/nagiosxi/includes/components/autodiscovery/index.php?mode=newjob
[Fri Dec 16 14:24:33 2016] [error] [client 10.132.164.185] PHP Warning: filemtime(): stat failed for /usr/local/nagiosxi/html/includes/components/autodiscovery/jobs/gmIRLS.out in /usr/local/nagiosxi/html/includes/components/autodiscovery/index.php on line 280, referer: https://<deleted>/nagiosxi/includes/components/autodiscovery/index.php?mode=newjob

The current context is:

[root@srtadinf0245 jobs]# pwd
/usr/local/nagiosxi/html/includes/components/autodiscovery/jobs
[root@srtadinf0245 jobs]# ls -laZ .
drwxrwx---. nagios nagios system_u:object_r:httpd_sys_rw_content_t:s0 .
drwxr-x---. nagios nagios system_u:object_r:httpd_sys_rw_content_t:s0 ..
-rw-r-----. nagios nagios system_u:object_r:httpd_sys_rw_content_t:s0 b2a27b.out
-rw-r-----. nagios nagios system_u:object_r:httpd_sys_rw_content_t:s0 b2a27b.watch
-rw-r-----. apache apache system_u:object_r:httpd_sys_rw_content_t:s0 s0ZiuH.out
-rw-r-----. apache apache system_u:object_r:httpd_sys_rw_content_t:s0 s0ZiuH.watch

Note that the s0ZiuH files (today's) are created by 'apache'. Can you give a hint as to what the proper SElinux context should be on the autodiscovery tree?

[dwhitfield]

What's the output of getenforce? Did this issue come about immediately following the upgrade? Could you go ahead and post your upgrade.log? What version of the auto-discovery component are you using now?

Does it work if you disable SELinux?

Code: Select all

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

We can try to get your system running with SELinux, but it's technically not supported. As of the moment though, I just want to disable SELinux as a troubleshooting step.

https://fportase.wordpress.com/selinux- ... x-enabled/ was written for Core, but it was written for RHEL 6, so you might find it of use.

Also, just so we have a complete picture of what is going on, can you PM me your 5.3.3 Profile? You can download it by going to Admin > System Config > System Profile and click the Download Profile button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+*, Show Profile if older) in the thread (that will at least get us some info).

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

*no offline installer for 5.3.4 yet and if I had to guess, there won't be.

[mvndnburg]

SElinux is enabled, the output of getenforce is

Code: Select all

Enforcing

. This has been the case since the initial installation, and also after the upgrade to 5.3.3.

When I switch SElinux to permissive and run the discovery job it does run and generates the .watch, .out and .xml files.
The .watch and .out files are owned by apache.apache. The .xml file is owned by root.root, like before.

When I change the ownership of the .xml file to apache.apache and click on Refresh job list, the finished job becomes visible and I can check its output.
When I leave the ownership at root.root and change the file permissions to 644 (from 640) and click on Refresh job list, the finished job becomes visible as well, and I can check its output.

So there seem to be two things going on:

- SElinux blocks the discovery job when in Enforcing mode.
- when SElinux is in permissive mode the autodiscovery job is not blocked, but one of the three output files is generated with a combination of ownership/file permissions that make it unreadable.

The content of the .out file is:
sudo: unable to open audit system: Permission denied

Can you give me a pointer to the name of script that does the autodiscovery? I'll check it to see what it is that it wants to do.

[tgriep]

It may be a missing sudo entry in the /etc/sudoers file.
Check it and see if the following 2 lines are in it.

Code: Select all

NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *

Also, if you disable selinux, are the permissions of the files correct?

[bheden]

The script you're looking for is in /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php.

Unfortunately, this script itself is source protected. The good news is that it calls a few other commands. The script that is called via this script is /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/run_fping.

Essentially, the call order is (loosely):

• autodiscover_new.php
  gather IPs with run_fping [/usr/sbin/fping] via a PHP popen() call
  *magic happens to ignore some old ips, etc.*
  nmap [/usr/bin/nmap] is ran to gather specific info related to each IP

I hope this is enough information for you!

[mvndnburg]

It may be a missing sudo entry in the /etc/sudoers file.
Check it and see if the following 2 lines are in it.

Code: Select all

NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *

Also, if you disable selinux, are the permissions of the files correct?

Those lines are in a local sudoers file, which is #included by /etc/sudoers:

Code: Select all

NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *

Also, if you disable selinux, are the permissions of the files correct?

No. The xml is owned by root.root, with 640. Should be more open.
The other two files are owned by apache.apache, with 640.