Nagios Support Forum

Hello,

We've run into a serious issue with your upgrade scrip for 2.1.3 and 2.1.4:

When running the upgrade script for either of these versions, the following commands are executed:
Please note the line deleting the system directory.

As the script continues, it tries to start PHP:
At this point, the upgrade fails.

When looking at /var/www/html/nagioslogserver/www/index.php, we see the following:

$system_path = dirname(__FILE__).'/../system';

As you'll note, this path contains the very directory the script previously deleted.

Now, we can't go forward with the uprgade nor can we fall back to the previous version. Our production log monitoring is down.

How do we fix this ASAP?

-- Mike Beebe

Hello,

We were able to fix this by dissecting the script and running the steps by hand to restore the missing files.

Now we can't log in to the instance. Trying to figure that out now.

-- Mike Beebe

Can you confirm for us that it was 2.1.3 that you were coming from?

Also can you send in a system profile and the upgrade.log file? The system profile will help us get you logged in and running again. And while you're grabbing data, if you could run tcpdump -i <interface_name> -w /tmp/output.pcap and then try to login, that could also be helpful.

I looked at the code, and the section for removing the system directory in the upgrade script goes back as far a 2.0.0, and the inclusion of the system directory in $system_path in index.php goes back further.

mbellerue wrote:Can you confirm for us that it was 2.1.3 that you were coming from?

Also can you send in a system profile and the upgrade.log file? The system profile will help us get you logged in and running again. And while you're grabbing data, if you could run tcpdump -i <interface_name> -w /tmp/output.pcap and then try to login, that could also be helpful.

I looked at the code, and the section for removing the system directory in the upgrade script goes back as far a 2.0.0, and the inclusion of the system directory in $system_path in index.php goes back further.

I'm attaching the upgrade.log to this message.

How do I get you the system profile?

-- Mike Beebe

Thank you for that. And my apologies, to get a profile from the command line run /usr/local/nagioslogserver/scripts/profile.sh 20200203 and it will place the system profile in /tmp.

mbellerue wrote:Thank you for that. And my apologies, to get a profile from the command line run /usr/local/nagioslogserver/scripts/profile.sh 20200203 and it will place the system profile in /tmp.

Please find attached to this message the system profile you requested.

-- Mike Beebe

Please find attached to this message the TCP dump that you requested.

Can you give me the output of the following commands,

ls -lh /var/www/html/nagioslogserver/:

total 16K
drwxr-xr-x 17 apache apache 4.0K Feb 3 12:56 application
-rw-r--r-- 1 root root 13 Feb 3 12:56 lsversion
drwxr-xr-x 8 apache apache 4.0K Dec 11 12:17 system
drwxr-xr-x 9 apache apache 4.0K Feb 3 12:56 www

/etc/sudoers (in our case, we source the Nagios permissions from its own file in /etc/sudoers.d):

[root@xxxxxx sudoers.d]# cat 10_nagios
# This file is managed by Puppet; changes may be overwritten
User_Alias NAGIOSXI=nagios
User_Alias NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_ssl_config.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/scripts/components/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/components/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

# NEEDED TO ALLOW NAGIOS TO CHECK SERVICE STATUS
Defaults:nagios !requiretty
nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_init_service

# ASTERISK-SPECIFIC CHECKS
# NOTE: You can uncomment the following line if you are monitoring Asterisk locally
#nagios ALL=NOPASSWD: /usr/local/nagios/libexec/check_asterisk_sip_peers.sh, /usr/local/nagios/libexec/nagisk.pl, /usr/sbin/asterisk

Defaults:svvsradm !requiretty
User_Alias NAGIOSLOGSERVER=nagios
User_Alias NAGIOSLOGSERVERWEB=apache
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl start httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl stop httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl restart httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl reload httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/bin/systemctl status httpd
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/change_timezone.sh
NAGIOSLOGSERVER ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/reconfigure_ncpa.sh
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_listener status
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive start
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive stop
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive restart
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive reload
NAGIOSLOGSERVER ALL = NOPASSWD:/etc/init.d/ncpa_passive status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/logstash status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/elasticsearch status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl start httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl stop httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl restart httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl reload httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/bin/systemctl status httpd
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/get_logstash_ports.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/profile.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/usr/local/nagioslogserver/scripts/reconfigure_ncpa.sh
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_listener status
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive start
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive stop
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive restart
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive reload
NAGIOSLOGSERVERWEB ALL = NOPASSWD:/etc/init.d/ncpa_passive status

grep nagios /etc/groups:

grep nagios /etc/group
nagios165:nagios,apache,s609150
apache48:nagios
nagcmd5003:nagios,apache

systemctl list-units --type=service
UNIT LOAD ACTIVE SUB DESCRIPTION
ajaxterm.service loaded active running SYSV: Ajaxterm is a web-based terminal
auditd.service loaded active running Security Auditing Service
crond.service loaded active running Command Scheduler
dbus.service loaded active running D-Bus System Message Bus
dm-event.service loaded active running Device-mapper event daemon
elasticsearch.service loaded active running LSB: This service manages the elasticsearch daemon
[email protected] loaded active running Getty on tty1
gssproxy.service loaded active running GSSAPI Proxy Daemon
httpd.service loaded active running The Apache HTTP Server
isecespd.service loaded active running McAfee Endpoint Security Platform for Linux
isectpd.service loaded active running McAfee Endpoint Security for Linux Threat Prevention
kdump.service loaded active exited Crash recovery kernel arming
kmod-static-nodes.service loaded active exited Create list of required static device nodes for the current kernel
logstash.service loaded active running LSB: Logstash
lvm2-lvmetad.service loaded active running LVM2 metadata daemon
lvm2-pvscan@8:113.service loaded active exited LVM2 PV scan on device 8:113
lvm2-pvscan@8:129.service loaded active exited LVM2 PV scan on device 8:129
lvm2-pvscan@8:17.service loaded active exited LVM2 PV scan on device 8:17
lvm2-pvscan@8:2.service loaded active exited LVM2 PV scan on device 8:2
lvm2-pvscan@8:33.service loaded active exited LVM2 PV scan on device 8:33
lvm2-pvscan@8:48.service loaded active exited LVM2 PV scan on device 8:48
lvm2-pvscan@8:65.service loaded active exited LVM2 PV scan on device 8:65
lvm2-pvscan@8:81.service loaded active exited LVM2 PV scan on device 8:81
lvm2-pvscan@8:97.service loaded active exited LVM2 PV scan on device 8:97
mariadb.service loaded active running MariaDB database server
mcafee.ma.service loaded active running McAfee Agent
monitor.service loaded active exited SYSV: sia
ncpa_listener.service loaded active running LSB: This manages the NCPA Listener service
ncpa_passive.service loaded active running LSB: This manages the NCPA Passive service
netbackup.service loaded active running LSB: NetBackup
network.service loaded active exited LSB: Bring up/down networking
NetworkManager-wait-online.service loaded active exited Network Manager Wait Online
NetworkManager.service loaded active running Network Manager
ntpd.service loaded active running Network Time Service
oddjobd.service loaded active running privileged operations for unprivileged applications
opsware-agent.service loaded active running Server Automation Agent Daemon
polkit.service loaded active running Authorization Manager
pxp-agent.service loaded active running PCP Execution Protocol (PXP) Agent
rhel-dmesg.service loaded active exited Dump dmesg to /var/log/dmesg
rhel-domainname.service loaded active exited Read and set NIS domainname from /etc/sysconfig/network
rhel-import-state.service loaded active exited Import network configuration from initramfs
rhel-readonly.service loaded active exited Configure read-only root support
rhnsd.service loaded active running LSB: Starts the Spacewalk Daemon
rhsmcertd.service loaded active running Enable periodic update of entitlement certificates.
rngd.service loaded active running Hardware RNG Entropy Gatherer Daemon
rpc-gssd.service loaded active running RPC security service for NFS client and server
rpc-statd.service loaded active running NFS status monitor for NFSv2/3 locking.
rpcbind.service loaded active running RPC bind service
rsyslog.service loaded active running System Logging Service
sendmail.service loaded active running Sendmail Mail Transport Agent
...skipping...
rhel-domainname.service loaded active exited Read and set NIS domainname from /etc/sysconfig/network
rhel-import-state.service loaded active exited Import network configuration from initramfs
rhel-readonly.service loaded active exited Configure read-only root support
rhnsd.service loaded active running LSB: Starts the Spacewalk Daemon
rhsmcertd.service loaded active running Enable periodic update of entitlement certificates.
rngd.service loaded active running Hardware RNG Entropy Gatherer Daemon
rpc-gssd.service loaded active running RPC security service for NFS client and server
rpc-statd.service loaded active running NFS status monitor for NFSv2/3 locking.
rpcbind.service loaded active running RPC bind service
rsyslog.service loaded active running System Logging Service
sendmail.service loaded active running Sendmail Mail Transport Agent
siasnmpd.service loaded active running LSB: start and stop Savvis SIA Net-SNMP daemon
sm-client.service loaded active running Sendmail Mail Transport Client
snmpd.service loaded active running Simple Network Management Protocol (SNMP) Daemon.
splunk.service loaded active exited SYSV: Splunk indexer service
sshd.service loaded active running OpenSSH server daemon
sssd.service loaded active running System Security Services Daemon
systemd-binfmt.service loaded active exited Set Up Additional Binary Formats
systemd-fsck-root.service loaded active exited File System Check on Root Device
systemd-fsck@dev-disk-by\x2duuid-1ef94454\x2dfd53\x2d4361\x2dbe1c\x2d1f23ceb786d3.service loaded active exited File System Check on /dev/disk/by-uuid/1ef94454-fd53-4361-be1c-1f23ceb786d3
systemd-fsck@dev-mapper-vg_root\x2dlv_data01.service loaded active exited File System Check on /dev/mapper/vg_root-lv_data01
systemd-fsck@dev-mapper-vg_root\x2dlv_tmp.service loaded active exited File System Check on /dev/mapper/vg_root-lv_tmp
systemd-fsck@dev-mapper-vg_root\x2dlv_usr.service loaded active exited File System Check on /dev/mapper/vg_root-lv_usr
systemd-fsck@dev-mapper-vg_root\x2dlv_var.service loaded active exited File System Check on /dev/mapper/vg_root-lv_var
systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-random-seed.service loaded active exited Load/Save Random Seed
systemd-remount-fs.service loaded active exited Remount Root and Kernel File Systems
systemd-sysctl.service loaded active exited Apply Kernel Variables
systemd-tmpfiles-setup-dev.service loaded active exited Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories
systemd-udev-trigger.service loaded active exited udev Coldplug all Devices
systemd-udevd.service loaded active running udev Kernel Device Manager
systemd-update-utmp.service loaded active exited Update UTMP about System Boot/Shutdown
systemd-user-sessions.service loaded active exited Permit User Sessions
systemd-vconsole-setup.service loaded active exited Setup Virtual Console
tuned.service loaded active running Dynamic System Tuning Daemon
vgauthd.service loaded active running VGAuth Service for open-vm-tools
vmtoolsd.service loaded active running Service for virtual machines hosted on VMware
vxpbx_exchanged.service loaded active running SYSV: Provides Symantec Private Branch Exchange
xagt.service loaded active running xagt
xinetd.service loaded active running Xinetd A Powerful Replacement For Inetd
z3_inst_opsw_agent.service loaded active exited SYSV: Installs Opsware agent.

Hello,

Is there any further update on this issue? Is there any additional information we can supply?

Thank you,

-- Mike Beebe