Error Backing up

[matt.lilek]

Is this the reason why my local backups wont run and when i try to do a manual run it just fill my disk and crashes my system? Please take a look at the screen shot and let me know what i need to run to get permissions back to normal.

Thank you,

Matt

[ssax]

This is likely related to the free space while creating the temporary files for the backup filling up the filesystem.

How much free space do you have left currently?

Code: Select all

df -h
df -i

Additionally, the crashed tables messages are likely what's causing the issue (the password error will come up if that happens).

If you have enough free space, please repair the DBs:

https://assets.nagios.com/downloads/nag ... tabase.pdf

Please send me the output of this command:
- Adjust the 127.0.0.1 IP, username, and password if needed

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table -h 127.0.0.1

What do you have set for the 3 optimize intervals on the Admin > Performance Settings > Databases tab?

[matt.lilek]

Hey Sean!

So i actually got this when trying to upgrade through the GUI a while back but thought this is giving insight to why the local backups were not happening. So not a space issue or anything like that as you can see, just permissions i think.

[root@ATT5-NAG1 mlilek]# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagi osxi --table -h 10.200.10.85
ERROR 1130 (HY000): Host 'att5-nag1.global.amec.com' is not allowed to connect to this MySQL server

[root@ATT5-NAG1 mlilek]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
77G 50G 24G 68% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 477M 138M 315M 31% /boot
tmpfs 500M 20M 481M 4% /var/nagiosramdisk
[root@ATT5-NAG1 mlilek]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
5037552 177182 4860370 4% /
tmpfs 1006664 1 1006663 1% /dev/shm
/dev/sda1 128016 62 127954 1% /boot
tmpfs 1023176 11 1023165 1% /var/nagiosramdisk
[root@ATT5-NAG1 mlilek]#

[npolovenko]

@matt.lilek, Please run this command:

mysqlcheck -r -f -uroot -pnagiosxi --all-databases --use_frm

And then perform the manual upgrade from the command line:
https://assets.nagios.com/downloads/nag ... ctions.pdf
If upgrade fails please upload the /tmp/nagiosxi/upgrade.log file.

[matt.lilek]

Hello,
I think the point is being missed here, i have no problem with upgrading. The issue is there is no Local Backup Archive even though Local Backups are Enabled and scheduled to run daily. The only reason why i provided the original screen shot was to point to a possible root cause for the issue. None the less here is the output to what you asked for but if we could get to focusing on the problem i just re-described that would be greatly appreciated.

Thank you,


[root@att1-nag1 mlilek]# mysqlcheck -r -f -uroot -pnagiosxi --all-databases --use_frm
mysql.columns_priv OK
mysql.db
warning : Number of rows changed from 0 to 4
status : OK
mysql.event OK
mysql.func OK
mysql.general_log OK
mysql.help_category
warning : Number of rows changed from 0 to 40
status : OK
mysql.help_keyword
warning : Number of rows changed from 0 to 453
status : OK
mysql.help_relation
warning : Number of rows changed from 0 to 1010
status : OK
mysql.help_topic
warning : Number of rows changed from 0 to 509
status : OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user
warning : Number of rows changed from 0 to 7
status : OK
nagios.nagios_acknowledgements
warning : Number of rows changed from 0 to 37
status : OK
nagios.nagios_commands
warning : Number of rows changed from 0 to 109
status : OK
nagios.nagios_commenthistory
warning : Number of rows changed from 0 to 2175
status : OK
nagios.nagios_comments
warning : Number of rows changed from 0 to 31
status : OK
nagios.nagios_configfiles
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_configfilevariables
warning : Number of rows changed from 0 to 137
status : OK
nagios.nagios_conninfo
warning : Number of rows changed from 0 to 563
status : OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands
warning : Number of rows changed from 0 to 1056
status : OK
nagios.nagios_contactgroup_members
warning : Number of rows changed from 0 to 88
status : OK
nagios.nagios_contactgroups
warning : Number of rows changed from 0 to 2
status : OK
nagios.nagios_contactnotificationmethods
warning : Number of rows changed from 0 to 39994
status : OK
nagios.nagios_contactnotifications
warning : Number of rows changed from 0 to 39994
status : OK
nagios.nagios_contacts
warning : Number of rows changed from 0 to 88
status : OK
nagios.nagios_contactstatus
warning : Number of rows changed from 0 to 88
status : OK
nagios.nagios_customvariables
warning : Number of rows changed from 0 to 1833
status : OK
nagios.nagios_customvariablestatus
warning : Number of rows changed from 0 to 1833
status : OK
nagios.nagios_dbversion OK
nagios.nagios_downtimehistory
warning : Number of rows changed from 0 to 1680
status : OK
nagios.nagios_eventhandlers OK
nagios.nagios_externalcommands
warning : Number of rows changed from 0 to 9
status : OK
nagios.nagios_flappinghistory
warning : Number of rows changed from 0 to 907
status : OK
nagios.nagios_host_contactgroups
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_host_contacts
warning : Number of rows changed from 0 to 5835
status : OK
nagios.nagios_host_parenthosts
warning : Number of rows changed from 0 to 825
status : OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members
warning : Number of rows changed from 0 to 3250
status : OK
nagios.nagios_hostgroups
warning : Number of rows changed from 0 to 68
status : OK
nagios.nagios_hosts
warning : Number of rows changed from 0 to 950
status : OK
nagios.nagios_hoststatus
warning : Number of rows changed from 0 to 950
status : OK
nagios.nagios_instances
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_logentries OK
nagios.nagios_notifications OK
nagios.nagios_objects
warning : Number of rows changed from 0 to 3204
status : OK
nagios.nagios_processevents
warning : Number of rows changed from 0 to 3308
status : OK
nagios.nagios_programstatus
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_runtimevariables
warning : Number of rows changed from 0 to 18
status : OK
nagios.nagios_scheduleddowntime OK
nagios.nagios_service_contactgroups
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_service_contacts
warning : Number of rows changed from 0 to 4209
status : OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members OK
nagios.nagios_servicegroups OK
nagios.nagios_services
warning : Number of rows changed from 0 to 885
status : OK
nagios.nagios_servicestatus
warning : Number of rows changed from 0 to 885
status : OK
nagios.nagios_statehistory
warning : Number of rows changed from 0 to 37444
status : OK
nagios.nagios_systemcommands OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges
warning : Number of rows changed from 0 to 635
status : OK
nagios.nagios_timeperiods
warning : Number of rows changed from 0 to 94
status : OK
nagiosql.tbl_command
warning : Number of rows changed from 0 to 113
status : OK
nagiosql.tbl_contact
warning : Number of rows changed from 0 to 87
status : OK
nagiosql.tbl_contactgroup
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_contacttemplate
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_domain
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_host
warning : Number of rows changed from 0 to 950
status : OK
nagiosql.tbl_hostdependency OK
nagiosql.tbl_hostescalation OK
nagiosql.tbl_hostextinfo OK
nagiosql.tbl_hostgroup
warning : Number of rows changed from 0 to 68
status : OK
nagiosql.tbl_hosttemplate
warning : Number of rows changed from 0 to 43
status : OK
nagiosql.tbl_info
warning : Number of rows changed from 0 to 218
status : OK
nagiosql.tbl_lnkContactToCommandHost
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContactToCommandService
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContactToContactgroup OK
nagiosql.tbl_lnkContactToContacttemplate
warning : Number of rows changed from 0 to 87
status : OK
nagiosql.tbl_lnkContactToVariabledefinition
warning : Number of rows changed from 0 to 172
status : OK
nagiosql.tbl_lnkContactgroupToContact
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContactgroupToContactgroup OK
nagiosql.tbl_lnkContacttemplateToCommandHost
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkContacttemplateToCommandService
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkContacttemplateToContactgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkContacttemplateToContacttemplate OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact
warning : Number of rows changed from 0 to 5835
status : OK
nagiosql.tbl_lnkHostToContactgroup OK
nagiosql.tbl_lnkHostToHost
warning : Number of rows changed from 0 to 825
status : OK
nagiosql.tbl_lnkHostToHostgroup
warning : Number of rows changed from 0 to 1045
status : OK
nagiosql.tbl_lnkHostToHosttemplate
warning : Number of rows changed from 0 to 950
status : OK
nagiosql.tbl_lnkHostToVariabledefinition
warning : Number of rows changed from 0 to 949
status : OK
nagiosql.tbl_lnkHostdependencyToHost_DH OK
nagiosql.tbl_lnkHostdependencyToHost_H OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H OK
nagiosql.tbl_lnkHostescalationToContact OK
nagiosql.tbl_lnkHostescalationToContactgroup OK
nagiosql.tbl_lnkHostescalationToHost OK
nagiosql.tbl_lnkHostescalationToHostgroup OK
nagiosql.tbl_lnkHostgroupToHost
warning : Number of rows changed from 0 to 26
status : OK
nagiosql.tbl_lnkHostgroupToHostgroup
warning : Number of rows changed from 0 to 79
status : OK
nagiosql.tbl_lnkHosttemplateToContact OK
nagiosql.tbl_lnkHosttemplateToContactgroup
warning : Number of rows changed from 0 to 4
status : OK
nagiosql.tbl_lnkHosttemplateToHost OK
nagiosql.tbl_lnkHosttemplateToHostgroup OK
nagiosql.tbl_lnkHosttemplateToHosttemplate
warning : Number of rows changed from 0 to 41
status : OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition
warning : Number of rows changed from 0 to 3
status : OK
nagiosql.tbl_lnkServiceToContact
warning : Number of rows changed from 0 to 4209
status : OK
nagiosql.tbl_lnkServiceToContactgroup OK
nagiosql.tbl_lnkServiceToHost
warning : Number of rows changed from 0 to 885
status : OK
nagiosql.tbl_lnkServiceToHostgroup OK
nagiosql.tbl_lnkServiceToServicegroup OK
nagiosql.tbl_lnkServiceToServicetemplate
warning : Number of rows changed from 0 to 888
status : OK
nagiosql.tbl_lnkServiceToVariabledefinition
warning : Number of rows changed from 0 to 884
status : OK
nagiosql.tbl_lnkServicedependencyToHost_DH OK
nagiosql.tbl_lnkServicedependencyToHost_H OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H OK
nagiosql.tbl_lnkServicedependencyToService_DS OK
nagiosql.tbl_lnkServicedependencyToService_S OK
nagiosql.tbl_lnkServiceescalationToContact OK
nagiosql.tbl_lnkServiceescalationToContactgroup OK
nagiosql.tbl_lnkServiceescalationToHost OK
nagiosql.tbl_lnkServiceescalationToHostgroup OK
nagiosql.tbl_lnkServiceescalationToService OK
nagiosql.tbl_lnkServicegroupToService OK
nagiosql.tbl_lnkServicegroupToServicegroup OK
nagiosql.tbl_lnkServicetemplateToContact OK
nagiosql.tbl_lnkServicetemplateToContactgroup
warning : Number of rows changed from 0 to 1
status : OK
nagiosql.tbl_lnkServicetemplateToHost OK
nagiosql.tbl_lnkServicetemplateToHostgroup OK
nagiosql.tbl_lnkServicetemplateToServicegroup OK
nagiosql.tbl_lnkServicetemplateToServicetemplate
warning : Number of rows changed from 0 to 67
status : OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_lnkTimeperiodToTimeperiod OK
nagiosql.tbl_logbook
warning : Number of rows changed from 0 to 81
status : OK
nagiosql.tbl_mainmenu
warning : Number of rows changed from 0 to 7
status : OK
nagiosql.tbl_service
warning : Number of rows changed from 0 to 885
status : OK
nagiosql.tbl_servicedependency OK
nagiosql.tbl_serviceescalation OK
nagiosql.tbl_serviceextinfo OK
nagiosql.tbl_servicegroup OK
nagiosql.tbl_servicetemplate
warning : Number of rows changed from 0 to 69
status : OK
nagiosql.tbl_settings
warning : Number of rows changed from 0 to 12
status : OK
nagiosql.tbl_submenu
warning : Number of rows changed from 0 to 29
status : OK
nagiosql.tbl_timedefinition
warning : Number of rows changed from 0 to 641
status : OK
nagiosql.tbl_timeperiod
warning : Number of rows changed from 0 to 93
status : OK
nagiosql.tbl_user
warning : Number of rows changed from 0 to 2
status : OK
nagiosql.tbl_variabledefinition
warning : Number of rows changed from 0 to 3624
status : OK

[npolovenko]

Looks like you ran the command that Sean gave you with a space in the MySQL password. -pnagi osxi

[root@ATT5-NAG1 mlilek]# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagi osxi --table -h 10.200.10.85
ERROR 1130 (HY000): Host 'att5-nag1.global.amec.com' is not allowed to connect to this MySQL server

Thats probably why it couldn't connect.

Please run this command from the command line and show me the output:

su - nagios
/usr/local/nagiosxi/scripts/backup_xi.sh

Another good troubleshooting step is to run the following command in the XI ssh terminal while the local backup is supposed to be generated(according to the scheduled time in the GUI):

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

[matt.lilek]

Hey seems to be taking a while here as this is what it shows and its been well over a half hour. What are the chances we can get a remote going to dig into this? Please let me know.

Thank you,

Matt

[nagios@ATT5-NAG1 ~]$ /usr/local/nagiosxi/scripts/backup_xi.sh
Password:
su: incorrect password
ERROR: Could not create or update '/usr/local/nagios/var/nagios.configtest'
Backing up NagiosQL...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var: file changed as we read it
tar: /usr/local/nagios/share/perfdata/mdr-wan-rt1.global.amec.com: file changed as we read it
tar: /usr/local/nagios/share/perfdata/att5-ccfa1.global.amec.com: file changed a s we read it

[npolovenko]

@matt.lilek, We can do that.
*We're technically required to have a support ticket to schedule a remote session. Please open one here and include the link to this forum thread in the description.
https://support.nagios.com/tickets/
Thank you!