Unable to see login page

[timlee]

Dear support,

I have not been using the Nagios machine for sometime and when I go into the URL, it shows me this page

errorLogin.JPG

without seeing the login page.

I have tried to run the repair, however there was segmentation fault and mysqld couldn't start after.

runRepair.JPG

[jolson]

Let's take a look at your mysql logs:

Code: Select all

tail -n 30 /var/log/mysqld.log

Does mysql.sock exist?

Code: Select all

ls /var/lib/mysql/mysql.sock

SELinux is off?

Code: Select all

sestatus

Check on your disk space:

Code: Select all

df -h

Let us know the results of the above. Thanks!

[timlee]

Let's take a look at your mysql logs:

Code: Select all

tail -n 30 /var/log/mysqld.log

Does mysql.sock exist?

Code: Select all

ls /var/lib/mysql/mysql.sock

SELinux is off?

Code: Select all

sestatus

Check on your disk space:

Code: Select all

df -h

Let us know the results of the above. Thanks!

Hi Jolson,

Thanks for pointing out, I've run out of disk space. Is there a documentation on how I can archive or delete the old records?

Furthermore, I get these errors from the /var/log/messages:

Code: Select all

[root@NagiosCentOS6 ~]# tail /var/log/messages
Jul 20 16:12:58 NagiosCentOS6 abrt[2392]: Aborting dump: only 141MiB is available on /var/spool/abrt
Jul 20 16:12:58 NagiosCentOS6 abrt[2395]: Aborting dump: only 141MiB is available on /var/spool/abrt
Jul 20 16:15:29 NagiosCentOS6 snmptt-sys[1864]: MySQL error: Unable to connect to database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Jul 20 16:20:29 NagiosCentOS6 snmptt-sys[1864]: MySQL error: Unable to connect to database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)

[abrist]

Well, the tables should be truncated after the number of days specified in the Admin --> performance settings --> databases page. There really is no need to archive these rows as the data is always available in the nagios logs, and the old data is really only used for SLA/Availability reporting (which is parsed directly from the logs).
You need to hunt down where the disk usage is coming from - often we find the main culprits are system logs, XI backups, or unreapable spooler folders.
What is the output of:

Code: Select all

du -a / | sort -n -r | head -n 20
find / -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

[timlee]

Well, the tables should be truncated after the number of days specified in the Admin --> performance settings --> databases page. There really is no need to archive these rows as the data is always available in the nagios logs, and the old data is really only used for SLA/Availability reporting (which is parsed directly from the logs).
You need to hunt down where the disk usage is coming from - often we find the main culprits are system logs, XI backups, or unreapable spooler folders.
What is the output of:

Code: Select all

du -a / | sort -n -r | head -n 20
find / -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

Hi Abrist, there are quite a number of backup files. I have removed them from and stopped cron for now. However, I am not able to start nagios service.

Code: Select all

[root@NagiosCentOS6 ~]# service nagios status
No lock file found in /usr/local/nagios/var/nagios.lock
[root@NagiosCentOS6 ~]# /etc/rc.d/init.d/nagios restart
Running configuration check...
[root@NagiosCentOS6 ~]# /etc/rc.d/init.d/nagios reload
Running configuration check...
[root@NagiosCentOS6 ~]# ps axu | grep nagios
root     19317  0.0  0.0 103252   836 pts/1    S+   10:47   0:00 grep nagios
nagios   31023  0.0  0.0   8340   860 ?        S    10:41   0:00 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
nagios   31305  0.0  0.0  50296   656 ?        Ss   10:41   0:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg

Here is the info you requested

Code: Select all

[root@NagiosCentOS6 ~]# du -a / | sort -n -r | head -n 20
du: cannot access `/proc/6788/task/6788/fd/4': No such file or directory
du: cannot access `/proc/6788/task/6788/fdinfo/4': No such file or directory
du: cannot access `/proc/6788/fd/4': No such file or directory
du: cannot access `/proc/6788/fdinfo/4': No such file or directory
16750433        /
11259960        /store
11259956        /store/backups
11222932        /store/backups/nagiosxi
4259896 /usr
1619884 /usr/share
1290336 /usr/lib64
937080  /var
515732  /usr/lib
423912  /var/cache
422100  /usr/share/locale
419964  /var/cache/yum
419960  /var/cache/yum/x86_64
419956  /var/cache/yum/x86_64/6
418564  /usr/local
413684  /usr/lib64/libreoffice
285940  /usr/bin
270048  /var/lib
230828  /var/cache/yum/x86_64/6/updates
229988  /usr/lib64/libreoffice/program

Code: Select all

[root@NagiosCentOS6 ~]# find / -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find: `/proc/3284/task/3284/fd/5': No such file or directory
find: `/proc/3284/task/3284/fdinfo/5': No such file or directory
find: `/proc/3284/fd/5': No such file or directory
find: `/proc/3284/fdinfo/5': No such file or directory
du: cannot access `/proc/3304/task/3304/fd/4': No such file or directory
du: cannot access `/proc/3304/task/3304/fdinfo/4': No such file or directory
du: cannot access `/proc/3304/fd/4': No such file or directory
du: cannot access `/proc/3304/fdinfo/4': No such file or directory
du: cannot access `/proc/3308': No such file or directory
du: cannot access `/proc/3430/task/3430/fd/4': No such file or directory
du: cannot access `/proc/3430/task/3430/fdinfo/4': No such file or directory
du: cannot access `/proc/3430/fd/4': No such file or directory
du: cannot access `/proc/3430/fdinfo/4': No such file or directory
du: cannot access `/proc/3477': No such file or directory
du: cannot access `/proc/3304': No such file or directory
du: cannot access `/proc/3304/task': No such file or directory
du: cannot access `/proc/3304/task/3304': No such file or directory
du: cannot access `/proc/3304/task/3304/fd': No such file or directory
du: cannot access `/proc/3304/task/3304/fdinfo': No such file or directory
du: cannot access `/proc/3304/task/3304/ns': No such file or directory
du: cannot access `/proc/3304/task/3304/attr': No such file or directory
du: cannot access `/proc/3304/fd': No such file or directory
du: cannot access `/proc/3304/fdinfo': No such file or directory
du: cannot access `/proc/3304/ns': No such file or directory
du: cannot access `/proc/3304/net': No such file or directory
du: cannot access `/proc/3304/net/vlan': No such file or directory
du: cannot access `/proc/3304/net/dev_snmp6': No such file or directory
du: cannot access `/proc/3304/net/netfilter': No such file or directory
du: cannot access `/proc/3304/net/stat': No such file or directory
du: cannot access `/proc/3304/attr': No such file or directory
11G     /store/backups/nagiosxi
11G     /store/backups/nagiosxi
11G     /store/backups/nagiosxi
11G     /store/backups/nagiosxi
11G     /store/backups
11G     /store/backups
11G     /store/backups
11G     /store
11G     /store
du: cannot access `/proc/3968/task/3968/fd/4': No such file or directory
du: cannot access `/proc/3968/task/3968/fdinfo/4': No such file or directory
du: cannot access `/proc/3968/fd/4': No such file or directory
du: cannot access `/proc/3968/fdinfo/4': No such file or directory
du: cannot access `/proc/3971': No such file or directory
16G     /

Code: Select all

[root@NagiosCentOS6 ~]# find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find: `/proc/8425/task/8425/fd/5': No such file or directory
find: `/proc/8425/task/8425/fdinfo/5': No such file or directory
find: `/proc/8425/fd/5': No such file or directory
find: `/proc/8425/fdinfo/5': No such file or directory
find: `/proc/8465': No such file or directory
find: `/proc/8474': No such file or directory
du: cannot access `/proc/510/task/510/fdinfo/12': No such file or directory
du: cannot access `/proc/510/fdinfo/12': No such file or directory
du: cannot access `/proc/7102/task/7102/fdinfo/128': No such file or directory
du: cannot access `/proc/7102/fdinfo/128': No such file or directory
du: cannot access `/store/backups/nagiosxi/1435461903.tar.gz': No such file or directory
du: cannot access `/store/backups/nagiosxi/1435548302.tar.gz': No such file or directory
du: cannot access `/store/backups/nagiosxi/1435634703.tar.gz': No such file or directory
du: cannot access `/store/backups/nagiosxi/1435721103.tar.gz': No such file or directory
du: cannot access `/store/backups/nagiosxi/1435807503.tar.gz': No such file or directory
du: cannot access `/store/backups/nagiosxi/1435893902.tar.gz': No such file or directory
du: cannot access `/store/backups/nagiosxi/1435980303.tar.gz': No such file or directory
du: cannot access `/store/backups/nagiosxi/1436066702/mysql/nagios.sql': No such file or directory
du: cannot access `/usr/lib/locale/locale-archive': No such file or directory
du: cannot access `/var/cache/yum/x86_64/6/updates/05a18878446f93dd7c0a47880eda694778ed2bb7a412e2d5ada59be33ed841f6-other.sqlite': No such file or directory

[jolson]

Access your Nagios XI Web GUI and navigate to 'Configure -> Tools -> Write Config Files'. Once there, click the buttons in the following order:

1. Delete
2. Write
3. Verify
4. Restart

After doing so, run a service nagios status on the CLI and ensure that nagios is running. If not, you may have to start it with service nagios start.

Best,


Jesse

[timlee]

Access your Nagios XI Web GUI and navigate to 'Configure -> Tools -> Write Config Files'. Once there, click the buttons in the following order:

1. Delete
2. Write
3. Verify
4. Restart

After doing so, run a service nagios status on the CLI and ensure that nagios is running. If not, you may have to start it with service nagios start.

Best,


Jesse

Hi Jesse,
I didn't see the 'Configure -> Tools -> Write Config Files' can I find it in the free version?

Anyways, I have deleted the backups from trash to free up some space and now nagios is back to normal.

[jolson]

I didn't see the 'Configure -> Tools -> Write Config Files' can I find it in the free version?

My mistake - the actual path is 'Configure -> Core Config Manager -> Tools -> Write Config Files'. I'm glad to hear you have this working though - may I close this thread and mark it as resolved?

[timlee]

I didn't see the 'Configure -> Tools -> Write Config Files' can I find it in the free version?

My mistake - the actual path is 'Configure -> Core Config Manager -> Tools -> Write Config Files'. I'm glad to hear you have this working though - may I close this thread and mark it as resolved?

Yes, thank you.