disk space filling up

[snapon_admin]

Our disk space (100GB) is filling up, sitting at about 15% free now, is there a specific method for extending the disk space for network analyzer or can I just follow the same steps that I would for doing this in XI? I've done it in XI twice, but don't want to try it in NNA without confirmation that I won't blow something up.

[jdalrymple]

As long as you didn't use a custom disk layout resizing the filesystem should be the same. The typical install creates only one root partition on LVM, so that's the disk/volume/partition that would need to be resized. If you did something custom, then obviously some things will have to be customized

For reference:http://assets.nagios.com/downloads/nagi ... M-Disk.pdf

[snapon_admin]

Is there any way to start all sources? All of them are stopped after the partition re-size and manually starting all of them is....a pain.

Edit: hmmmm, have a problem here. Even clicking start one at a time isn't working. None of them are starting, which is kind of a big deal. Any ideas?

[jdalrymple]

nfcapd should send errors to /var/log/messages. Anything useful there?

Code: Select all

grep nfcapd /var/log/messages | tail -40

If not,maybe there is an httpd error?

Code: Select all

tail -40 /var/log/httpd/error_log

[snapon_admin]

Looking now, nothing that jumps out at me but you guys are always better at spotting issues in logs than me:

Code: Select all

[root@lisl-ngna-01-pv ~]# grep nfcapd /var/log/messages | tail -40
Mar 20 15:05:10 lisl-ngna-01-pv nfcapd[18108]: Signal launcher
Mar 20 15:05:10 lisl-ngna-01-pv nfcapd[18108]: Total ignored packets: 0
Mar 20 15:05:10 lisl-ngna-01-pv nfcapd[18722]: expire completed - nothing to expire.
Mar 20 15:05:10 lisl-ngna-01-pv nfcapd[18722]: laucher child exit 1 childs.
Mar 20 15:05:10 lisl-ngna-01-pv nfcapd[18722]: laucher waiting childs done. 0 childs
Mar 20 15:05:10 lisl-ngna-01-pv nfcapd[17123]: Ident: '17' Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
Mar 20 15:05:10 lisl-ngna-01-pv nfcapd[17123]: Signal launcher
Mar 20 15:05:10 lisl-ngna-01-pv nfcapd[17123]: Total ignored packets: 0
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18938]: Run expire on '/usr/local/nagiosna/var/BollnasSE-MPLS/flows'
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18938]: Limits: Filesize <none>, Lifetime 2419200 = 4.0 weeks, Watermark: 95%
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18938]: Current size: 183316480 = 174.8 MB, Current lifetime: 2388900 = 3.9 weeks, Number of files: 7964
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]: Run expire on '/usr/local/nagiosna/var/MiamiFL-Sentry/flows'
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]: Limits: Filesize <none>, Lifetime 86400 = 1.0 days, Watermark: 95%
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]: Current size: 1560576 = 1.5 MB, Current lifetime: 86100 = 23.9 hours, Number of files: 288
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18938]: expire completed - nothing to expire.
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18938]: laucher child exit 1 childs.
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18938]: laucher waiting childs done. 0 childs
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]: expire completed
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]:    expired files: 15
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]:    expired time slot: 4500 = 1.2 hours
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]:    expired file size: 86016 = 84.0 KB
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]: New size: 1474560 = 1.4 MB, New lifetime: 81600 = 22.7 hours, Number of files: 273
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]: laucher child exit 1 childs.
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17232]: laucher waiting childs done. 0 childs
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]: Run expire on '/usr/local/nagiosna/var/BeijingCN-Sentry/flows'
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18109]: Run expire on '/usr/local/nagiosna/var/RobesoniaPA-MPLS/flows'
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18109]: Limits: Filesize <none>, Lifetime 2419200 = 4.0 weeks, Watermark: 95%
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]: Limits: Filesize <none>, Lifetime 86400 = 1.0 days, Watermark: 95%
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18109]: Current size: 112201728 = 107.0 MB, Current lifetime: 2349000 = 3.9 weeks, Number of files: 7831
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]: Current size: 1179648 = 1.1 MB, Current lifetime: 86100 = 23.9 hours, Number of files: 288
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]: expire completed
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]:    expired files: 15
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]:    expired time slot: 4500 = 1.2 hours
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]:    expired file size: 61440 = 60.0 KB
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]: New size: 1118208 = 1.1 MB, New lifetime: 81600 = 22.7 hours, Number of files: 273
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]: laucher child exit 1 childs.
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[17124]: laucher waiting childs done. 0 childs
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18109]: expire completed - nothing to expire.
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18109]: laucher child exit 1 childs.
Mar 20 15:05:11 lisl-ngna-01-pv nfcapd[18109]: laucher waiting childs done. 0 childs
[root@lisl-ngna-01-pv ~]# 
[root@lisl-ngna-01-pv ~]# tail -40 /var/log/httpd/error_log
Warning: SSL error ignored
Warning: SSL error ignored
Warning: SSL error ignored                                        
Warning: SSL error ignored                                        
Warning: SSL error ignored
Warning: SSL error ignored                                        
Warning: SSL error ignored                                        
Warning: SSL error ignored                                        
Warning: SSL error ignored                                        
Warning: SSL error ignored
Warning: SSL error ignored
Warning: SSL error ignored
Warning: SSL error ignored                                        
Warning: SSL error ignored
Warning: SSL error ignored
Warning: SSL error ignored
Warning: SSL error ignored
Warning: SSL error ignored                                        
Warning: SSL error ignored                                        
Counting pages (2/6)                                               
Resolving links (4/6)                                                       
Loading headers and footers (5/6)                                           
Printing pages (6/6)
Done                                                                            
[Fri Mar 20 15:07:38 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 20 15:07:38 2015] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 20 15:07:38 2015] [notice] Digest: done
[Fri Mar 20 15:07:39 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Fri Mar 20 15:08:26 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 20 15:08:26 2015] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 20 15:08:26 2015] [notice] Digest: done
[Fri Mar 20 15:08:26 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Fri Mar 20 15:11:28 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 20 15:11:28 2015] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 20 15:11:28 2015] [notice] Digest: done
[Fri Mar 20 15:11:29 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Fri Mar 20 15:25:04 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 20 15:25:04 2015] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 20 15:25:04 2015] [notice] Digest: done
[Fri Mar 20 15:25:05 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[root@lisl-ngna-01-pv ~]#

[jdalrymple]

To answer the first question - if possible the quickest way I know of to restart all sources is to reboot. I don't know of a simple way in the GUI. I'm sure it could be scripted, but I don't know of anything in place right off-hand.

[jdalrymple]

This appears fine - are the nfcapds running?

Code: Select all

ps -ef | grep nfcapd

Are you determining that the sources are down from the GUI? I'm not sure where that data actually comes from, it may be incorrect. Not to say that it's acceptable for it to be incorrect, but maybe this is just an aesthetics problem and not a function problem.

[snapon_admin]

Well the server did just reboot. I should clarify on the re-size. Our server is set up a little differently for some reason. We used the VM image from the website, but for whatever reason the server is not using LVM so there's only 1 partition. In order to re-size it, we had to mount gparted to the VM and then boot into that and extend the partition . We've done it this way once before, which is why I knew that was the process. Once the partition was re-sized the server was obviously rebooted to boot into centOS, but none of the sources were started and now none of them will restart.

And to answer your other questions, yes I'm seeing them as not started in the GUI.

Code: Select all

[root@lisl-ngna-01-pv ~]# ps -ef | grep nfcapd
root      7142  7118  0 15:58 pts/0    00:00:00 grep nfcapd

[snapon_admin]

Also, just rebooted to be sure and still have the same issue.

[jdalrymple]

Can you tail the messages again - a bit differently this time though? I noticed that you had a running nfcapd about an hour ago. What changed after that?

Code: Select all

grep nfcapd /var/log/messages | grep -A5 Startup