Nagios NA - bandwidth.rrd does not update
Nagios NA - bandwidth.rrd does not update
Hello,
I have a new Nagios Network Analyzer installation running version 2.4.2 on RHEL8.4.
The nfcapd files are populating and confirmed with nfdump -r <nfcapd.???>. I can also see top 5 talkers in the web UI.
The bandwidth.rrd files are not updating unless I stop/start the sources from the web UI. After using the web UI to stop/start, I see the bandwidth.rrd for the specific sources I restarted continues to update and see data in the web UI posted. If I reboot or restart nagiosna.service, they stop working until I go into the web UI and stop/start the sources. I have 74 sources configured, so this is not a feasible workaround. The backend.log file shows no errors and is empty until I stop/start through web UI. I have compared the processes and see no difference.
An additional troubleshooting step - I disabled nagiosna.service for boot, restarted server, and manually started nagiosna.service - still no update to the bandwidth.rrd files until I stop/start the sources in web UI.
Looking at other forum posts, I checked the time zone configuration and found they match.
Where do I go from here to get the sources working correctly?
Thanks,
Brandon
I have a new Nagios Network Analyzer installation running version 2.4.2 on RHEL8.4.
The nfcapd files are populating and confirmed with nfdump -r <nfcapd.???>. I can also see top 5 talkers in the web UI.
The bandwidth.rrd files are not updating unless I stop/start the sources from the web UI. After using the web UI to stop/start, I see the bandwidth.rrd for the specific sources I restarted continues to update and see data in the web UI posted. If I reboot or restart nagiosna.service, they stop working until I go into the web UI and stop/start the sources. I have 74 sources configured, so this is not a feasible workaround. The backend.log file shows no errors and is empty until I stop/start through web UI. I have compared the processes and see no difference.
An additional troubleshooting step - I disabled nagiosna.service for boot, restarted server, and manually started nagiosna.service - still no update to the bandwidth.rrd files until I stop/start the sources in web UI.
Looking at other forum posts, I checked the time zone configuration and found they match.
Where do I go from here to get the sources working correctly?
Thanks,
Brandon
Re: Nagios NA - bandwidth.rrd does not update
Are you out of semaphores? (check /var/log/messages to see what it shows)
https://support.nagios.com/kb/article/n ... rt-31.html
See anything in this output that could be related?
If you run this command, do you see any errors in /var/log/messages or output to the console?
https://support.nagios.com/kb/article/n ... rt-31.html
See anything in this output that could be related?
Code: Select all
journalctl -xe
dmesg
Code: Select all
systemctl restart nagiosna
Re: Nagios NA - bandwidth.rrd does not update
Hi - Thanks for responding.
I checked /var/log/messages and there are no entries indicating issues with semaphores. The nfcapd processes starting and capturing data, just not updating the bandwidth.rrd files until I restart the sources through the web UI.
I checked journalct -xe and dmesg and nothing of interest related to Nagios or nfcapd.
Brandon
I checked /var/log/messages and there are no entries indicating issues with semaphores. The nfcapd processes starting and capturing data, just not updating the bandwidth.rrd files until I restart the sources through the web UI.
I checked journalct -xe and dmesg and nothing of interest related to Nagios or nfcapd.
Brandon
ssax wrote:Are you out of semaphores? (check /var/log/messages to see what it shows)
https://support.nagios.com/kb/article/n ... rt-31.html
See anything in this output that could be related?
If you run this command, do you see any errors in /var/log/messages or output to the console?Code: Select all
journalctl -xe dmesg
Code: Select all
systemctl restart nagiosna
Re: Nagios NA - bandwidth.rrd does not update
What is the output of these commands:
- Change XXXX to the sourcename
Do you see any errors in the log files in this directory? (if so, please attach or PM them)
- Change XXXX to the sourcename
Code: Select all
ls -ld /usr/local/nagiosna/var/XXXX
ls -l /usr/local/nagiosna/var/XXXX
grep nna /etc/group
Code: Select all
/usr/local/nagiosna/var
Re: Nagios NA - bandwidth.rrd does not update
Here are two sources I have been testing with. I have seen different permission requirements, so I will give you output of both.
[root@stp-app8a ~]# ls -ld /usr/local/nagiosna/var/fw1
drwsrwsr-t 3 nna nnacmd 56 Aug 18 11:03 /usr/local/nagiosna/var/fw1
[root@stp-app8a ~]# ls -l /usr/local/nagiosna/var/fw1
total 1248
-rw-r--r-- 1 nna nnacmd 7 Aug 18 11:03 2078.pid
-rwxr-xr-x 1 nna nnacmd 1255360 Aug 18 11:02 bandwidth.rrd
drwxr-xr-x 2 nna nnacmd 12288 Aug 19 13:50 flows
[root@stp-app8a ~]# ls -ld /usr/local/nagiosna/var/fw.alma
drwsrwsr-t 3 nna users 56 Aug 18 11:04 /usr/local/nagiosna/var/fw.alma
[root@stp-app8a ~]# ls -l /usr/local/nagiosna/var/fw.alma
total 1248
-rw-r--r-- 1 nna users 7 Aug 18 11:04 3003.pid
-rw-r--r-- 1 nna users 1255360 Aug 18 11:02 bandwidth.rrd
drwxr-sr-x 2 nna users 12288 Aug 19 13:50 flows
[root@stp-app8a var]# ps aux | egrep "fw.alma|fw1"
nna 361593 0.0 0.0 32688 5356 ? S Aug18 1:09 /usr/local/bin/nfcapd -I 25 -l /usr/local/nagiosna/var/fw1/flows -p 2078 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw1/2078.pid -D -e -w -z -T all
nna 361595 0.0 0.0 11424 2012 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 25 -l /usr/local/nagiosna/var/fw1/flows -p 2078 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw1/2078.pid -D -e -w -z -T all
nna 363288 0.0 0.0 32688 4480 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 50 -l /usr/local/nagiosna/var/fw.alma/flows -p 3003 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw.alma/3003.pid -D -e -w -z -T all
nna 363292 0.0 0.0 11424 1960 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 50 -l /usr/local/nagiosna/var/fw.alma/flows -p 3003 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw.alma/3003.pid -D -e -w -z -T all
root 543433 0.0 0.0 12136 2380 pts/0 S+ 13:56 0:00 grep -E --color=auto fw.alma|fw1
[root@stp-app8a var]#
-rw-r--r-- 1 nna nnacmd 1544498 Aug 19 13:25 nfcapd.202108191320
-rw-r--r-- 1 nna nnacmd 1611039 Aug 19 13:30 nfcapd.202108191325
-rw-r--r-- 1 nna nnacmd 1647867 Aug 19 13:35 nfcapd.202108191330
-rw-r--r-- 1 nna nnacmd 1626121 Aug 19 13:40 nfcapd.202108191335
-rw-r--r-- 1 nna nnacmd 1660592 Aug 19 13:45 nfcapd.202108191340
-rw-r--r-- 1 nna nnacmd 1498233 Aug 19 13:50 nfcapd.202108191345
-rw-r--r-- 1 nna nnacmd 1561032 Aug 19 13:55 nfcapd.202108191350
drwxr-xr-x 2 nna nnacmd 12288 Aug 19 13:55 .
-rw-r--r-- 1 nna nnacmd 108 Aug 19 13:55 .nfstat
-rw-r--r-- 1 nna nnacmd 1002046 Aug 19 13:57 nfcapd.current.361590
[root@stp-app8a flows]# pwd
/usr/local/nagiosna/var/fw1/flows
[root@stp-app8a flows]#
-rw-r--r-- 1 nna users 9015 Aug 19 13:30 nfcapd.202108191325
-rw-r--r-- 1 nna users 5825 Aug 19 13:35 nfcapd.202108191330
-rw-r--r-- 1 nna users 34525 Aug 19 13:40 nfcapd.202108191335
-rw-r--r-- 1 nna users 10541 Aug 19 13:45 nfcapd.202108191340
-rw-r--r-- 1 nna users 3764 Aug 19 13:50 nfcapd.202108191345
-rw-r--r-- 1 nna users 276 Aug 19 13:55 nfcapd.current.363286
-rw-r--r-- 1 nna users 11141 Aug 19 13:55 nfcapd.202108191350
drwxr-sr-x 2 nna users 12288 Aug 19 13:55 .
-rw-r--r-- 1 nna users 106 Aug 19 13:55 .nfstat
[root@stp-app8a flows]# pwd
/usr/local/nagiosna/var/fw.alma/flows
[root@stp-app8a flows]#
[root@stp-app8a ~]# grep nna /etc/group
apache48:nna
nnacmd1005:nna,apache
[root@stp-app8a ~]#
[root@stp-app8a var]# grep -i error *.log
[root@stp-app8a var]# ls -al *.log
-rw-rw-r--. 1 nna nnacmd 0 Aug 18 11:35 backend.log
-rw-rw-r--. 1 nna nnacmd 0 Aug 19 13:00 cache.log
-rw-rw-r--. 1 nna nnacmd 61 Aug 19 13:55 cmdsubsys.log
[root@stp-app8a var]# date
Thu Aug 19 13:56:00 CDT 2021
[root@stp-app8a var]#
Thanks,
Brandon
[root@stp-app8a ~]# ls -ld /usr/local/nagiosna/var/fw1
drwsrwsr-t 3 nna nnacmd 56 Aug 18 11:03 /usr/local/nagiosna/var/fw1
[root@stp-app8a ~]# ls -l /usr/local/nagiosna/var/fw1
total 1248
-rw-r--r-- 1 nna nnacmd 7 Aug 18 11:03 2078.pid
-rwxr-xr-x 1 nna nnacmd 1255360 Aug 18 11:02 bandwidth.rrd
drwxr-xr-x 2 nna nnacmd 12288 Aug 19 13:50 flows
[root@stp-app8a ~]# ls -ld /usr/local/nagiosna/var/fw.alma
drwsrwsr-t 3 nna users 56 Aug 18 11:04 /usr/local/nagiosna/var/fw.alma
[root@stp-app8a ~]# ls -l /usr/local/nagiosna/var/fw.alma
total 1248
-rw-r--r-- 1 nna users 7 Aug 18 11:04 3003.pid
-rw-r--r-- 1 nna users 1255360 Aug 18 11:02 bandwidth.rrd
drwxr-sr-x 2 nna users 12288 Aug 19 13:50 flows
[root@stp-app8a var]# ps aux | egrep "fw.alma|fw1"
nna 361593 0.0 0.0 32688 5356 ? S Aug18 1:09 /usr/local/bin/nfcapd -I 25 -l /usr/local/nagiosna/var/fw1/flows -p 2078 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw1/2078.pid -D -e -w -z -T all
nna 361595 0.0 0.0 11424 2012 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 25 -l /usr/local/nagiosna/var/fw1/flows -p 2078 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw1/2078.pid -D -e -w -z -T all
nna 363288 0.0 0.0 32688 4480 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 50 -l /usr/local/nagiosna/var/fw.alma/flows -p 3003 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw.alma/3003.pid -D -e -w -z -T all
nna 363292 0.0 0.0 11424 1960 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 50 -l /usr/local/nagiosna/var/fw.alma/flows -p 3003 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw.alma/3003.pid -D -e -w -z -T all
root 543433 0.0 0.0 12136 2380 pts/0 S+ 13:56 0:00 grep -E --color=auto fw.alma|fw1
[root@stp-app8a var]#
-rw-r--r-- 1 nna nnacmd 1544498 Aug 19 13:25 nfcapd.202108191320
-rw-r--r-- 1 nna nnacmd 1611039 Aug 19 13:30 nfcapd.202108191325
-rw-r--r-- 1 nna nnacmd 1647867 Aug 19 13:35 nfcapd.202108191330
-rw-r--r-- 1 nna nnacmd 1626121 Aug 19 13:40 nfcapd.202108191335
-rw-r--r-- 1 nna nnacmd 1660592 Aug 19 13:45 nfcapd.202108191340
-rw-r--r-- 1 nna nnacmd 1498233 Aug 19 13:50 nfcapd.202108191345
-rw-r--r-- 1 nna nnacmd 1561032 Aug 19 13:55 nfcapd.202108191350
drwxr-xr-x 2 nna nnacmd 12288 Aug 19 13:55 .
-rw-r--r-- 1 nna nnacmd 108 Aug 19 13:55 .nfstat
-rw-r--r-- 1 nna nnacmd 1002046 Aug 19 13:57 nfcapd.current.361590
[root@stp-app8a flows]# pwd
/usr/local/nagiosna/var/fw1/flows
[root@stp-app8a flows]#
-rw-r--r-- 1 nna users 9015 Aug 19 13:30 nfcapd.202108191325
-rw-r--r-- 1 nna users 5825 Aug 19 13:35 nfcapd.202108191330
-rw-r--r-- 1 nna users 34525 Aug 19 13:40 nfcapd.202108191335
-rw-r--r-- 1 nna users 10541 Aug 19 13:45 nfcapd.202108191340
-rw-r--r-- 1 nna users 3764 Aug 19 13:50 nfcapd.202108191345
-rw-r--r-- 1 nna users 276 Aug 19 13:55 nfcapd.current.363286
-rw-r--r-- 1 nna users 11141 Aug 19 13:55 nfcapd.202108191350
drwxr-sr-x 2 nna users 12288 Aug 19 13:55 .
-rw-r--r-- 1 nna users 106 Aug 19 13:55 .nfstat
[root@stp-app8a flows]# pwd
/usr/local/nagiosna/var/fw.alma/flows
[root@stp-app8a flows]#
[root@stp-app8a ~]# grep nna /etc/group
apache48:nna
nnacmd1005:nna,apache
[root@stp-app8a ~]#
[root@stp-app8a var]# grep -i error *.log
[root@stp-app8a var]# ls -al *.log
-rw-rw-r--. 1 nna nnacmd 0 Aug 18 11:35 backend.log
-rw-rw-r--. 1 nna nnacmd 0 Aug 19 13:00 cache.log
-rw-rw-r--. 1 nna nnacmd 61 Aug 19 13:55 cmdsubsys.log
[root@stp-app8a var]# date
Thu Aug 19 13:56:00 CDT 2021
[root@stp-app8a var]#
Thanks,
Brandon
Re: Nagios NA - bandwidth.rrd does not update
What is the output of these commands?
Code: Select all
sestatus
cat /usr/lib/systemd/system/nagiosna.service
cat /etc/sudoers.d/nagiosna
Re: Nagios NA - bandwidth.rrd does not update
[root@stp-app8a ~]# sestatus
SELinux status: disabled
[root@stp-app8a ~]# cat /usr/lib/systemd/system/nagiosna.service
[Unit]
Description=NagiosNA Daemon
Requires=httpd.service mariadb.service
After=mariadb.service
[Service]
Type=forking
ExecStart=/etc/rc.d/init.d/nagiosna start
ExecStop=/etc/rc.d/init.d/nagiosna stop
TimeoutStartSec=600
[Install]
WantedBy=multi-user.target
[root@stp-app8a ~]# cat /etc/sudoers.d/nagiosna
Defaults:%nnacmd !requiretty
Defaults:nna !requiretty
nna ALL = NOPASSWD:/usr/local/nagiosna/scripts/change_timezone.sh
nna ALL = NOPASSWD:/usr/local/nagiosna/scripts/upgrade_to_latest.sh
%nnacmd ALL=(ALL) NOPASSWD:/bin/kill *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/bin/rc.py *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/scripts/manage_firewall.sh *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/scripts/remove_source.sh *
%nnacmd ALL=(ALL) NOPASSWD:/usr/bin/systemctl restart httpd
[root@stp-app8a ~]#
SELinux status: disabled
[root@stp-app8a ~]# cat /usr/lib/systemd/system/nagiosna.service
[Unit]
Description=NagiosNA Daemon
Requires=httpd.service mariadb.service
After=mariadb.service
[Service]
Type=forking
ExecStart=/etc/rc.d/init.d/nagiosna start
ExecStop=/etc/rc.d/init.d/nagiosna stop
TimeoutStartSec=600
[Install]
WantedBy=multi-user.target
[root@stp-app8a ~]# cat /etc/sudoers.d/nagiosna
Defaults:%nnacmd !requiretty
Defaults:nna !requiretty
nna ALL = NOPASSWD:/usr/local/nagiosna/scripts/change_timezone.sh
nna ALL = NOPASSWD:/usr/local/nagiosna/scripts/upgrade_to_latest.sh
%nnacmd ALL=(ALL) NOPASSWD:/bin/kill *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/bin/rc.py *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/scripts/manage_firewall.sh *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/scripts/remove_source.sh *
%nnacmd ALL=(ALL) NOPASSWD:/usr/bin/systemctl restart httpd
[root@stp-app8a ~]#
Re: Nagios NA - bandwidth.rrd does not update
I'm wondering if this is actually causing it:
Please edit this file:
Remove that line, and then run these commands:
You can also test the reboot after that.
If it still has issues on boot, you may need to add this under the [Unit] section:
You can also test the reboot after that.
Code: Select all
TimeoutStartSec=600
Code: Select all
/usr/lib/systemd/system/nagiosna.service
Code: Select all
systemctl daemon-reload
systemctl restart nagiosna
If it still has issues on boot, you may need to add this under the [Unit] section:
Code: Select all
After=network.target local-fs.target
Code: Select all
systemctl daemon-reload
systemctl restart nagiosna
Re: Nagios NA - bandwidth.rrd does not update
Hi,
I added "TimeoutStartSec=600" because systemd was not giving the this unit file enough time to start up all the nfcapd processes and complete, therefore at boot or doing a systemctl restart, it would not complete successfully leaving me with no nfcapd processes running. I have also started Nagios NA using the /etc/rc.d/init.d/nagiosna script and still do not see the bandwidth.rrd files update.
It appears the web UI starts the sources differently from the nagiosna script. I was not able to look at the PHP code and figure out the difference. Can you provide the actions the web UI uses and we can try that from the command line?
Thanks,
Brandon
I added "TimeoutStartSec=600" because systemd was not giving the this unit file enough time to start up all the nfcapd processes and complete, therefore at boot or doing a systemctl restart, it would not complete successfully leaving me with no nfcapd processes running. I have also started Nagios NA using the /etc/rc.d/init.d/nagiosna script and still do not see the bandwidth.rrd files update.
It appears the web UI starts the sources differently from the nagiosna script. I was not able to look at the PHP code and figure out the difference. Can you provide the actions the web UI uses and we can try that from the command line?
Thanks,
Brandon
Re: Nagios NA - bandwidth.rrd does not update
When you click start it initializes it like this:
Then it starts the exact commands you see in the ps aux output:
vs the init script method which uses the python capd module:
You don't see anything in here? That's where it should be logged:
Please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:
https://support.nagios.com/tickets/
Code: Select all
/usr/local/nagiosna/bin/initialize_source.py --directory /usr/local/nagiosna/var/XXXX/flows --port YYYY
Code: Select all
/usr/local/bin/nfcapd -I 1 -l /usr/local/nagiosna/var/XXXX/flows -p YYYY -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/XXXX/YYYY.pid -D -e -w -z -T all
/usr/local/bin/nfcapd -I 1 -l /usr/local/nagiosna/var/XXXX/flows -p YYYY -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/XXXX/YYYY.pid -D -e -w -z -T all
Code: Select all
handler = logging.handlers.RotatingFileHandler('/usr/local/nagiosna/var/backend.log', 'a', 1048576, 10)
handler.setLevel(logging.INFO)
logger.addHandler(handler)
def start(sourcename):
try:
name, directory, port, flowtype, lifetime, sid = get_source(sourcename)
except Exception as e:
print(str(e))
raise Exception('Unable to find %s in the database, cannot start it.' % sourcename)
capd.start(name, directory, port, flowtype, lifetime, sid)
Code: Select all
/usr/local/nagiosna/var/backend.log
https://support.nagios.com/tickets/