Nagios NA - bandwidth.rrd does not update

This support forum board is for support questions relating to Nagios Network Analyzer, our network traffic and bandwidth analysis solution.
Locked
sbsbstout
Posts: 9
Joined: Thu Aug 05, 2021 9:45 am

Nagios NA - bandwidth.rrd does not update

Post by sbsbstout »

Hello,

I have a new Nagios Network Analyzer installation running version 2.4.2 on RHEL8.4.

The nfcapd files are populating and confirmed with nfdump -r <nfcapd.???>. I can also see top 5 talkers in the web UI.

The bandwidth.rrd files are not updating unless I stop/start the sources from the web UI. After using the web UI to stop/start, I see the bandwidth.rrd for the specific sources I restarted continues to update and see data in the web UI posted. If I reboot or restart nagiosna.service, they stop working until I go into the web UI and stop/start the sources. I have 74 sources configured, so this is not a feasible workaround. The backend.log file shows no errors and is empty until I stop/start through web UI. I have compared the processes and see no difference.

An additional troubleshooting step - I disabled nagiosna.service for boot, restarted server, and manually started nagiosna.service - still no update to the bandwidth.rrd files until I stop/start the sources in web UI.

Looking at other forum posts, I checked the time zone configuration and found they match.

Where do I go from here to get the sources working correctly?

Thanks,
Brandon
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Nagios NA - bandwidth.rrd does not update

Post by ssax »

Are you out of semaphores? (check /var/log/messages to see what it shows)

https://support.nagios.com/kb/article/n ... rt-31.html

See anything in this output that could be related?

Code: Select all

journalctl -xe
dmesg 
If you run this command, do you see any errors in /var/log/messages or output to the console?

Code: Select all

systemctl restart nagiosna
sbsbstout
Posts: 9
Joined: Thu Aug 05, 2021 9:45 am

Re: Nagios NA - bandwidth.rrd does not update

Post by sbsbstout »

Hi - Thanks for responding.

I checked /var/log/messages and there are no entries indicating issues with semaphores. The nfcapd processes starting and capturing data, just not updating the bandwidth.rrd files until I restart the sources through the web UI.

I checked journalct -xe and dmesg and nothing of interest related to Nagios or nfcapd.


Brandon
ssax wrote:Are you out of semaphores? (check /var/log/messages to see what it shows)

https://support.nagios.com/kb/article/n ... rt-31.html

See anything in this output that could be related?

Code: Select all

journalctl -xe
dmesg 
If you run this command, do you see any errors in /var/log/messages or output to the console?

Code: Select all

systemctl restart nagiosna
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Nagios NA - bandwidth.rrd does not update

Post by ssax »

What is the output of these commands:
- Change XXXX to the sourcename

Code: Select all

ls -ld /usr/local/nagiosna/var/XXXX
ls -l /usr/local/nagiosna/var/XXXX
grep nna /etc/group
Do you see any errors in the log files in this directory? (if so, please attach or PM them)

Code: Select all

/usr/local/nagiosna/var
sbsbstout
Posts: 9
Joined: Thu Aug 05, 2021 9:45 am

Re: Nagios NA - bandwidth.rrd does not update

Post by sbsbstout »

Here are two sources I have been testing with. I have seen different permission requirements, so I will give you output of both.

[root@stp-app8a ~]# ls -ld /usr/local/nagiosna/var/fw1
drwsrwsr-t 3 nna nnacmd 56 Aug 18 11:03 /usr/local/nagiosna/var/fw1
[root@stp-app8a ~]# ls -l /usr/local/nagiosna/var/fw1
total 1248
-rw-r--r-- 1 nna nnacmd 7 Aug 18 11:03 2078.pid
-rwxr-xr-x 1 nna nnacmd 1255360 Aug 18 11:02 bandwidth.rrd
drwxr-xr-x 2 nna nnacmd 12288 Aug 19 13:50 flows
[root@stp-app8a ~]# ls -ld /usr/local/nagiosna/var/fw.alma
drwsrwsr-t 3 nna users 56 Aug 18 11:04 /usr/local/nagiosna/var/fw.alma
[root@stp-app8a ~]# ls -l /usr/local/nagiosna/var/fw.alma
total 1248
-rw-r--r-- 1 nna users 7 Aug 18 11:04 3003.pid
-rw-r--r-- 1 nna users 1255360 Aug 18 11:02 bandwidth.rrd
drwxr-sr-x 2 nna users 12288 Aug 19 13:50 flows

[root@stp-app8a var]# ps aux | egrep "fw.alma|fw1"
nna 361593 0.0 0.0 32688 5356 ? S Aug18 1:09 /usr/local/bin/nfcapd -I 25 -l /usr/local/nagiosna/var/fw1/flows -p 2078 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw1/2078.pid -D -e -w -z -T all
nna 361595 0.0 0.0 11424 2012 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 25 -l /usr/local/nagiosna/var/fw1/flows -p 2078 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw1/2078.pid -D -e -w -z -T all
nna 363288 0.0 0.0 32688 4480 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 50 -l /usr/local/nagiosna/var/fw.alma/flows -p 3003 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw.alma/3003.pid -D -e -w -z -T all
nna 363292 0.0 0.0 11424 1960 ? S Aug18 0:00 /usr/local/bin/nfcapd -I 50 -l /usr/local/nagiosna/var/fw.alma/flows -p 3003 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/fw.alma/3003.pid -D -e -w -z -T all
root 543433 0.0 0.0 12136 2380 pts/0 S+ 13:56 0:00 grep -E --color=auto fw.alma|fw1
[root@stp-app8a var]#

-rw-r--r-- 1 nna nnacmd 1544498 Aug 19 13:25 nfcapd.202108191320
-rw-r--r-- 1 nna nnacmd 1611039 Aug 19 13:30 nfcapd.202108191325
-rw-r--r-- 1 nna nnacmd 1647867 Aug 19 13:35 nfcapd.202108191330
-rw-r--r-- 1 nna nnacmd 1626121 Aug 19 13:40 nfcapd.202108191335
-rw-r--r-- 1 nna nnacmd 1660592 Aug 19 13:45 nfcapd.202108191340
-rw-r--r-- 1 nna nnacmd 1498233 Aug 19 13:50 nfcapd.202108191345
-rw-r--r-- 1 nna nnacmd 1561032 Aug 19 13:55 nfcapd.202108191350
drwxr-xr-x 2 nna nnacmd 12288 Aug 19 13:55 .
-rw-r--r-- 1 nna nnacmd 108 Aug 19 13:55 .nfstat
-rw-r--r-- 1 nna nnacmd 1002046 Aug 19 13:57 nfcapd.current.361590
[root@stp-app8a flows]# pwd
/usr/local/nagiosna/var/fw1/flows
[root@stp-app8a flows]#

-rw-r--r-- 1 nna users 9015 Aug 19 13:30 nfcapd.202108191325
-rw-r--r-- 1 nna users 5825 Aug 19 13:35 nfcapd.202108191330
-rw-r--r-- 1 nna users 34525 Aug 19 13:40 nfcapd.202108191335
-rw-r--r-- 1 nna users 10541 Aug 19 13:45 nfcapd.202108191340
-rw-r--r-- 1 nna users 3764 Aug 19 13:50 nfcapd.202108191345
-rw-r--r-- 1 nna users 276 Aug 19 13:55 nfcapd.current.363286
-rw-r--r-- 1 nna users 11141 Aug 19 13:55 nfcapd.202108191350
drwxr-sr-x 2 nna users 12288 Aug 19 13:55 .
-rw-r--r-- 1 nna users 106 Aug 19 13:55 .nfstat
[root@stp-app8a flows]# pwd
/usr/local/nagiosna/var/fw.alma/flows
[root@stp-app8a flows]#


[root@stp-app8a ~]# grep nna /etc/group
apache:x:48:nna
nnacmd:x:1005:nna,apache
[root@stp-app8a ~]#


[root@stp-app8a var]# grep -i error *.log
[root@stp-app8a var]# ls -al *.log
-rw-rw-r--. 1 nna nnacmd 0 Aug 18 11:35 backend.log
-rw-rw-r--. 1 nna nnacmd 0 Aug 19 13:00 cache.log
-rw-rw-r--. 1 nna nnacmd 61 Aug 19 13:55 cmdsubsys.log
[root@stp-app8a var]# date
Thu Aug 19 13:56:00 CDT 2021
[root@stp-app8a var]#


Thanks,
Brandon
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Nagios NA - bandwidth.rrd does not update

Post by ssax »

What is the output of these commands?

Code: Select all

sestatus
cat /usr/lib/systemd/system/nagiosna.service
cat /etc/sudoers.d/nagiosna
sbsbstout
Posts: 9
Joined: Thu Aug 05, 2021 9:45 am

Re: Nagios NA - bandwidth.rrd does not update

Post by sbsbstout »

[root@stp-app8a ~]# sestatus
SELinux status: disabled


[root@stp-app8a ~]# cat /usr/lib/systemd/system/nagiosna.service
[Unit]
Description=NagiosNA Daemon
Requires=httpd.service mariadb.service
After=mariadb.service
[Service]
Type=forking
ExecStart=/etc/rc.d/init.d/nagiosna start
ExecStop=/etc/rc.d/init.d/nagiosna stop
TimeoutStartSec=600
[Install]
WantedBy=multi-user.target



[root@stp-app8a ~]# cat /etc/sudoers.d/nagiosna

Defaults:%nnacmd !requiretty
Defaults:nna !requiretty

nna ALL = NOPASSWD:/usr/local/nagiosna/scripts/change_timezone.sh
nna ALL = NOPASSWD:/usr/local/nagiosna/scripts/upgrade_to_latest.sh

%nnacmd ALL=(ALL) NOPASSWD:/bin/kill *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/bin/rc.py *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/scripts/manage_firewall.sh *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/scripts/remove_source.sh *
%nnacmd ALL=(ALL) NOPASSWD:/usr/bin/systemctl restart httpd
[root@stp-app8a ~]#
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Nagios NA - bandwidth.rrd does not update

Post by ssax »

I'm wondering if this is actually causing it:

Code: Select all

TimeoutStartSec=600
Please edit this file:

Code: Select all

/usr/lib/systemd/system/nagiosna.service
Remove that line, and then run these commands:

Code: Select all

systemctl daemon-reload
systemctl restart nagiosna
You can also test the reboot after that.

If it still has issues on boot, you may need to add this under the [Unit] section:

Code: Select all

After=network.target local-fs.target

Code: Select all

systemctl daemon-reload
systemctl restart nagiosna
You can also test the reboot after that.
sbsbstout
Posts: 9
Joined: Thu Aug 05, 2021 9:45 am

Re: Nagios NA - bandwidth.rrd does not update

Post by sbsbstout »

Hi,

I added "TimeoutStartSec=600" because systemd was not giving the this unit file enough time to start up all the nfcapd processes and complete, therefore at boot or doing a systemctl restart, it would not complete successfully leaving me with no nfcapd processes running. I have also started Nagios NA using the /etc/rc.d/init.d/nagiosna script and still do not see the bandwidth.rrd files update.

It appears the web UI starts the sources differently from the nagiosna script. I was not able to look at the PHP code and figure out the difference. Can you provide the actions the web UI uses and we can try that from the command line?

Thanks,
Brandon
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Nagios NA - bandwidth.rrd does not update

Post by ssax »

When you click start it initializes it like this:

Code: Select all

/usr/local/nagiosna/bin/initialize_source.py --directory /usr/local/nagiosna/var/XXXX/flows --port YYYY
Then it starts the exact commands you see in the ps aux output:

Code: Select all

/usr/local/bin/nfcapd -I 1 -l /usr/local/nagiosna/var/XXXX/flows -p YYYY -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/XXXX/YYYY.pid -D -e -w -z -T all
/usr/local/bin/nfcapd -I 1 -l /usr/local/nagiosna/var/XXXX/flows -p YYYY -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/XXXX/YYYY.pid -D -e -w -z -T all
vs the init script method which uses the python capd module:

Code: Select all

handler = logging.handlers.RotatingFileHandler('/usr/local/nagiosna/var/backend.log', 'a', 1048576, 10)
handler.setLevel(logging.INFO)
logger.addHandler(handler)

def start(sourcename):
    try:
        name, directory, port, flowtype, lifetime, sid = get_source(sourcename)
    except Exception as e:
        print(str(e))
        raise Exception('Unable to find %s in the database, cannot start it.' % sourcename)

    capd.start(name, directory, port, flowtype, lifetime, sid)
You don't see anything in here? That's where it should be logged:

Code: Select all

/usr/local/nagiosna/var/backend.log
Please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/
Locked