nfcapd not capturing any data

This support forum board is for support questions relating to Nagios Network Analyzer, our network traffic and bandwidth analysis solution.
sergey-frontier
Posts: 8
Joined: Fri Feb 11, 2022 11:32 am

nfcapd not capturing any data

Post by sergey-frontier »

Hello,

I have a fresh NagionNA installation running on AlmaLinux 8.5.

I see nfcapd daemon up and running:

Code: Select all

udp        0      0 0.0.0.0:2055            0.0.0.0:*                           1001       552755774  1588220/nfcapd
With 2 processes:

Code: Select all

nna      1588220  0.0  0.0  32688  4048 ?        S    11:36   0:00 /usr/local/bin/nfcapd -I 1 -l /usr/local/nagiosna/var/test/flows -p 2055 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/test/2055.pid -D -e -w -z -T all
nna      1588221  0.0  0.0  11556  2272 ?        S    11:36   0:00 /usr/local/bin/nfcapd -I 1 -l /usr/local/nagiosna/var/test/flows -p 2055 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/test/2055.pid -D -e -w -z -T all
Firewalld/iptables are disabled.
When I run tcpdump on port 2055, I see udp netflow packets constantly coming from two configured Cisco routers.

However, all nfcapd.* files in /usr/local/nagiosna/var/test/flows/ appear to be empty (276 bytes in size):

Code: Select all

-rw-r--r-- 1 nna nnacmd 276 Feb 11 12:10 nfcapd.202202111205
-rw-r--r-- 1 nna nnacmd 276 Feb 11 12:15 nfcapd.202202111210
-rw-r--r-- 1 nna nnacmd 276 Feb 11 12:20 nfcapd.202202111215
-rw-r--r-- 1 nna nnacmd 276 Feb 11 12:25 nfcapd.202202111220
-rw-r--r-- 1 nna nnacmd 276 Feb 11 12:25 nfcapd.current.1588217
The only thing from nfcapd in the logs is this (repeated every 5 minutes), doesn't indicate any problems:

Code: Select all

Feb 11 12:15:01 localhost nfcapd[1588220]: Ident: '1' Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
Feb 11 12:15:01 localhost nfcapd[1588220]: Signal launcher
Feb 11 12:15:01 localhost nfcapd[1588220]: Total ignored packets: 0
Feb 11 12:15:01 localhost nfcapd[1588221]: Launcher: fork child.
Feb 11 12:15:01 localhost nfcapd[1588221]: Launcher: child exec done.
Feb 11 12:15:01 localhost nfcapd[1588221]: Run expire on '/usr/local/nagiosna/var/test/flows'
Feb 11 12:15:01 localhost nfcapd[1588221]: Limits: Filesize <none>, Lifetime 259200 = 3.0 days, Watermark: 95%
Feb 11 12:15:01 localhost nfcapd[1588221]: Current size: 3457024 = 3.3 MB, Current lifetime: 253800 = 2.9 days, Number of files: 844
Feb 11 12:15:01 localhost nfcapd[1588221]: expire completed - nothing to expire.
Feb 11 12:15:01 localhost nfcapd[1588221]: launcher child exit 1 children.
Feb 11 12:15:01 localhost nfcapd[1588221]: launcher child 1749761 exit status: 1
Feb 11 12:15:01 localhost nfcapd[1588221]: launcher waiting children done. 0 children
We tried changing netflow version on the routers from v9 to ipfix (these are only two versions supported), but it didn't have any effect.
Moreover, same netflow data is being sent to FastNetMon server, which has no issues parsing it.

What could be the cause of this? Is there any way to enable debug logging mode for nfcapd?
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: nfcapd not capturing any data

Post by pbroste »

Hello @sergey-frontier

Thanks for reaching out, want to get some more information from you on this so we can see what is going on.

want to see the output on these:

Code: Select all

sestatus

Code: Select all

journalctl -u nagiosna.service > /tmp/nnaresults.txt
And:

Code: Select all

tar -czvf /tmp/nnaresults.tar.gz /tmp/nnaresults.txt /usr/local/nagiosna/var/
Please send over the '/tmp/nnaresults.tar.gz' when you get a chance.

Thanks,
Perry
sergey-frontier
Posts: 8
Joined: Fri Feb 11, 2022 11:32 am

Re: nfcapd not capturing any data

Post by sergey-frontier »

Hi Perry,

Here you go:

Code: Select all

# sestatus
SELinux status:                 disabled
You do not have the required permissions to view the files attached to this post.
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: nfcapd not capturing any data

Post by pbroste »

Hello @sergey-frontier

Thanks for following up, and sending over the info.

We see that that there is another connector running,
Another collector with pid 503065 is already running, and configured for '/usr/local/nagiosna/var/test/flows'
initialize bookkeeper failed.
Software error in bookkeeper.c line 402Entry not found in list
Signal launcher[589479] to terminate.
let's take a look:

Find the 'PID':

Code: Select all

journalctl -u nagiosna.service | grep -Ei 'pid'
Find the line with; "Another collector with pid xxxxx is already running, and configured for '/usr/local/nagiosna/var/test/flows'

Code: Select all

ps -aux | grep -Ei 'xxxxxx'
Observe investigate the running process and kill it if necessary

Code: Select all

kill xxxxxx
And then restart the nagiosna.service by:

Code: Select all

systemctl restart nagiosna.service
Then review the service journal:

Code: Select all

journalctl -u nagiosna.service
Also, review this support article that references similar issues as well:
https://support.nagios.com/kb/article/n ... rt-31.html

Thanks,
Perry
sergey-frontier
Posts: 8
Joined: Fri Feb 11, 2022 11:32 am

Re: nfcapd not capturing any data

Post by sergey-frontier »

Thanks Perry,
But I resolved "Another collector already running" issue before sending the logs.
Only one collector is running with one child process. Still no data is being written.

Here's journalctl log after another restart:

Code: Select all

Feb 15 20:07:35 nagios2022 systemd[1]: Stopping NagiosNA Daemon...
Feb 15 20:07:35 nagios2022 sudo[2778884]:     root : TTY=unknown ; PWD=/ ; USER=nna ; COMMAND=/bin/kill 2982931
Feb 15 20:07:35 nagios2022 nfcapd[2982931]: Ident: '1' Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
Feb 15 20:07:35 nagios2022 nfcapd[2982931]: Signal launcher
Feb 15 20:07:35 nagios2022 nfcapd[2982931]: Total ignored packets: 0
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: Launcher: fork child.
Feb 15 20:07:35 nagios2022 nfcapd[2982931]: Signal launcher[2982940] to terminate.
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: Launcher: child exec done.
Feb 15 20:07:35 nagios2022 nagiosna[2778880]: NagiosNA backend stopped.
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: Run expire on '/usr/local/nagiosna/var/test/flows'
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: Limits: Filesize <none>, Lifetime 259200 = 3.0 days, Watermark: 95%
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: Current size: 3530752 = 3.4 MB, Current lifetime: 258300 = 3.0 days, Number of files: 862
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: expire completed - nothing to expire.
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: launcher child exit 1 children.
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: launcher child 2778886 died due to signal 15
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: launcher waiting children done. 0 children
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: Launcher: Terminating.
Feb 15 20:07:35 nagios2022 nfcapd[2982940]: Launcher: exit.
Feb 15 20:07:35 nagios2022 nfcapd[2982931]: launcher exit status: 0
Feb 15 20:07:35 nagios2022 nfcapd[2982931]: Terminating nfcapd.
Feb 15 20:07:35 nagios2022 systemd[1]: nagiosna.service: Succeeded.
Feb 15 20:07:35 nagios2022 systemd[1]: Stopped NagiosNA Daemon.
Feb 15 20:07:35 nagios2022 systemd[1]: Starting NagiosNA Daemon...
Feb 15 20:07:35 nagios2022 sudo[2778895]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/nagiosna/scripts/manage_firewall.sh -t udp -p 2055 --add
Feb 15 20:07:35 nagios2022 sudo[2778895]: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 15 20:07:36 nagios2022 nagiosna[2778889]: FirewallD is not running
Feb 15 20:07:36 nagios2022 nagiosna[2778889]: FirewallD is not running
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 2 byte input/output interface index
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 4 byte input/output interface index
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 2 byte src/dst AS number
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 4 byte src/dst AS number
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: dst tos, direction, src/dst mask
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: IPv4 next hop
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: IPv6 next hop
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: IPv4 BGP next IP
Feb 15 20:07:36 nagios2022 nagiosna[2778889]: NagiosNA backend started...
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: IPv6 BGP next IP
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: src/dst vlan id
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 4 byte output packets
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 8 byte output packets
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 4 byte output bytes
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 8 byte output bytes
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 4 byte aggregated flows
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: 8 byte aggregated flows
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: in src/out dst mac address
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: in dst/out src mac address
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: MPLS Labels
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: IPv4 router IP addr
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: IPv6 router IP addr
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: router ID
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: BGP adjacent prev/next AS
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: time packet received
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NSEL Common block
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NSEL xlate ports
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NSEL xlate IPv4 addr
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NSEL xlate IPv6 addr
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NSEL ACL ingress/egress acl ID
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NSEL username
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NSEL max username
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: nprobe/nfpcapd latency
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NEL Common block
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: Compat NEL IPv4
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Add extension: NAT Port Block Allocation
Feb 15 20:07:36 nagios2022 nfcapd[2778910]: Bound to IPv4 host/IP: any, Port: 2055
Feb 15 20:07:36 nagios2022 nfcapd[2778912]: Launcher[2778913] forked
Feb 15 20:07:36 nagios2022 nfcapd[2778913]: Launcher: Startup. auto-expire enabled
Feb 15 20:07:36 nagios2022 nfcapd[2778912]: Startup.
Feb 15 20:07:36 nagios2022 nfcapd[2778912]: Init IPFIX: Max number of IPFIX tags: 72
Feb 15 20:07:36 nagios2022 systemd[1]: Started NagiosNA Daemon.
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: nfcapd not capturing any data

Post by pbroste »

Hello @sergey-frontier

Thanks for following up and clarifying the subject logs.

Want to look at the packets:

Code: Select all

tcpdump port 2055
or write it to pcap:

Code: Select all

tcpdump port 2055 -w /tmp/tcpdump.pcap
Appears that when the service is coming up you are setting port 2055 open in the firewall. And does not appear to be bound to ipv6 and we are expecting ipv4 traffic.

Next that's check to see what the current flows look like:

Code: Select all

nfdump -r /usr/local/nagiosna/var/yourhost_flows/flows/nfcapd.current.xxxx
or

Code: Select all

nfdump -r /usr/local/nagiosna/var/yourhost/flows/nfcapd.current.xxxx
Let me know how things look,
Perry
sergey-frontier
Posts: 8
Joined: Fri Feb 11, 2022 11:32 am

Re: nfcapd not capturing any data

Post by sergey-frontier »

Code: Select all

# nfdump -r /usr/local/nagiosna/var/test/flows/nfcapd.current.2778910
Date first seen          Event  XEvent Proto      Src IP Addr:Port          Dst IP Addr:Port     X-Src IP Addr:Port        X-Dst IP Addr:Port   In Byte Out Byte
No matched flows
Attached is a tcpdump capture sample.

One thing that I didn't mention before - netflow traffic comes in on a vlan interface that is attached to the main non-vlan one. But I even tried starting nfcapd on this specific interface (by adding -b our.internal.ip.address option in bin/capd.py), and still it doesn't write anything.

Couple more things I did:

1. Stopped nfcapd and ran netcat test with user nna on udp 2055:

Code: Select all

sudo -u nna  nc -l our.internal.ip.address 2055 -u
then connected from another server and was able to send/receive test messages.

2. ran strace on both nfcapd processes
From what I can tell parent process only rotates the flow files, child process doesn't do anything.
Last edited by sergey-frontier on Tue Feb 22, 2022 2:22 pm, edited 1 time in total.
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: nfcapd not capturing any data

Post by pbroste »

Hello @sergey-frontier

Thanks for following up and providing the extra info. By the looks of the pcap and the netcat packets are cruising between. Appears that the Analyzer is not reading/capturing that data. At least we know that the flows are working, now to focus on the reason why the Analyzer is not picking it up.

Want to verify the date/time/timezone are synced across:

Code: Select all

date
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
grep "date.timezone =" /etc/php.ini
grep date.timezone /etc/php.ini
Let us know how that looks,
Perry
sergey-frontier
Posts: 8
Joined: Fri Feb 11, 2022 11:32 am

Re: nfcapd not capturing any data

Post by sergey-frontier »

Code: Select all

#     date
Thu Feb 17 23:20:25 EST 2022
#     ls -l /etc/localtime
lrwxrwxrwx 1 root root 30 Jan 28 16:14 /etc/localtime -> /usr/share/zoneinfo/US/Eastern
#     php -r 'echo date("D M j G:i:s T Y")."\n";'
Thu Feb 17 23:20:25 EST 2022
#     grep "date.timezone =" /etc/php.ini
date.timezone = US/Eastern
#     grep date.timezone /etc/php.ini
; http://php.net/date.timezone
date.timezone = US/Eastern
#
time is synced by chronyd.
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: nfcapd not capturing any data

Post by pbroste »

Hello @sergey-frontier

Thanks for following up, let's make sure that permissions and user accounts are active:

Code: Select all

chage -l nna
chage -l apache
grep nna /etc/group
We should see active process:

Code: Select all

ps -ef --cols=300 | grep -Ei 'reap_files.py'
Should look like this:
/usr/local/bin/nfcapd -I 2 -l /usr/local/nagiosna/var/temp/flows -p *9915 -x /usr/local/nagiosna/bin/reap_files.py %d %f %i -P /usr/local/nagiosna/var/temp/9915.pid -D -e -w -z -T all
  • *==> port number
Verify permission:

Code: Select all

ls -lR /usr/local/nagiosna/
Should look like this:
-rw-r--r-- 1 nna nnacmd 276 Feb 18 13:35 nfcapd.202202181330
-rw-r--r-- 1 nna nnacmd 276 Feb 18 13:40 nfcapd.202202181335
-rw-r--r-- 1 nna nnacmd 276 Feb 18 13:45 nfcapd.202202181340
-rw-r--r-- 1 nna nnacmd 276 Feb 18 13:50 nfcapd.202202181345

Code: Select all

python -V

Code: Select all

cat /etc/sudoers.d/nagiosna
Results:
Defaults:%nnacmd !requiretty
Defaults:nna !requiretty

nna ALL = NOPASSWD:/usr/local/nagiosna/scripts/change_timezone.sh
nna ALL = NOPASSWD:/usr/local/nagiosna/scripts/upgrade_to_latest.sh

%nnacmd ALL=(ALL) NOPASSWD:/bin/kill *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/bin/rc.py *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/scripts/manage_firewall.sh *
%nnacmd ALL=(ALL) NOPASSWD:/usr/local/nagiosna/scripts/remove_source.sh *
%nnacmd ALL=(ALL) NOPASSWD:/usr/bin/systemctl restart httpd
let us know how things look.

Thanks,
Perry
Locked