Sources stopping unexpectedly

[btgnrw]

Hi,

Our system dashboard is showing as healthy with plenty of available free disk space.

A number of sources are stopping unexpectedly, with the status displaying as stopped.

When trying to start the problematic stopped sources they sometimes start but then stop again.

Have you experienced this behaviour before?

Are there any logs that can be inspected to aid troubleshooting?

thanks

[cdienger]

Check for any messages regarding nfcapd in /var/log/messages. nfcapd is the process that opens the listening port for the source and when it is running should be visible with:

yum -y install net-tools
netstat -nap | grep nfcapd

[btgnrw]

If a number of sources are stopped this allows the previously stopped sources to be started and remain started.

This leads me to suspect that there is max connections issue somewhere.

Can you provide the password for MariaDB root user on Network Analyzer to replace the bold password below:
https://support.nagios.com/kb/article.php?id=513

mysql -uroot -pnagiosxi -e "show variables like 'max_connections';"
mysql -uroot -pnagiosxi -e "show global status like 'Max_used_connections';"

Are there any other settings that may be restricting the maxium number of concurrently started sources?

Thanks

[cdienger]

The credentials are nagiosna:nagiosna

mysql -unagiosna -pnagiosna -e "show variables like 'max_connections';"
mysql -unagiosna -pnagiosna -e "show global status like 'Max_used_connections';"

How many sources do you have on the machine? I wonder if you may be running into an open file limit - do you see anything logged in /var/log/messages? What is the output of "su - nna --command "ulimit -a""?

[btgnrw]

Ok might not be anything to do with max db connections:

Code: Select all

+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 151   |
+-----------------+-------+

Code: Select all

+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Max_used_connections | 7     |
+----------------------+-------+

Code: Select all

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 255762
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Sources Count = 141

fs.file-max = 6505764

Nothing obvious in messages

[tgriep]

One cause of the sources stopping after a few minutes.
It could be a Kernel Semaphore limit that needs to be increased on the NNA server.

To do that edit this file

Code: Select all

/etc/sysctl.conf

Add the following option to the file and reboot the server for the change to take effect.

Code: Select all

kernel.sem = 1024 256000 256 32767

Let us know if this fixed the issue or not.

[btgnrw]

Sorry, sources are still stopping unexpectedly. There are only 11 sources running and some are stopping.

[tgriep]

Run the following as root on the NNA server to restart the nfcapd processes.

Code: Select all

service nagiosna restart

Then, if the sources stop running, get the following file from the server and post it here so we can view it.

Code: Select all

/usr/local/nagiosna/var/backend.log

Then run this to display the bottom of the /var/log/messages file and post the output here.

Code: Select all

tail -200 /var/log/messages

Thanks