Delay in updation of Nagios Front end screen

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
jatindeepsharma
Posts: 70
Joined: Wed Apr 04, 2018 10:52 am

Re: Delay in updation of Nagios Front end screen

Post by jatindeepsharma »

Hi @benjaminsmith,
Can you please help me with point 1 mentioned in above comment, cause i want to implement this solution ASAP.


Regards
JD
benjaminsmith
Posts: 5324
Joined: Wed Aug 22, 2018 4:39 pm
Location: saint paul

Re: Delay in updation of Nagios Front end screen

Post by benjaminsmith »

Hi JD,

You can try adding more CPUs and RAM to boost performance, but my recommendation would be to add another XI server and distribute the check load across multiple systems.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.

Be sure to check out our Knowledgebase for helpful articles and solutions!
jatindeepsharma
Posts: 70
Joined: Wed Apr 04, 2018 10:52 am

Re: Delay in updation of Nagios Front end screen

Post by jatindeepsharma »

Hi @Benjaminsmith

Thanks for suggestion, Can you please guide me what should i need after getting my new VM machine?
Should i take snap shots of current server & get same installed on other server & put new nagios licence on other machine. Please suggest me, I am making my assumptions on same.
Please suggest.


Regards
Jatindeep
jatindeepsharma
Posts: 70
Joined: Wed Apr 04, 2018 10:52 am

Re: Delay in updation of Nagios Front end screen

Post by jatindeepsharma »

Hello @ Benjaminsmith,
I am quite confused cause right now i am using NAGIOS XI Maintenance GRANDFATHERED PRICING (UNLIMITED NODE LICENSE) (QOOUPQ) . then if i will increase memory , CPU of current machine, will it run fine?
why would i need new licence then?


Regards
Jatindeep
benjaminsmith
Posts: 5324
Joined: Wed Aug 22, 2018 4:39 pm
Location: saint paul

Re: Delay in updation of Nagios Front end screen

Post by benjaminsmith »

Hi JD,

The license is for a single instance, unlimited nodes. If you have any further questions about your license, please contact our sales team.

I cannot give you a definitive answer as every system is set up differently. We recommend moving to a physical server after 5000 services. We also don't provide hardware recommendations for servers running 149,000 service checks. After 20,000 services, we recommend adding additional Nagios XI servers.

Nagios XI Hardware Requirements
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.

Be sure to check out our Knowledgebase for helpful articles and solutions!
jatindeepsharma
Posts: 70
Joined: Wed Apr 04, 2018 10:52 am

Re: Delay in updation of Nagios Front end screen

Post by jatindeepsharma »

Hi Benjaminsmith,
I have shared my latest system profile with you, Can you please take a look ?
Right now we have 26000 services running with 1600 hosts.
i have extended my CPU cores(8 Cores) & Memory(32GB), still system is not performing good, ndo2db service is getting stuck after 30 minutes.

Regards
JD
benjaminsmith
Posts: 5324
Joined: Wed Aug 22, 2018 4:39 pm
Location: saint paul

Re: Delay in updation of Nagios Front end screen

Post by benjaminsmith »

Hi JD,

OK. The top command output looks good, but let's check the kernel message queues. What is the output of the following command?

Code: Select all

ipcs -q
Also, this error message is showing up in the nagios.log.
[1576269535] External command error: Command failed
Please open the /usr/local/nagios/etc/nagios.cfg file and make sure that following entry is set to 1:

Code: Select all

check_external_commands=1
Then restart the nagios service:

Code: Select all

service nagios restart
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.

Be sure to check out our Knowledgebase for helpful articles and solutions!
jatindeepsharma
Posts: 70
Joined: Wed Apr 04, 2018 10:52 am

Re: Delay in updation of Nagios Front end screen

Post by jatindeepsharma »

Hi @Benjaminsmith,
Below is output of ipcs -q command.
[root@NagiosXI ~]# service nagios restart
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.
[root@NagiosXI ~]# ipcs -q

------ Message Queues --------
key msqid owner perms used-bytes messages
0xb4000002 753664 nagios 600 85329920 83330

Can you please take a look why front end is updating late now?

Regards
JD
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: Delay in updation of Nagios Front end screen

Post by tgriep »

Most of the time when the Kernel Message Queue dies not clear out is that the system cannot write the data fast enough to the MYSQL database.
When that happens, it takes a lot longer to update the data and the XI interface takes longer to update as well.

Couple of things to get the server to run faster is to remove any other applications from the server.
I see that the server is running logstash, elasticsearch and splunkd. Stop those from running on the server.
The server is running XWindows and Gnome. Disable that as well.

Your MYSQL data is offloaded to a remote server. Make sure the network connection is as fast as possible and if possible, enable Jumbo Frames on the Network interfaces of the Nagios server and MYSQL server.


Disable sysloging in nagios by editing the /usr/local/nagios/etc/nagios.cfg file and change thos from

Code: Select all

use_syslog=1
to

Code: Select all

use_syslog=0
Save the change and restart Nagios

Code: Select all

service nagios restart
Please get this file and post it here.

Code: Select all

/etc/sysctl.conf
Run this command and post the output were so we can view the Nagios processes stats.

Code: Select all

/usr/local/nagios/bin/nagiostats
Be sure to check out our Knowledgebase for helpful articles and solutions!
jatindeepsharma
Posts: 70
Joined: Wed Apr 04, 2018 10:52 am

Re: Delay in updation of Nagios Front end screen

Post by jatindeepsharma »

Hello @ tgriep,
Good afternoon, Thanks for replying back, Please find below output of commands & suggest.

[root@NagiosXI snmp]# /etc/sysctl.conf
/etc/sysctl.conf: line 7: net.ipv4.ip_forward: command not found
/etc/sysctl.conf: line 10: net.ipv4.conf.default.rp_filter: command not found
/etc/sysctl.conf: line 13: net.ipv4.conf.default.accept_source_route: command not found
/etc/sysctl.conf: line 16: kernel.sysrq: command not found
/etc/sysctl.conf: line 20: kernel.core_uses_pid: command not found
/etc/sysctl.conf: line 23: net.ipv4.tcp_syncookies: command not found
/etc/sysctl.conf: line 26: net.bridge.bridge-nf-call-ip6tables: command not found
/etc/sysctl.conf: line 27: net.bridge.bridge-nf-call-iptables: command not found
/etc/sysctl.conf: line 28: net.bridge.bridge-nf-call-arptables: command not found
/etc/sysctl.conf: line 31: kernel.msgmnb: command not found
/etc/sysctl.conf: line 35: kernel.msgmax: command not found
/etc/sysctl.conf: line 39: kernel.shmmax: command not found
/etc/sysctl.conf: line 42: kernel.shmall: command not found
/etc/sysctl.conf: line 44: kernel.msgmni: command not found


Command 2:-

[root@NagiosXI snmp]# /usr/local/nagios/bin/nagiostats

Nagios Stats 4.0.6
Copyright (c) 2003-2008 Ethan Galstad (www.nagios.org)
Last Modified: 04-29-2014
License: GPL

CURRENT STATUS DATA
------------------------------------------------------
Status File: /var/nagiosramdisk/status.dat
Status File Age: 0d 0h 0m 1s
Status File Version: 4.0.6

Program Running Time: 0d 0h 2m 22s
Nagios PID: 20076

Total Services: 24705
Services Checked: 6225
Services Scheduled: 0
Services Actively Checked: 18480
Services Passively Checked: 6225
Total Service State Change: 0.000 / 100.000 / 5.648 %
Active Service Latency: 0.000 / 0.000 / 0.000 sec
Active Service Execution Time: 0.000 / 0.000 / 0.000 sec
Active Service State Change: 0.000 / 0.000 / 0.000 %
Active Services Last 1/5/15/60 min: 0 / 0 / 0 / 0
Passive Service Latency: 0.181 / 631318.139 / 14327.458 sec
Passive Service State Change: 0.000 / 100.000 / 22.415 %
Passive Services Last 1/5/15/60 min: 0 / 15 / 45 / 116
Services Ok/Warn/Unk/Crit: 20350 / 2886 / 0 / 1469
Services Flapping: 0
Services In Downtime: 0

Total Hosts: 1600
Hosts Checked: 0
Hosts Scheduled: 1599
Hosts Actively Checked: 1600
Host Passively Checked: 0
Total Host State Change: 0.000 / 0.000 / 0.000 %
Active Host Latency: 0.000 / 0.000 / 0.000 sec
Active Host Execution Time: 0.000 / 0.000 / 0.000 sec
Active Host State Change: 0.000 / 0.000 / 0.000 %
Active Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0
Passive Host Latency: 0.000 / 0.000 / 0.000 sec
Passive Host State Change: 0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0
Hosts Up/Down/Unreach: 1599 / 0 / 1
Hosts Flapping: 0
Hosts In Downtime: 0

Active Host Checks Last 1/5/15 min: 0 / 0 / 0
Scheduled: 0 / 0 / 0
On-demand: 0 / 0 / 0
Parallel: 0 / 0 / 0
Serial: 0 / 0 / 0
Cached: 0 / 0 / 0
Passive Host Checks Last 1/5/15 min: 0 / 0 / 0
Active Service Checks Last 1/5/15 min: 0 / 0 / 0
Scheduled: 0 / 0 / 0
On-demand: 0 / 0 / 0
Cached: 0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 0 / 0 / 0

External Commands Last 1/5/15 min: 0 / 0 / 0
Locked