Page 1 of 5
Delay in updation of Nagios Front end screen
Posted: Wed Dec 04, 2019 10:39 am
by jatindeepsharma
Front end of nagios making an delay while updation, first it is letting all queue to filled up to 256000 & then messages are getting reduced in a go(like storm). rather than updating after every minute it is taking delay up to 20 to 25 minutes for updation, Can you please suggest?
Re: Delay in updation of Nagios Front end screen
Posted: Wed Dec 04, 2019 12:28 pm
by benjaminsmith
Hello
@jatindeepsharma,
Ok, I believe I have an understanding of the issue - messages queue filling up and Nagios taking to long to update check results. How long has this been happening on the server?
Follow this article to increase the kernel message queue settings on your server, restart and let it run for a while.
NDOUtils - Message Queue Exceeded
Next, PM your system profile so we can review the logs? Thanks.
To send us your system profile.
Login to the Nagios XI GUI using a web browser
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share this in a private message and then
reply to this post to bring it up in the queue.
Re: Delay in updation of Nagios Front end screen
Posted: Wed Dec 04, 2019 4:19 pm
by jatindeepsharma
Hello @Benjaminsmith,
this issue is happening from last 1.5 month. I have followed this document & increased kernel messages queue settings.
please check i have PM you profile.zip file.
Regards
Jatindeep
Re: Delay in updation of Nagios Front end screen
Posted: Wed Dec 04, 2019 5:31 pm
by benjaminsmith
Hello Jatindeep,
Thanks for the profile. Even though many are passive, you're running a large number of service checks (Total Services: 149424 ). We usually recommend adding an additional XI server after 20K checks.
1. Is this a physical server or a VM?
2. I noticed the nagios database has crashed tables, run the following command to repair.
Code: Select all
service mysqld stop
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
Once the script has completed, start the MySQL database server with the following command:
3. Post the output of the following commands:
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
Code: Select all
echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi
4. Please take a look at the following tips for boosting system performance.
Maximizing Performance In Nagios X
Re: Delay in updation of Nagios Front end screen
Posted: Thu Dec 05, 2019 8:50 am
by jatindeepsharma
Hello @Benjaminsmith,
Thanks for your reply, i am repairing my database right now & after repairing, I will get back to you with output of given commands.
Regards
Jatindeep
Re: Delay in updation of Nagios Front end screen
Posted: Thu Dec 05, 2019 10:43 am
by benjaminsmith
Hi Jatindeep,
Sounds good. We'll wait for your update.
Re: Delay in updation of Nagios Front end screen
Posted: Thu Dec 05, 2019 5:25 pm
by jatindeepsharma
Hello Benjamin,
i have repaired SQL database tables but when i am putting query i am getting below errors. please suggest.
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
ERROR 1045 (28000): Access denied for user 'root'@'127.0.0.1' (using password: YES)
echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Re: Delay in updation of Nagios Front end screen
Posted: Thu Dec 05, 2019 5:42 pm
by benjaminsmith
Hello,
Those commands are to check the size of the database tables. Have you changed the default passwords?
With regards to the initial problem, has the performance improved after repairing the database and increasing the default kernel message queue settings?
What is the output of the following command?
Take a look at the guide below for tips on improving system performance.
Maximizing Performance In Nagios XI
Re: Delay in updation of Nagios Front end screen
Posted: Fri Dec 06, 2019 9:20 am
by jatindeepsharma
Hello Benjamin,
Problem is still there after repairing tables.it is getting delay of about 35 minutes now. right now current time is 9:19 AM & it is showing data of 8:36 AM at front end. please suggest.
I will look into password of SQL database.
output of ipcs is given below.
[root@NagiosXI ~]# ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0052e2c1 0 postgres 600 37879808 34
0x00000000 32769 gdm 600 393216 2 dest
0x00000000 65538 gdm 600 393216 2 dest
0x00000000 98307 gdm 600 393216 2 dest
0x00000000 131076 gdm 600 393216 2 dest
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 0 root 600 1
0x00000000 65537 root 600 1
0x0052e2c1 131074 postgres 600 17
0x0052e2c2 163843 postgres 600 17
0x0052e2c3 196612 postgres 600 17
0x0052e2c4 229381 postgres 600 17
0x0052e2c5 262150 postgres 600 17
0x0052e2c6 294919 postgres 600 17
0x0052e2c7 327688 postgres 600 17
0x00000000 360457 apache 600 1
0x00000000 393226 apache 600 1
0x00000000 425995 apache 600 1
------ Message Queues --------
key msqid owner perms used-bytes messages
0x90000002 163840 nagios 600 261919744 255781
Re: Delay in updation of Nagios Front end screen
Posted: Fri Dec 06, 2019 9:28 am
by jatindeepsharma
Hello Benjamin,
I am using VM machine, is that the number of hosts & passive checks that are causing this problem?
right now we have around 1700 hosts on this server & in coming months i will increase more hosts on it.(i.e. it will also increase passive checks on it).
Regards
Jatindeep