NagiosXI with external DB : Error "Object does not exist"

[dberlenda]

Hello ,
when I change the thresholds of a service , for example :
check_xi_ncpa!-t 'xxxxxxxx' -P 5693 -M 'disk/logical/M:|' -w '90' -c '95'
after apply I often receive the error : "Object does not exist "

For information my MYSQL server is offloaded to a remote server.

I suppose that we problem is on the database but i don't know how to check the problem .
I read the topic : https://support.nagios.com/forum/viewto ... =6&t=51819 but does not contain the solution ....

I checked the database tables with the follow command and all tables are OK :

mysqlcheck --check -u nagios -p<pwd> --databases nagios -h <ip_address_DB_server>
mysqlcheck: [Warning] Using a password on the command line interface can be insecure.
nagios.nagios_acknowledgements OK
nagios.nagios_commands OK
nagios.nagios_commenthistory OK
nagios.nagios_comments OK
nagios.nagios_configfiles OK
nagios.nagios_configfilevariables OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands OK
nagios.nagios_contactgroup_members OK
nagios.nagios_contactgroups OK
nagios.nagios_contactnotificationmethods OK
nagios.nagios_contactnotifications OK
nagios.nagios_contacts OK
nagios.nagios_contactstatus OK
nagios.nagios_customvariables OK
nagios.nagios_customvariablestatus OK
nagios.nagios_dbversion OK
nagios.nagios_downtimehistory OK
nagios.nagios_eventhandlers OK
nagios.nagios_externalcommands OK
nagios.nagios_flappinghistory OK
nagios.nagios_host_contactgroups OK
nagios.nagios_host_contacts OK
nagios.nagios_host_parenthosts OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members OK
nagios.nagios_hostgroups OK
nagios.nagios_hosts OK
nagios.nagios_hoststatus OK
nagios.nagios_instances OK
nagios.nagios_logentries OK
nagios.nagios_notifications OK
nagios.nagios_objects OK
nagios.nagios_processevents OK
nagios.nagios_programstatus OK
nagios.nagios_runtimevariables OK
nagios.nagios_scheduleddowntime OK
nagios.nagios_service_contactgroups OK
nagios.nagios_service_contacts OK
nagios.nagios_service_parentservices OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members OK
nagios.nagios_servicegroups OK
nagios.nagios_services OK
nagios.nagios_servicestatus OK
nagios.nagios_statehistory OK
nagios.nagios_systemcommands OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges OK
nagios.nagios_timeperiods OK


Could you please help me ?


Thanks
Davide

[pbroste]

Hello @dberlenda

Thanks for reaching out, want to run the 'check_ncpa.py' command with the arguments to verify results:

Code: Select all

/usr/local/nagios/libexec/check_ncpa.py -H yourhostaddresshere -t yourtokenhere -M 'memory/virtual/percent' -vvv

Code: Select all

/usr/local/nagios/libexec/check_ncpa.py -H yourhostaddresshere -t yourtokenhere-M 'disk/logical/M:|' -w '90' -c '95'

Secondly, the option to pull-up the 'ncpa' info via web browser api; please verify:

Code: Select all

https://yourhostaddresshere:5693

Please let us know how that looks,
Perry

[dberlenda]

Hello Perry,
I have the problem "Object does not exist" with all the services if I change the thresholds from the nagios services home page and not with specific satellite system .

The problem happen when I edit a service then tab "Configure" ---> Re-configure this service --> Monitoring and here I change for example the thresholds . After the "update" when I push the botton "Continue" , I receive the message ""Object does not exist" .
This problem does not happen always , but the often .

The output of requested commands are the follow :

/usr/local/nagios/libexec/check_ncpa.py -H xxx.xxx.xxx.xxx -t mytoken -M 'memory/virtual/percent' -vvv
Connecting to: https://172.25.1.116:5693/api/memory/vi ... 21&check=1
File returned contained:
{
"returncode": 0,
"stdout": "OK: Percent was 83.50 % | 'percent'=83.50%;;;"
}
OK: Percent was 83.50 % | 'percent'=83.50%;;;

/usr/local/nagios/libexec/check_ncpa.py -H xxx.xxx.xxx.xxx -t mytoken -M 'disk/logical/M:|' -w '90' -c '95'
OK: Used disk space was 35.20 % (Used: 14.07 GiB, Free: 25.92 GiB, Total: 39.99 GiB) | 'used'=14.07GiB;;; 'free'=25.92GiB;;; 'total'=39.99GiB;;;
[nagios@tegp-nag-ia1 ~]$

The ncpa web access works fine ( https://yourhostaddresshere:5693 ) , i don't have any error , all the option work .

Best Regards
Davide

[pbroste]

Hello @dberlenda

Thanks for following up with the details, we ruled out that "Object does not exist" does not correspond with the check commands. Want to take a look at the System Profile and fetch so logs from the database server.

On the database server let's get:

Code: Select all

tar -czvf /tmp/datalogs.tar.gz /var/log/syslog /var/log/messages /var/log/mysql/mysqld.log /etc/my.cnf /etc/mycnf.d/

• *One or more might error that is okay

To send us your system profile.

• Login to the Nagios XI GUI using a web browser.
• Click the "Admin" > "System Profile" Menu
• Click the "Download Profile" button
• Save the profile.zip file and send via Private Message [PM]

Please send over the System Profile and '/tmp/datalogs.tar.gz'

Thanks,
Perry

[dberlenda]

Hello Perry ,
thanks for your support .
I sent to you a privare message with nagiosXI System Profile .

I don't have access to the nagios database server ( this is a DB server in AWS ) , I requested the logs to my collegues , I wil send you also these logs as soon as possible .

Best Regards
Davide

[pbroste]

Hello @dberlenda

Thanks for sending over the System Profile, after review we see the following details logged in the system log.

[1642084885] WARNING: RLIMIT_NPROC is 61236, total max estimated processes is 67134! You should increase your limits (ulimit -u, or limits.conf)

Let's take a look at the current configs for ulimit nofile and nproc values set and compare:

Code: Select all

cat /etc/security/limits.conf | grep -Ei 'nproc|nofile | grep -v ^#

Add or edit the following lines to /etc/security/limits.conf

Code: Select all

* soft nofile 10000
* hard nofile 10000
* hard nproc 4000
* soft nproc 70000

Edit this file:

Code: Select all

/etc/httpd/conf.d/php.conf

Change this specific part (around line 46):

Code: Select all

<IfModule !mod_php5.c>
  <IfModule !mod_php7.c>
    # Enable http authorization headers
    SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1

    <FilesMatch \.(php|phar)$>
        SetHandler "proxy:unix:/run/php-fpm/www.sock|fcgi://localhost"
    </FilesMatch>
  </IfModule>
</IfModule>

Add this:

Code: Select all

    Timeout 600
    RequestReadTimeout header=600,minrate=50
    RequestReadTimeout body=600,minrate=50
    <Proxy "unix:/run/php-fpm/www.sock|fcgi://localhost">
        ProxySet timeout=600
    </Proxy>

So it looks like this:

Code: Select all

<IfModule !mod_php5.c>
  <IfModule !mod_php7.c>
    # Enable http authorization headers
    SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1

    Timeout 600
    RequestReadTimeout header=600,minrate=50
    RequestReadTimeout body=600,minrate=50
    <Proxy "unix:/run/php-fpm/www.sock|fcgi://localhost">
        ProxySet timeout=600
    </Proxy>

    <FilesMatch \.(php|phar)$>
        SetHandler "proxy:unix:/run/php-fpm/www.sock|fcgi://localhost"
    </FilesMatch>
  </IfModule>
</IfModule>

Then edit this file:

Code: Select all

/etc/php-fpm.d/www.conf

Set:

Code: Select all

pm.max_requests = 50

Then restart apache/php-fpm:

Code: Select all

systemctl restart httpd php-fpm

We also see that '...libexec/SAP/check_sap_cons <args>' service(s) check is process intensive, and bring this up so you can take a look at ways to dial back before adding additional hosts/services to your environment.

Thanks,
Perry

[dberlenda]

Hello Perry ,
the limits on file /etc/security/limits.conf was already increased as follow :
root hard nofile 10000
root soft nofile 10000
root hard nproc 250000
root soft nproc 250000
nagios hard nproc 250000
nagios soft nproc 250000

We applied your suggestion for the setting of http and php and not we don't have anymore the error " Object does not exist" .
At the moment we have definined around of 10.000 check in the nagiosXI server . Could you please confirm that the parameters that you suggested are also OK to monitor 20.000 - 25.000 services ?

About the database server we use the follow DB in AWS :
RDS Aurora MySQL
Engine version
5.7.mysql_aurora.2.10.1

and we don't have access to database server , we anabled all available logs in the AWS console but during the weekend no errors are wrote in the logfiles.

Now we stil have only this problem in nagiosXI web interface : when we force the check of a service , in the nagios core the new state of the service is displayed very fast , in the nagiosXI web interface the new state of the services is displayed 1 o 2 mins after .

Could you please help me also to solve this problem ?


Thanks again
Davide

[pbroste]

Hello @dberlenda

Thanks for following up; typically, the optimizations are based on the environment and the processes and services running. There really are no hard-coded set-in-stone optimizations for the number of service checks.

For the ongoing web console load/rendering issue, we want to load the page while observing the Network Tab in the Development tools to see what is causing the most time delays when loading. Also, verify that status column with nothing other than 200's. Please see example:

devetools.png

Thanks,
Perry

[dberlenda]

Hello Perry ,
I'm sorry but I'm not able to arrive in the form in your screenshot . Could you please give me the "path" to arrive to " Network Tab in the Development tools " .

Thanks in advance
Davide

[pbroste]

Hello @dberlenda

Thanks for following up, you can either watch the logs *(typically /var/log/httpd/ or /var/log/apache2) or:

dev_network.png

Thanks,
Perry