Multiple Nagios session in the database

[RIDS_I2MP]

Hi Team,

We have setup monitoring on oracle database.

Yesterday database team informed that there are 200+ active Nagios session can be seen in one of the database node.

Please help us in understanding about the multiple sessions and how this can be avoided in future.

[rkennedy]

How many checks are you running against the database? Additionally, what scripts are you using to check it?

[RIDS_I2MP]

We have set 11 services against database and other are OS related services.

Below listed are few database related services
connection-time
corrupted-blocks
process-usage
rman-backup-problems etc


check_xi_oracleserverspace script is run for all services.

----- Tue Jan 12, 2016 8:26 am -----

Hi Team,

We have provided the details as requested. Please check and give further inputs on the same.

[rkennedy]

What scripts are the 11 services using?

One thing that comes to mind is these scripts may not be closing the SQL connection properly.

[RIDS_I2MP]

Hi,

The command run is check_xi_oracleserverspace.

script: check_oracle_health

mode used is
1)connection-time
2)corrupted-blocks
3)process-usage
4)rman-backup-problems
5)session-usage
6)sga-data-buffer-hit-ratio
7)sga-library-cache-hit-ratio
8)Stale-Statistics
9)tablespace-fragmentation
10)tablespace-usage
11) TNSPING

[jolson]

Yesterday database team informed that there are 200+ active Nagios session can be seen in one of the database node.

These active connections were happening _at the same time_ correct? If so, you should check your Nagios XI server and see if there are any hanging plugins:

Code: Select all

ps -ef | grep oracle

My guess is that the plugins were hanging for some reason - perhaps the DB load was too high to respond to Nagios or something similar. In any case, I'm curious if the number of active connections on your Oracle Server coincides with the number of plugins being run simultaneously on the Nagios box. I don't see any way to set a timeout - perhaps the plugin runs until it's allowed space to finish, which could explain the active connections.

In order to troubleshoot this effectively, we'll need to reproduce the problem - is there any chance you could reproduce this and check on the amount of plugins Nagios is running?

Please help us in understanding about the multiple sessions and how this can be avoided in future.

Hopefully I've answered your question with my above response. You can always monitor the amount of active session with Nagios as well

[RIDS_I2MP]

Hi Team,

Thanks for your response.
we have run the command now and can see just one row as a output.
Unfortunately we will not be able to reproduce the issue. So we need support for below two points :

1) how we can monitor the active sessions in Nagios?

2) Any suggestion to take any preventive steps which allow not to happen this issue again.

[rkennedy]

Without being to replicate it, it's hard to tell what was hanging on your system. As for monitoring, you can create a check to monitor how many oracle scripts are running using the command @jolson mentioned above, and piping it to wc.

Create a command definition that uses this, and then assign the command to a new service on localhost.

Code: Select all

ps -ef|grep oracle|wc -l

While this won't alert you at all, it will help you identify how many scripts are running against the Oracle DB. You could always convert this into a script that generates warning / critical alerts.

[RIDS_I2MP]

Hi team,
Thanks for your reply.

Please confirm in which side( Oracle DB or Nagios server), we have to run this command.
When we ran this command in Nagios server, we got only 4 rows. which should not be the case.
we have around 50 DB servers in monitoring under Nagios, so I believe there should be minimum 50 rows of output when we run ps-ef|grep oracle
Please clarify when the session will be created with the DB from Nagios.

When we will ran this command in DB, there will be many other oracle process as well which will come as part of count.
Please help me in understanding all the things so that I can work out on monitoring the active sessions.

[rkennedy]

I would run that on the Nagios server, if you're monitoring a lot of different database servers though that one liner will not work properly.

Can you run a netstat instead? Change the IP and port to the one that Nagios uses to connect to the DB. This should always tell you how many connections are open to the server.

Code: Select all

netstat|grep ip:port|wc -l