Informix Monitoring

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Informix Monitoring

Post by donnyforbes »

Does anyone know of a way to monitor "informix"? Basically what I am looking for is a way to get alerted when informix goes down

I placed the following script below in the following directory on the remote server (check_informix.sh)
/usr/local/nagios/libexec

Code: Select all

#!/bin/bash
#
# check_ifx  - plugin to validate that Informix is online # # parameters # 1) Informix directory # 2) Informix server

print_use(){
   echo -e "USE: check_ifx informixdir informixserver\\n"
   echo -e "\t informixdir    = Informix directory"
   echo -e "\t informixserver = server name"
}

STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3

if [ $# -lt 2 ]
then
   print_use
   exit $STATE_UNKNOWN
fi

if [ $# -gt 2 ]
then
   print_use
   exit $STATE_UNKNOWN
fi

export INFORMIXDIR=$1
export INFORMIXSERVER=$2

result=`$INFORMIXDIR/bin/onstat -`

for cmd in $result
do
  if [ $cmd = On-Line ]
  then
     echo $result
     exit 0
  fi
done
echo $result
exit 3
chmod +x /usr/local/nagios/libexec/check_informix.sh
chown nagios.nagios /usr/local/nagios/libexec/check_informix.sh

Then added this to the bottom of your /usr/local/nagios/etc/nrpe.cfg:
command[check_informix]=/usr/local/nagios/libexec/check_informix.sh

Then restart the nrpe service:
systemctl restart xinetd

I then tried to test the script by running this command from the remote server and got this error
[root@huey libexec]# /usr/local/nagios/libexec/check_informix.sh /usr/informix hueynet
-bash: /usr/local/nagios/libexec/check_informix.sh: Permission denied

Here are the permission
[root@huey libexec]# ls -la
total 4
drwxr-xr-x. 2 root root 31 Nov 15 06:58 .
drwxr-xr-x. 3 root root 21 Nov 14 22:36 ..
-rw-r-----. 1 nagios nagios 684 Nov 15 06:58 check_informix.sh
[root@huey libexec]#

From the Nagios server I tried to run this command and got this error
[root@nag etc]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42 -c check_informix.sh
connect to address 192.168.1.42 port 5666: No route to host

Please advise so I can get this working. At one time I had this working and worked with one of your reps, however since then we had to wipe and rebuild the servers. Here is the ticket reference "Need to Monitor Informix" and this was back in July 2017. Waiting for assistance with this. Need to get this working again.

Thank you,
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: Informix Monitoring

Post by donnyforbes »

I did change the permission to 777 on the /usr/local/nagios/libexec/check_informix.sh on all server and now it ran and gave me the following and it seems okay see below..

[root@huey libexec]# /usr/local/nagios/libexec/check_informix.sh /usr/informix hueynet
IBM Informix Dynamic Server Version 12.10.FC8W2WE -- Updatable (Sec) -- Up 16 days 12:25:10 -- 1138808 Kbytes
[root@huey libexec]#

[root@duey libexec]# /usr/local/nagios/libexec/check_informix.sh /usr/informix dueynet
IBM Informix Dynamic Server Version 12.10.FC8W2WE -- On-Line (Prim) -- Up 19 days 09:11:42 -- 1171576 Kbytes
[root@duey libexec]#

[root@louie libexec]# /usr/local/nagios/libexec/check_informix.sh /usr/informix louienet
IBM Informix Dynamic Server Version 12.10.FC8W2WE -- Updatable (RSS) -- Up 19 days 09:11:55 -- 1106040 Kbytes
[root@louie libexec]#

This is expected.... Please help me get this added to nagiosxi and everything else working correctly...

Thanks for your help and assistance.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Informix Monitoring

Post by scottwilkerson »

donnyforbes wrote:

Code: Select all

connect to address 192.168.1.42 port 5666: No route to host
this error implies that your Nagios server cannot reach 192.168.1.42 and doesn't have a route to know how to get there. This would be a networking issue and outside of the scope of Nagios support.

However once you get the networking fugored out, based on the command you defined in nrpe, you would want to change the command you run from this

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 192.168.1.42 -c check_informix.sh
to this

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 192.168.1.42 -c check_informix
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: Informix Monitoring

Post by donnyforbes »

Thanks for the quick response. I am not sure what is going on. I can ping from nag server to remote and from the remote server to the nag server see below..

[root@nag etc]# ping huey
PING huey.sg1.allamericanasphalt.com (192.168.1.42) 56(84) bytes of data.
64 bytes from huey.sg1.allamericanasphalt.com (192.168.1.42): icmp_seq=1 ttl=64 time=0.336 ms
64 bytes from huey.sg1.allamericanasphalt.com (192.168.1.42): icmp_seq=2 ttl=64 time=0.260 ms
64 bytes from huey.sg1.allamericanasphalt.com (192.168.1.42): icmp_seq=3 ttl=64 time=0.283 ms

[root@huey libexec]# ping nag
PING nag.sg1.allamericanasphalt.com (192.168.1.186) 56(84) bytes of data.
64 bytes from nag.sg1.allamericanasphalt.com (192.168.1.186): icmp_seq=1 ttl=64 time=0.303 ms
64 bytes from nag.sg1.allamericanasphalt.com (192.168.1.186): icmp_seq=2 ttl=64 time=0.276 ms
64 bytes from nag.sg1.allamericanasphalt.com (192.168.1.186): icmp_seq=3 ttl=64 time=61.7 ms

I checked the port status on the remote server and I have the following

[root@huey libexec]# netstat -an |grep 5666
tcp 0 0 192.168.1.42:10022 192.168.1.49:55666 ESTABLISHED
[root@huey libexec]#

Then on the nag server I have this

[root@nag etc]# netstat -an |grep 5666
tcp 0 0 192.168.1.186:34345 192.168.1.12:5666 TIME_WAIT
tcp 0 0 192.168.1.186:34354 192.168.1.12:5666 TIME_WAIT
tcp 0 0 192.168.1.186:34352 192.168.1.12:5666 TIME_WAIT
tcp 0 0 192.168.1.186:34357 192.168.1.12:5666 TIME_WAIT
tcp 0 0 192.168.1.186:34359 192.168.1.12:5666 TIME_WAIT
tcp6 0 0 :::5666 :::* LISTEN
You have mail in /var/spool/mail/root
[root@nag etc]#

This appears to be correct.... Any thoughts????

Thanks
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: Informix Monitoring

Post by donnyforbes »

From the remote server I did the following as well

Code: Select all

[root@huey libexec]# netstat -na | grep 5666
tcp        0      0 192.168.1.42:10022      192.168.1.49:55666      ESTABLISHED
And this....

Code: Select all

[root@huey libexec]# iptables -L -v -n > test
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 824M  242G ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
1568K  118M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
  14M 1402M INPUT_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  14M 1402M INPUT_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  14M 1402M INPUT_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  947 41742 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
  14M 1382M REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 FORWARD_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 FORWARD_IN_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 FORWARD_IN_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 FORWARD_OUT_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 FORWARD_OUT_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 625M packets, 2562G bytes)
 pkts bytes target     prot opt in     out     source               destination         
 625M 2562G OUTPUT_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD_IN_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 FWDI_public  all  --  eno1   *       0.0.0.0/0            0.0.0.0/0           [goto] 
    0     0 FWDI_public  all  --  +      *       0.0.0.0/0            0.0.0.0/0           [goto] 

Chain FORWARD_IN_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD_OUT_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 FWDO_public  all  --  *      eno1    0.0.0.0/0            0.0.0.0/0           [goto] 
    0     0 FWDO_public  all  --  *      +       0.0.0.0/0            0.0.0.0/0           [goto] 

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain FWDI_public (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 FWDI_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 FWDI_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 FWDI_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FWDI_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain FWDI_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain FWDI_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain FWDO_public (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 FWDO_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 FWDO_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 FWDO_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FWDO_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain FWDO_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain FWDO_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  14M 1402M IN_public  all  --  eno1   *       0.0.0.0/0            0.0.0.0/0           [goto] 
    0     0 IN_public  all  --  +      *       0.0.0.0/0            0.0.0.0/0           [goto] 

Chain INPUT_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain IN_public (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  14M 1402M IN_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  14M 1402M IN_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  14M 1402M IN_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
26558 2549K ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain IN_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  759 45060 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW
   46  2760 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9102 ctstate NEW
93605 4979K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW
 197K   10M ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10022 ctstate NEW
31119 1867K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10023 ctstate NEW
  386 19852 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10030 ctstate NEW
   16   810 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10031 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10032 ctstate NEW
 2630  158K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10033 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10025 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2381 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2301 ctstate NEW

Chain IN_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain IN_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination         
Any other thoughts on this. Also I did a grep for nrpe to make sure it was running and it seems as it is.
[root@huey libexec]# ps -ef |grep nrpe
root 36500 8067 0 08:27 pts/3 00:00:00 grep --color=auto nrpe
[root@huey libexec]#
Thanks,
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: Informix Monitoring

Post by donnyforbes »

I have ran this again

[root@nag etc]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42 -c check_informix
connect to address 192.168.1.42 port 5666: No route to host

Still getting "No route to host" I do have a question. is there a way to verify that this server is being monitored. This was wipe and rebuilt and puppetized. I do see in the nagiosXI when I search for the server it shows the following (screenshot attached)


Which by the looks of this it looks like it is being monitored.

I also verify that xinetd was running on that remote sever. See the outcome below.

[root@huey /]# service xinetd status
Redirecting to /bin/systemctl status xinetd.service
● xinetd.service - Xinetd A Powerful Replacement For Inetd
Loaded: loaded (/usr/lib/systemd/system/xinetd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-09-24 20:58:02 PDT; 1 months 21 days ago
Main PID: 1779 (xinetd)
CGroup: /system.slice/xinetd.service
└─1779 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid

Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing discard
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing discard
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing echo
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing echo
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing puppet-kick
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing tcpmux
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing time
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing time
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: xinetd Version 2.3.15 started with libwrap loadavg labeled-ne... in.
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: Started working: 0 available services
Hint: Some lines were ellipsized, use -l to show in full.
[root@huey /]#
You do not have the required permissions to view the files attached to this post.
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: Informix Monitoring

Post by donnyforbes »

Can someone please work on this with me. I need to get this working for our production environment.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Informix Monitoring

Post by scottwilkerson »

curious you can ping huey but get a no route to host trying to connect to 192.168.1.42

Does the following work?

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H huey -c check_informix
or

Code: Select all

ping 192.168.1.42
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: Informix Monitoring

Post by donnyforbes »

Please see results

[root@nag etc]# /usr/local/nagios/libexec/check_nrpe -H Huey -c check_informix
connect to address 192.168.1.42 port 5666: No route to host
connect to host Huey port 5666: No route to host[root@nag etc]#

[root@nag etc]# ping 192.168.1.42
PING 192.168.1.42 (192.168.1.42) 56(84) bytes of data.
64 bytes from 192.168.1.42: icmp_seq=1 ttl=64 time=183 ms
64 bytes from 192.168.1.42: icmp_seq=2 ttl=64 time=0.254 ms
64 bytes from 192.168.1.42: icmp_seq=3 ttl=64 time=0.252 ms
64 bytes from 192.168.1.42: icmp_seq=4 ttl=64 time=0.256 ms
64 bytes from 192.168.1.42: icmp_seq=5 ttl=64 time=0.270 ms
64 bytes from 192.168.1.42: icmp_seq=6 ttl=64 time=0.294 ms

What am I missing.. Something just seems strange.. I sent you output of iptables as well.
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: Informix Monitoring

Post by donnyforbes »

From my server running nagios I did a traceroute to the remote server and here is the output

[root@nag etc]# traceroute 192.168.1.42
traceroute to 192.168.1.42 (192.168.1.42), 30 hops max, 60 byte packets
1 huey.sg1.allamericanasphalt.com (192.168.1.42) 78.315 ms !X 78.284 ms !X 78.266 ms !X
[root@nag etc]#

This seems to be okay .... I am not sure what else this could be, however I really need to get this issues resolved...
Locked