Page 1 of 8
Informix Monitoring
Posted: Thu Nov 15, 2018 10:08 am
by donnyforbes
Does anyone know of a way to monitor "informix"? Basically what I am looking for is a way to get alerted when informix goes down
I placed the following script below in the following directory on the remote server (check_informix.sh)
/usr/local/nagios/libexec
Code: Select all
#!/bin/bash
#
# check_ifx - plugin to validate that Informix is online # # parameters # 1) Informix directory # 2) Informix server
print_use(){
echo -e "USE: check_ifx informixdir informixserver\\n"
echo -e "\t informixdir = Informix directory"
echo -e "\t informixserver = server name"
}
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3
if [ $# -lt 2 ]
then
print_use
exit $STATE_UNKNOWN
fi
if [ $# -gt 2 ]
then
print_use
exit $STATE_UNKNOWN
fi
export INFORMIXDIR=$1
export INFORMIXSERVER=$2
result=`$INFORMIXDIR/bin/onstat -`
for cmd in $result
do
if [ $cmd = On-Line ]
then
echo $result
exit 0
fi
done
echo $result
exit 3
chmod +x /usr/local/nagios/libexec/check_informix.sh
chown nagios.nagios /usr/local/nagios/libexec/check_informix.sh
Then added this to the bottom of your /usr/local/nagios/etc/nrpe.cfg:
command[check_informix]=/usr/local/nagios/libexec/check_informix.sh
Then restart the nrpe service:
systemctl restart xinetd
I then tried to test the script by running this command from the remote server and got this error
[root@huey libexec]# /usr/local/nagios/libexec/check_informix.sh /usr/informix hueynet
-bash: /usr/local/nagios/libexec/check_informix.sh: Permission denied
Here are the permission
[root@huey libexec]# ls -la
total 4
drwxr-xr-x. 2 root root 31 Nov 15 06:58 .
drwxr-xr-x. 3 root root 21 Nov 14 22:36 ..
-rw-r-----. 1 nagios nagios 684 Nov 15 06:58 check_informix.sh
[root@huey libexec]#
From the Nagios server I tried to run this command and got this error
[root@nag etc]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42 -c check_informix.sh
connect to address 192.168.1.42 port 5666: No route to host
Please advise so I can get this working. At one time I had this working and worked with one of your reps, however since then we had to wipe and rebuild the servers. Here is the ticket reference "Need to Monitor Informix" and this was back in July 2017. Waiting for assistance with this. Need to get this working again.
Thank you,
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 10:10 am
by donnyforbes
I did change the permission to 777 on the /usr/local/nagios/libexec/check_informix.sh on all server and now it ran and gave me the following and it seems okay see below..
[root@huey libexec]# /usr/local/nagios/libexec/check_informix.sh /usr/informix hueynet
IBM Informix Dynamic Server Version 12.10.FC8W2WE -- Updatable (Sec) -- Up 16 days 12:25:10 -- 1138808 Kbytes
[root@huey libexec]#
[root@duey libexec]# /usr/local/nagios/libexec/check_informix.sh /usr/informix dueynet
IBM Informix Dynamic Server Version 12.10.FC8W2WE -- On-Line (Prim) -- Up 19 days 09:11:42 -- 1171576 Kbytes
[root@duey libexec]#
[root@louie libexec]# /usr/local/nagios/libexec/check_informix.sh /usr/informix louienet
IBM Informix Dynamic Server Version 12.10.FC8W2WE -- Updatable (RSS) -- Up 19 days 09:11:55 -- 1106040 Kbytes
[root@louie libexec]#
This is expected.... Please help me get this added to nagiosxi and everything else working correctly...
Thanks for your help and assistance.
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 10:44 am
by scottwilkerson
donnyforbes wrote:Code: Select all
connect to address 192.168.1.42 port 5666: No route to host
this error implies that your Nagios server cannot reach 192.168.1.42 and doesn't have a route to know how to get there. This would be a networking issue and outside of the scope of Nagios support.
However once you get the networking fugored out, based on the command you defined in nrpe, you would want to change the command you run from this
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H 192.168.1.42 -c check_informix.sh
to this
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H 192.168.1.42 -c check_informix
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 11:07 am
by donnyforbes
Thanks for the quick response. I am not sure what is going on. I can ping from nag server to remote and from the remote server to the nag server see below..
[root@nag etc]# ping huey
PING huey.sg1.allamericanasphalt.com (192.168.1.42) 56(84) bytes of data.
64 bytes from huey.sg1.allamericanasphalt.com (192.168.1.42): icmp_seq=1 ttl=64 time=0.336 ms
64 bytes from huey.sg1.allamericanasphalt.com (192.168.1.42): icmp_seq=2 ttl=64 time=0.260 ms
64 bytes from huey.sg1.allamericanasphalt.com (192.168.1.42): icmp_seq=3 ttl=64 time=0.283 ms
[root@huey libexec]# ping nag
PING nag.sg1.allamericanasphalt.com (192.168.1.186) 56(84) bytes of data.
64 bytes from nag.sg1.allamericanasphalt.com (192.168.1.186): icmp_seq=1 ttl=64 time=0.303 ms
64 bytes from nag.sg1.allamericanasphalt.com (192.168.1.186): icmp_seq=2 ttl=64 time=0.276 ms
64 bytes from nag.sg1.allamericanasphalt.com (192.168.1.186): icmp_seq=3 ttl=64 time=61.7 ms
I checked the port status on the remote server and I have the following
[root@huey libexec]# netstat -an |grep 5666
tcp 0 0 192.168.1.42:10022 192.168.1.49:55666 ESTABLISHED
[root@huey libexec]#
Then on the nag server I have this
[root@nag etc]# netstat -an |grep 5666
tcp 0 0 192.168.1.186:34345 192.168.1.12:5666 TIME_WAIT
tcp 0 0 192.168.1.186:34354 192.168.1.12:5666 TIME_WAIT
tcp 0 0 192.168.1.186:34352 192.168.1.12:5666 TIME_WAIT
tcp 0 0 192.168.1.186:34357 192.168.1.12:5666 TIME_WAIT
tcp 0 0 192.168.1.186:34359 192.168.1.12:5666 TIME_WAIT
tcp6 0 0 :::5666 :::* LISTEN
You have mail in /var/spool/mail/root
[root@nag etc]#
This appears to be correct.... Any thoughts????
Thanks
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 11:27 am
by donnyforbes
From the remote server I did the following as well
Code: Select all
[root@huey libexec]# netstat -na | grep 5666
tcp 0 0 192.168.1.42:10022 192.168.1.49:55666 ESTABLISHED
And this....
Code: Select all
[root@huey libexec]# iptables -L -v -n > test
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
824M 242G ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
1568K 118M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
14M 1402M INPUT_direct all -- * * 0.0.0.0/0 0.0.0.0/0
14M 1402M INPUT_ZONES_SOURCE all -- * * 0.0.0.0/0 0.0.0.0/0
14M 1402M INPUT_ZONES all -- * * 0.0.0.0/0 0.0.0.0/0
947 41742 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
14M 1382M REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_direct all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_IN_ZONES_SOURCE all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_IN_ZONES all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_OUT_ZONES_SOURCE all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_OUT_ZONES all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 625M packets, 2562G bytes)
pkts bytes target prot opt in out source destination
625M 2562G OUTPUT_direct all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD_IN_ZONES (1 references)
pkts bytes target prot opt in out source destination
0 0 FWDI_public all -- eno1 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all -- + * 0.0.0.0/0 0.0.0.0/0 [goto]
Chain FORWARD_IN_ZONES_SOURCE (1 references)
pkts bytes target prot opt in out source destination
Chain FORWARD_OUT_ZONES (1 references)
pkts bytes target prot opt in out source destination
0 0 FWDO_public all -- * eno1 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all -- * + 0.0.0.0/0 0.0.0.0/0 [goto]
Chain FORWARD_OUT_ZONES_SOURCE (1 references)
pkts bytes target prot opt in out source destination
Chain FORWARD_direct (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_public (2 references)
pkts bytes target prot opt in out source destination
0 0 FWDI_public_log all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDI_public_deny all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDI_public_allow all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
Chain FWDI_public_allow (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_public_deny (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_public_log (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_public (2 references)
pkts bytes target prot opt in out source destination
0 0 FWDO_public_log all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDO_public_deny all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDO_public_allow all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FWDO_public_allow (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_public_deny (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_public_log (1 references)
pkts bytes target prot opt in out source destination
Chain INPUT_ZONES (1 references)
pkts bytes target prot opt in out source destination
14M 1402M IN_public all -- eno1 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all -- + * 0.0.0.0/0 0.0.0.0/0 [goto]
Chain INPUT_ZONES_SOURCE (1 references)
pkts bytes target prot opt in out source destination
Chain INPUT_direct (1 references)
pkts bytes target prot opt in out source destination
Chain IN_public (2 references)
pkts bytes target prot opt in out source destination
14M 1402M IN_public_log all -- * * 0.0.0.0/0 0.0.0.0/0
14M 1402M IN_public_deny all -- * * 0.0.0.0/0 0.0.0.0/0
14M 1402M IN_public_allow all -- * * 0.0.0.0/0 0.0.0.0/0
26558 2549K ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
Chain IN_public_allow (1 references)
pkts bytes target prot opt in out source destination
759 45060 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate NEW
46 2760 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9102 ctstate NEW
93605 4979K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 ctstate NEW
197K 10M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10022 ctstate NEW
31119 1867K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10023 ctstate NEW
386 19852 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10030 ctstate NEW
16 810 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10031 ctstate NEW
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10032 ctstate NEW
2630 158K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10033 ctstate NEW
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10025 ctstate NEW
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2381 ctstate NEW
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2301 ctstate NEW
Chain IN_public_deny (1 references)
pkts bytes target prot opt in out source destination
Chain IN_public_log (1 references)
pkts bytes target prot opt in out source destination
Chain OUTPUT_direct (1 references)
pkts bytes target prot opt in out source destination
Any other thoughts on this. Also I did a grep for nrpe to make sure it was running and it seems as it is.
[root@huey libexec]# ps -ef |grep nrpe
root 36500 8067 0 08:27 pts/3 00:00:00 grep --color=auto nrpe
[root@huey libexec]#
Thanks,
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 11:51 am
by donnyforbes
I have ran this again
[root@nag etc]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.42 -c check_informix
connect to address 192.168.1.42 port 5666: No route to host
Still getting "No route to host" I do have a question. is there a way to verify that this server is being monitored. This was wipe and rebuilt and puppetized. I do see in the nagiosXI when I search for the server it shows the following (screenshot attached)
Which by the looks of this it looks like it is being monitored.
I also verify that xinetd was running on that remote sever. See the outcome below.
[root@huey /]# service xinetd status
Redirecting to /bin/systemctl status xinetd.service
● xinetd.service - Xinetd A Powerful Replacement For Inetd
Loaded: loaded (/usr/lib/systemd/system/xinetd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-09-24 20:58:02 PDT; 1 months 21 days ago
Main PID: 1779 (xinetd)
CGroup: /system.slice/xinetd.service
└─1779 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing discard
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing discard
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing echo
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing echo
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing puppet-kick
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing tcpmux
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing time
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: removing time
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: xinetd Version 2.3.15 started with libwrap loadavg labeled-ne... in.
Sep 24 20:58:02 huey.sg1.allamericanasphalt.com xinetd[1779]: Started working: 0 available services
Hint: Some lines were ellipsized, use -l to show in full.
[root@huey /]#
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 12:20 pm
by donnyforbes
Can someone please work on this with me. I need to get this working for our production environment.
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 1:31 pm
by scottwilkerson
curious you can ping huey but get a no route to host trying to connect to 192.168.1.42
Does the following work?
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H huey -c check_informix
or
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 1:40 pm
by donnyforbes
Please see results
[root@nag etc]# /usr/local/nagios/libexec/check_nrpe -H Huey -c check_informix
connect to address 192.168.1.42 port 5666: No route to host
connect to host Huey port 5666: No route to host[root@nag etc]#
[root@nag etc]# ping 192.168.1.42
PING 192.168.1.42 (192.168.1.42) 56(84) bytes of data.
64 bytes from 192.168.1.42: icmp_seq=1 ttl=64 time=183 ms
64 bytes from 192.168.1.42: icmp_seq=2 ttl=64 time=0.254 ms
64 bytes from 192.168.1.42: icmp_seq=3 ttl=64 time=0.252 ms
64 bytes from 192.168.1.42: icmp_seq=4 ttl=64 time=0.256 ms
64 bytes from 192.168.1.42: icmp_seq=5 ttl=64 time=0.270 ms
64 bytes from 192.168.1.42: icmp_seq=6 ttl=64 time=0.294 ms
What am I missing.. Something just seems strange.. I sent you output of iptables as well.
Re: Informix Monitoring
Posted: Thu Nov 15, 2018 2:32 pm
by donnyforbes
From my server running nagios I did a traceroute to the remote server and here is the output
[root@nag etc]# traceroute 192.168.1.42
traceroute to 192.168.1.42 (192.168.1.42), 30 hops max, 60 byte packets
1 huey.sg1.allamericanasphalt.com (192.168.1.42) 78.315 ms !X 78.284 ms !X 78.266 ms !X
[root@nag etc]#
This seems to be okay .... I am not sure what else this could be, however I really need to get this issues resolved...