Using Mod_Gearman2 in Nagios XI

[SteveBeauchemin]

From a question I asked in a previous post, I learned that Mod_Gearman2 would work with Nagios XI.

Knowing that it could be done, I did the following to get it setup on my system.
This can be used as a guide, it is not intended to be instructions followed by rote.
This post reflects much of what I learned.

Start on the Nagios XI server, (worker is below)

Stop the existing gearman before you remove it, if you are upgrading to ver 2.

Code: Select all

service mod_gearman_worker stop
service gearmand stop
ps -ef | grep gear

Kill off any gearman processes that do not stop on their own

Uninstall the old gearman code. Look for the program names, and remove them.

Code: Select all

rpm -qa | grep gear
yum remove gearmand libgearman mod_gearman

Install the new Mod_Gearman2. Use the versions of gearman that the developer includes in the source location directory...
Take a look at http://mod-gearman.org/download/v2.1.2/rhel6/x86_64/

Code: Select all

wget http://mod-gearman.org/download/v2.1.2/rhel6/x86_64/gearmand-0.33-2.rhel6.x86_64.rpm
wget http://mod-gearman.org/download/v2.1.2/rhel6/x86_64/gearmand-devel-0.33-2.rhel6.x86_64.rpm
wget http://mod-gearman.org/download/v2.1.2/rhel6/x86_64/gearmand-server-0.33-2.rhel6.x86_64.rpm
wget http://mod-gearman.org/download/v2.1.2/rhel6/x86_64/mod_gearman2-2.1.2-1.rhel6.x86_64.rpm

yum install -y gcc autoconf automake libtool gcc-c++ libevent-devel \
libtool-ltdl libtool-ltdl-devel ncurses-devel e2fsprogs-devel \
boost-devel boost-program-options libevent perl

yum install -y ./gearmand-0.33-2.rhel6.x86_64.rpm \
./gearmand-devel-0.33-2.rhel6.x86_64.rpm \
./gearmand-server-0.33-2.rhel6.x86_64.rpm \
./mod_gearman2-2.1.2-1.rhel6.x86_64.rpm

Edit nagios.cfg and get the NEB Module to look at the new location ( my preference is no to eventhandler )

Code: Select all

vi /usr/local/nagios/etc/nagios.cfg
broker_module=/usr/lib64/mod_gearman2/mod_gearman2.o server=localhost eventhandler=no hosts=yes services=yes config=/etc/mod_gearman2/module.conf

Be aware that while you have the gearman NEB module in place that Gearman has to be started Before Nagios starts,
and also has to stop After nagios stops. There is a definite order to starting and stopping the services.

Change the service startup init code to use the proper user name.

Code: Select all

sed -i '/USER=/c\USER=nagios' /etc/rc.d/init.d/mod-gearman2-worker

Then change ownerships and permissions and set the services to start at boot

Code: Select all

chown -R nagios:root /var/mod_gearman2
chown -R nagios:root /var/log/mod_gearman2
chown -R apache:nagios /etc/mod_gearman2
chmod -R 775 /etc/mod_gearman2
chgrp -R nagios /etc/mod_gearman2/
chmod -R g+w /etc/mod_gearman2/

chkconfig gearmand on
chkconfig --add mod-gearman2-worker
chkconfig  --level 35 mod-gearman2-worker on

service gearmand start
service mod-gearman2-worker start
service nagios restart

I like the application to survive a reboot

Next the server needs its workers.
First thing again is to remove the older gearman code

Code: Select all

service mod_gearman_worker stop
ps -ef | grep gear

Kill off any worker processes that do not stop on their own

Remove the old code (been there, done that once? this is the remote worker)

Code: Select all

yum remove -y libgearman gearmand mod_gearman
yum install -y gcc autoconf automake libtool gcc-c++ libevent-devel \
libtool-ltdl libtool-ltdl-devel ncurses-devel e2fsprogs-devel \
boost-devel boost-program-options libevent perl

If this is a new worker, then this is the time to install the Nagios Plugins and get the nagios user created. (different doc for that somewhere I'm sure)

Change to the location where you have the RPM files and install them

Code: Select all

yum install -y ./gearmand-0.33-2.rhel6.x86_64.rpm \
./gearmand-devel-0.33-2.rhel6.x86_64.rpm \
./gearmand-server-0.33-2.rhel6.x86_64.rpm \
./mod_gearman2-2.1.2-1.rhel6.x86_64.rpm

Change the init script

Code: Select all

sed -i '/USER=/c\USER=nagios' /etc/rc.d/init.d/mod-gearman2-worker

Then a few more changes

Code: Select all

chown -R nagios:root /var/mod_gearman2
chown -R nagios:root /var/log/mod_gearman2
chown -R apache:nagios /etc/mod_gearman2
chmod -R 775 /etc/mod_gearman2
chgrp -R nagios /etc/mod_gearman2/
chmod -R g+w /etc/mod_gearman2/

chkconfig --add mod-gearman2-worker
chkconfig  --level 35 mod-gearman2-worker on

service mod-gearman2-worker start

I like the application to survive a reboot here too

I did not cover changing settings in the mod_gearman2 configuration files.
The settings are up to you and your requirements.
The Gearman configuration files are located in /etc/mod_gearman2/

Install the Mod Gearman 2 component
Last but not least, there is a component you can use to help manage your Mod_Gearman setup.
It is written for the previous version of Mod_Gearman whice I call version 1.
I have a version 2 of that component that I modified from version 1
It works for me. You still need to manually edit in your primary gearman server
and the workers into the PHP file. You still need to make sure your SSH keys work.

Install the component, then edit the following php file and setup your names and IP addresses

Code: Select all

cd /usr/local/nagiosxi/html/includes/components/modgearman2
vi modgearman2.php

The way I dealt with the component was to install it, then edit the php file, then setup the ssh keys.
That is totally up to you.

I am attaching the version 2 component, and am including my notes on setting up shared keys below even though you may use the component script to setup the keys.

Setup SSH Keys
The apache user needs to have a shared key to the remote gearman worker
in order for the Mod_Gearman2 component to work. I set them up manually
as opposed to using the script from the component. I do not have the root
passwords to all my remote workers, so this did the trick for me.

Create the key
On the Nagios XI core system that runs Gearmand server change the apache user to allow logins

Code: Select all

chsh -s /bin/bash apache

Login as the apache user via su, sudo, or whatever means you have.
As the apache user, cd to the home directory and create the .ssh directory

Code: Select all

cd
mkdir .ssh

Change the owner to ensure the apache web process owns that directory and generate the keys.
I just accept the defaults, up to you. Protect the private key.

Code: Select all

chown apache .ssh
ssh-keygen -t dsa
chmod 0500 .ssh/id_dsa

Copy the public key to the remote workers.

Code: Select all

scp .ssh/id_dsa.pub [username]@[remote-gear-server]:/tmp/apache.pub

Login to the remote host and make sure that the nagios user has a password.

Code: Select all

passwd nagios

Become the nagios user, cd to it's home location, make the .ssh directory, put the key in place, and delete the /tmp file.

Code: Select all

cd
mkdir .ssh
cat /tmp/apache.pub >> .ssh/authorized_keys
rm -f /tmp/apache.pub

Make the key work for you
Back on the Nagios XI gearmand server, as the apache user, attach to the remote worker as the nagios user. So you are apache attaching as nagios.

Code: Select all

whoami
ssh nagios@[remotehost]

If asked to add the host to a list the answer is yes. Exit from the remote host and ssh to it again.
You should get attached with no need to type password or answer questions.

Reset your temporary changes
Once that process is clean, exit from the apache user account and set it back to the original state where you are not allowed to login.

Code: Select all

chsh -s /sbin/nologin apache

Install the Component
That done, you can get the component installed now.
The Gearman component from Brian Heyden is the foundation for the version 2 component I have attached.
Please extract the files and read them. Make sure you understand it somewhat. Please...

I hope this helps someone avoid the learning cure I had to go thru.

Thanks

Steve B

[WillemDH]

Thanks Steve. I'll use your notes in the coming weeks! Can anyone confirm that it should be possible to use two mod gearman versions? So that I don't have to migrate my production 1.5 node untill I have a working Gearman 2 node.

[tmcdonald]

Christmas was a few weeks ago, Steve :)

Awesome write-up! We've been dealing with some mod_gearman-based memory leaks these past few weeks, and were considering re-evaluating our MG docs. With your permission, do you mind if we absorb some of your post into our official documentation?

[SteveBeauchemin]

I post this stuff with that in mind. Use any and all you want from the write up. I attached the Component too.

Thanks

Steve B

[tmcdonald]

We'll definitely be taking a look at this in the coming weeks. For the sake of our daily support workflow I'm going to close this up now, but thanks again!