ssh vs nrpe

[BanditBBS]

Ok, so I am having some weird ssh issues that we just can't grasp. I posted about it before and we attempted the fix(maxstartups and maxsessions) but that didn't fix it. That being said, I'd like input on this:

Capture.JPG

What other ones can you think of? The one con for NRPE about needing to modify the cfg and restart across 1000+ servers if I ever want to make a change is a pretty big con....but then again, this ssh issue is a pain in the butt.

[WillemDH]

Bandit,

Fyi, in case you would be concerned with encryption of traffic, NRPE can also be secured with ssl certs and NSClient++.

http://www.medin.name/blog/2012/12/02/s ... ntication/

I never came to trying it though. Anyone using certificate based authentication with NRPE?

Grtz

Willem

[mp4783]

Doesn't NRPE encrypt by default? I certainly thought it did. I assumed the exchange of certificates occurred when the Nagios XI server contact an NRPE agent configured to accept requests from it.

I'm not sure what it is being asked here. All NRPE is doing is executing plugins for you and returning the results. If you want to use SSH, you're going to get the same results. The only "advantage" of using SSH is that you don't have to install an agent. Beyond that, a properly configured NRPE agent can run any command that you could run through SSH.

[BanditBBS]

Doesn't NRPE encrypt by default? I certainly thought it did. I assumed the exchange of certificates occurred when the Nagios XI server contact an NRPE agent configured to accept requests from it.

I'm not sure what it is being asked here. All NRPE is doing is executing plugins for you and returning the results. If you want to use SSH, you're going to get the same results. The only "advantage" of using SSH is that you don't have to install an agent. Beyond that, a properly configured NRPE agent can run any command that you could run through SSH.

By default NRPE just uses ssl, no certs or anything.

I am currently running 15000 checks every 5 minutes with at least half of them being check_by_ssh checks. We happen to have an ssh issue every so often spread through our network. We are working to investigate and solve, but I'm considering switching to nrpe(which I've used extensively in the past). I'm just trying to get others input on pros/cons list so when I talk to others in my company I can be as up front as possible. The only con I care about so far is having to modify the nrpe.cfg if I want to add a new check or something. Has anyone automated that? If so, then that could make my decision easy as ever!

[mp4783]

I see what you're saying, you want to go beyond encryption to authentication. Good point.

I'll briefly describe something I set up (although it's not yet rigorously tested) to deal with NRPE maintenance. I apologize for the obtuse description, I'm not sure how much I'm allowed to reveal about things we're doing.

- Create a chrooted user account on a "repository" server, set it's shell to nologin, and lock it's password
- Configure SSHD so that user account can only perform SCP/SFTP, if possible
- Create a directory off the home directory of user that is writable only be root, but readable my the user
- Generate a public/private, unencrypted key pair for that user
- Place the public key into the /home/<user>/.ssh/authorized_keys file of the user on the repository server
- Copy the private/public key pair to a location which is secure and only accessible to the nagios user on the monitored host
- Create a script to be placed on all NRPE agent servers that when executed, SCP/SFTP files from the repository serer directory created above, using the private key, and places it in the appropriate nagios user controlled directory (e.g. /usr/local/nagios/libexec)
- Create an entry in the nrpe.cfg file to permit execution of that check
- On the Nagios XI server, create a check_by_nrpe service check that points the name of the script you created on the NRPE agent monitored host
- On the repository server, place any new plugin or check script files in the appropriate directory
- On the Nagios XI server, schedule the check to run regularly or force its execution when needed

When the check executes, the following will happen:
- The monitored host's NRPE agent will execute the check
- The check will initiate an SCP/SFTP session to the user created on the repository server
- It will pull down any files it finds or those it is directed to
- It will place them in the appropriate directory on the monitored host that is writable to the nagios user
- The configuration on the monitored host is now updated without requiring you to access it directly

These updates cannot require a restart of the inetd or xinetd daemon (NRPE agent). If a restart of a process is required that falls outside of the permissions of the agent user, then sudo permissions will need to be granted.

Benefits:
• All configuration files pulled from a central, secure source repository
• This check can, theoretically, be used to maintain any configuration file on any server with SFTP access to the repository server.
• This check restricts what can be run on a host by providing “customized” NRPE configuration files for each.
• Configuration files are separated into directories on the repository server, so all you need to know is which directory contains the agent configuration file for your server type.
• Check could be extended to the deployment of new service checks and plugins.
• By maintaining a central repository of plugin archives (tar.gz) files, another agent configuration update check would ensure that all of the necessary tooling is present on the monitored hosts.

If you think through this outline, you'll see additional efficiencies like using service or host groups and the environment variables to drive the check, repository directories tied to specific groups, etc.

You should also consider the security ramifications of this and take appropriate measures to secure the repository server.

[cmerchant]

Thank you mp4783 for your suggestions, I will go ahead and leave this thread open until Banditt says to close. BTW, Banditt, did you clear up the nfs issues on this thread:

http://support.nagios.com/forum/viewtop ... fs#p131705

[BanditBBS]

Thank you mp4783 for your suggestions, I will go ahead and leave this thread open until Banditt says to close. BTW, Banditt, did you clear up the nfs issues on this thread:

http://support.nagios.com/forum/viewtop ... fs#p131705

Hey cmerchant, leave this open just a bit more, I"d like to get others opinions if anyone else cares to speak.

My reason for this post really has nothing to do with security. I'm thinking about ease of management. Looking at the pros/cons list I included, the only big con for me is the need to update nrpe.cfg and restart xinetd across 1000+ servers if I need to add or change something. mp4783 gave a good idea about the nrpe check to once per day grab any new files(plugins), but that is only part of the issue. We could even use that for updating nrpe.cfg, but then how do I automate restarting xinetd when needed.

Other than that, does anyone else have anymore pros/cons to either ssh or nrpe usage?

[abrist]

I like check_by_ssh much more than I like nrpe. Nrpe has pretty weak security and has all the issues with local configs. I like running check_by_ssh from a gearman worker if ssh session load/connections is a worry. Key management is stupid simple if you already have ways to push keys or a unified key management system. The remote plugins directory could even be a netmount, so you don't have to copy new plugins to every box. Also, you do not have the same issues with payload size, etc, that you have with nrpe.

Check by ssh allows the entire configuration to reside on the nagios server. Of course, the one thing neither of these do well are passive checks.

Just my 2 cents.

[BanditBBS]

Thousands of ssh connections can be a big load hog though...that plus this darn ssh issue we are facing here is just killing me. We can not find the cause or resolution! But yeah, having keys in place already makes ssh so nice and not having to actually install anything on the remote servers is great.

I'm pretty sure I'd thought out a complete automation for nrpe config update and xinetd restarting process....so I am really leaning towards nrpe right now unless someone can give me a pretty big con that I am not thinking about.

[abrist]

nrpe = weak df cipher?
But your environment is so secure, that is not an issue, right?