Nagios Log Server - Backup and Maintenance

[roddergreg]

I've tried both. /tmp and /usr/local/nagioslogserver/backuprepo

Backup_Maintenance_•_Nagios_Log_Server_-_2014-12-17_16.27.19.png

[lgroschen]

OK, I talked with the other devs and we think you should wait a day and see if these are being written since the maintenance jobs will take up to 24 hours to run from the time you replaced the cmdsubsys_helper file.

[roddergreg]

No Luck once again. I waited 5 days. I was running out of disk space since the maintenance wasn't running. I had to close and delete indexes manually in the Web UI.

How should I proceed?

[lgroschen]

Let's check that directories write permissions. ls -la your repository:

Code: Select all

ls -la /usr/local/nagioslogserver/backuprepo

Have you tried offloading your backup onto a shared file system? If you load them locally you're going to use resources on your local machine, where as sending the backup to a shared file system will allow log server more resources separate from your backups.

[roddergreg]

Code: Select all

ls -al /usr/local/nagioslogserver/backuprepo/
total 8
drwxrwxr-x  2 nagios nagios 4096 Oct 29 16:25 .
drwxrwxr-x 11 nagios nagios 4096 Dec 12 11:29 ..

This is a test environment so the backups arent really important at this time. We've put in a purchase order for 3 paid nagios logserver licenses. I'd like to get the bugs worked out before this goes into production. The production server is going to be designed with much more storage. I think around 5TB

[lgroschen]

Can you send us another file. That file that you edited earlier in the topic: cmdsubsys_helper.php where you replaced the file path with this: /usr/lib/python2.6/site-packages/curator/curator.py

Just zip and attach the file here. It's likely the cause, but at the very least a good place to start.

We'll definitely try to get this cleared up in your dev environment before you launch prod instances.

[roddergreg]

Here is the zip file you requested.

[jomann]

Can you verify that the user nagios can run the command with the full path? The commands in the cmdsubsys get ran as the nagios user since it's done on a cron.

[scottwilkerson]

Also, for testing on this machine, here is a procedure I would like you to run, from the CLI start

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log > /tmp/jobs_output.txt

then in a browser logged into your log server instance navigate to

Code: Select all

http://YOUR_SERVER_ADDRESS/nagioslogserver/index.php/api/system/schedule_job/?job[id]=do_maintenance_create_snapshot&job[command]=do_maintenance_create_snapshot&job[node]=global&job[type]=system&job[run_time]=1

Wait about a minute or 2 and then stop the running tail and post the output of /tmp/jobs_output.txt

Thanks!

[roddergreg]

When i try executing the command exactly how it is in the cmd

Code: Select all

-bash-4.1$ /usr/lib/python2.6/site-packages/curator/curator.py bloom --older-than 1
-bash: /usr/lib/python2.6/site-packages/curator/curator.py: Permission denied

I get permission denied. However when i the curator command prefacing it with python it works correctly.

Code: Select all

-bash-4.1$ python /usr/lib/python2.6/site-packages/curator/curator.py bloom --older-than 1
2014-12-26 12:28:25,940 INFO      Job starting...
2014-12-26 12:28:26,182 INFO      Beginning BLOOM operations...
2014-12-26 12:28:26,455 INFO      Attempting to disable bloom filter for index logstash-2014.12.25.
2014-12-26 12:28:34,719 INFO      logstash-2014.12.25: Successfully bloom filter disabled.
2014-12-26 12:28:34,720 INFO      logstash-2014.12.26 is within the threshold period (1 days).
2014-12-26 12:28:34,720 INFO      DISABLE BLOOM FILTER index operations completed.
2014-12-26 12:28:34,720 INFO      Done in 0:00:08.802540.

I ran this as nagios and cleared the bloom filter.