Page 4 of 5
Re: Nagios Log Server - Backup and Maintenance
Posted: Wed Dec 17, 2014 4:29 pm
by roddergreg
I've tried both. /tmp and /usr/local/nagioslogserver/backuprepo
Backup_Maintenance_•_Nagios_Log_Server_-_2014-12-17_16.27.19.png
Re: Nagios Log Server - Backup and Maintenance
Posted: Wed Dec 17, 2014 4:42 pm
by lgroschen
OK, I talked with the other devs and we think you should wait a day and see if these are being written since the maintenance jobs will take up to 24 hours to run from the time you replaced the cmdsubsys_helper file.
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 22, 2014 11:25 am
by roddergreg
No Luck once again. I waited 5 days. I was running out of disk space since the maintenance wasn't running. I had to close and delete indexes manually in the Web UI.
How should I proceed?
Re: Nagios Log Server - Backup and Maintenance
Posted: Mon Dec 22, 2014 3:51 pm
by lgroschen
Let's check that directories write permissions. ls -la your repository:
Code: Select all
ls -la /usr/local/nagioslogserver/backuprepo
Have you tried offloading your backup onto a shared file system? If you load them locally you're going to use resources on your local machine, where as sending the backup to a shared file system will allow log server more resources separate from your backups.
Re: Nagios Log Server - Backup and Maintenance
Posted: Tue Dec 23, 2014 9:10 am
by roddergreg
Code: Select all
ls -al /usr/local/nagioslogserver/backuprepo/
total 8
drwxrwxr-x 2 nagios nagios 4096 Oct 29 16:25 .
drwxrwxr-x 11 nagios nagios 4096 Dec 12 11:29 ..
This is a test environment so the backups arent really important at this time. We've put in a purchase order for 3 paid nagios logserver licenses. I'd like to get the bugs worked out before this goes into production. The production server is going to be designed with much more storage. I think around 5TB
Re: Nagios Log Server - Backup and Maintenance
Posted: Tue Dec 23, 2014 3:56 pm
by lgroschen
Can you send us another file. That file that you edited earlier in the topic: cmdsubsys_helper.php where you replaced the file path with this: /usr/lib/python2.6/site-packages/curator/curator.py
Just zip and attach the file here. It's likely the cause, but at the very least a good place to start.
We'll definitely try to get this cleared up in your dev environment before you launch prod instances.
Re: Nagios Log Server - Backup and Maintenance
Posted: Wed Dec 24, 2014 2:52 pm
by roddergreg
Here is the zip file you requested.
Re: Nagios Log Server - Backup and Maintenance
Posted: Fri Dec 26, 2014 12:16 pm
by jomann
Can you verify that the user nagios can run the command with the full path? The commands in the cmdsubsys get ran as the nagios user since it's done on a cron.
Re: Nagios Log Server - Backup and Maintenance
Posted: Fri Dec 26, 2014 12:29 pm
by scottwilkerson
Also, for testing on this machine, here is a procedure I would like you to run, from the CLI start
Code: Select all
tail -f /usr/local/nagioslogserver/var/jobs.log > /tmp/jobs_output.txt
then in a browser logged into your log server instance navigate to
Code: Select all
http://YOUR_SERVER_ADDRESS/nagioslogserver/index.php/api/system/schedule_job/?job[id]=do_maintenance_create_snapshot&job[command]=do_maintenance_create_snapshot&job[node]=global&job[type]=system&job[run_time]=1
Wait about a minute or 2 and then stop the running tail and post the output of /tmp/jobs_output.txt
Thanks!
Re: Nagios Log Server - Backup and Maintenance
Posted: Fri Dec 26, 2014 12:33 pm
by roddergreg
When i try executing the command exactly how it is in the cmd
Code: Select all
-bash-4.1$ /usr/lib/python2.6/site-packages/curator/curator.py bloom --older-than 1
-bash: /usr/lib/python2.6/site-packages/curator/curator.py: Permission denied
I get permission denied. However when i the curator command prefacing it with python it works correctly.
Code: Select all
-bash-4.1$ python /usr/lib/python2.6/site-packages/curator/curator.py bloom --older-than 1
2014-12-26 12:28:25,940 INFO Job starting...
2014-12-26 12:28:26,182 INFO Beginning BLOOM operations...
2014-12-26 12:28:26,455 INFO Attempting to disable bloom filter for index logstash-2014.12.25.
2014-12-26 12:28:34,719 INFO logstash-2014.12.25: Successfully bloom filter disabled.
2014-12-26 12:28:34,720 INFO logstash-2014.12.26 is within the threshold period (1 days).
2014-12-26 12:28:34,720 INFO DISABLE BLOOM FILTER index operations completed.
2014-12-26 12:28:34,720 INFO Done in 0:00:08.802540.
I ran this as nagios and cleared the bloom filter.