Backup & Maintenance not working automatically

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
TEWLS
Posts: 33
Joined: Wed Dec 28, 2016 3:53 pm

Re: Backup & Maintenance not working automatically

Post by TEWLS »

So I have nagios as a user now but I still get the jsonselect issue when trying to run the jobs automatically.
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Backup & Maintenance not working automatically

Post by mcapra »

The errors we're running into are actually separate from the backup_maintenance job. The backups job runs create_backup.sh which references the jsonselect module directly rather than importing it.

Can you run the following as root:

Code: Select all

curl -s -XPOST 'http://localhost:9200/_export/state' > /tmp/state.json
python -m jsonselect.__main__ .count < /tmp/state.json
python -c "help('modules');" | grep jsonselect
Then again as the nagios user (use sudo if needed), sharing outputs along the way?
Former Nagios employee
https://www.mcapra.com/
TEWLS
Posts: 33
Joined: Wed Dec 28, 2016 3:53 pm

Re: Backup & Maintenance not working automatically

Post by TEWLS »

Code: Select all

curl -s -XPOST 'http://localhost:9200/_export/state' > /tmp/state.json                                                    
python -m jsonselect.__main__ .count < /tmp/state.json
2
python -c "help('modules');" | grep jsonselect
DLFCN               cPickle             jsonselect          sgmllib

Code: Select all

curl -s -XPOST 'https://localhost:9200/_export/state' > /tmp/state.json
$ python -m jsonselect.__main__ .count < /tmp/state.json
/bin/python: No module named jsonselect
$ python -c "help('modules');" | grep jsonselect
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Backup & Maintenance not working automatically

Post by mcapra »

Can you share the outputs of these commands as both root and nagios:

Code: Select all

python -V
which python
Former Nagios employee
https://www.mcapra.com/
TEWLS
Posts: 33
Joined: Wed Dec 28, 2016 3:53 pm

Re: Backup & Maintenance not working automatically

Post by TEWLS »

Both are the same

Code: Select all

$python -V
Python 2.7.5

$which python
/bin/python
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Backup & Maintenance not working automatically

Post by mcapra »

Pretty baffling so far. How about the full output of this from nagios and root:

Code: Select all

python -c "help('modules');"
It's almost like nagios and root are using 2 different python installations.
Former Nagios employee
https://www.mcapra.com/
TEWLS
Posts: 33
Joined: Wed Dec 28, 2016 3:53 pm

Re: Backup & Maintenance not working automatically

Post by TEWLS »

Okay these are actually different, and from what I saw the only thing different is jsonselect.

Nagios

Code: Select all

$ python -c "help('modules');"

Please wait a moment while I gather a list of all available modules...

BaseHTTPServer      backports           imputil             selinux
Bastion             base64              iniparse            semanage
CDROM               bdb                 inspect             seobject
CGIHTTPServer       binascii            io                  sepolgen
ConfigParser        binhex              itertools           sepolicy
Cookie              bisect              javapackages        sets
DLFCN               bsddb               json                setuptools
DocXMLRPCServer     bz2                 keyword             sgmllib
HTMLParser          cPickle             lib2to3             sha
IN                  cProfile            liblzma             shelve
IPy                 cStringIO           linecache           shlex
MimeWriter          calendar            linuxaudiodev       shutil
OpenSSL             cgi                 locale              signal
Queue               cgitb               logging             site
SimpleHTTPServer    chunk               lxml                slip
SimpleXMLRPCServer  cmath               lzma                smtpd
SocketServer        cmd                 macpath             smtplib
StringIO            code                macurl2path         snack
TYPES               codecs              mailbox             sndhdr
UserDict            codeop              mailcap             socket
UserList            collections         markupbase          spwd
UserString          colorsys            marshal             sqlite3
_LWPCookieJar       commands            math                sqlitecachec
_MozillaCookieJar   compileall          md5                 sre
__builtin__         compiler            mhlib               sre_compile
__future__          configobj           mimetools           sre_constants
_abcoll             contextlib          mimetypes           sre_parse
_ast                cookielib           mimify              ssl
_audit              copy                mmap                stat
_bisect             copy_reg            modulefinder        statvfs
_bsddb              crypt               multifile           string
_codecs             csv                 multiprocessing     stringold
_codecs_cn          ctypes              mutex               stringprep
_codecs_hk          curl                netrc               strop
_codecs_iso2022     curses              new                 struct
_codecs_jp          datetime            nis                 subprocess
_codecs_kr          dbhash              nntplib             sunau
_codecs_tw          dbm                 ntpath              sunaudio
_collections        dbus                nturl2path          symbol
_crypt              decimal             numbers             symtable
_csv                decorator           opcode              sys
_ctypes             difflib             operator            sysconfig
_curses             dircache            optparse            syslog
_curses_panel       dis                 os                  tabnanny
_dbus_bindings      distutils           os2emxpath          tarfile
_dbus_glib_bindings dl                  ossaudiodev         telnetlib
_elementtree        doctest             parser              tempfile
_functools          dumbdbm             pdb                 termios
_hashlib            dummy_thread        perf                test
_heapq              dummy_threading     pickle              textwrap
_hotshot            easy_install        pickletools         this
_io                 email               pipes               thread
_json               encodings           pkg_resources       threading
_locale             errno               pkgutil             time
_lsprof             exceptions          platform            timeit
_markerlib          fcntl               plistlib            timing
_multibytecodec     filecmp             policycoreutils     toaiff
_multiprocessing    fileinput           popen2              token
_osx_support        firewall            poplib              tokenize
_pyio               fnmatch             posix               trace
_random             formatter           posixfile           traceback
_semanage           fpformat            posixpath           tty
_snack              fractions           pprint              tuned
_socket             ftplib              profile             types
_sqlite3            functools           pstats              unicodedata
_sqlitecache        future_builtins     pty                 unittest
_sre                gc                  pwd                 urlgrabber
_ssl                gdbm                py_compile          urllib
_strptime           genericpath         pyclbr              urllib2
_struct             getopt              pycurl              urlparse
_symtable           getpass             pydoc               user
_sysconfigdata      gettext             pydoc_data          uu
_threading_local    gi                  pyexpat             uuid
_warnings           glob                pygtkcompat         validate
_weakref            gpgme               pyparsing           warnings
_weakrefset         grp                 pyudev              wave
abc                 gzip                quopri              weakref
acutil              hashlib             random              webbrowser
aifc                heapq               re                  whichdb
antigravity         hmac                readline            wsgiref
anydbm              hotshot             repr                xattr
argparse            htmlentitydefs      resource            xdrlib
array               htmllib             rexec               xml
ast                 httplib             rfc822              xmllib
asynchat            idlelib             rlcompleter         xmlrpclib
asyncore            ihooks              robotparser         xxsubtype
atexit              imageop             rpm                 yum
audiodev            imaplib             rpmUtils            zipfile
audioop             imghdr              runpy               zipimport
audit               imp                 sched               zlib
auparse             importlib           select

Enter any module name to get more help.  Or, type "modules spam" to search
for modules whose descriptions contain the word "spam".
Root:

Code: Select all

python -c "help('modules');"

Please wait a moment while I gather a list of all available modules...

BaseHTTPServer      bdb                 iniparse            semanage
Bastion             binascii            inspect             seobject
CDROM               binhex              io                  sepolgen
CGIHTTPServer       bisect              itertools           sepolicy
ConfigParser        bsddb               javapackages        sets
Cookie              bz2                 json                setuptools
DLFCN               cPickle             jsonselect          sgmllib
DocXMLRPCServer     cProfile            keyword             sha
HTMLParser          cStringIO           lib2to3             shelve
IN                  calendar            liblzma             shlex
IPy                 cgi                 linecache           shutil
MimeWriter          cgitb               linuxaudiodev       signal
OpenSSL             chunk               locale              site
Queue               click               logging             slip
SimpleHTTPServer    cmath               lxml                smtpd
SimpleXMLRPCServer  cmd                 lzma                smtplib
SocketServer        code                macpath             snack
StringIO            codecs              macurl2path         sndhdr
TYPES               codeop              mailbox             socket
UserDict            collections         mailcap             spwd
UserList            colorsys            markupbase          sqlite3
UserString          commands            marshal             sqlitecachec
_LWPCookieJar       compileall          math                sre
_MozillaCookieJar   compiler            md5                 sre_compile
__builtin__         configobj           mhlib               sre_constants
__future__          contextlib          mimetools           sre_parse
_abcoll             cookielib           mimetypes           ssl
_ast                copy                mimify              stat
_audit              copy_reg            mmap                statvfs
_bisect             crypt               modulefinder        string
_bsddb              csv                 multifile           stringold
_codecs             ctypes              multiprocessing     stringprep
_codecs_cn          curator             mutex               strop
_codecs_hk          curl                netrc               struct
_codecs_iso2022     curses              new                 subprocess
_codecs_jp          datetime            nis                 sunau
_codecs_kr          dbhash              nntplib             sunaudio
_codecs_tw          dbm                 ntpath              symbol
_collections        dbus                nturl2path          symtable
_crypt              decimal             numbers             sys
_csv                decorator           opcode              sysconfig
_ctypes             difflib             operator            syslog
_curses             dircache            optparse            tabnanny
_curses_panel       dis                 os                  tarfile
_dbus_bindings      distutils           os2emxpath          telnetlib
_dbus_glib_bindings dl                  ossaudiodev         tempfile
_elementtree        doctest             parser              termios
_functools          dumbdbm             pdb                 test
_hashlib            dummy_thread        perf                tests
_heapq              dummy_threading     pickle              textwrap
_hotshot            easy_install        pickletools         this
_io                 elasticsearch       pip                 thread
_json               email               pipes               threading
_locale             encodings           pkg_resources       time
_lsprof             errno               pkgutil             timeit
_markerlib          exceptions          platform            timing
_multibytecodec     fcntl               plistlib            toaiff
_multiprocessing    filecmp             policycoreutils     token
_osx_support        fileinput           popen2              tokenize
_pyio               firewall            poplib              trace
_random             fnmatch             posix               traceback
_semanage           formatter           posixfile           tty
_snack              fpformat            posixpath           tuned
_socket             fractions           pprint              types
_sqlite3            ftplib              profile             unicodedata
_sqlitecache        functools           pstats              unittest
_sre                future_builtins     pty                 urlgrabber
_ssl                gc                  pwd                 urllib
_strptime           gdbm                py_compile          urllib2
_struct             genericpath         pyclbr              urllib3
_symtable           getopt              pycurl              urlparse
_sysconfigdata      getpass             pydoc               user
_threading_local    gettext             pydoc_data          uu
_warnings           gi                  pyexpat             uuid
_weakref            glob                pygtkcompat         validate
_weakrefset         gpgme               pyparsing           warnings
abc                 grp                 pyudev              wave
acutil              gzip                quopri              weakref
aifc                hashlib             random              webbrowser
antigravity         heapq               re                  whichdb
anydbm              hmac                readline            wsgiref
argparse            hotshot             repr                xattr
array               htmlentitydefs      resource            xdrlib
ast                 htmllib             rexec               xml
asynchat            httplib             rfc822              xmllib
asyncore            idlelib             rlcompleter         xmlrpclib
atexit              ihooks              robotparser         xxsubtype
audiodev            imageop             rpm                 yum
audioop             imaplib             rpmUtils            zipfile
audit               imghdr              runpy               zipimport
auparse             imp                 sched               zlib
backports           importlib           select
base64              imputil             selinux

Enter any module name to get more help.  Or, type "modules spam" to search
for modules whose descriptions contain the word "spam".
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Backup & Maintenance not working automatically

Post by mcapra »

Are the perms on that module's files in site-packages allowing the nagios user to read the jsonselect path? That's one way I was able to reproduce this:

Code: Select all

ls -al /usr/lib/python2.7/site-packages
If it's not 744, try changing it to 744. Or 755.
Former Nagios employee
https://www.mcapra.com/
TEWLS
Posts: 33
Joined: Wed Dec 28, 2016 3:53 pm

Re: Backup & Maintenance not working automatically

Post by TEWLS »

I had to change jsonselect to 755 to get the errors to subside. However, I still don't see anything getting closed or deleted and there are no additional snapshots.

Code: Select all

/usr/lib/python2.7/site-packages/jsonselect:
total 64
-rwxr-xr-x.  1 root root 15910 Oct 21  2013 jsonselect.py
-rwxr-xr-x.  1 root root  1253 Oct 21  2013 __main__.py
-rwxr-xr-x.  1 root root   178 Oct 24  2013 __init__.py
-rwxr-xr-x.  1 root root   394 Nov 17 10:22 __init__.pyc
-rwxr-xr-x.  1 root root  1598 Nov 17 10:22 __main__.pyc
-rwxr-xr-x.  1 root root 21578 Nov 17 10:22 jsonselect.pyc
drwxr-xr-x.  2 root root  4096 Nov 17 10:22 .
drwxr-xr-x. 34 root root  4096 Jan 24 12:08 ..

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
PHP Warning:  Module 'SourceGuardian' already loaded in Unknown on line 0
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command update_check with args ' ' for job id: run_update_check
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
PHP Warning:  Module 'SourceGuardian' already loaded in Unknown on line 0
Running command create_backup with args ' ' for job id: AVs_k0f-cJMl7DjS3OF1
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
PHP Warning:  Module 'SourceGuardian' already loaded in Unknown on line 0
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
PHP Warning:  Module 'SourceGuardian' already loaded in Unknown on line 0
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Backup & Maintenance not working automatically

Post by mcapra »

That particular job (backup_maintenance) only gets run every 24 hours unless you're specifically telling it to run sooner via the GUI.

Based on this:
TEWLS wrote:

Code: Select all

tail -f /usr/local/nagioslogserver/var/jobs.log
PHP Warning:  Module 'SourceGuardian' already loaded in Unknown on line 0
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command update_check with args ' ' for job id: run_update_check
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
PHP Warning:  Module 'SourceGuardian' already loaded in Unknown on line 0
Running command create_backup with args ' ' for job id: AVs_k0f-cJMl7DjS3OF1
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
PHP Warning:  Module 'SourceGuardian' already loaded in Unknown on line 0
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
Processed 0 node jobs.
Processed 2 global jobs.
tail: /usr/local/nagioslogserver/var/jobs.log: file truncated
PHP Warning:  Module 'SourceGuardian' already loaded in Unknown on line 0
Running command run_alerts with args ' ' for job id: run_all_alerts
SUCCESS
It doesn't look like the backup_maintenance job has ran yet.
Former Nagios employee
https://www.mcapra.com/
Locked